tag:blogger.com,1999:blog-73983793227537246272024-03-09T18:46:16.982-08:00Ingenious MalarkeyThoughts on concepts across suitable subjects.Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.comBlogger76125tag:blogger.com,1999:blog-7398379322753724627.post-86871958771180571252022-08-10T08:37:00.003-07:002022-08-10T08:43:42.902-07:00Done When? <p><span style="font-family: Arial; font-size: 20pt; white-space: pre-wrap;">Dangers of not being done</span></p><span id="docs-internal-guid-39e7f344-7fff-9e1a-8078-a7c6de652897"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Successful software teams share many similar team norms. One such norm is the ability to quickly answer the following question by reflex – </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">What does it mean to be DONE with my task?</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At first glance, it seems easy to answer this. Doesn’t it just mean that I wrote the code to do the thing the product owner asked? Good question. Unfortunately, that’s only half the battle. That code has to run in a production environment and it needs to do the right thing. Ensuring both of those things will go as expected actually involves a whole list of other activities by the developer. Failing to complete those activities results in a bad day for many folks involved with the product. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So, what’s a developer to do? Two things can easily help,</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"></p><ol style="text-align: left;"><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Each team agrees upon a set of criteria implicitly required of all tasks in order to be considered not just Done, but DONE. </span></li><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Each team agrees upon how to write Task Descriptions in order to capture the intent of the task unambiguously </span></li></ol><p></p><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Sample implicit criteria shared by all work</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The lists below outline some implicit “Done” criteria shared by successful product development teams that I’ve observed throughout my career. A developer should be able to answer “yes” to each item. </span></p><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Feature complete</span></h2><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have run the code at least once locally in my dev environment end to end</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have run the unit tests and they pass</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have added any additional unit tests where applicable</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have run front-end unit tests where applicable. In the absence of unit tests, I have done my best to navigate through the UI to uncover possible regressions</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have worked through a full understanding of the feature with the Product Owner and identified the various use and edge cases</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have engaged the stakeholder, demoed, and gotten approval</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Onboarding new clients onto this feature is automated & documented</span></p></li></ul><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Systems complete</span></h2><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have performed any infrastructure changes necessary in production</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have automated all those tasks as much as is reasonable – at the least, I talked to the Infrastructure (or SRE) team about it and possibly submitted an Infrastructure request if applicable. </span></p></li></ul><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Sustainability</span></h2><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have added a new item in the OWNERS file or relevant other location </span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have documented the feature</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have added relevant metrics</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have added appropriate alerting & associated runbooks (e.g. internal documentation site)</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I have sent out a PSA and/or Chat and/or engaged the team appropriately</span></p></li></ul><div><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Writing Unambiguous Tasks</span></h1><div><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">In my experience, the quickest way to get unambiguous task descriptions is adopting <a href="https://www.mountaingoatsoftware.com/agile/user-stories" target="_blank">the User Story format</a>. </span></span></div></span>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-62253190018299088852022-08-10T07:50:00.005-07:002022-08-10T07:50:50.204-07:00Logging in the Code and out<span id="docs-internal-guid-49121168-7fff-d8f0-fa10-bef5cead0e4c"><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">1. Why Logs?</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Logs are a way to log what's going on in your application. Logs show up in the console or in log files on disk. See</span><a href="https://en.wikipedia.org/wiki/Logbook" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;"> history of the term</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">2. Logging Levels</span></h2><div align="left" dir="ltr" style="margin-left: 0pt;"><table style="border-collapse: collapse; border: none;"><colgroup><col width="312"></col><col width="312"></col></colgroup><tbody><tr style="height: 33pt;"><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-right: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Level</span></p></td><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">When?</span></p></td></tr><tr style="height: 74.25pt;"><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-right: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">ERROR</span></p></td><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Something troubling has happened that will break something else. This almost always precedes an exception or an end to execution</span></p></td></tr><tr style="height: 135.75pt;"><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-right: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">WARN</span></p></td><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Warning is often used for handled 'exceptions' or other important log events. For example, if your application requires a configuration setting but has a default in case the setting is missing, then the Warning level should be used to log the missing configuration setting.</span></p></td></tr><tr style="height: 156pt;"><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-right: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">INFO</span></p></td><td style="border-bottom: solid #c1c7d0 0.75pt; border-left: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Output information that is useful to the running and management of your system. Information would also be the level used to log Entry and Exit points in key areas of your application. However, you may choose to add more entry and exit points at Debug level for more granularity during development and testing.</span></p></td></tr><tr style="height: 135.75pt;"><td style="border-left: solid #c1c7d0 0.75pt; border-right: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 2.0568; margin-bottom: 12pt; margin-top: 18pt;"><span style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">DEBUG</span></p></td><td style="border-left: solid #c1c7d0 0.75pt; border-top: solid #c1c7d0 0.75pt; overflow-wrap: break-word; overflow: hidden; padding: 6pt 6pt 6pt 6pt; vertical-align: top;"><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Debug is considered out-of-bounds for a production system and used it only for development and testing. I prefer to aim to get my logging levels just right so I have just enough information and endeavor to log this at the Information level or above.</span></p></td></tr></tbody></table></div><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">3. How to write a useful log message</span></h1><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">3.1 SMART</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">All log messages should be:</span></p><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Specific</span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> -- easy to pinpoint where</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Meaningful</span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> -- provide discerning information to identify what's wrong & some context</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Actionable</span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> -- provide information with which to attempt to fix the problem or further debug / recreate</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Relevant</span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> -- provide information pertinent to the situation, not redundant or superfluous</span></p></li><li aria-level="1" dir="ltr" style="color: #172b4d; font-family: Roboto, sans-serif; font-size: 12pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Terse</span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> -- Get to the point and be clear about it</span></p></li></ol><br /></span>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-31922722807864386122022-04-27T18:25:00.004-07:002022-04-27T18:26:42.294-07:00Don't Do Nothing<p><span style="background-color: white;">An owner cares about all aspects of their product and organization. That means that when you see something, you say something. This mentality is beautifully summed up by the "Don't Do Nothing" principle.</span></p><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;"><span face="-apple-system, system-ui, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji">I learned about the "Don't Do Nothing" principle from a great article on the </span><a href="https://blog.trello.com/avoid-social-loafing" rel="nofollow" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; text-decoration-line: none; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s;">Trello Blog</a><span face="-apple-system, system-ui, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji">. I immediately incorporated it into my daily life and preached it to my team.</span></span></p><blockquote style="border-left: 0.25em solid var(--color-border-default); box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin: 0px 0px 16px; padding: 0px 1em;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 0px; margin-top: 0px;"><span style="background-color: white;">>> In short, if you see a problem outside your area, <em style="box-sizing: border-box;">don't do nothing about it.</em></span></p></blockquote><h2 dir="auto" style="border-bottom: 1px solid var(--color-border-muted); box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/papaben/battlescars/blob/master/Ownership.md#a-simple-example" id="user-content-a-simple-example" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span style="background-color: white;"><span style="color: black;"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></span></span></a><span style="background-color: white;">A simple example</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">A simple example -- if you see a typo on the marketing blog, then:</span></p><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">don't assume that the marketing team already knows about it</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">don't assume someone else will report it</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">don't assume that it's not important enough to bother someone</span></li></ul><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">--> Tell the marketing team!</span></p><h2 dir="auto" style="border-bottom: 1px solid var(--color-border-muted); box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/papaben/battlescars/blob/master/Ownership.md#takeaways" id="user-content-takeaways" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span style="background-color: white;"><span style="color: black;"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></span></span></a><span style="background-color: white;">Takeaways</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">What makes the Don't Do Nothing principle so powerful is that in these exact types of situations it provides you the license to be the 105th person to report the typo or the "annoying" person who reported a very insignificant problem. It provides you this license because there just might be the chance that if <em style="box-sizing: border-box;">everyone did nothing</em>, then the marketing team would never know about the typo, and that blog post could be part of a crucial launch.</span></p><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, "system-ui", "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"><span style="background-color: white;">A team or company that abides by this principle will always welcome that 105th or low priority typo report because they know that everyone is here for the same reasons: to build something amazing.</span></p>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-74146986739442445832022-02-20T07:35:00.003-08:002022-02-20T07:35:39.254-08:00Looking for a Code Review? 4 Tips to Set Yourself Up For Success<p>A lot goes into making an effective code review. It starts with setting the appropriate expectations between the author and the reviewer(s) and ends with the effort the author puts in. This article is a list of helpful questions to ask yourself as a Pull Request author.</p><h2 style="text-align: left;">1. What are you expecting from each reviewer</h2><p>Each of the following is important, but it's not reasonable to expect that one single reviewer will do them all. It's important as both the code author and reviewer to clarify the role expected by each reviewer.</p><p></p><ul style="text-align: left;"><li>A semantic or style review?</li><li>A quick check for simple errors?</li><li>A review of your code layout design decisions?</li><li>A review of the business logic?</li><li>A review of the (missing) error handling?</li><li>A review of the scalability implications of your choices?</li><li>A check for security vulnerabilities?</li><li>A review of your comments & confluence docs?</li></ul><p></p><h2 style="text-align: left;">2. What level of diligence are you expecting to get</h2><p></p><ul style="text-align: left;"><li>A pairing session?</li><li>Asynchronous 5 minutes? 15 minutes?</li><li>Review of each line of code changed?</li><li>Review the efficacy of each unit test?</li><li>Check out of the Pull Request locally and run it?</li><li>Check out of the Pull Request locally and tracing through of related parts of the code?</li><li>Reading the sprint task or Product Requirements and consideration of the use cases?</li><li>Reading the sprint task or Product Requirements and consideration of missed use cases?</li><li>Reading the sprint task or Product Requirements and consideration of the landscape of possible data inputs?</li><li>Review the proper use of any new dependent modules?</li></ul><p></p><h2 style="text-align: left;">3. When do you want to get feedback?</h2><p></p><ul style="text-align: left;"><li>Today</li><li>Tomorrow</li><li>Before End of Sprint</li><li>Do you expect a back and forth?</li></ul><p></p><h2 style="text-align: left;">4. What have you done to improve chances of a good review?</h2><p></p><ul style="text-align: left;"><li>Did you add useful comments?</li><li>Did you review the PR yourself and add comments where applicable?</li><li>Did you limit the size of the change to a small enough amount? See Tip #3 in this checklist.</li><li>Did you describe your goal for the change in the description?</li><li>Did you run the unit tests & linters before requesting review?</li><li>Did you clarify the state of the code (draft, WIP, ready for merge)?</li></ul><p></p><p><br /></p><p><br /></p>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-82660652703626983172022-02-20T07:23:00.006-08:002022-02-20T07:24:41.363-08:00Writing effective documentationDo you struggle to find the answers you need from your documentation? Have you ever said “This documentation page sucks” or complained about your documentation platform in general?<p style="text-align: left;">If so, you might be excited to know that the solution to your problems is within your power to fix! From comments in code, to internal documentation, to writing emails to your colleagues - there are some quick best practices that can help you be more effective at communicating.</p><h2 style="text-align: left;">My quick tips</h2><ol style="text-align: left;"><li>Always have a table of contents</li><li>Number your sections and use section headers appropriately so that folks can keep track of where they are in the page </li><li>Avoid long-winded (or even short winded) explanations about history or conversational asides anywhere that you are sharing how-to or instructions. This type of information is very distracting and can even be confusing to readers, who will be unaware about the relevance & significance to what they need to know. </li><ol><li>If you feel compelled to include such content in your documentation, then it can be placed in an <span style="font-family: courier;">Appendix A</span> section at the bottom.</li></ol><li>As an alternative to several repetitive sub-sections, consider if your information could be better presented via a <b>Table</b>.</li><li>Use numbered <b>Lists</b>.</li><li>If you have a list with multiple sub-lists, then it’s likely that you need to create new sections for each of those sub-lists. Your top level section can have a high level list – but don’t forget, so does your Table of Contents!</li><li>Stay focused on the type of information you are sharing. Is this a Runbook or an instructional page? I.e. is the reader looking to learn about a feature so they can add new code to it, or are they just solving a problem with no desire to ever think about this again?</li></ol><div><h2 style="text-align: left;">Learn More!</h2><p style="text-align: left;">I did some searching in Google and thought this post covers a lot of useful topics</p><ol style="text-align: left;"><li><a href="https://bethaitman.com/posts/tld-2019/">Writing effective documentation - the Lead Developer Berlin 2019 | Beth Aitman</a> (there’s also a<a href="https://www.youtube.com/watch?v=R6zeikbTgVc"> YouTube video</a> for people who prefer video).</li><li>She links to another page,<a href="https://www.gov.uk/guidance/content-design"> Content design: planning, writing and managing content</a> , which I found to be amazing, but has possibly more information to peruse than you’re ready for.</li></ol><h2 style="text-align: left;">Give back!</h2><p style="text-align: left;">Improving at this skill set will make you a stronger engineer, and it will help your team be better too. If you have your own thoughts to share, please comment on this post! </p></div>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com1tag:blogger.com,1999:blog-7398379322753724627.post-1977471115504818462022-02-19T14:38:00.003-08:002022-02-19T14:38:44.850-08:00Give your tests a story<p style="text-align: left;">One thing that I remember distinctly from my college classes was how purely academic our assignments and lectures were. Why couldn’t we get “real world” examples? This was a real problem for me because I struggled to connect the dots about the topic when all the principal points were only abstract theory. There were no concrete, tangible things in my life with which to associate the knowledge. In fact, programming didn’t really click for me until my first (college) job when I had to write code to solve – you guessed it – real world problems.</p><h2 style="text-align: left;">So, what does this have to do with tests?</h2><div><p style="text-align: left;">Much the same as those theoretical situations in my college classes, the unit, integration, and functional tests that we write tend to be generic & ambiguous. When we come back to them after some time has passed, it's often not clear what we were testing. Even worse, when we're <i>writing </i>the tests we can miss crucial pieces because the code is so abstract. </p><p style="text-align: left;">So, let's take that learning from above and apply it here. When setting up a test situation, you have the opportunity to be an author. You can create a storyline about the user who is using your product and trying to get value from it. Not only will this make your test more <b>memorable</b>, it will make it more <b>comprehensible</b>.</p><h2 style="text-align: left;">An example</h2><div><p style="text-align: left;">Perhaps you’re writing code for marketing software and testing a new Push Notification action. To do this, you need to create a marketing campaign, configure a campaign action, and create some dynamic content (we’ll use Jinja in this example).</p><p style="text-align: left;">The naive way to approach testing your new Push Notification action would be something along the lines of “<span style="font-family: courier;">test_action = …</span>” and “<span style="font-family: courier;">assert test_action …</span>”. However, that name doesn't give us much context to work with; nor does that assertion have a real world expectation. So, how can we make this more memorable and furthermore how can we make it easier for future readers to understand?</p>The answer is that we can give our tests a <b>story</b>.</div></div><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;">Marcia is a marketer who works at Acme, Inc. She uses your product to help increase the LTV of their shoe purchasing customers. Marcia is planning a big event for the launch of a new shoe design. The Push Notification action you just built is going to be critical for Marcia to engage with the younger crowd that Acme is targeting for the launch.</p></blockquote><div><p style="text-align: left;">The test variables and values might look like the following:</p><ul style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 16pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Campaign Name: <span style="color: #172b4d; font-family: Roboto Mono, monospace;"><span style="font-size: 10.5pt; white-space: pre-wrap;">Acme Shoe Launch, testing Push Action</span></span></span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Campaign Action<span style="color: #172b4d; font-family: Roboto, sans-serif;"><span style="font-size: 12pt; white-space: pre-wrap;">: </span></span></span>Your Push action</p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Push subject<span style="color: #172b4d; font-family: Roboto, sans-serif;"><span style="font-size: 12pt; white-space: pre-wrap;">: </span></span></span><span style="color: #172b4d; font-family: "Roboto Mono", monospace; font-size: 10.5pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">👟Get excited for your next shoe!</span></p></li><li aria-level="1" dir="ltr" style="font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Push body<span style="color: #172b4d; font-family: Roboto, sans-serif;"><span style="font-size: 12pt; white-space: pre-wrap;">:</span></span></span><span style="color: #172b4d; font-family: "Roboto Mono", monospace; font-size: 10.5pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> Is it time to replace your {{ customer.acme_most_recent_shoe_name }}?</span></p></li></ul><p style="text-align: left;">Wow, that was fun! You got to use a realistic contact property in your jinja and it’s obvious to anyone who encounters this test what you were doing. Most importantly, if you need to dig this up in a few months, it’s going to be a lot easier for you to (a) remember it and (b) distinguish it from your other test campaigns.</p><h2 style="text-align: left;">Takeaways</h2><ol style="text-align: left;"><li>Consider the ability for others (and even your future self) to understand what you’re trying to test based on the scenario and names you choose.</li><li>Consider the likelihood that you’ll spot problems or edge cases by choosing a scenario and set of names that more closely reflect the reality of an actual user.</li><li>Consider the ease with which you’ll be able to dig up an old test case or functional test object later down the road when you need to refresh or remind yourself of something.</li></ol></div>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-13662004185916047092019-02-14T11:42:00.000-08:002019-11-30T11:44:58.946-08:00How do I prepare for my tech job interview?A candidate reached out to me today to ask for advice on how to prepare for her job search. After sending my reply, I realized that this is not an uncommon question engineers ask themselves. The resources out there are good and bad, so I thought I'd share what I said. For some reference, I'm coming at this after reviewing thousands of resumés in the last 12 months and conducting hundreds of first touch phone calls and technical assessment coding calls.<br />
<ol>
<li>Stay very focused during the open dialog. Pause every few sentences to ensure the other person is still following and that you're staying on topic. </li>
<li>Run through some practice algorithms & coding questions. Maybe Project Euler or Hacker Rank. This will just get your brain used to solving the types of things you'll see in an interview situation. Typically, once you're past the first phone assessment, it's less "Problem then Solution," and more architecture / big picture. </li>
<li>Pick a few industries or companies that you'd love to work at. Do some research on them. Then make a special version of your resume for those jobs. </li>
<ol>
<li>If you're still not getting any traction, figure out who the hiring manager is and see if they have some blog articles or public presence in the industry and then add something about that to your cover letter -- don't be creepy about it! but you can typically find something that resonates with you and make a remark on it. </li>
</ol>
<li>Cover letters <i>do</i> make a difference. Short & sweet.</li>
<ol>
<li>Just get the point across that you read the job post, </li>
<li>Five (no more than five!) sentence elevator pitch about yourself. </li>
</ol>
</ol>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-70077388090601743092016-12-16T12:58:00.003-08:002016-12-16T12:58:32.550-08:00How to safely kill your NTP serverNTP is that service that typically quietly runs in the background on almost <i>every single </i>computer in the world. When a computer boots up for the first time, NTP gets configured and is almost never considered again. Why? Because for most people, it just works. Until the end of this century (Y2K + 100) or the next leap year second, most system administrators will happily not need to think about the wonders that NTP gives us. There are, of course, always <a href="http://support.ntp.org/bin/view/Main/SecurityNotice">security considerations to make</a>, but again, once those are addressed, NTP again fades to the back of your mind. <div>
<br /></div>
<div>
If you are running your servers in Amazon, it's possible that your VPC doesn't permit access to external NTP peers. In this case, you may follow directions to <a href="https://www.import.io/post/sync-up-keeping-aws-servers-inside-a-vpc-on-the-clock/">set up your own NTP server.</a> These were the exact directions my sys admin predecessor followed for the VPC I inherited at my current job. </div>
<div>
<br /></div>
<div>
As it turns out, every one of the servers in our VPC needs external internet access and we configure that to route through a NAT. So, other than the security considerations of allowing traffic over port 123, there was no good reason to not rely on the public NTP peers. Great! So, that meant we could kill our internal NTP and save time and money by not hosting it.</div>
<div>
<br /></div>
<div>
So, how do you safely stop and shutdown your NTP server?</div>
<div>
<br /></div>
<div>
First up, the basics:</div>
<div>
<ol>
<li>Run a loop across all your servers and <span style="font-family: Courier New, Courier, monospace;">`grep 'server' /etc/ntp.conf | grep <ip|name>`</span>. I just used a bash for-loop. </li>
<ol>
<li>If you have any matches. Use a similar bash loop to replace the conf file and `<span style="font-family: Courier New, Courier, monospace;">sudo service ntpd stop; sudo service ntpd stop</span>`. </li>
<li>Then run `<span style="font-family: Courier New, Courier, monospace;">ntpq -p</span>` to confirm connections could be established with upstream peers.</li>
</ol>
<li>Check your image and the scripts that build them. If necessary, rebuild those.</li>
</ol>
<div>
<br /></div>
</div>
<div>
<div>
<div>
Now you can feel pretty confident that everything <i>you know about </i>is configured properly. ...but what about what you <i>don't </i>know about?? Here's what I did:</div>
</div>
</div>
<div>
<br /></div>
<div>
<ol>
<li><span style="font-family: Courier New, Courier, monospace;">ssh ntp-server</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">sudo service ntpd stop</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">screen</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">sudo nc -vulk -w 1 123</span></li>
</ol>
<div>
This sets up a dumb netcat UDP listener on NTP port 123. It will log all connections to the console. The `<span style="font-family: Courier New, Courier, monospace;">-w 1</span>` is the only way I could contrive prevent the first connection from commandeering the process. It will time out the connection after 1 second; then combined with <span style="font-family: Courier New, Courier, monospace;">`-k`</span>, it will keep listening for new connections. The screen is simply for the safety of knowing that this will keep running when I close my laptop, a.k.a. screen daemon. </div>
</div>
<div>
<br /></div>
<div>
Let this run for a day and if you see any traffic in there, you'll know the IP address of the server trying to get at you for NTP info.</div>
<div>
<br /></div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-20787419616833698472016-07-12T14:14:00.000-07:002016-07-12T14:14:26.644-07:00git log exit code 141?Something's that been bugging me for a while now is that when I do a `<span style="font-family: Courier New, Courier, monospace;">git log</span>` command and quit before scrolling through all the output, my terminal prompt shows my last command (the git log) exited non-zero with status code <span style="font-family: Courier New, Courier, monospace;">141</span>.<br />
<br />
I have git configured to use `<span style="font-family: Courier New, Courier, monospace;">less -+S</span>` as the pager.<br />
<br />
The answer turns out to be really straightforward, but took me a little bit of internet searching. I am capturing the learning here to save my future self the research and possibly anyone else typing the exact same thing into Google that I did ;-)<br />
<br />
The explanation:<br />
<br />
<ul>
<li>Git pipes the logs into less</li>
<li>I quit the less process, which sends a <span style="font-family: Courier New, Courier, monospace;">SIGPIPE</span> signal (<span style="font-family: Courier New, Courier, monospace;">13</span>) to the underlying git process streaming the logs</li>
<li>git catches the interrupt and exits prematurely and per POSIX convention returns 128 + the <span style="font-family: Courier New, Courier, monospace;">SIGPIPE</span> status ==> <span style="font-family: Courier New, Courier, monospace;">141</span> to indicate that it was terminated by signal <span style="font-family: Courier New, Courier, monospace;">13</span>. </li>
</ul>
<div>
The background:</div>
<div>
<ol>
<li>http://stackoverflow.com/questions/19120263/why-exit-code-141-with-grep-q</li>
<li>http://www.pixelbeat.org/programming/sigpipe_handling.html</li>
<li>http://unix.stackexchange.com/questions/99112/default-exit-code-when-process-is-terminated/99134#99134</li>
</ol>
</div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com3tag:blogger.com,1999:blog-7398379322753724627.post-59776111741212822592016-06-02T12:19:00.000-07:002016-06-02T12:19:44.165-07:00Yes, git, I did mean thatGit comes out of the box with a nice "auto correct" feature, but sometimes it's not correct. And other times, the first suggestion isn't the one you intended, but the second or third.<br />
<br />
Here's a short, fun shell function (tested in zsh and bash) that will run your last git command and figure out what you meant to do and then run it for you.<br />
<br />
<script src="https://gist.github.com/papaben/463d72bb4ea329754566fa82ef0c949e.js"></script>
<a href="https://gist.github.com/papaben/463d72bb4ea329754566fa82ef0c949e">https://gist.github.com/papaben/463d72bb4ea329754566fa82ef0c949e</a><br />
<br />
<h2 style="background-color: white; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
Parity with auto-correct</h2>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 14px; line-height: 22.4px; margin-bottom: 16px; overflow: visible !important;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 11.9px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ git dif
git: <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>dfif<span class="pl-pds" style="box-sizing: border-box;">'</span></span> is not a git <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">command</span>. See <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>git --help<span class="pl-pds" style="box-sizing: border-box;">'</span></span>.
Did you mean this<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>
diff
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> idid <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!!</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span>shows diff<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span></pre>
</div>
<h2 style="background-color: white; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
Choose non-default auto-correct suggestion</h2>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 14px; line-height: 22.4px; margin-bottom: 16px; overflow: visible !important;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 11.9px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ git pu
git: <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>pu<span class="pl-pds" style="box-sizing: border-box;">'</span></span> is not a git <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">command</span>. See <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>git --help<span class="pl-pds" style="box-sizing: border-box;">'</span></span>.
Did you mean one of these<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>
pull
push
p4
$ idid -2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!!</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span>runs git push<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span></pre>
</div>
<br />
<br />Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-14890313649391774712016-05-19T14:02:00.000-07:002016-05-19T14:02:08.530-07:00Starting an Open Source Office within your CompanyI had the privilege of sharing the stage yesterday with other members of the TODO Group to share our reflections from running Open Source Offices within our respective companies. See the post on the <a href="http://todogroup.org/blog/oscon-talk/">TODO blog</a>.<br />
<br />
In my portion of the talk, I focused on the questions and challenges that might face someone attempting to start up an Open Source presence for their company.<br />
<br />
<h3>
Grass Roots?</h3>
<br />
The first thing you're faced with deciding is if you should start from the bottom up or the top down. Ultimately, you'll want the executive support from the top, but when you're just getting started you can function without it. Instead, find a good first project to be the poster child. Use this to build excitement and draw attention. Within your engineering team, it's very likely that most folks will be interested in Open Source and want to be involved. It's also just as likely that they'll have no idea how to do that. The poster child will serve as an example of not only how to do it, but that it is possible.<br />
<br />
Start working on how to fit thinking about Open Source into the Software Development Life Cycle, for both outgoing open source and incoming open source software.<br />
<br />
Find someone who will champion OSS for you, be that person, or share the role.<br />
<br />
<h3>
Good Intentions</h3>
<br />
The percentage of folks who tell you they are interested in sharing their software is far greater than the percentage of folks who will put in the time to share it. Realizing this upfront will save you a lot of disappointment and undelivered work.<br />
<br />
Figure out how to embed OSS into the DNA of your new projects. Learn the build systems that your developers use and make it easy for them to fork small, cohesive modules into open source ready projects that can be composed back into their larger, internal work. For example, break out small python libraries into pypi projects.<br />
<br />
<h3>
Straight Outta Compton</h3>
<br />
Open sourcing a software project successfully as a company is more than just creating amazing code. There no trivial amount of red tape to traverse from legal, to security, to IP. And after that there is promotion. Imagine if Eazy E had never met Jerry Heller. There might never have been an NWA as we know it or modern day rap. The role of the Open Source Office is to be the guide through all the red tape and promotional process to your developers that Jerry Heller was to NWA.<br />
<br />
<br />
<h3>
Who Cares?</h3>
<br />
Success with OSS is a living thing and it needs to be tended to. It won't be on the roadmap for most teams and it won’t be a behavior for which time is allotted. In other words: it's unlikely that most developers will be gratuitously spending their time on OSS.<br />
<br />
The role of the OSS office is to continually drive engagement within the company. Regularly promote successes in the community or within the company. Maybe you just got a great new hire because of an OSS project or maybe one of your engineers was just recognized in the tech news. Maybe a talented employee previously on the verge of leaving renewed their commitment to the company because of engagement with OSS. Share these successes with the whole engineering team or within the leadership group.<br />
<br />
Make the time to meet with team managers to see what projects are coming up in their backlogs. Brainstorm how OSS could be a part of that.<br />
<br />
<h3>
Community</h3>
<br />
Finally, it's critical to keep your community happy. Never release a project without thinking about promotion and adoption, easy install, good README, etc. Keep an eye on the metrics on your projects. Celebrate healthy projects. Respond to GitHub Issues in a timely manner. Encourage pull requests! Coach your developers to be polite, be thoughtful, and to engage with the community. Create mailing lists. Consider adopting a code of conduct. Be a responsible member of the OSS community and remember that as an enterprise / company, you are setting an example and representing more than simply your company.<br />
<br />
<br />
<h3>
Final Note</h3>
<br />
These were the five points that I found most prominent from my time running <a href="http://opensource.box.com/">Open Source at Box</a>. However, the list of other reflections is much longer. Please see the presentation for some of the other takeaways from other members of the TODO Group. The role of Open Source Office is very critical to any company interested in being part of the OSS community, whether as a consumer or producer.<br />
<br />Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0Austin, TX, USA30.267153 -97.74306079999996729.828484 -98.388507799999971 30.705822 -97.097613799999962tag:blogger.com,1999:blog-7398379322753724627.post-13878467396502873042016-04-02T17:34:00.000-07:002016-04-02T18:06:19.774-07:00Building Successful Teams (it's not just your manager's job)<div>
Over the past year and a half I've been managing a small team of software developers. This past week I was reflecting on some of the practices we've developed as a team that I feel have contributed to our success as both a team and as individuals. Notwithstanding the likelihood that these are not new ideas, I wanted to collect my own reflections in a single, memorable-to-me place.</div>
<div>
<br /></div>
<div>
Success of the team is not measured only by the projects we have completed, but also by the steps each of the team members has made in their careers, personally and professionally. I feel that the practices outlined below are designed to achieve both of these aspects of success.</div>
<div>
<br /></div>
<h2>
<span style="font-size: x-large;">Our Practices</span></h2>
<div>
A rough overview of our formula is below. I've tried to keep the descriptions concise, but several of these points do merit their own blog posts.<br />
<br /></div>
<h4>
Sidenote On Applicability</h4>
<div>
These practices evolved in a team of 3 people in an org of > 100 people in a department of > 300 people in a company of > 1000. There's a completely fair chance that some of these practices won't make sense in small companies (i.e. less than Dunbar's number) or very young companies.</div>
<div>
<br /></div>
<h4>
Recurring Events</h4>
<div>
<div>
<ul>
<li>Weekly 1:1's with manager. Time for the manager to listen and the IC to talk. Check in on short term career development goals. Check in on impediments. Check in on any drama with other employees. See some more ideas, <a href="https://www.quora.com/What-are-some-good-tips-for-1-1s-with-your-employees">Good Tips for Your 1 on 1</a>.</li>
<li>6-weekly 1:1 offsite lunch to discuss career fit and plan. A deeper dive into long term career goals. A time to build rapport. Again, the IC should do most of the talking; though this is also a chance for the manager to provide guidance and reflections from their own experiences.</li>
<li>4-weekly team offsite lunch. A chance to get out of the office and just be regular people. The fresh air and less familiar surroundings will be good for your psychology.</li>
<li>4-weekly team lunch onsite. On the in between weeks from the offsite lunch. This just makes sure you are getting together and seeing each others faces.</li>
<li>6-weekly team scrum retrospective. It is amazing what you learn about your team when you do this, both good things and bad things. The regular cadence also helps with accountability.</li>
<li>Weekly status emails for celebration, retrospection, and future (i.e. next week). At the least, these go to the team. I like to include my boss and other parties who helped play a part in recent success or have some stake (approvers, consulted, informed, etc.) in ongoing work. This is adopted from the <a href="https://en.wikipedia.org/wiki/Progress,_plans,_problems">PPP Concept</a>.</li>
</ul>
<div>
<br /></div>
</div>
</div>
<h4>
Infrastructure</h4>
<div>
These tie a team together and provide them the necessary context to self-navigate.</div>
<div>
<ul>
<li>Team vision & purpose and company fit</li>
<li>Clear short term priorities. This is generally something visible like a Scrum or Kanban board. This is good for the team itself to use, but also for anyone else interested in what your team is doing.</li>
<li>Clear long term goals. These should fit with team vision and purpose. Typically, these will be the overarching epics comprising the short term priorities.</li>
<li>Rubric for roles on the team. At its worst, absence of clearly defined expectations for each role and level, can lead to feelings of unfairness or favoritism. At its best, it provides career goals for team members and acts as a guide for what is and (as importantly) is <i>not </i>expected of them. </li>
</ul>
</div>
<h4>
Ongoing</h4>
<div>
<ul>
<li>Constant feedback, both good & bad. This builds respect in both directions. This also helps ensure transparency.</li>
<li>Fairness. Managers should be transparent about decisions made regarding other employees. If you feel you are being preferentially treated, speak up. If you notice it, your teammates probably do and doubtless will not appreciate it.</li>
<li>Outward communication to other teams about what you've accomplished. This is a step beyond the weekly status email. These happen anytime a major goal is achieved or an epic is completed. There is so much going on in most companies, that achievements get lost in the noise and too often only failures get any attention. If a manager isn't shouting from the rooftops about the team's success, then it's unlikely that anyone else will be.</li>
<li>Information pass downs from the manager's manager. Lots of decisions get made on high and awareness of that information about those can help your team be more successful.</li>
<li>Commitment to finish your short term priorities. New needs will come up that may be more important and vie for the team's attention & time. It's critical that you remain committed to complete your short term priorities, without this you lose trust and faith in the team. Furthermore, knowing you'll complete your commitments makes you more thoughtful about what you'll commit to.</li>
<li>You are held accountable. Whether you're a manager or an IC, if you're not held accountable then your work will suffer.</li>
</ul>
</div>
<h4>
Outside the Team</h4>
<div>
<ul>
<li>You should ask a manager for help finding a mentor.</li>
<li>You should identify someone in upper leadership who shares your view points and is driven by the same things as you. There will be situations when you need a second opinion or someone to back you.</li>
<li>Try to cultivate a <a href="http://www.forbes.com/sites/danschawbel/2013/09/10/sylvia-ann-hewlett-find-a-sponsor-instead-of-a-mentor/">sponsor</a>.</li>
</ul>
<div>
<br /></div>
</div>
<h2>
<span style="font-size: x-large;">
Parting Thoughts</span></h2>
<div>
Never sacrifice these things. Skipping one or two of them once or twice might be OK, but it will add up and accumulate into discontent and dissatisfaction and poor performance. You must keep yourselves accountable and hold those around you accountable.</div>
<div>
<br /></div>
<div>
Those lucky ones of us will find a manager who will naturally do these things, but for the rest of us we can't simply claim misfortune or "being a victim" as excuse and then give up. When it's your career on the line, it doesn't matter whose fault it is; what matters is the results you are going to get.</div>
<div>
<br /></div>
<h4>
Additional Learning</h4>
<div>
<ol>
<li><a href="https://en.wikipedia.org/wiki/The_Five_Dysfunctions_of_a_Team">5 Dysfunctions of a Team</a></li>
<li><a href="https://www.youtube.com/watch?v=ReRcHdeUG9Y">Leaders Eat Last</a></li>
<li><a href="http://www.jimcollins.com/article_topics/articles-leadership.html">Built to Last</a></li>
</ol>
</div>
<div>
<br /></div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com1tag:blogger.com,1999:blog-7398379322753724627.post-16691666899108916942015-11-11T22:37:00.000-08:002015-11-11T22:37:35.516-08:00What Makes a Good Engineering Culture?<span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">This question was asked several years ago <a href="https://www.quora.com/What-makes-a-good-engineering-culture/">on Quora</a> and has gotten some great answers! I was going through some of my old drafts and came across an answer that I had almost completed writing and decided to finally push through (better late than never!). I've copied the full text of the answer below.</span><br />
<span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><br /></span>
<span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">--</span><br />
<span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><br /></span>
<span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">In my experience, a company's engineering culture is possibly the most important thing a person can consider when evaluating a job offer. I've seen many Quora questions touch on similar topics (e.g. "Where should I work?", etc.), and I feel like the root of almost all of the answers stems from the culture. Thank you for asking this question.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">My answer to this question is rooted in my ten years of software engineering, numerous blogs & books on the topic, and countless conversations with engineers. It also comes with the expectation that no answer to this question can speak for all software engineers. Everyone has different and changing needs, and with them different and changing views of what makes a good culture.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">So, here goes -- the prevailing concepts that have stuck with me are as follows.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Teams</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">High quality software engineering is the product of a team. No one individual can be expected to deliver, nor take credit for, a successful product on his or her own. This gets fuzzy in small startups where there may only be one engineer, but otherwise holds true. A culture that celebrates one individual at the cost of another is making a grievous mistake.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">There is an important distinction to note here about what comprises a *</span><b class="" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">team</b><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">* of engineers versus a group. The distinction is that a group is not a team until everyone in the group is committed the purpose [1]. In my experience, this commitment comes from inspirational leadership and transparency. The fact that an engineer is employed by a company is not reason enough to incite the determination, dedication, and thoughtfulness necessary to produce quality code. Committed engineers are engineers who are proud to work where they work and excited to talk about their jobs and their company's products.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Stake in the Product</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">A healthy culture builds a product that means something to them. This can happen in a lot of ways, one of which is by connecting the engineers with the users. Some examples are providing engineering the opportunity to sit with the customer support team, to join sales or product folks on customer visits, or to attend company conventions. As an engineer, it is such a rewarding feeling to meet a customer and hear their story about how a feature you built makes their life better or how the bug you fixed made their day. This kind of connection is what makes the software you build at your job significantly more meaningful than any college project or homework assignment.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">The next critical piece of this is a healthy relationship between the product and engineering teams. I've always been baffled by product management teams that treat engineers like they are incapable of contributing to product design. Product and engineering need to function together like two sides of a brain; and respect needs to flow in both directions. Giving engineers a say in the product design simultaneously gives them stake in outcome. People are much more likely to care about the success of something they helped design versus something they were told how to do.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Equally, a culture that does not respect the purpose of its product team is no better off than one that does respecting engineering. The truly successful cultures of which I have been a part were made up of cross functional teams of product & engineering talent working together to set short term and long term goals; write achievable design specifications; and build software that makes both groups proud.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Experiments</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Next up: Experiments! Happy cultures promote experimenting with features over endlessly debating them [2]. One company where I've seen this really done well is Yammer, where each feature has a clear definition of success, but is assumed to be unnecessary until proven otherwise by objective usage metrics and customer feedback [3]. This precludes endless debates or analysis paralysis, and effectively lets product and engineering teams focus on what matters: building software. A positive side effect of writing software that inherently supports experimenting is that it encourages a culture that commits to small, manageable deliverables.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Encourage Learning</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Another hallmark is a culture that encourages establishing a *</span><b class="" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">deep understanding</b><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">* of the tools, work flows, and responsibilities that go into producing production ready software. A team that knows __</span><b class="" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">why</b><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">__ something must be done in a certain fashion will encounter substantially less mistakes than a team kept ignorant by relying too heavily on process scripts. Intuitively it seems natural to reduce mistakes by automating processes and restricting influence. In my experience, it is a fine balance of the safety in automated process and the danger in freedom that produces a well rounded team more apt to optimize the whole rather than their own piece of it. Automated process saves time and avoids unintended errors, but that intentionally dangerous freedom lets the engineers know that they are respected and responsible for the success of their own products. That responsibility provides the motivation to go the extra mile to truly understand, for example, how a service is going to function in a production environment, or how version control really works.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Deployment Is Just the Beginning</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Engineering is more than development, it's also deployment and support. Time spent in development is just a fraction of a project's lifetime. The majority of a product's life is in production. Engineering teams that fail to structure priorities around this concept will either produce poor quality products or endlessly miss deadlines because they are fixing bugs.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Learn From Failure</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Nothing is perfect. Even rocket scientists at NASA make mistakes [4]. Expect failure and plan for it, in your code and in your culture. Learning from failures and *</span><b class="" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">improving</b><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">* from them makes a team stronger. I won't say too much about this because much has already been said elsewhere. Etsy has published some great material on this subject [5] and [6]. The Pragmatic Programmers also published a great book on ways to mitigate failures in production [7]. For my part, I'm proud of how we learned to learn from failure at Box [8].</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br />
<h2 class="" style="color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 19px; margin: 0px; padding: 0px;">
Finally</h2>
<br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">Ultimately, all of this represents my own opinion based on my time spent engineering software. One prescription does not fit all and any prescription should be an ever changing vision.</span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[1] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" data-tooltip="attached" href="http://www.amazon.com/exec/obidos/ASIN/0321620704/poppendieckco-20" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">Leading Lean Software Development: Results Are not the Point: Mary Poppendieck, Tom Poppendieck: 9780321620705: Amazon.com: Books</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[2] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" href="https://news.ycombinator.com/item?id=1463751" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">Always ship trunk: Managing change in complex websites</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[3] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" data-tooltip="attached" href="https://pando.com/2013/02/28/why-yammer-believes-the-traditional-engineering-organizational-structure-is-dead/" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">Why Yammer believes the traditional engineering organizational structure is dead</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[4] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" href="http://www.goodreads.com/book/show/18007564-the-martian" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">The Martian</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[5] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" href="http://conferences.oreilly.com/velocity/devops-web-performance-2015/public/schedule/detail/41680" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">Failure is an option - Velocity 2015</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[6] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" data-tooltip="attached" href="http://www.kitchensoap.com/2013/09/30/learning-from-failure-at-etsy/" rel="noreferrer nofollow" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">Kitchen Soap – Learning from Failure at Etsy</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[7] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" href="https://pragprog.com/book/mnee/release-it" rel="noreferrer" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">The Pragmatic Bookshelf | Release It!</a></span><br style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;" /><span style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;">[8] </span><span class="qlink_container" style="color: #333333; font-family: Georgia, Times, 'Times New Roman', serif; font-size: 15px; line-height: 21px;"><a class="external_link" data-tooltip="attached" href="https://www.box.com/blog/three-letters-saved-my-tech-debt/" rel="noreferrer nofollow" style="background-attachment: initial; background-clip: initial; background-image: url(data:image/gif; background-origin: initial; background-position: 100% 5px; background-repeat: no-repeat; background-size: initial; color: #2b6dad; padding-right: 12px; text-decoration: none;" target="_blank">The Three Letters that Saved My Tech Debt</a></span>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-44817292881560192902015-11-08T20:08:00.001-08:002015-11-08T20:08:26.771-08:00Managing Runtime Configurations<h1 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.05714rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
Configuration Headaches</h1>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Managing runtime application configurations in large scale, heterogeneous environments is a total pain. Over the last five years I have attacked this problem in various ways, each with its own grace and flaws. The goal of this post is to sum up the evolution of my experience and hopefully impart some insight to any folks in similar plight. </div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
It starts with the challenges:</div>
<ol style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Different environments get different configurations</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Different services in the same environment get different configurations</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configuration files get massive and treacherous</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configuration files get complex quickly with cross references</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configurations have no safety against typos in keys (or values)</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configurations have essentially no type safety guarantees</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configurations have no README indicating their intent or usage</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Configurations can be accessed from <em style="box-sizing: border-box;">anywhere</em> in a code base (or outside) in inconsistent manners</li>
</ol>
<h4 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
Sidebar: Defining Runtime Application Configurations</h4>
<h2 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Sidebar_Defining_Runtime_Application_Configurations_13" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a></h2>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
For the context of this article, I use the term “runtime application configurations” to represent the configurations used by an application at runtime. I do <em style="box-sizing: border-box;">not</em> mean configuration as code type things like Puppet, Chef, or Ansible. For example, I would consider the log configurations in a<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">logger.xml</code> file for a JVM application that I own to be runtime app configs. I would not consider the configuration of Apache on a web server a runtime app config in this context, I would consider that a <em style="box-sizing: border-box;">system</em> configuration. More specifically, I would consider it configuration for an application whose source code I'm not writing.</div>
<h1 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.05714rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Evolution_of_a_solution_17" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Evolution of a solution</h1>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
Compound keys</h3>
<h2 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Compound_keys_19" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a></h2>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The most popular convention I've observed for managing configurations is by flat file such as ini, json, yaml, xml; very rarely is it in the same language as the code consuming it. My assumption is that this accomplishes a few things:</div>
<ol style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">They can be edited by non-programmers</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">They can be consumed by multiple programming languages, thus freeing operations teams from managing multiple manifestations of the same configuration values</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">In the case of compiled code, configurations do not need to be compiled (or recompiled when they change)</li>
</ol>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
This restriction is very powerful in the guarantees of simplicity, but also limiting. One major limiting factor is that without a separate management system, it’s impossible to have different values of configurations per runtime environment. This was a major problem for us since we shared configuration files between all of our environments, but need to set different values for the same configuration key based on the environment.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Our initial solution was to encode the environmental context onto the keys such that we could effectively define unique values for a given key based on the environment that would use it. So instead of a single key-value pair, we would have multiple key-value pairs differentiated by environment. See an example of log levels for different environments.</div>
<pre style="background: linear-gradient(rgb(255, 255, 255) 0px, rgb(255, 255, 255) 0.75rem, rgb(245, 247, 250) 0.75rem, rgb(245, 247, 250) 2.75rem, rgb(255, 255, 255) 2.75rem, rgb(255, 255, 255) 4rem) rgb(255, 255, 255); border-radius: 4px; border: 1px solid rgb(211, 218, 234); box-sizing: border-box; color: #333333; font-family: monospace, monospace; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; overflow: auto; padding: 0.66001rem 9.5px 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-ini" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: monospace, monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[logger]
level<dev> = DEBUG
level<staging> = INFO
level<prod> = WARN
</code></pre>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
When the INI gets parsed natively, each of the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">level</code>'s will be parsed as unique keys. The next step is custom code in the application to split the environments (e.g. <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;"><dev></code>) off of the keys and then figure out which environment and value applies in its running context.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Sound complicated? It is. Making this work involved writing some complicated INI parsing <a href="https://github.com/box/bart/blob/master/src/Bart/ConfigResolver.php" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">code</a> as well as figuring out a way to tell an application the environment in which it’s running. The result was a brittle system that was error prone and slowed down onboarding. Furthermore, if any configurations needed to be shared between projects, those projects also needed to solve the parsing and environment setting problems.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
Moving away from flat files</h3>
<h2 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Moving_away_from_flat_files_39" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a></h2>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I asked myself, “Why <em style="box-sizing: border-box;">are</em> configurations always in flat files anyway?” I couldn’t come up with a convincing answer, so my next attempt was to write a straightforward configuration framework in PHP that expected different values per each key based on environmental context. All configuration files were PHP files that returned a large array. I experimented with model objects that could build configurations, but ultimately discarded the idea because I felt the time to solve the edge cases would outweigh the incremental value.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
With this system, the mind bending key-to-environment relationship was slightly more clear. Another gain was removing cross references to other keys within the INI values. Since the values were set with PHP, it was possible to reuse values or base other values off each other, e.g. <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">url = "{$scheme}{$domain}{$path}"</code>.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Although the system was less brittle and easier to understand, the configuration files themselves were still rather large and it wasn’t clear how defaults worked between environments. Worse, the files could only be used by PHP applications.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
A separate configuration management system</h3>
<h2 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Separate_configuration_management_system_47" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a></h2>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
My biggest problem was the size and complications of putting configurations for every environment in the configuration files. So I decided to look into using a separate system to manage the differences configurations, which could then deliver only the necessary content to a given consumer. I landed on Puppet since we already use it extensively elsewhere. This allowed me to write files containing <em style="box-sizing: border-box;">only</em> the configurations that mattered in the respective environments. Based on this, going back to flat files was possible. I could also tie together configuration values in Puppet before writing the files, so there was no need for cross references within the configuration file itself.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
This was great, my first 5 problems were solved. But still, type safety was not enforced, configuration files had no guaranteed documentation, and understanding how they were used within an application was a grep nightmare.</div>
<h3 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
Configuration model objects</h3>
<h2 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.95312rem; line-height: 3rem; margin-bottom: 0.183584rem; margin-top: 0px; padding-top: 0.816416rem;">
<a href="https://www.blogger.com/null" id="Configuration_model_objects_53" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a></h2>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I decided to revisit configuration model objects. However this time not as builders, but instead as accessors. By funneling all access of configurations through a single point, I figured that I could enforce a few things.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I buried the configuration parsing code inside an abstract class (aptly named <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;"><a href="https://github.com/box/bart/blob/master/src/Bart/Configuration/Configuration.php">Configuration</a></code>) with protected methods for getting at the parsed values. Configuration model classes could then extend the base and expose relevant methods for their configurations. For example, a logging configuration class would have a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">getLevel()</code> method, which behind the scenes would parse the configurations file and return back the value. Any user that wanted to load configurations could only do so by using a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Configuration</code>class, or writing a new one. Next, I tied the name of the configuration file to the name of the class, such that if a class were named <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">LogConfigurations</code>, the configuration file behind the scenes needed to be named <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">log.conf</code>. Finding usages of a given configuration file became trivial.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I added type expectations methods to the parsing code so that invalid values would raise exceptions. The<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Configuration</code> class requires that all extending classes implement a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">README()</code> method, so that users understand the classes intention and expected INI contents. This is coupled with a generic unit test that parses the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">README()</code> output and ensures it can actually be used by the class. Furthermore, each of the accessor methods has its own documentation.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Configuration model classes also opened the door for more sophisticated configurations. Since access is inside a class, and not just array referencing, the class can do smart things like tie multiple values together or call out to other classes. One really great manifestation of this was the ability to create the <a href="https://github.com/box/bart/blob/master/src/Bart/Configuration/ProjectConfiguration.php" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">ProjectConfiguration</a> class which uses the project version control to load a configuration file’s contents.</div>
<h1 style="-webkit-font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.05714rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Get_involved_63" style="background: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Get involved</h1>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
That’s where I am today. It’s been a fun journey and I’m happy (atm) with where things are. I’m excited to see where this goes next and to learn other ways folks have found to solve this problem.</div>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
One key component that I’d like to see next is the configuration delivery mechanism. Currently, Puppet solves the problem reasonably for short lived PHP processes. However, the two areas that I’d like to improve are:</div>
<ol style="background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">Automatic reloading by long lived processes. Maybe this is just from a file watch.</li>
<li style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; box-sizing: border-box; margin-left: 1rem;">A better interface for managing the configurations. This could get extensive. One tool here would be validation. This could be much more than just type checking, but semantics and plugins for special configuration groups (e.g. validate a database in a slave section is up not a master). Typos in keys would be completely avoided. Another would be the ability to <em style="box-sizing: border-box;">see</em> what values a given environment would get.</li>
</ol>
<div style="-webkit-font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; background-color: white; box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; line-height: 28px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Please post back with comments or pull requests!</div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-60873435022305369772015-05-28T20:52:00.000-07:002015-05-28T21:20:19.738-07:00Make doing the right things easy enough that the doing the wrong things is embarrassing<div class="separator" style="clear: both; text-align: center;">
<a href="https://farm2.staticflickr.com/1009/1406434416_780880c682_o.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="265" src="https://farm2.staticflickr.com/1009/1406434416_780880c682_o.jpg" width="400" /></a></div>
<br />
<h2>
Make the right things easy</h2>
<br />
It's not a new concept: doing the right thing should be easy, doing the wrong thing should be hard. In blogs, books, or management guides that's the advice you get for preventing people from doing the wrong thing. In practice, this is a subtle science and I believe this little apothegm needs some tweaking. My recommendation is "<i>Make doing the right things easy enough that the doing the wrong things is embarrassing.</i>"<br />
<br />
<h3>
What's the difference?</h3>
<br />
In practice, making the wrong things hard can of itself be hard. Often the wrong thing is an established practice or convention. It may be the only way some people know how to get things done. Even worse, it may just be easy. When you're faced with this kind of ill luck, you need to look at the problem from a different angle. Making the right things easy becomes a cultural endeavor.<br />
<br />
<br />
<h3>
The subtle part</h3>
<br />
It starts with the leadership. Messaging from the leadership has a direct impact on the goals and priorities of the engineering workforce. If engineers know that taking longer to do the right thing is encouraged, then there is a much higher likelihood they will take that path than if that encouragement is absent. Leadership can reinforce the message by rewarding the engineers who go above and beyond to do the recognized right thing in challenging situations. They can sponsor development of tools that will make doing the right things easier.<br />
<br />
All of this will *encourage* good behavior, but it won't do too much to discourage bad behavior.<br />
<br />
<h3>
For every encouragement there is an equal and opposite discouragement</h3>
<br />
It's important to me to be clear about what I mean when I say "embarrass." I do not mean that leadership or any team member should explicitly embarrass another team member. A culture that publicly shames or embarrasses its team is caustic, foments fear, and dampers collaboration [1]. You don't want that kind of culture, but you don't want to hide the bad stuff either. My suggestion is that the trick is in believing in good intentions.<br />
<br />
Unless your hiring process is extremely dysfunctional, you should be able to assume your engineers have good intentions. That means that by encouraging the desired behavior, people will naturally be discouraged from doing otherwise. As your tools and processes continue to make the encouraged behaviors easier, it will become increasingly difficult to justify doing the wrong things.<br />
<br />
An example of this is using leader boards to send a message. Consider a situation in which you want to encourage your engineers to write more unit tests. First, you invest in making writing tests *really* easy. Then, you start tracking unit test code coverage per commit. Soon thereafter, you can publish a leader board of "person to % code coverage per commit." You won't be explicitly calling anyone out for habitually not adding providing reasonable code coverage, but it will be clear over time that certain individuals may not be doing so [2].<br />
<br />
<h2>
Conclusion</h2>
<br />
I hope this opens up a healthy discussion about how we can stop complaining about bad behaviors and start feasibly changing them today.<br />
<br />
<h2>
Footnotes</h2>
<br />
[1] See <a href="https://www.startwithwhy.com/tabid/81/Default?ProductID=lelbook">Leaders East Last</a> by Simon Sinek for a great exposition of this.<br />
<br />
[2] Note I don't believe in metrics that track raw quantity. I don't feel that such measures accurately capture the intention. Metrics that people can be proud of should track the quality of their work over time, not "gamable" counts that are easily subverted.<br />
<div>
<br /></div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-83045930523773129672014-07-02T23:51:00.001-07:002014-07-21T21:24:46.667-07:00Hey, Don't Fire that GuyA few weeks ago at Silicon Valley DevOps Days, in an open spaces discussion about outages, somebody asked a question that has been on my mind ever since. The question was as follows, "has anyone ever fired someone for causing an outage?" It's not a new question, but the verdict is still out on the right answer.<br />
<br />
Some of my favorite insight on the topic comes from a post by <a href="http://www.kitchensoap.com/2013/09/30/learning-from-failure-at-etsy/">John Allspaw</a>. In the post, he states that in order to learn from a mistake without inhibition, there must be not only complete trust but also no fear of retribution. I take this view to heart and I'm a devoted defender of the concept.<br />
<br />
However, I feel like the view comes with a few provisos. There is possibly a greater philosophy at play in the question. What really got me thinking about this was one of the stories I heard at the conference. The story was about a site outage caused by a technician on the customer support team running a tool written by the dev team, which was known to have a very high risk of causing site outages. The technician had been trained in how to use the tool and, according to the story teller, knew full well the consequences of his actions. So, for the company it was a straightforward decision to fire him. ...Or was it?<br />
<br />
<span style="font-size: large;">Human Error</span><br />
<br />
If you read about human error, you learn that situations within a system and organization can foster human errors. This is often the first explanation of a failure induced by a human. In our investigation here, it's difficult to imagine any resident situation in the system or org which would have fostered a technician to *knowingly* induce an adverse outage of the application he was paid to support. So, that lead me next to consider that perhaps the technician felt running the script would have no connection to an outage. But again, having been fully trained in the consequence of the script, this felt unlikely. Perhaps a final explanation might be that the technician felt the risk of outage justified the outcome of running the tool. Maybe; but from the storyline, it didn't seem like that was a likely decision.<br />
<br />
So, were the theories about human error just somehow inapplicable to this situation?<br />
<br />
<span style="font-size: large;">Theory Prerequisites</span><br />
<br />
That's when I realized something about each of the questions. I realized that they were posited on the assumption that the technician was qualified for and satisfied with his job. I realized that if the technician didn't care about his job or hated his boss, then all of my thinking was flawed. All the explanations are based on the belief that the technician wants what is best for the company. If he had no thought for the success of the company, then it would make easy sense for him to take the shortest path to accomplishing an immediate goal despite negative side effects. So, rethinking the first question -- could an organizational, <i>non-technical</i> situation foster human errors? Yes, I believe so. A non-technical situation could certainly foster feelings of dissatisfaction or frustration resulting in a technician who didn't care about the outcome of running a tool that would solve his immediate problem, but potentially cause a site outage.<br />
<br />
Excellent! So, the key to success is maintaining an inspired work force, right? Partly, yes; but not exactly.<br />
<br />
<span style="font-size: large;">For the Workforce</span><br />
<br />
If the company is fostering a disgruntled workplace, then the solution is straightforward: keep them happy and inspired. But, what if your employees <i>are </i>happy? How could the tech's actions be explained?<br />
<br />
The next idea that occurred to me was perhaps it wasn't the organizational situation at all, but actually the employee himself. What if the employee was generally just a disgruntled or careless person? DevOps and other strong cultures are built upon trust, but that trust needs to be built on the prerequisite that you hire the right people. If poor hiring choices are made, the whole ecosystem breaks down.<br />
<br />
<span style="font-size: large;">A Different Type of Outage</span><br />
<br />
As it turns out, although hiring the right people is such an important task to do well, it's often underdone and rushed. And why? For the exact same reasons we end up with technical debt: urgency. If we've learned anything from technical debt, we know that we create it to solve an immediate problem, but that solution leaves us in a much worse long term situation. If we know this is true, then maybe we should treat firing an employee the same way we treat a site outage. Specifically, we should postmortem all the steps that lead up to hiring: the phone screen, the recruiter's notes, the interviews, the round-up, the closing. I think that if firing were treated as a much costlier action, then perhaps hiring would be conducted with a much higher bar. Ultimately, in the case of the fired technician, perhaps the fault lays more with the hiring manager for putting the tech in the situation to use the tool at all.<br />
<br />
So, to bring it back to the original question: would you ever fire anyone who caused an outage? My answer is a lot more complicated now. I will continue to apply the concepts of human error and would never fire an employee <i>only</i> because of an outage. Would I fire an employee who consistently underperforms, is habitually unhappy and careless, costs more than he or she produces, continues to fail at personal performance improvement plans, and is uninterested in other roles within the company? Very possibly. Whatever the case, if an engineer is fired for <i>any </i>reason, I want to postmortem how we came to the situation, starting with the initial phone screen in the hiring process.<br />
<br />
<span style="font-size: large;">Open Questions</span><br />
<br />
What metrics can drive improvement?<br />
<div>
<br /></div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com2tag:blogger.com,1999:blog-7398379322753724627.post-80797224320165116782014-05-04T11:04:00.001-07:002014-05-04T11:04:08.156-07:00Debugging Gitosis "Read Access Denied"I recently set up <a href="http://git-scm.com/book/en/Git-on-the-Server-Gitosis">gitosis</a> to serve some side projects that I'd like to share with a few friends. I've used it in the past professionally and really enjoy the sanity it brings to managing users and permissions.<br />
<br />
Things started off pretty well and I was committing and pushing changes in no time. A week or so passed and I wanted to add a new user to a project. I made the necessary changes to my clone of the <span style="font-family: Courier New, Courier, monospace;">gitosis-admin</span> project, but when I tried to push my changes upstream, I suddenly I was unable to push! This was a major issue since the admin project is the heart of the configuration.<br />
<br />
I put on my spelunking hat and ssh-ed to the box and switched to the git user to start debugging. The first thing I did was revert the <span style="font-family: Courier New, Courier, monospace;">gitosis.conf</span> file back to it's original state. You can find this file in <span style="font-family: Courier New, Courier, monospace;">~/git/repositories/gitosis-admin.git/gitosis.conf</span>. Changing it had no effect.<br />
<br />
I took a closer look at the error message from my failed push command, and noticed that it was complaining "Read Access Denied," but for a<i> different user name</i> (I could see this because I had <span style="font-family: 'Courier New', Courier, monospace;">loglevel = DEBUG</span>). There are a total of three users involved in the projects, one of which I'd just added and only locally. So, on the server, there were only two users at play. OK, so maybe that user is causing issues. I next removed his key file from the server. This file was at <span style="font-family: Courier New, Courier, monospace;">~git/repositories/gitosis-admin.git/gitosis-export/keydir/</span>. <br />
<br />
I tried to push again. No luck.<br />
<br />
Hmm, next I looked into <span style="font-family: 'Courier New', Courier, monospace;">~/git/.ssh/authorized_keys</span>. I found there was a still a reference to the user there, so I deleted that line.<br />
<br />
I tried to push again. It worked!<br />
<br />
Ok, so are things working now? I tried to fetch. No dice.<br />
<br />
So, when I pushed, gitosis re-applied the configuration and undid all of my debugging steps. Essentially reverting the system back to the previous state, including new additions for the new user.<br />
<br />
At this point, it dawned on me to check my ssh agent identities. Lo and behold, I had <i>two </i>identities and one of them was for the other user! Oops! This was completely my mistake. I had generated his keys a few weeks ago and tested them to ensure they worked. Apparently I had not been so thoughtful as to delete the identity when done.<br />
<br />
After running <span style="font-family: 'Courier New', Courier, monospace;">ssh-add -D</span>, things started working again.Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-17332159220244806712014-03-31T16:48:00.000-07:002014-03-31T16:48:28.311-07:00Use Jenkins REST API to Update Job Configurations AutomaticallyI manage several Jenkins jobs for a single git repository that has several modules (i.e. directories). Each of the modules has the same structure and the job for each of the modules is essentially the same, differentiated only by the name of the directory.<br />
<br />
I recently needed to make a change to each of the job configurations and rather than do so by hand, I thought I'd investigate the possibility of doing so automatically using the Jenkins REST API.<br />
<br />
I had originally created all the jobs using the API by posting to the generic the <span style="font-family: Courier New, Courier, monospace;">http://jenkins.example.com/createItem</span> URI. I had since moved the jobs into <a href="https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Folders+Plugin">folders</a> and hitting <span style="font-family: Courier New, Courier, monospace;">createItem</span> only created new jobs and did not update the proper jobs.<br />
<br />
I did some searching on the internet and couldn't find any help. I eventually found my answer on the api page for the folder. I'm posting my code below for anyone else with similar questions.<br />
<br />
<pre class="brush:bash"># First, get the http://jenkins.example.com/job/folder-name/job/sample-job--template/configure looking like you want
read -s token
# type token from http://jenkins.example.com/user/$userName/configure
# Download the configuration XML for the template job (which will be our model template)
curl -v -u "bvanevery:$token" http://jenkins.example.com/job/folder-name/job/sample-job--template/config.xml > generic-config.xml
# My modules
declare modules=('module1' 'module2' 'module3')
# POST the updated configuration XML to Jenkins
for m in ${modules[@]}; do
echo "module $m";
sed "s/MODULE/$m/g" generic-config.xml > $m-config.xml;
curl -v -X POST --data-binary @$m-config.xml -u "bvanevery:$token" \
-H 'Content-Type: application/xml' \
"http://jenkins.example.com/job/folder-name/job/$m/config.xml ;
done
</pre>
<br />
<br />Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com1tag:blogger.com,1999:blog-7398379322753724627.post-23603425472298718992014-01-07T13:04:00.000-08:002014-01-17T11:54:12.403-08:00Script to Fix Gerrit: LDAP floods log for gerrit-only usersWe recently upgraded to Gerrit 2.7 and started to see lots of LDAP related errors in the logs. We tracked it down to this bug report: <a href="https://code.google.com/p/gerrit/issues/detail?id=1640">https://code.google.com/p/gerrit/issues/detail?id=1640</a>.<br />
<br />
I wrote a quick script to fix the issue and thought I'd share it.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">read -s pwd</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">echo "SELECT external_id FROM account_external_ids WHERE external_id LIKE 'gerrit:%';" | mysql -h db.example.com -u gerrit -p${pwd} reviewdb | sed 's/^gerrit://' > usernames.txt</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for u in $(< usernames.txt); do</span><br />
<span style="font-family: Courier New, Courier, monospace;">if ! id $u > /dev/null 2>1; then</span><br />
<span style="font-family: Courier New, Courier, monospace;"> echo "DELETE FROM account_external_ids WHERE external_id = 'gerrit:$u' LIMIT 1;" | mysql -h </span><span style="font-family: 'Courier New', Courier, monospace;">db.example.com</span><span style="font-family: Courier New, Courier, monospace;"> -u gerrit -p${pwd} reviewdb</span><br />
<span style="font-family: Courier New, Courier, monospace;">fi; </span><br />
<span style="font-family: Courier New, Courier, monospace;">done</span>Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-16926006451403321342013-12-04T01:19:00.000-08:002013-12-04T01:19:14.404-08:00How To Build Gerrit Replication PluginAfter far too long of a wait, we've finally upgraded to Gerrit Code Review 2.7.<br />
<br />
In versions prior to 2.5, the replication feature was packaged with the main war file. <a href="http://www.blogger.com/+%20https://groups.google.com/forum/#!topic/repo-discuss/n6u1xEjFGxU">No longer is this the case</a>. Now, if you want Gerrit to replicate your changes upstream to other repositories, you'll need to add the <a href="https://review.typo3.org/plugins/replication/Documentation/index.html">replication plugin</a> using the command line tool `plugin add`. Sadly, I could not find the replication jar hosted anywhere and it appears that you need to build it by hand.<br />
<br />
I ran into some confusion [1] with this end to end process and didn't find a sufficiently succinct answer online, so I'm writing up my own =) Here are the steps that lead me to successfully building the replication jar and installing it.<br />
<br />
<ol>
<li><span style="font-family: Courier New, Courier, monospace;">git clone <b>--recursive</b> https://gerrit.googlesource.com/gerrit</span></li>
<ul>
<li>You need the <b>--recursive</b> here because the plugins are actually git <i>submodules </i>and won't otherwise be cloned along with your repo.</li>
<li>If you're already cloned, you can run `<span style="font-family: Courier New, Courier, monospace;">git submodule init; git submodule update</span>`</li>
</ul>
<li><span style="font-family: Courier New, Courier, monospace;">cd gerrit</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">git checkout -b stable-2.7 origin/stable-2.7</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">mvn install -DskipTests=true -Dmaven.javadoc.skip=true</span></li>
<ul>
<li>It's not necessary to skip the tests or generating Java Doc, but it will greatly improve your compile time and decrease the amount of memory maven uses</li>
</ul>
<li><span style="font-family: Courier New, Courier, monospace;">cd gerrit-plugin-api</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">mvn package -Dmaven.javadoc.skip=true</span></li>
<ul>
<li><span style="font-family: inherit;">This creates the jar that will be necessary for the replication plugin to get built</span></li>
</ul>
<li><span style="font-family: Courier New, Courier, monospace;">cd plugins/replication</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">mvn package -Dmaven.javadoc.skip=true</span></li>
<li><span style="font-family: inherit;">At this point, you have compiled and packaged the replication jar! All you need to do now is register it with your Gerrit server. For simplicity, I'll pretend your gerrit server is running at gerrit.example.com.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">scp target/replication-2.7.jar gerrit.example.com:/tmp/</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">ssh -p 29418 gerrit.example.com gerrit plugin install -n replication /tmp/replication-2.7.jar</span></li>
</ol>
<br />
<div>
I hope this helps out anyone who was struggling with the same issues as I!</div>
<div>
<br /></div>
<h3>
PS</h3>
<div>
Our Gerrit Code Review server runs inside an environment with no outside internet access. When upgrading Gerrit, the service assumes that it has internet access and tries to download any jars that are not packaged into its bundle. In my upgrade situation, it tried to download <span style="font-family: Courier New, Courier, monospace;">mysql-connector-java-5.1.21.jar </span>from <span style="font-family: Courier New, Courier, monospace;">http://repo2.maven.org/maven2/</span>. It obviously failed.</div>
<div>
<br /></div>
<div>
In order to resolve this issue, I <a href="http://repo2.maven.org/maven2/mysql/mysql-connector-java/5.1.21/mysql-connector-java-5.1.21.jar">downloaded</a> it to a system that had external access and then scp-ed the jar to <span style="font-family: Courier New, Courier, monospace;">$review_site/lib</span> and restarted the <span style="font-family: Courier New, Courier, monospace;">gerrit.war init</span> upgrade process.</div>
<h3>
FOOTNOTES</h3>
<div>
<div>
[1] -- Some errors I saw:</div>
<div>
<ul>
<li><span style="background-color: #f1c232; font-family: Courier New, Courier, monospace;">Maven out of memory</span></li>
<li><span style="background-color: #f1c232; font-family: Courier New, Courier, monospace;">Child module gerrit/plugins/commit-message-length-validator/pom.xml of gerrit/pom.xml does not exist</span></li>
<li><span style="background-color: #f1c232; font-family: Courier New, Courier, monospace;">Child module gerrit/plugins/replication/pom.xml of gerrit/pom.xml does not exist</span></li>
<li><span style="background-color: #f1c232; font-family: Courier New, Courier, monospace;">Child module gerrit/plugins/reviewnotes/pom.xml of gerrit/pom.xml does not exist</span></li>
</ul>
<div>
<br /></div>
</div>
</div>
<div>
<br /></div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-60070784560344216602013-11-22T15:05:00.000-08:002013-11-22T15:05:14.024-08:00Talkin’ ’bout my Generation, or How I Learned More Than I Ever Wanted About JVM Memory<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Profiling a Java application is an experience many developers may never encounter. Identifying the source of a memory leak is probably even more rare. Those kinds of investigations are typically handled by teams dedicated to the subject, or just deferred by throwing more memory at the problem. Until recently, I’ve been one of those lucky developers, blissfully ignorant of what goes on within a JVM. In this article, I seek to share my journey in tracking down a performance issue and what I learned along the way.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
The journey begins with an open source scala project built on the play framework running on a 64 bit SL6 box with Java hotspot 1.6. The code base has a small footprint and is not overly complex. Its external resources consist of a mysql database, an internal solr index for searching, and minimal file system interaction. The critical feature of the app is a REST endpoint that handles very large HTTP PUT requests.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
We’ve been running the app for some time in production under light load with no complaints. Just recently we started importing a large amount of records into the system via the aforementioned REST endpoint. That’s when we started to observe problems. Periodically the service would crash. The error was consistently “java.lang.OutOfMemoryError: PermGen space”. We observed that given enough time, this error was guaranteed to occur. Critically, it wasn’t simply time: it was after enough requests. I determined that the issue wasn’t going to go away and we needed to face it head on. And thus I embarked on my journey.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
The first step was understanding what “PermGen” even means. Research revealed that PermGen stands for “Permanent Generation,” but before that could make sense, I needed to know a little bit more about JVM garbage collection (GC), which is whence the term “generation” comes. Put simply, garbage collection is the process by which old objects get removed from memory (aka “the heap”). Old objects consist of things like class instances that have been created within a function or class scope. When those scopes go away then so should those instances. That makes sense. So in more detail, the actual process of garbage collection consists of sweeping through the heap and determining each object’s classification. The GC has four major classifications for objects named Eden, Survivor 1, Survivor 2, and Old. These classifications are known as Generations. Objects are born into Eden and progressively promoted into the Old Generation at which point they are referred to as tenured [Footnote 1].</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
OK, so where does the Permanent Generation fit in? The Permanent Generation is a space outside the heap reserved for storing information about the Java classes your app uses. The PermGen has a fixed size that can be set with a JVM option. Class information in the PermGen is managed by ClassLoader instances, the most common of which is the Java system class loader or <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">sun.misc.launcher.AppClassLoader</code>. Generally, the loaded class information takes up a small, fixed memory footprint and you don’t need to put much thought into it.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Classes get loaded into the PermGen on demand, such that only the classes used by your app will take up space. The twist is when you start using Java Reflection. The Java Reflection package is an extremely powerful meta programming tool that gives you the ability to query class information and even create new classes at runtime. Reflection works in Java by calling through the Java Native Interface to get back the loaded class information from the JVM. By default, if information about the same class is requested more than 15 times [Footnote 2], then a <strong style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">new</strong> class will be created to hold that information. This process is called <strong style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">inflation</strong> and the new class information is stored in a <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">DelegatingClassLoader</code>.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Application frameworks that make heavy use of reflection will observe several instances of <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">DelegatingClassLoaders</code> in their PermGen. One such framework is the scala play framework.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Now that we have a pretty good idea of the PermGen’s role, it’s important to understand how garbage collection works in the PermGen space. I read in a few places that “classes are forever,” but in fact that is not the case. When garbage collection runs in the PermGen, it is true that it will not collect classes. However, what it will collect are ClassLoader instances that no longer hold references to any active classes. So, if all the classes in a given ClassLoader instance have expired, then that instance and all classes it references will go away. This is pretty much never going to happen with the Java system class loader [Footnote 3], but it could very well happen with a <span style="font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal;">DelegatingClassLoader</span>.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
With all this knowledge in hand, I finally felt ready to start investigating what <em style="border: 0px; font-family: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">might</em> be going on. The JDK comes with a tool called <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap</code>. <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap</code> is an excellent tool for analyzing what’s happening with the memory in a JVM. The first command I tried out was <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap -permstat</code>[Footnote 4]. Wow! Pretty cool; this shows me every class loader in the PermGen, the number of classes for which its responsible, the space it’s occupying, and its health. The first thing to jump out at me was that I observed a good deal of dead DelegatingClassLoader instances.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Weird. So, why aren’t those getting garbage collected if they are dead? After some more researching into JVM options, I discovered the following two options: <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC</code>. In the absence of these two flags, garbage collection will not remove unused class loader instances from the PermGen space [Footnote 5].</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Alright! Now we’re getting somewhere. These flags might be just what I’m missing. In order to find out, I set up a test system on which I could simulate the initial problem and then apply the new JVM options, rerun the simulation, and see how it holds up.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
For the simulation, I wanted to start up my app with a small PermGen and keep my eye on the usage. The PermGen size is configurable at JVM startup with the option: <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">-XX:MaxPermSize=68M</code>. I choose 68M due to my observation that typically on startup the app immediately used up just under 60MB and quickly grew to near 68MB after a few requests. To keep my eye on the PermGen usage, I used <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap -heap</code>, which shows both the total PermGen allocated as well as the current PermGen usage. Both of these values are important because although the PermGen has a max size, it will not use it all right away. Next up, <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap</code> displays a whole load of useful information, but I was only interested in PermGen, so a little grep filtering brought me to <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap -heap $pid | grep -A 4 'Perm Gen'</code>. Finally, I want to watch this as it changes, so throwing <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">watch</code> into the mix produced </div>
<blockquote class="tr_bq">
<code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">declare pid=$(pidof java);</code> </blockquote>
<blockquote class="tr_bq">
<code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">watch -n 2 "sudo /usr/java/jdk1.6.0_26/bin/jmap -heap $pid 2>&1 | grep -A 4 'Perm' "</code></blockquote>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
With my watch in place, I started simulating load with a while loop in bash to hit the web endpoint with curl requests. The PermGen quickly rose and eventually peaked at 100% and then crashed after about 500 total curl requests. I set 500 as my base line for comparison and restarted the JVM with the GC flags <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC</code>. Re-running my load simulation script, I watched the PermGen rise just like before. This time however, at around 500 requests, the PermGen usage dropped. The load simulation continued and I reached around 2000 requests before the PermGen finally ran out.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
My conclusion from the experiment is that the GC flags successfully configured the JVM to clean up the PermGen space and consequently improved performance over time. The new parameters have been deployed into production and we haven’t seen a crash!</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
My journey isn’t over yet. I’m not 100% satisfied with this solution because I still see dead DelegatingClassLoader instances in my <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">jmap -permstat</code> output and I’m mildly suspicious there may be a memory leak with the framework. I will continue to monitor PermGen usage and see how things progress. As an ultimate fall back option, I can disable inflation.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
…and that’s it! Hopefully this can serve a helpful starting point for anyone facing similar situations.</div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: large;">Some More Notes</span></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
I’m also including some more notes here that I learned along the way, but weren’t really pertinent to the discussion about heap dump and analysis. </div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<ul>
<li>The JVM can dump your heap to a file when it crashes. To do this enable <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">+XX:HeapDumpOnOutOfMemoryError</code> and set a path for the dump file, <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">-XX:HeapDumpPath=/path/for/dumps/java-.hprof</code> [Footnote 6]. </li>
<li>I used the Eclipse MAT tool to analyze my dump file and had a great experience. It’s worth reading the manual on its basic functions before using it. The default reports available gave me an immense amount of actionable information. If the OOM errors recur, I will likely be turning to MAT to dig to the bottom of the (potential) leak.</li>
<li>This article was substantially valuable in wrapping my head around how the JVM interacts with native memory, <a href="http://www.ibm.com/developerworks/java/library/j-nativememory-linux/">http://www.ibm.com/developerworks/java/library/j-nativememory-linux/</a></li>
</ul>
<br />
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-size: large;">FOOTNOTES</span><span style="font-size: 15px;">:</span></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 1</u>: To see details of tenure calculations, enable <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">-XX:+PrintTenuringDistribution</code></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 2</u>: See a quick description of how to control the process with system properties (sun.reflect.noInflation and inflationThreshold = 15), <a href="http://anshuiitk.blogspot.com/2010/11/excessive-full-garbage-collection.html">http://anshuiitk.blogspot.com/2010/11/excessive-full-garbage-collection.html</a></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 3</u>: This actually not completely out of the question. Application frameworks that employ the “hot deploy” method of swapping in new jars without stopping the JVM will ideally recycle the Java system class loader. See <a href="http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html">http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html</a> for a good description how this can get you into undesirable situations. To read about more people experiencing this problem, <a href="http://stackoverflow.com/questions/5066044/java-lang-outofmemoryerror-permgen-space-on-web-app-usage">http://stackoverflow.com/questions/5066044/java-lang-outofmemoryerror-permgen-space-on-web-app-usage</a></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 4</u>: The permstat information is expensive to calculate, and I’d suggest redirecting the output to a file so you can manipulate it. <a href="http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jmap.html">http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jmap.html</a></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 5</u>: See the bug report at <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8000325">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8000325</a> and a discussion on StackOverflow <a href="http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do">http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do</a></div>
<div class="FocusMe" style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
<u>Footnote 6</u>: JVM Options <a href="http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html">http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html</a> WARNING You must ensure there is no file with the same name at the dump path or the JVM will refuse to overwrite it. To increase entropy in the dump file name, use the <code style="border: 0px; font-family: Monaco, Consolas, 'Andale Mono', 'DejaVu Sans Mono', monospace; font-size: 13px; line-height: normal; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"></code>placeholder in the path you pass.</div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-78456430901624248052013-10-29T14:47:00.001-07:002013-10-29T14:47:35.785-07:00There's No Room for HeroesI've been thinking lately about how to successfully grow & mature an engineering team. What are the changes and shifts in mindset and practice that need to occur, naturally or synthetically, to remain effective, productive, and happy? One idea that's caught my attention is the shift from a culture of all-nighter heroes to a culture of on-time delivery of 100% finished projects.<br />
<br />
A culture of all-nighter heroes revolves around the star players. The ones who get all the glory consistently saving the day. Every startup at which I've worked has had at least one person like this. You know you can turn to them whenever there's a problem and be confident that no matter the issue, it will get fixed. These are the guys or gals who're answering questions in the chat room 24*7, know every intricacy of your architecture, and have memorized the root password. These are the people the whole team looks up to and for whom you say a prayer every night hoping that they won't quit.<br />
<br />
In a bootstrapped startup, the time for long term thinking is brief. Solutions are slapped together and shoved out the door just in time to evade catastrophe. Work is never quite finished. Instead the Pareto principle of 80% good enough is put to the test. Tech debt is an ever looming, always growing shadow of which no one wants to speak. "We'll fix it later" is the general motto. This is the haven of the hero: an undocumented, uncontrolled environment in which only the native knows their way around. Business becomes increasingly reliant on your heroes' ability to keep the technical ship afloat and battle scars are signs of prestige.<br />
<br />
Some might argue that this is unavoidable, and perhaps even necessary. Yet as business becomes more stable and clients become less tolerant of failure, this method of operation becomes unsustainable. Suddenly, 80%-done work isn't good enough, tech debt has become a substantial hindrance, and that missing documentation is causing daily problems. The architecture has reached a point in which no group of heroes can reasonably hold it together.<br />
<br />
This is the tipping point.<br />
<br />
Simply acknowledging this is critical, but the biggest challenge to survival is the cultural shift. The heroes of this new business environment are the types that recognize long term goals. They realize that good solutions aren't invented over night, but via healthy design, debate, experimentation, and measurement. They create simple solutions that eliminate tech debt and avoid gotchas. They share their approaches and engage the rest of the team. They automate every possible touch point. Ultimately, the heroes of this new field do everything they can to be replaceable.<br />
<br />
So, your challenge is to embrace automation, celebrate diligently crafted and deployed solutions, and put a damper on rewarding superhuman efforts riddled with tech debt. In my opinion, this a major inflection point in the professional development of a young team of engineers to a senior team. It must be closely shepherded and properly incentivized.Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-62328240883972712462013-07-25T14:40:00.001-07:002013-07-25T14:40:55.607-07:00Add Try-Catch Block in IntelliJ for PHP Using Live TemplatesChoosing a powerful IDE and investing the time to learn its many features is a distinctly rewarding experience, saving you countless hours of typing of and brain cycles on boiler plate code.<br />
<br />
I find the "Live Templates" feature of IntelliJ to be a major time saver. Today, I wanted to add a missing template for PHP: "try catch." I read through the docs and wasn't immediately educated on how it worked. I had to mess around a bit to get it working and thought I'd share my findings.<br />
<br />
<br />
<ol>
<li>Get into the preferences pane for editing live templates by following these directions http://www.jetbrains.com/idea/webhelp/creating-and-editing-template-variables.html</li>
<li>Find the "PHP" section.</li>
<li>Click the "+" symbol. In my version (Ultimate 12, Darcula), the plus symbol is to the upper right in the preferences pane.</li>
<li>Follow the instructions for naming listed in the IntelliJ docs linked above.</li>
<li>Paste the code outlined below (Exhibit A) into the "Template Text" box.</li>
<li>In the box below the template text, be sure to choose "PHP" as an applicable context</li>
</ol>
<div>
Exhibit A, the live template:</div>
<div>
<br /></div>
<div>
<pre class="brush:php">try
{
$SELECTION$
}
catch (\Exception $$e)
{
$$this->logger->warn("$END$");
}
</pre>
<br /></div>
<div>
<br /></div>
<div>
Now, when you want to use the template,<br />
<br />
<ol>
<li>Highlight the text you'd like to wrap in the try-catch</li>
<li>Hit the keyboard combo for "Surround with Live Template"</li>
<li>Hit Enter.</li>
<ol>
<li>IntelliJ will wrap the inject the template and replace "$SELECTION$" with the highlighted text.</li>
<li>It will then place your cursor between the double quotes at "$END$"</li>
<li>There is a way to have it stop along the way at "Exception" and "$e" allowing you to change the defaults, but I haven't gotten there yet =)</li>
</ol>
<li>Start typing your log message. If you don't typically have a $logger field defined in your class, you can customize this to fit your pattern.</li>
</ol>
</div>
Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-38669844607383570652013-05-31T19:22:00.000-07:002013-05-31T19:23:50.949-07:00Log4PHP not logging?Do you use log4php? I am a big fan of the log4-* suite. It's great to have a consistent approach to logging across languages.<br />
<br />
I've been using Log4PHP for almost a year now and aside from one issue with <a href="https://github.com/box/bart/blob/master/src/Bart/Log4PHP/LoggerLayoutPatternWithException.php" target="_blank">logging exceptions</a>, I've had no problems with it and, in general, enjoy any time I've spent reading the source.<br />
<br />
I had a bit of a problem today that was my own fault, but seems very possible for another to make. I've been happily using <span style="font-family: Courier New, Courier, monospace;">LoggerLayoutTTCC</span> for some time. It was deprecated in the 2.3.0 release last October in favor of <span style="font-family: 'Courier New', Courier, monospace;">LoggerLayoutPattern</span>. So I switched to <span style="font-family: 'Courier New', Courier, monospace;">LoggerLayoutPattern</span>. Things have been humming along just fine, but something changed recently. I noticed today that none of my log entries were getting written. Only errors were getting written out, and they weren't using any identifiable format.<br />
<br />
After digging into the source, I discovered that <span style="font-family: 'Courier New', Courier, monospace;">LoggerLayoutPattern</span> had no parsed pattern. Effectively this means that when my appenders asked the layout class to format the message, it was returning an empty string, which was being thrown away and not logged.<br />
<br />
How could this happen? The culprit was that although the <span style="font-family: 'Courier New', Courier, monospace;">LoggerLayoutPattern</span> class has a default pattern, it is not applied by the constructor. Out of the box this would not be an issue, but since I had previously been using <span style="font-family: 'Courier New', Courier, monospace;">LoggerLayoutTTCC</span>, which takes no constructor arguments, I was not passing in <span style="font-family: 'Courier New', Courier, monospace;">conversionPattern</span> for the <span style="font-family: 'Courier New', Courier, monospace;">params</span> argument to <span style="font-family: 'Courier New', Courier, monospace;">Logger::configure()</span> when defining my appender.<br />
<br />
The fix was straightforward: supply a conversion pattern! This feels like a minor bug with the Log4PHP code that should warn about the missing param or just use the defined default.<br />
<br />
<pre class="brush:php">'appenders' => array(
'default' => array(
'class' => 'LoggerAppenderDailyFile',
'params' => array('fileName' => 'logs/default.log'),
'layout' => array(
'class' => 'Bart\Log4PHP\LoggerLayoutTTCCWithException',
'params' => array(
'conversionPattern' => '%d %p pid:%t %c %m%n',
)
),
),
</pre>
<br />Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0tag:blogger.com,1999:blog-7398379322753724627.post-65322193513335884972013-05-21T19:55:00.001-07:002013-05-21T19:55:12.611-07:00Say Goodbye to PHP shell_exec() and PHP exec()Dealing with the shell in PHP can be a pain. If you are executing anything more complicated than a no argument command with a one line string result, things start getting complicated.<br />
<br />
To get started, if you want to check the return status of your commands, you need to <i>pass a reference parameter</i> to <span style="font-family: Courier New, Courier, monospace;">exec()</span>! Ok, so it's not returning the command exit status, well then you might expect that it returns the command's output instead? Nope; the return value of <span style="font-family: 'Courier New', Courier, monospace;">exec()</span>is actually the last line from the command's output. If you want to get the full output of the command, then you're looking at yet another reference parameter. And wait, which order do those references parameters go?<br />
<br />
After you've got your parameters sorted out, what about escaping arguments for security and accuracy? For that, you're left manually using PHP's other global functions.<br />
<br />
Alright, so now you've got a safe, accurate command. How do you know it works? You might want to test it, right? Well, if you want to test any of your code that uses <span style="font-family: 'Courier New', Courier, monospace;">exec()</span>, you're looking at a rough road ahead because PHPUnit has no facility to let you make assertions against parameters passed by reference. Furthermore, if you have more than one call to <span style="font-family: 'Courier New', Courier, monospace;">exec</span>, then you really have no way to stub results out of the box because you can't differentiate between calls to <span style="font-family: 'Courier New', Courier, monospace;">exec</span>. I previously blogged about one approach I've taken to solving this -- <a href="http://asheepapart.blogspot.com/2012/10/testing-shell-methods-with-phpunit.html">http://asheepapart.blogspot.com/2012/10/testing-shell-methods-with-phpunit.html</a>. <span style="font-family: 'Courier New', Courier, monospace;">MockShell</span> works alright, but it doesn't help us with escaping the command for security or accuracy.<br />
<br />
Enter the Bart <span style="font-family: Courier New, Courier, monospace;">Command</span> class. <span style="font-family: Courier New, Courier, monospace;">Command</span> is a simple class that acts as a facade over a single shell command. It takes a variable number of arguments to its constructor, representing arguments to the shelled command. The first constructor arg is reserved for the actual command itself. Each shell argument is escaped and then substituted for placeholders in the shell command.<br />
<br />
A command can be passed around and executed on demand. The results are returned as a full string or an array of each line. If the command fails, a custom exception of type is <span style="font-family: 'Courier New', Courier, monospace;">CommandException </span>thrown.<br />
<br />
We're using <span style="font-family: 'Courier New', Courier, monospace;">Command</span> extensively in our internal code bases with a lot of success. Currently, you can see it used in a few places in the <span style="font-family: 'Courier New', Courier, monospace;"><a href="https://github.com/box/bart/blob/master/src/Bart/Git.php#L90" target="_blank">Git</a></span> class.<br />
<br />
I also created the gist below,<br />
<br />
<script src="https://gist.github.com/asheepapart/5624868.js"></script>
<br />
Testing is easy. There isn't any pass-by-reference magic, you can just apply all your standard testing know-how. You can use <span style="font-family: 'Courier New', Courier, monospace;">Diesel</span> to inject your stub into your system under test. You can also mock the results of the shorthand <span style="font-family: 'Courier New', Courier, monospace;">Shell->command()</span> method.<br />
<br />Ben VanEveryhttp://www.blogger.com/profile/11437593840142482845noreply@blogger.com0