Sunday, November 8, 2015

Managing Runtime Configurations

Configuration Headaches

Managing runtime application configurations in large scale, heterogeneous environments is a total pain. Over the last five years I have attacked this problem in various ways, each with its own grace and flaws. The goal of this post is to sum up the evolution of my experience and hopefully impart some insight to any folks in similar plight. 
It starts with the challenges:
  1. Different environments get different configurations
  2. Different services in the same environment get different configurations
  3. Configuration files get massive and treacherous
  4. Configuration files get complex quickly with cross references
  5. Configurations have no safety against typos in keys (or values)
  6. Configurations have essentially no type safety guarantees
  7. Configurations have no README indicating their intent or usage
  8. Configurations can be accessed from anywhere in a code base (or outside) in inconsistent manners

Sidebar: Defining Runtime Application Configurations

For the context of this article, I use the term “runtime application configurations” to represent the configurations used by an application at runtime. I do not mean configuration as code type things like Puppet, Chef, or Ansible. For example, I would consider the log configurations in alogger.xml file for a JVM application that I own to be runtime app configs. I would not consider the configuration of Apache on a web server a runtime app config in this context, I would consider that a system configuration. More specifically, I would consider it configuration for an application whose source code I'm not writing.

Evolution of a solution

Compound keys

The most popular convention I've observed for managing configurations is by flat file such as ini, json, yaml, xml; very rarely is it in the same language as the code consuming it. My assumption is that this accomplishes a few things:
  1. They can be edited by non-programmers
  2. They can be consumed by multiple programming languages, thus freeing operations teams from managing multiple manifestations of the same configuration values
  3. In the case of compiled code, configurations do not need to be compiled (or recompiled when they change)
This restriction is very powerful in the guarantees of simplicity, but also limiting. One major limiting factor is that without a separate management system, it’s impossible to have different values of configurations per runtime environment. This was a major problem for us since we shared configuration files between all of our environments, but need to set different values for the same configuration key based on the environment.
Our initial solution was to encode the environmental context onto the keys such that we could effectively define unique values for a given key based on the environment that would use it. So instead of a single key-value pair, we would have multiple key-value pairs differentiated by environment. See an example of log levels for different environments.
[logger]
level<dev> = DEBUG
level<staging> = INFO
level<prod> = WARN
When the INI gets parsed natively, each of the level's will be parsed as unique keys. The next step is custom code in the application to split the environments (e.g. <dev>) off of the keys and then figure out which environment and value applies in its running context.
Sound complicated? It is. Making this work involved writing some complicated INI parsing code as well as figuring out a way to tell an application the environment in which it’s running. The result was a brittle system that was error prone and slowed down onboarding. Furthermore, if any configurations needed to be shared between projects, those projects also needed to solve the parsing and environment setting problems.

Moving away from flat files

I asked myself, “Why are configurations always in flat files anyway?” I couldn’t come up with a convincing answer, so my next attempt was to write a straightforward configuration framework in PHP that expected different values per each key based on environmental context. All configuration files were PHP files that returned a large array. I experimented with model objects that could build configurations, but ultimately discarded the idea because I felt the time to solve the edge cases would outweigh the incremental value.
With this system, the mind bending key-to-environment relationship was slightly more clear. Another gain was removing cross references to other keys within the INI values. Since the values were set with PHP, it was possible to reuse values or base other values off each other, e.g. url = "{$scheme}{$domain}{$path}".
Although the system was less brittle and easier to understand, the configuration files themselves were still rather large and it wasn’t clear how defaults worked between environments. Worse, the files could only be used by PHP applications.

A separate configuration management system

My biggest problem was the size and complications of putting configurations for every environment in the configuration files. So I decided to look into using a separate system to manage the differences configurations, which could then deliver only the necessary content to a given consumer. I landed on Puppet since we already use it extensively elsewhere. This allowed me to write files containing only the configurations that mattered in the respective environments. Based on this, going back to flat files was possible. I could also tie together configuration values in Puppet before writing the files, so there was no need for cross references within the configuration file itself.
This was great, my first 5 problems were solved. But still, type safety was not enforced, configuration files had no guaranteed documentation, and understanding how they were used within an application was a grep nightmare.

Configuration model objects

I decided to revisit configuration model objects. However this time not as builders, but instead as accessors. By funneling all access of configurations through a single point, I figured that I could enforce a few things.
I buried the configuration parsing code inside an abstract class (aptly named Configuration) with protected methods for getting at the parsed values. Configuration model classes could then extend the base and expose relevant methods for their configurations. For example, a logging configuration class would have a getLevel() method, which behind the scenes would parse the configurations file and return back the value. Any user that wanted to load configurations could only do so by using a Configurationclass, or writing a new one. Next, I tied the name of the configuration file to the name of the class, such that if a class were named LogConfigurations, the configuration file behind the scenes needed to be named log.conf. Finding usages of a given configuration file became trivial.
I added type expectations methods to the parsing code so that invalid values would raise exceptions. TheConfiguration class requires that all extending classes implement a README() method, so that users understand the classes intention and expected INI contents. This is coupled with a generic unit test that parses the README() output and ensures it can actually be used by the class. Furthermore, each of the accessor methods has its own documentation.
Configuration model classes also opened the door for more sophisticated configurations. Since access is inside a class, and not just array referencing, the class can do smart things like tie multiple values together or call out to other classes. One really great manifestation of this was the ability to create the ProjectConfiguration class which uses the project version control to load a configuration file’s contents.

Get involved

That’s where I am today. It’s been a fun journey and I’m happy (atm) with where things are. I’m excited to see where this goes next and to learn other ways folks have found to solve this problem.
One key component that I’d like to see next is the configuration delivery mechanism. Currently, Puppet solves the problem reasonably for short lived PHP processes. However, the two areas that I’d like to improve are:
  1. Automatic reloading by long lived processes. Maybe this is just from a file watch.
  2. A better interface for managing the configurations. This could get extensive. One tool here would be validation. This could be much more than just type checking, but semantics and plugins for special configuration groups (e.g. validate a database in a slave section is up not a master). Typos in keys would be completely avoided. Another would be the ability to see what values a given environment would get.
Please post back with comments or pull requests!

91 comments:

  1. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  2. I am agree with above point. You are such a nice author. I have already bookmark your domain for share it everywhere.

    ReplyDelete
  3. myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

    ReplyDelete
  4. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.
    Advanced AWS Course Interview Questions And Answers, Top 250+AWS Jobs Interviews Questions and Answers 2018
    Advanced AWS Jobs Interview questions and answers |Best Top 110 AWS Interview Question and Answers – india

    ReplyDelete
  5. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article. thank you for sharing such a great blog with us.
    microsoft azure training in bangalore
    rpa training in bangalore
    rpa training in pune
    best rpa training in bangalore

    ReplyDelete
  6. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Best Devops training in sholinganallur
    Devops training in velachery
    Devops training in annanagar
    Devops training in tambaram

    ReplyDelete
  7. Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…
    python Course in Pune
    python Course institute in Chennai
    python Training institute in Bangalore

    ReplyDelete
  8. Attend The Python Training in Hyderabad From ExcelR. Practical Python Training Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Python Training in Hyderabad.
    python training in bangalore

    ReplyDelete
  9. I have express a few of the articles on your website now, and I really like your style of Python classes in pune blogging. I added it to my favorite’s blog site list and will be checking back soon…

    ReplyDelete
  10. Home Mart is a site about Home Improvement, Furniture, Home Appliances and many more.
    Check out the best
    bedroom furniture nz
    entertainment unit
    shoe rack

    ReplyDelete
  11. celR offers data science course in hyderabad , the most comprehensive Data Science course in the market, covering the complete Data Science lifecycle concepts from Data Collection, Data Extraction, Data Cleansing, Data Exploration,

    ReplyDelete
  12. Nice Blog
    For Data Science training in Bangalore, Visit:
    Data Science training in Bangalore

    ReplyDelete
  13. Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
    ExcelR Course in Data Analytics

    ReplyDelete
  14. Amazing post and sure I will share the post among my friends.​ Machine Learning Training In Bangalore

    ReplyDelete
  15. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    data analytics courses

    ReplyDelete
  16. We are always happy to learn more information from you! It is nice to be able to read a good quality article with useful information on topics of interest. I look forward to your future updates.

    ReplyDelete
  17. Wow it is really wonderful and awesome thus it is veWow, it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.

    python training in bangalore

    python courses in bangalore

    python classes in bangalore

    python training institute in bangalore

    python course syllabus

    best python training

    python training centers

    ReplyDelete
  18. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information.

    aws training in bangalore

    aws courses in bangalore

    aws classes in bangalore

    aws training institute in bangalore

    aws course syllabus

    best aws training

    aws training centers

    ReplyDelete
  19. Amazing blog. Amazing stuff contained on this website. great admin continue.

    Best Training Institute in Bangalore BTM. My Class Training Bangalore training center for certified course, learning on Software Training Course by expert faculties, also provides job placement for fresher, experience job seekers.
    Software Training Institute in Bangalore

    ReplyDelete
  20. Really very happy to say, your post is very interesting to read. I never stop myself to say something about it. You’re doing a great job. Keep it up...

    Learn Hadoop Training from the Industry Experts we bridge the gap between the need of the industry. Softgen Infotech provide the Best Hadoop Training in Bangalore with 100% Placement Assistance. Book a Free Demo Today.
    Big Data Analytics Training in Bangalore
    Tableau Training in Bangalore
    Data Science Training in Bangalore
    Workday Training in Bangalore

    ReplyDelete
  21. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts.

    sap abap training in bangalore

    sap abap courses in bangalore

    sap abap classes in bangalore

    sap abap course syllabus

    best sap abap training

    sap abap training center

    sap abap training institute in bangalore

    ReplyDelete
  22. This post is very simple to read and appreciate without leaving any details out. Great work! data science courses

    ReplyDelete
  23. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    ExcelR Data Analytics Course
    Data Science Interview Questions

    ReplyDelete
  24. Nice post...Thanks for sharing...
    RPA Training in Bangalore - Marathahalli | RPA Training Institutes | RPA Course Fees and Content | RPA Interview Questions
    - ECare Technologies located in Marathahalli - Bangalore, is one of the best RPA Training institutes with 100% Placement support. RPA Training in Bangalore provided by RPA Certified Experts and real-time Working Professionals with handful years of experience in real time RPA Projects.

    ReplyDelete
  25. Awesome blog. I enjoyed reading your articles. This is truly a great read for me.


    data analytics course

    data science course

    business analytics course

    ReplyDelete
  26. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our data analytics course in mumbai
    data analytics course in mumbai | https://www.excelr.com/data-analytics-certification-training-course-in-mumbai

    ReplyDelete
  27. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    Digital marketing course mumbai

    ReplyDelete
  28. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

    ReplyDelete
  29. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    data scientist course in pune

    ReplyDelete
  30. Hi, your article was of great help. I loved the way you shared the information, thanks.
    Amazing article, I highly appreciate your efforts, it was highly helpful. Thank you CEH Training ,CEH Certification, CEH Online Course, Ethicalhacking

    ReplyDelete
  31. Hi, This is your awesome article , I appreciate your effort, thanks for sharing us.
    cism training
    cism certification

    cisa training,
    cisa certification
    cisa exam

    ReplyDelete
  32. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions
    Data Science Course

    ReplyDelete
  33. This was really one of my favorite website. ExcelR Machine Learning Course In Pune Please keep on posting.

    ReplyDelete
  34. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete

  35. The information you have posted is very useful. The sites you have referred was good. Thanks for sharing. ExcelR Data Science Course In Pune

    ReplyDelete

  36. This was really one of my favorite website. ExcelR Machine Learning Course Pune Please keep on posting.

    ReplyDelete
  37. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  38. Attend The Business Analytics Course From ExcelR. Practical Business Analytics Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Course.
    Business Analytics Course

    ReplyDelete
  39. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  40. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.

    Electric Toothbrush

    ReplyDelete
  41. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  42. This post is really helpful for us. I certainly love this website, keep on it.
    Data Science Course in Hyderabad

    ReplyDelete
  43. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  44. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  45. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites! data science training in coimbatore

    ReplyDelete
  46. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete


  47. This post is great. I reallly admire your post. Your post was awesome.
    data science course in Hyderabad

    ReplyDelete
  48. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    data science interview questions

    ReplyDelete
  49. I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this sitebest data science courses in mumbai.

    ReplyDelete
  50. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  51. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  52. This is my first time visit here. From the tons of comments on your articles.I guess I am not only one having all the enjoyment right here! ExcelR Data Scientist Courses

    ReplyDelete
  53. This is my first time visit here. From the tons of comments ExcelR Machine Learning Course In Pune on your articles.I guess I am not only one having all the enjoyment right here!

    ReplyDelete
  54. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    https://360digitmg.com/india/pmp-certification-course-training-in-guduvanchery

    ReplyDelete
  55. The content that I normally go through nowadays is not at all in parallel to what you have written. It has concurrently raised many questions that most readers have not yet considered.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata

    ReplyDelete
  56. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    workday studio online training
    best workday studio online training
    top workday studio online training

    ReplyDelete
  57. Amazing post found to be very impressive while going through this post. Thanks for sharing and keep posting such an informative content. data science courses

    ReplyDelete
  58. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog. data science training in Hyderabad

    ReplyDelete
  59. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    360DigiTMG

    ReplyDelete
  60. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  61. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    360DigiTMG Data Science Course In Pune
    360DigiTMG Data Science Training In Pune

    Thank you..

    ReplyDelete
  62. Awesome information and enjoyed reading this blog learnt new things thanks for sharing.
    Data Science Training 360DigiTMG

    ReplyDelete
  63. I have bookmarked your blog because this site contains unique and valuable information in it thank you.
    Data Analytics Certification

    ReplyDelete
  64. Found your post interesting to read. I cant wait to see your post soon. Good Luck for the upcoming update. This article is really very interesting and effective, data sciecne course in hyderabad

    ReplyDelete
  65. Very interesting article. Many articles I come across these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome. I will instantly grab your rss feed to stay informed of any updates you make and as well take the advantage to share some latest information about

    CREDIT CARD HACK SOFTWARE which many are not yet informed of, the recent technology and advantages of it.

    Thank so much for the great job.

    ReplyDelete
  66. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  67. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  68. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more. 360DigiTMG

    ReplyDelete
  69. Truly, this article is really one of the very best with unique information thank you.
    Data Science Course in Hyderabad 360DigiTMG

    ReplyDelete
  70.  You are really performing grand work. I must articulate that you really have done a great research before writing. Keep up the good work!
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  71. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work. data science training in Hyderabad

    ReplyDelete
  72. Data Analytics Course Pune wow... what a great blog, this writter who wrote this article it's realy a great blogger, this article so inspiring me to be a better person in some of them hope you will give more information on this topics in your next articles.
    I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    ReplyDelete
  73. Fantastic blog with unique content and information provided was very valuable waiting for next blog update thank you .
    Ethical Hacking Course in Bangalore 360DigiTMG

    ReplyDelete
  74. I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts.
    a href="https://www.excelr.com/data-analytics-certification-training-course-in-pune/"> Data Analytics Course in Pune/">Took me time to understand all of the comments, but I seriously enjoyed the write-up. It proved being really helpful to me and Im positive to all of the commenters right here! Its constantly nice when you can not only be informed, but also entertained! I am certain you had enjoyable writing this write-

    ReplyDelete