Dynamic configurations avoid Butterfly-Effect of App Changes

Introduction

When an app gets developed, it comes with a configuration file. The configuration files need to be a part of application-build packages in-order to ship the applications. However, the most common problem faced by teams is a simple change in one configuration property, which requires a new build. A configuration management system needs to be implemented to simplify the whole process of effecting changes in configuration properties. It usually works with the proper use of a configuration server.

Configuration Management System

The configuration management system, a software component, is responsible to manage application configurations. The configurations get used by applications in an ecosystem to carry out various responsibilities. It provides the capability to store and retrieve configurations to other services.

In a configuration management system, multiple applications in the software ecosystem communicate with the configuration system to retrieve various configurations. However, the mode of communication varies from one configuration system to another. Here, some systems have the potential capability to expose RESTful Application Programming Interfaces (APIs), while others use specific protocols to enable full duplex communication. A few configuration systems use messages to broadcast various configuration change events. Nowadays, configuration management systems get developed in-house in various organizations.

Incidentally, configuration systems must be singular across all environments. These environments include DEV, QA, Stage, UAT, and Prod; and they should use the same deployment of the configuration management system that stores their configurations with ease.

The teams involved should stay away from having environment-specific deployments of configuration management systems as it avoids the overhead of configuration promotions from one environment to another. Here, a single deployment also brings down or reduces the infrastructure cost.

Storage of Configurations

The dynamic configurations of apps get stored as key-value pairs. Here, the key is string and the value ranges from simple strings to complex objects. These include JSON or YAML, and it is a good idea to leave the interpretation of values to consuming-applications. The configuration system should be flexible to include the key-value pairs given to it.

Storage of Secrets

The secret information to be stored includes data and information such as passwords, API keys, and secrets. Ideally, configuration management systems are not the right choice to store the secrets. Hence, they get stored in the Secrets Management Tool. Some of the good secrets management tools include Azure Key Vault, HashiCorp Vault, AWS Key Management Services, and Google Cloud’s Secret Manager. Rarely, do these secrets get strongly encrypted and stored in configuration management systems. However, one should take absolute care not to store the encryption keys in the configuration management system in such cases. This approach is highly discouraged as it is less secure than the Secrets Management Tool.

Features of Configurations

The feature to look for includes dynamic updates of configurations wherein the configuration management system should perform CRUD operations on configurations without any downtime or service interruptions.

The second feature is the real-time propagation of changes where the configuration system ensures that any updates are readily available for app consumption. It gets achieved through various ways that include developing a client library, which gets bundled into each client application. The client library is capable of exposing interfaces to get the latest value of a given configuration from the configuration system. Another approach is to use messaging systems like Kafka and RabbitMQ. In the second approach, the server takes upon the responsibility of broadcasting updates on the messaging system as soon as the configuration gets updated. Here, the clients consume the broadcasted messages to update the locally cached values to the latest ones. A major challenge is the additional infrastructure cost of a messaging system with more code in client applications to consume updates.

The third feature includes environment-based definitions. The configuration system is designed to intuitively represent values based on the app environment. It allows better value-partitioning by the environment with lesser chances of human error in updates. An example includes feature-flag getting turned on in production while it is supposed to be turned on in Stage.

The fourth feature is the leader election and failure recovery as the configuration system should be resistant to failure. Here, the developers can choose a distributed configuration system instead of one that can become an SPOF or Single Point of Failure. The examples include configuration systems like ZooKeeper and Redis. These can be easily scaled up to thousands of nodes that are smart enough to handle touch problems like leader election and split-brain networks.

The fifth feature, version, keeps track of all configuration values by versioning them. Thus, it helps to revert configuration values. For instance, if a configuration change causes a break in an existing functionality, the DevOps team can quickly revert the configuration to an earlier version.

The sixth feature of the configuration management system is audit that tracks the capability of What, When, Who, and Why. It comes in handy when things break due to various configuration changes and also when compliance is of utmost importance.

Configuration Options

The best configuration options to choose for setting up configuration management systems include Apache ZooKeeper, HashiCorp Consul, Redis, etcd, Spring Cloud Config Server, and Company proprietary solutions.

Apache ZooKeeper: Reference to how Configuration Management System works

The Apache ZooKeeper system runs on a cluster of servers called an ensemble. It shares the state of the data, and whenever a change gets made, it is not considered successful until it is written to a quorum of the servers in the ensemble. The ZooKeeper-quorum is the minimum number of nodes a ZooKeeper needs to start functioning for a ZooKeeper-cluster to work. Here, a leader gets selected in the ensemble and all write-requests from clients get forwarded to a single server. The rest of the ZooKeeper servers get called the followers. The leader processes all the write-requests, which get forwarded to followers, and waits for their feedback. When half of the followers answer, the write gets considered to be successful. Here, the ZooKeeper allows distributed processes to coordinate with each other. It gets done through a shared hierarchical namespace that gets organized similarly to standard file systems.

The namespace contains data registers known as Z-nodes, which are similar to files and directories. The Z-nodes get identified by unique and absolute paths that are delimited Unicode Strings. All the Z-nodes maintain a data structure, which includes version numbers for data changes, and each time the data of the Z-node changes, the version number increases indicating the change. Also, each Z-node has an Access Control List (ACL) that keeps restrictions on who can do what.

ZooKeeper is quite fast and simple, and works to fulfill its goal of synchronization. The guarantees it provides include sequential consistency, atomicity, single system image, reliability, and timeliness.

Dynamic configurations avoid Butterfly-Effect of App changes

It has been observed that configuration changes on complex apps aid companies with faster reactions to changing conditions. Also, it creates less stressful features and hassle-free product launches. Butterfly-Effect is the crux and fulcrum of Chaos Theory. It states that small changes in single states of complex systems result in large differences in later states. Avoidance of the butterfly-effect gets applied to app development as well. For instance, a typo or error in a single line in the code can result in catastrophic results in later stages of app development. Companies have dealt with the butterfly-effect with proper use of DevOps through Continuous Integration (CI) and Continuous Development (CD). However, the application configuration methods are difficult to manage with many configurations getting loaded during the process of application-initialization. The use of configuration management systems can be powerful while properly responding to dynamic live-environment changes. For example, Amazon has developed a process to handle configuration changes from thousands of developers and hundreds of internal organizations. It gets done through AWS AppConfig, which got launched in 2019.

AWS AppConfig: Reference

The AppConfig of AWS supports three major capabilities. These allow companies to adopt continuous configuration process. Here, configuration data gets separated from main codes. It has the ability to update quickly while facilitating separate data storage. Through AppConfig, the applications can fetch configurations from external services. The configuration changes get checked and validated before getting deployed to avoid catastrophic results.

Conclusion

The main benefits of continuous configuration tools such as AWS AppConfig is that it allows for nearly instant release of a feature, without requiring the risk and time cost of a full application redeploy. For example, a common use case for feature flags is the ability to turn a feature on or off depending on a precise event – like the start of Black Friday. As volume explodes on Black Friday, vendors have to enable and configure features like analytics, payment validation and processing, inventory management, social media integrations, and much more. For example, a vendor may have to limit the requests made to an inventory database to avoid resources from being overloaded. Or prioritize deliveries to locations that can be fulfilled in time for the holidays.

More than anything else, having a configuration system that not only can instantly update configuration but also checks and validates changes before deployment occurs, can be less stressful during those big events.

“In that launch moment, when the stakes and stress levels are high, it’s scary to have to redeploy the whole software stack,” says Steve Rice, Principal Product Manager of AWS AppConfig. “It’s simpler to make a tiny little configuration change that turns on that feature.”

In this way, small changes that can be validated, deployed gradually and monitored can help companies avoid a catastrophic butterfly effect down the road.