Hierarchical Environment Variable Management

31 Aug, 2023

I still spend too much time managing environment variables!

The “Twelve-Factor” methodology prescribes that cloud-native applications should be configured via environment variables.

This recommendation separates configuration from code, which is essential for scalability and security. But it also results in an extensive array of environment variables.

Having to manage these variables for test, integration, end-to-end, and production environments becomes a daunting task. Often, these variable sets intersect. For instance, while Github credentials might be consistent across all environments, AWS accounts may vary.

Duplicating variables across environments is error-prone due to the necessary, often manual synchronization.

Small errors can lead to subtle bugs and tedious debug sessions.

Problem To Solve

How to manage the set of environment variables without duplication and synchronization.

Solution: rs-env

rs-env crafts a definitive variable set for each environment.

By utilizing different definition files, it prevents duplication and sync errors. It starts with global variables, gradually augmenting and refining the set for each level of specification.

Demonstration

Given a set of files which declare variables on various levels, the files can be linked and the resulting set is read from top to bottom. Hereby overwrite later variable definitions earlier ones.

This set of files is being linked together:

The resulting variable set then would be:

# final variable set
export VAR_1=var_11
export VAR_2=var_21
export VAR_3=var_31
export VAR_4=cloud_42
export VAR_5=var_53
export VAR_6=local_64
export VAR_7=local_74

Advantage

No more sync mishaps. Each variable is declared just once and integrated into a dependency graph. Any modifications in the graph flow seamlessly to the definitive variable set.

How It Works

File comments, # rsenv: <previous-file>, link variable definitions.
No limit to structuring variable definitions or the number of parallel dependency chains.
The dependency graph is recursively assembled.
Subsequent definitions override preceding ones.

Usage

Hierarchical environment variable management

Usage: rsenv [OPTIONS] [NAME] [COMMAND]

Commands:
  build        Build the resulting set of environment variables (DAG/Tree)
  envrc        Write the resulting set of variables to .envrc (requires direnv, DAG/Tree)
  files        Show all files involved in resulting set (DAG/Tree)
  edit-leaf    Edit the given environment file and all its parents (DAG/Tree)
  edit         Edit the FZF selected branch/DAG
  select-leaf  select environment/branch and update .envrc file (requires direnv, DAG/Tree)
  select       FZF based selection of environment/branch and update of .envrc file (requires direnv, DAG/Tree)
  link         Link files into a linear dependency branch (root -> parent -> child)
  branches     Show all branches (linear representation)
  tree         Show all trees (hierarchical representation)
  tree-edit    Edit branches of all trees side-by-side (vim required in path)
  leaves       Output leaves as paths (Tree)
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [NAME]  Optional name to operate on

Options:
  -d, --debug...              Turn debugging information on
      --generate <GENERATOR>  [possible values: bash, elvish, fish, powershell, zsh]
      --info
  -h, --help                  Print help
  -V, --version               Print version

Activate a set of variable definitions using:

source <(rsenv build <path-to-leaf.env>)

Seamless Integrations

direnv

direnv is a gem for environment activation. It seamlessly toggles variable activation without side-effects and remains my go-to tool for environment management. Pairing it with rs-env seems to be an excellent idea.

rs-env updates the .envrc file with the definite set of variables. .envrc is the single point of documentation for environment settings. No guessing where a particular environment variable value is coming from. It is right where you expect it to be.

JetBrains

Having always the correct set of variables active in JetBrains IDEs is challenging.

Yes, starting the IDE from the terminal where the variables are active does work, but any changes in the environment go un-noticed in the IDE unless it is being restarted. Environment settings are only picked up during startup time. Not good.

Alternatively, variables can be defined in the IDE’s run configurations. But then we are back at having several places to update and keep in sync. Laborious and error-prone. Not good.

However, there is an elegant solution to this problem:

Using the plugin EnvFile allows to specify an executable which injects the variables live. rs-env has a command which builds the environment set. This can be easily parametrized to pull the correct environment variable tree and inject it into the IDE runtime configuration.

By setting just one variable RUN_ENV=local the script rsenv.sh injects the correct full set of variables into the IDE. And this works live, i.e. changes in the environment files will be picked up automatically. Nice! No redundant configuration, always up-to-date, less headache.

Summary

While managing environment variables inherently has its complexities, it shouldn’t be a tedious chore.

By using rs-env, especially in combination with direnv and JetBrain’ IDE the process can be much smoother and be less error-prone.

#software development #rust