Config Management | Haobin Tan

🔖 Config Manaegment

Thu, 23 Feb 2023 00:00:00 +0000

YACS

Wed, 12 May 2021 00:00:00 +0000

YACS stands for Yet Another Configuration System. It helps define and manage system configurations such as hyperparameters and architecture/module choices for training a model. A tool like this one is essential to reproducibility and is a fundamental component of the system.

Usage

YACS can be used in a variety of flexible ways. There are two main paradigms:

Configuration as local variable (recommend)
Configuration as a global singleton

Step 1: Create Project Config File

Create a folder called config

In config folder, create config file, typically called config.py or default.py

my_project
|- config
 |- __init__.py
 |- default.py
|- experiment.yaml
|- main.py

This file is the one-stop reference point for all configurable options. It should be very well documented and provide sensible defaults for all options.

E.g.

# my_project/config/default.py

from yacs.config import CfgNode as CN


_C = CN()

_C.SYSTEM = CN()
# Number of GPUS to use in the experiment
_C.SYSTEM.NUM_GPUS = 8
# Number of workers for doing things
_C.SYSTEM.NUM_WORKERS = 4

_C.TRAIN = CN()
# A very important hyperparameter
_C.TRAIN.HYPERPARAMETER_1 = 0.1
# The all important scales for the stuff
_C.TRAIN.SCALES = (2, 4, 8, 16)


def get_cfg_defaults():
 """Get a yacs CfgNode object with default values for my_project."""
 # Return a clone so that the defaults will not be altered
 # This is for the "local variable" use pattern
 return _C.clone()

# Alternatively, provide a way to import the defaults as a global singleton
# cfg = _C # users can `from config import cfg`

For global singleton, another way is to declare cfg in __init__.py:
# my_project/config/__init__.py

from .default import _C as cfg
Then, in other script, we can import it like this:
from config import cfg

Step 2: Create YAML Configuration Files

Typically you’ll make one for each experiment. Each configuration file only overrides the options that are changing in that experiment.

# my_project/experiment.yaml

SYSTEM:
 NUM_GPUS: 2
TRAIN:
 SCALES: (1, 2)

Step 3: Use Config in Actual Project Code

Local variable usage pattern:

# my_project/main.py

from config import get_cfg_defaults

if __name__ == "__main__":
 cfg = get_cfg_defaults()

 cfg.defrost()

 cfg.merge_from_file("experiment.yaml")

 # Override from a list
 opts = ["SYSTEM.NUM_WORKERS", 8]
 cfg.merge_from_list(opts)

 # Freeze the config to prevent further modification
 cfg.freeze()
 print(cfg)

 # Further code using config settings
 # ...

SYSTEM:
 NUM_GPUS: 2
 NUM_WORKERS: 8
TRAIN:
 HYPERPARAMETER_1: 0.1
 SCALES: (1, 2)

We can also use global singleton usage pattern:

from config import cfg

if __name__ == "__main__":
 cfg.defrost()

 cfg.merge_from_file("experiment.yaml")

 # Override from a list
 opts = ["SYSTEM.NUM_WORKERS", 8]
 cfg.merge_from_list(opts)

 # Freeze the config to prevent further modification
 cfg.freeze()
 print(cfg)

 # Further code using config settings
 # ...

Reference

rbgirshick/yacs
Building A Flexible Configuration System For Deep Learning Models - A good guideline for organizing Deep Learning projects 👍

Hydra: Basics

Thu, 23 Feb 2023 00:00:00 +0000

Hydra is an open-source Python framework that simplifies the development of research and other complex applications. The name Hydra comes from its ability to run multiple similar jobs - much like a Hydra with multiple heads.

Key features

Hierarchical configuration composable from multiple sources
Configuration can be specified or overridden from the command line
Dynamic command line tab completion
Run your application locally or launch it to run remotely
Run multiple jobs with different arguments with a single command

Installation:

$ pip install hydra-core --upgrade

Specifying a Config File

Hydra configuration files are yaml files and should have the .yaml file extension.
Specify the config name by passing a config_name parameter to the @hydra.main() decorator. Note that you should omit the .yaml extension.
Specify the directory containing it relative to the application by passing config_path.
When running the script, use ++ to override a config value if it’s already in the config, or add it otherwise.

Example

|- config.yaml
|- my_app.py

# config.yaml
db:
 driver: mysql
 user: omry
 password: secret

from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(version_base=None, config_path=".", config_name="config")
def my_app(cfg):
 print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
 my_app()

Hydra creates an empty cfg object and passes it to the function annotate with @hydra.main

config.yaml is loaded automatically when you run your application.

$ python my_app.py
db:
 driver: mysql
 user: omry
 password: secret

Note: config_path and config_name can also be specified in command line instead of in code.
$ python my_app.py --config-path=. --config-name="config"

We can use ++ to override a config value if it’s already in the config, or add it otherwise:

# Override an existing item
$ python my_app.py ++db.password=1234
db:
 driver: mysql
 user: omry
 password: 1234


# Add a new item
$ python my_app.py ++db.timeout=5
db:
 driver: mysql
 user: omry
 password: secret
 timeout: 5

Using the Config Object

Hydra’s configuration object is an instance of OmegaConf’s DictConfig. More about OmegaConf see here.

Example

# config.yaml
node: # Config is hierarchical
 loompa: 10 # Simple value
 zippity: ${node.loompa}  # Value interpolation
 do: "oompa ${node.loompa}" # String interpolation
 waldo: ??? # Missing value, must be populated prior to access

# main.py


from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(version_base=None, config_path=".", config_name="config")
def my_app(cfg: DictConfig):
 assert cfg.node.loompa == 10 # attribute style access
 assert cfg["node"]["loompa"] == 10 # dictionary style access

 assert cfg.node.zippity == 10 # Value interpolation
 assert isinstance(cfg.node.zippity, int) # Value interpolation type
 assert cfg.node.do == "oompa 10" # string interpolation

 cfg.node.waldo # raises an exception

if __name__ == "__main__":
 my_app()

Grouping Config Files

A Config Group is a named group with a set of valid options. Selecting a non-existent config option generates an error message with the valid options.

Creating config groups

To create a config group, create a directory to hold a file for each configuration option.

For example, suppose we want to benchmark your application on each of PostgreSQL and MySQL. We create a directory, e.g. db, to hold a file for each database config option.

Directory layout

├─ conf
│ └─ db
│ ├─ mysql.yaml
│ └─ postgresql.yaml
└── my_app.py

db/mysql.yaml

driver: mysql
user: omry
password: secret

db/postgresql.yaml

driver: postgresql
user: postgres_user
password: drowssap
timeout: 10

Using config groups

Since we moved all the configs into the conf directory, we need to tell Hydra where to find them using the config_path parameter. config_path is a directory relative to my_app.py.

# my_app.py

from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(version_base=None, config_path="conf")
def my_app(cfg: DictConfig) -> None:
 print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
 my_app()

Running my_app.py without requesting a configuration will print an empty config.

$ python my_app.py
{}

Select an item from a config group with +GROUP=OPTION:

$ python my_app.py +db=postgresql
db:
 driver: postgresql
 pass: drowssap
 timeout: 10
 user: postgres_user

You can still override individual values in the resulting config:

$ python my_app.py +db=postgresql db.timeout=20
db:
 driver: postgresql
 pass: drowssap
 timeout: 20
 user: postgres_user

Selecting default configs

You can add a Default List to your config file. A Defaults List is a list telling Hydra how to compose the final config object. By convention, it is the first item in the config.

The defaults are ordered:

If multiple configs define the same value, the last one wins.
If multiple configs contribute to the same dictionary, the result is the combined dictionary.

Example:

We extend the example above by adding a default config file named default.yaml in the conf directory:

├─ conf
│ └─ db
│ ├─ mysql.yaml
│ └─ postgresql.yaml
| └─ default.yaml
└── my_app.py

default.yaml:

defaults:
 - db: mysql

from omegaconf import DictConfig, OmegaConf
import hydra

@hydra.main(version_base=None, config_path="conf", config_name="default")
def my_app(cfg: DictConfig) -> None:
 print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
 my_app()

When you run the updated application, MySQL is loaded by default.

$ python my_app.py
db:
 driver: mysql
 pass: secret
 user: omry

Overriding a config group default

We can still load PostgreSQL, and override individual values.

$ python my_app.py db=postgresql db.timeout=20
db:
 driver: postgresql
 pass: drowssap
 timeout: 20
 user: postgres_user

We can remove a default entry from the defaults list by prefixing it with ~:

$ python my_app.py ~db
{}

Composition order of primary config

Your primary config can contain both config values and a Defaults List. In such cases, you should add the _self_ keyword to your defaults list to specify the composition order of the config file relative to the items in the defaults list.

If you want your primary config to override the values of configs from the Defaults List, append _self_ to the end of the Defaults List.

# config.yaml
defaults:
 - db: mysql
 - _self_  # primary overrides default 

db:
 user: root

 # Result config: db.user from config.yaml
db:
 driver: mysql # db/mysql.yaml
 pass: secret # db/mysql.yaml 
 user: root # config.yaml (the value is overriden by primary confi)

If you want the configs from the Defaults List to override the values in your primary config, insert _self_ as the **first **item in your Defaults List.

# config.yaml
defaults:
 - _self_  # default overrides primary
 - db: mysql

db:
 user: root

 # Result config: db.user from config.yaml
db:
 driver: mysql # db/mysql.yaml
 pass: secret # db/mysql.yaml 
 user: omry # db/mysql.yaml

Leveraging Modularity and Composition

Resorting to modularity and composition, we can keep the configs manageable. Instead of creating a numerb of different config files that fully specify each config, create a single config that specifies the different configuration dimensions, and the default for each.

Example

Directory layout:

├── conf
│ ├── config.yaml
│ ├── db
│ │ ├── mysql.yaml
│ │ └── postgresql.yaml
│ ├── schema
│ │ ├── school.yaml
│ │ ├── support.yaml
│ │ └── warehouse.yaml
│ └── ui
│ ├── full.yaml
│ └── view.yaml
└── my_app.py

configy.yaml

defaults:
 - db: mysql
 - ui: full
 - schema: school

The resulting default configuration is a composition of the mysql database, the full ui, and the school schema

Running Hydra

Multirun from the command-line

You can configure hydra.mode in any supported way. The legal values are RUN and MULTIRUN.

Setting hydra.mode=MULTIRUN in your input config would make your application multi-run by default. For example, the following shows how to override from the command-line and sweep over all 6 combinations of the dbs and schemas.

$ python my_app.py hydra.mode=MULTIRUN db=mysql,postgresql schema=warehouse,support,school
[2021-01-20 17:25:03,317][HYDRA] Launching 6 jobs locally
[2021-01-20 17:25:03,318][HYDRA] #0 : db=mysql schema=warehouse
[2021-01-20 17:25:03,458][HYDRA] #1 : db=mysql schema=support
[2021-01-20 17:25:03,602][HYDRA] #2 : db=mysql schema=school
[2021-01-20 17:25:03,755][HYDRA] #3 : db=postgresql schema=warehouse
[2021-01-20 17:25:03,895][HYDRA] #4 : db=postgresql schema=support
[2021-01-20 17:25:04,040][HYDRA] #5 : db=postgresql schema=school

You can also specify the multirun mode by using --multirum (-m) argument:

$ python my_app.py --multirun db=mysql,postgresql schema=warehouse,support,school

python my_app.py -m db=mysql,postgresql schema=warehouse,support,school

Sweeping via `hydra.sweeper.params`

You can also define sweeping in the input configs by overriding hydra.sweeper.params.

The multirun example above can be achieved via the following config:

hydra:
 sweeper:
 params:
 db: mysql,postgresql
 schema: warehouse,support,school

If a sweep is specified in both an input config and at the command line, then the commandline sweep will take precedence over the sweep defined in the input config.

Additional sweep types

See the Extended Override syntax for details.

Output/Working directory

Hydra creates a directory for each run and executes your code within that working directory.

The working directory is used to:

Store the output for the application (For example, a database dump file)
Store the Hydra output for the run (Configuration, Logs etc)

Every time you run the app, a new working directory is created.

# my_app.py


import hydra
from omegaconf import DictConfig
from pathlib import Path

@hydra.main(version_base=None)
def my_app(cfg: DictConfig) -> None:
 print(f"Current working directory : {Path.cwd()}")

$ python my_app.py hydra.job.chdir=True
Current working directory : outputs\2023-02-20\10-00-36

Note: By default hydra.job.chdir=False, which means Path.cwd() will output the directory where the python script locates. To get the log directory, you need to explicitly set hydra.job.chdir to True.

Structure of the working directory:

outputs/2023-02-20/10-00-36
├── .hydra
│ ├── config.yaml
│ ├── hydra.yaml
│ └── overrides.yaml
└── my_app.log

Hydra output directory (.hydra by default)
- config.yaml: A dump of the user specified configuration
- hydra.yaml: A dump of the Hydra configuration
- overrides.yaml: The command line overrides used
main_app.log: Application log file created for this run

Changing or disabling Hydra’s output subdir

You can change the .hydra subdirectory name by overriding hydra.output_subdir.
You can disable its creation by overriding hydra.output_subdir to null.

Accessing the original working directory in your application

With hydra.job.chdir=True, you can still access the original working directory by importing get_original_cwd() and to_absolute_path() in hydra.utils.

Customizing working directory pattern

The output directory can be configured by

setting hydra.run.dir (for single hydra runs) or
hydra.sweep.dir/hydra.sweep.subdir (for multirun sweeps).

Configuration for single run

Run output directory grouped by date:

hydra:
 run:
 dir: ./outputs/${now:%Y-%m-%d}/${now:%H-%M-%S}

Run output directory grouped by job name:

hydra:
 run:
 dir: outputs/${hydra.job.name}/${now:%Y-%m-%d_%H-%M-%S}

Run output directory can contain user configuration variables:

hydra:
 run:
 dir: outputs/${now:%Y-%m-%d_%H-%M-%S}/opt:${optimizer.type}

More see: Customizing working directory pattern

Logging

By default, Hydra logs at the INFO level to both the console and a log file in the automatic working directory.

Example

import logging
from omegaconf import DictConfig
import hydra

# A logger for this file
logger = logging.getLogger(__name__)

@hydra.main()
def my_app(_cfg: DictConfig) -> None:
 logger.info("Info level message")
 logger.debug("Debug level message")

if __name__ == "__main__":
 my_app()

We can easily integrate loguru.logger for customized logging.

You can enable DEBUG level logging from the command line by overriding hydra.verbose.

hydra.verbose can be a Boolean, a String or a List:

Examples:

hydra.verbose=true : Sets the log level of all loggers to DEBUG
hydra.verbose=NAME : Sets the log level of the logger NAME to DEBUG. Equivalent to import logging; logging.getLogger(NAME).setLevel(logging.DEBUG).
hydra.verbose=[NAME1,NAME2]: Sets the log level of the loggers NAME1 and NAME2 to DEBUG

Debug

Printing the configuration

Print the config for your app without running your function by adding --cfg or -c to the command line.

The --cfg option takes one argument indicating which part of the config to print:

job: Your config
hydra: Hydra’s config
all: The full config, which is a union of job and hydra.

You can use –package or -p to display a subset of the configuration:

$ python my_app.py --cfg hydra --package hydra.job

Reference

Hydra official tutorial

Hydra: Advanced

Thu, 23 Feb 2023 00:00:00 +0000

Extend Configs

A common pattern is to extend an existing config, overriding and/or adding new config values to it. The extension is done by

including the base configuration, and then
overriding the chosen values in the current config.

Example: Extending a config from the same config group

config.yaml

defaults:
 - db: mysql

db/mysql.yaml

defaults:
 - base_mysql  # We extend mysql on the base of base_mysql

user: omry
password: secret
port: 3307
encoding: utf8

db/base_mysql.yaml

host: localhost
port: 3306
user: ???
password: ???

$ python my_app.py
db:
 host: localhost # from db/base_mysql
 port: 3307 # overridden by db/mysql.yaml 
 user: omry # populated by db/mysql.yaml
 password: secret # populated by db/mysql.yaml
 encoding: utf8 # added by db/mysql.yaml

Configuring Experiments

To clearly support multiple configurations, each configuration file only specifies the changes to the master (default) configuration.

Example:

The default configuration is:

The benchmark config files specify the deltas from the default configuration:

Key concepts:

# @package _global_
Changes specified in this config should be interpreted as relative to the _global_ package.
We could instead place nglite.yaml and aplite.yaml next to config.yaml and omit this line.
The overrides of /db and /server are absolute paths.
This is necessary because they are outside of the experiment directory.

Running the experiments from the command line requires prefixing the experiment choice with a +.

$ python .\train.py +experiment=aplite

Sweeping over experiments

This approach also enables sweeping over those experiments:

$ python my_app.py --multirun +experiment=aplite,nglite

To run all the experiments, use the glob syntax:

$ python my_app.py --multirun '+experiment=glob(*)'

Specializing Configuration

In some cases the desired configuration should depend on other configuration choices.

Example:

You may want to use only 5 layers in your Alexnet model if the dataset of choice is cifar10, and the default 7 otherwise.

We can start with a config that looks like this:

# config.yaml
defaults:
 - dataset: imagenet
 - model: alexnet

We want to specialize the config based on the choice of the selected dataset and model.

OmegaConf supports value interpolation, we can construct a value that would - at runtime - be a function of other values. The idea is that we can add another element to the defaults list that would load a file name that depends on those two values.

Modify config.yaml:

defaults:
 - dataset: imagenet
 - model: alexnet
 - optional dataset_model: ${dataset}_${model}

The key dataset_model is an arbitrary directory, it can be anything unique that makes sense, including nested directory like dataset/model.
${dataset}_${model} is using OmegaConf’s variable interpolation syntax. At runtime, that value would resolve to imagenet_alexnet, or cifar_resnet - depending on the values of defaults.dataset and defaults.model.
optional: By default, Hydra fails with an error if a config specified in the defaults does not exist. In this case we only want to specialize cifar10 + alexnet, not all 4 combinations. the keyword optional tells Hydra to just continue if it can’t find this file.

Code example see: hydra/examples/patterns/specializing_config at main · facebookresearch/hydra · GitHub

Configuring Hydra

Hydra is highly configurable. Many of its aspects and subsystems can be configured, including:

The Launcher
The Sweeper
Logging
Output directory patterns
Application help (–help and –hydra-help)

You can include some Hydra config snippet in your own config to override it directly, or compose in different configurations provided by plugins or by your own code. You can also override everything in Hydra from the command line just like with your own configuration.

Accessing the Hydra config

Hydra is passing to the function annotated by @hydra.main(). Two ways to access the Hydra config:

In your config, using the hydra resolver:
```
config_name: ${hydra:job.name}
```
The resolver name is hydra, and the key is passed after the colon.

In code, using the HydraConfig singleton

from hydra.core.hydra_config import HydraConfig

@hydra.main()
def my_app(cfg: DictConfig) -> None:
 print(HydraConfig.get().job.name)

The following variables are populated at runtime:

hydra.job: used for configuring some aspects of your job (more information see: Job Configuration)
hydra.run: Used in single-run mode (i.e. when the --multirun command-line flag is omitted). See configuration for run.
hydra.sweep: Used in multi-run mode (i.e. when the --multirun command-line flag is given) See configuration for multirun.
hydra.runtime: Fields under hydra.runtime are populated automatically and should NOT be overridden.
hydra.overrides: Fields under hydra.overrides are populated automatically and should not be overridden.
hydra.mode

For other fields that are present also at the top level of the Hydra Config see: Other Hydra settings.

Resolvers provided by Hydra

hydra: Interpolates into the hydra config node. e.g. Use ${hydra:job.name} to get the Hydra job name.
now: Creates a string representing the current time using strftime. e.g. for formatting the time you can use something like${now:%H-%M-%S}.
python_version: Return a string representing the runtime python version by calling sys.version_info. Takes an optional argument of a string with the values major, minor or macro. e.g:

Hydra + wandb

See: Configuring W&B Projects with Hydra

Reference

Hydra official tutorial

Config Management | Haobin Tan

🔖 Config Manaegment

YACS

Usage

Step 1: Create Project Config File

Step 2: Create YAML Configuration Files

Step 3: Use Config in Actual Project Code

Reference

Hydra: Basics

Specifying a Config File

Example

Using the Config Object

Example

Grouping Config Files

Creating config groups

Using config groups

Selecting default configs

Overriding a config group default

Composition order of primary config

Leveraging Modularity and Composition

Example

Running Hydra

Multirun from the command-line

Sweeping via hydra.sweeper.params

Additional sweep types

Output/Working directory

Changing or disabling Hydra’s output subdir

Accessing the original working directory in your application

Customizing working directory pattern

Configuration for single run

Logging

Debug

Printing the configuration

Reference

Hydra: Advanced

Extend Configs

Configuring Experiments

Sweeping over experiments

Specializing Configuration

Configuring Hydra

Accessing the Hydra config

Hydra + wandb

Reference

Sweeping via `hydra.sweeper.params`