expansion

The expansion script allows you to expand the experiment.json files accepted by the --experiment_config flags in subcommands like caliban run and caliban cloud .

expansion supports the following arguments:

usage: expansion [-h] [--helpfull] [--version] [--pprint] [--print_flags]
                 experiment_config

Experiment config expander. For documentation, visit https://github.com/google/caliban

positional arguments:
  experiment_config  Path to an experiment config, or 'stdin' to read from
                     stdin.

optional arguments:
  -h, --help         show this help message and exit
  --helpfull         show full help message and exit
  --version          show program's version number and exit
  --pprint           Pretty-print the config to stdout.
  --print_flags      Print the actual flags generated by each experiment in
                     the expansion, one per line.

Given a file called experiment.json with these contents:

[    {
        "epochs": [1,2, 3, 4],
        "batch_size": [64, 128],
        "constant_arg": "something",
        "important_toggle": [true, false]
    },
    {
        "epochs": 1000,
        "batch_size": 1
    }
]

Running expansion experiment.json will generate a NON-pretty-printed string containing every combination of flag key and value described by the structure above (See the Experiment Broadcasting page for details):

[{"epochs": 1, "batch_size": 64, "constant_arg": "something", "important_toggle": true}, {"epochs": 1, "batch_size": 64, "constant_arg": "something", "important_toggle": false}, {"epochs": 1, "batch_size": 128, "constant_arg": "something", "important_toggle": true}, {"epochs": 1, "batch_size": 128, "constant_arg": "something", "important_toggle": false}, {"epochs": 2, "batch_size": 64, "constant_arg": "something", "important_toggle": true}, {"epochs": 2, "batch_size": 64, "constant_arg": "something", "important_toggle": false}, {"epochs": 2, "batch_size": 128, "constant_arg": "something", "important_toggle": true}, {"epochs": 2, "batch_size": 128, "constant_arg": "something", "important_toggle": false}, {"epochs": 3, "batch_size": 64, "constant_arg": "something", "important_toggle": true}, {"epochs": 3, "batch_size": 64, "constant_arg": "something", "important_toggle": false}, {"epochs": 3, "batch_size": 128, "constant_arg": "something", "important_toggle": true}, {"epochs": 3, "batch_size": 128, "constant_arg": "something", "important_toggle": false}, {"epochs": 4, "batch_size": 64, "constant_arg": "something", "important_toggle": true}, {"epochs": 4, "batch_size": 64, "constant_arg": "something", "important_toggle": false}, {"epochs": 4, "batch_size": 128, "constant_arg": "something", "important_toggle": true}, {"epochs": 4, "batch_size": 128, "constant_arg": "something", "important_toggle": false}, {"epochs": 1000, "batch_size": 1}]

You can also pipe the experiment.json file in via stdin by passing stdin instead of a filename:

cat experiment.json | expansion stdin

The Experiment Broadcasting page, specifically its section on pipes, has more information on how to use this function with other caliban commands to generate very complex sets of experiments with ease.

expansion –pprint

The --pprint argument forces a more sane expansion. Here are the results of expansion experiment.json --pprint:

[
  {
    "epochs": 1,
    "batch_size": 64,
    "constant_arg": "something",
    "important_toggle": true
  },
  {
    "epochs": 1,
    "batch_size": 64,
    "constant_arg": "something",
    "important_toggle": false
  },
  {
    "epochs": 1,
    "batch_size": 128,
    "constant_arg": "something",
    "important_toggle": true
  },
  // etc etc
]

expansion –print_flags

The --print_flags argument goes one step further and prints the actual argparse flags that correspond to each of the expanded JSON objects. Here are the results of expansion experiment.json --print_flags:

--epochs 1 --batch_size 64 --constant_arg something --important_toggle
--epochs 1 --batch_size 64 --constant_arg something
--epochs 1 --batch_size 128 --constant_arg something --important_toggle
--epochs 1 --batch_size 128 --constant_arg something
--epochs 2 --batch_size 64 --constant_arg something --important_toggle
--epochs 2 --batch_size 64 --constant_arg something
--epochs 2 --batch_size 128 --constant_arg something --important_toggle
--epochs 2 --batch_size 128 --constant_arg something
--epochs 3 --batch_size 64 --constant_arg something --important_toggle
--epochs 3 --batch_size 64 --constant_arg something
--epochs 3 --batch_size 128 --constant_arg something --important_toggle
--epochs 3 --batch_size 128 --constant_arg something
--epochs 4 --batch_size 64 --constant_arg something --important_toggle
--epochs 4 --batch_size 64 --constant_arg something
--epochs 4 --batch_size 128 --constant_arg something --important_toggle
--epochs 4 --batch_size 128 --constant_arg something
--epochs 1000 --batch_size 1