Caliban

Caliban is a tool for developing research workflow and notebooks in an isolated Docker environment and submitting those isolated environments to Google Compute Cloud.

For a short tutorial introduction to Caliban, see the GitHub page.

Overview

Caliban provides five subcommands that you run inside some directory on your laptop or workstation:

  • caliban shell generates a Docker image containing any dependencies you’ve declared in a requirements.txt and/or setup.py in the directory and opens an interactive shell in that directory. The caliban shell environment is ~identical to the environment that will be available to your code when you submit it to AI Platform; the difference is that your current directory is live-mounted into the container, so you can develop interactively.

  • caliban notebook starts a Jupyter notebook or lab instance inside of a docker image containing your dependencies; the guarantee about an environment identical to AI Platform applies here as well.

  • caliban run packages your directory’s code into the Docker image and executes it locally using docker run. If you have a workstation GPU, the instance will attach to it by default - no need to install the CUDA toolkit. The docker environment takes care of all that. This environment is truly identical to the AI Platform environment. The docker image that runs locally is the same image that will run in AI Platform.

  • caliban cloud allows you to submit jobs to AI Platform that will run inside the same docker image you used with caliban run. You can submit hundreds of jobs at once. Any machine type, GPU count, and GPU type combination you specify will be validated client side, so you’ll see an immediate error with suggestions, rather than having to debug by submitting jobs over and over.

  • caliban build builds the docker image used in caliban cloud and caliban run without actually running the container or submitting any code.

  • caliban cluster creates GKE clusters and submits jobs to GKE clusters.

  • caliban status displays information about all jobs submitted by Caliban, and makes it easy to interact with large groups of experiments. Use caliban status when you need to cancel pending jobs, or re-build a container and resubmit a batch of experiments after fixing a bug.

These all work from your Macbook Pro. (Yes, you can build and submit GPU jobs to Cloud from your Mac!)

The only requirement for the directory where you run these commands is that it declare some set of dependencies in either a requirements.txt or setup.py file. See the requirements docs for more detail.

The rest of this document contains detailed information and guides on Caliban’s various modes. If you want to get started in a more interactive way, head over to the Caliban tutorials directory.

Caliban’s code lives on Github.

Using Caliban

If you want to practice using Caliban with a proper getting-started style guide, head over to Caliban’s tutorials (Coming Soon!).

See the sidebar for information on the subcommands exposed by Caliban and a whole series of tutorials and guides that you might find interesting as you work with Caliban.

Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more.

For an introduction to Caliban, start at the Caliban GitHub page.

Indices and tables