When I joined GitHub in 2021, they were on the cusp of launching GitHub Codespaces. Codespaces is one part cloud IDE, one part VM; the key selling point is that you can go to any Codespaces-enabled repo in your browser, click “New Codespace,” and you’ll be launched into an environment where you can write, test, and commit code, without a lot of setup. This is powered primarily by Visual Studio Code’s devcontainer feature.

After the public launch, there was a push to get teams using Codespaces internally, and as the New Guy, I started refactoring our codebase to work in a devcontainer (and thus, Codespaces.) I’m not normally someone who jumps on every hype train that passes by, but the value proposition of Codespaces and devcontainers was immediately evident to me. The setup process for GitHub at the time was extremely convoluted and seemed to always require one more CPU fan than my laptop had available.

Early Infatuation

Outside of work, Codespaces was not free at the levels I required, so I didn’t really use it. However, devcontainers are just a feature, and I still found them useful. I added one to geneac which was useful when I solicited contributions for Hacktoberfest. One particularly awesome use case for a containerized development environment is the ability to pin to certain versions of an environment, like I did for ProjectBlazt, so I would be able to build it against the toolchain I needed in the future.

Papercuts

When everything is set up properly, and the stars are aligned, everything works great. Contributors start up VS Code, get a little message to open the folder in a devcontainer, and within a minute or so they’re ready to code. Unfortunately, this is rarely the case…

Docker

Setting up Docker is, in theory, pretty easy. On Windows and Mac, there is https://www.docker.com/products/docker-desktop/, which is free with some restrictions. On Linux, it should be even easier since it is usually in the distro’s packages, or, if not there, you can install it from the upstream packages directly.

In practice, the best experience I’ve had as a developer by far was using Docker Desktop on a Mac. Docker Desktop creates a Linux virtual machine in order to run Docker, and however that works under the hood on Mac1 was very reliable and rarely caused issues. On Windows, it uses Hyper-V, which is… fine… but there are quirks you need to learn about here to. For example, did you know that mounting a folder from Windows into a Docker container is dog slow? The “workaround” here is to use the WSL2 integration, clone your repo in your distro there, and then run the devcontainer from WSL2.

On Linux, where the overhead is much lower (no need for a separate VM!) the situation is unfortunately worse. If your container setup is doing funky stuff with permissions (like BLIS is) then you need to be careful, since changes in the mounted filesystem directly affect the underlying files too. Other than that, things worked as expected in Ubuntu… but then I switched to Fedora, which uses SELinux. If Docker and SELinux were in a relationship, their Facebook status would be “it’s complicated.” I think it’s possible to reconcile them, but rather than spend the time doing that, I now just use an Ubuntu VM, running Docker, from which I can run my devcontainers.

Features

As devcontainers matured, “Devcontainer Features” were added. These are supposed to be one-liners you can add to your configuration that setup things like SSH, Python, Java, Docker (that’s right: you can run Docker in Docker!), etc. Although I do use them, I am a little confused about the niche they fill. At best, it moves a pile of bash from something that you maintain to something that somebody else maintains. At worst, it’s another external dependency that becomes a vector for breakage when you change your base container image to a distro it doesn’t support.

Speed

Pulling Docker images can be pretty slow by itself, but the premise of devcontainers is that you can shove a bunch of logic into a script that runs and sets up the environment. So, in order to get going, you have to:

  1. Pull the base image (probably large if it’s something like Ubuntu)
  2. Run a bunch of scripts to install additional packages and do other setup

Codespaces solves this (kind of) with their prebuild feature which packages 1 & 2 together as much as possible. Devcontainers doesn’t have that luxury, although what I tend to do is fake it by cramming as much as possible into the Docker container image and pushing it to the container registry. Then, when you start a devcontainer, it pulls the image rather than building a fresh copy2. Still not ideal, but faster.

Hell is Other Peoples’ Computers

All of this is manageable… extremely annoying at times, but manageable - if you’re comfortable getting your hands dirty. My biggest mistake with devcontainers is trying to use them for projects where people without a lot of experience in this area are regularly contributing. There are so many places where things could go wrong:

  • Docker might not be set up correctly, and it might not be telling you what’s wrong
  • Maybe Docker works, but VS Code repeatedly fails to build or run the image (this happens often to me without a clear explanation)
  • The container starts, but it can’t be accessed by VS Code
  • The devcontainer starts properly, but one of the sidecar containers (eg. the database) died for some reason
  • Wait, how do I access one of the sidecar containers? Instructions unclear, built a toaster
  • Something broke for no clear reason. What layer is it in? If I’m helping someone, how do I tell them what to check?
  • If you have an ARM-based machine (M1+ Mac), I don’t even know what’s going to happen

Debugging issues with the whole stack across a variety of computers, remotely, is leading me away from devcontainers and towards good old fashioned virtual machines. Vagrant can get me most of the way there for every platform that matters (ie. I can tell people “just install VirtualBox”) and it seems like there is less to go wrong.

The Promised Land

I dream of a day when a one-click development environment is a reality, and this is so close, but it’s not there for complicated projects yet. For simple projects, I likely will continue to use devcontainers, but with extremely conservative options that minimize the dependency on their specific quirks.

  1. x86, not M1/M2+
  2. Building a fresh copy of the image also requires a functional Docker setup, and, well…