Container Build Alternatives

Docker decided to charge a per user fee for companies and universities using their Docker Desktop for Windows package. If you are actually developing “Windows Containers”, this is the only way they can be built. If you want to develop Linux Containers using a Windows development platform, there are other fairly simple solutions that are completely free. When this paper was first written, the alternatives were in early stages of development, worked poorly, and required a lot of effort to install and use. Today all those problems have been solved, and there are several options that require almost no effort.

What We Are Replacing (Docker Desktop for Windows)

Docker Desktop for Windows installs a set of Windows programs including docker.exe in the Windows executable path. It also creates a custom Windows Subsystem for Linux “distro” containing a version of the Moby Linux system optimized for running containers and a preinstalled version of the “docker engine” component (dockerd/containerd).

While WSL has its own private virtual network and its own protocol for communication between Windows and Linux programs, Docker already had its own mechanism through which the “docker” command talks to the “docker engine” running in the background. Rather than converting to an entirely different approach, Docker tries to maintain in Windows/WSL as much of the same code and protocol as it used in Linux, with a local “unix socket” or through the network from one machine to another. The result is something that is not exactly the same as Linux but is also completely different from any other WSL package.

The alternatives to Docker Desktop also install in WSL, but either they use a completely standard WSL distribution and standard Windows to WSL communication, or else they also create a custom distribution with their own Linux-of-choice image (Fedora instead of Moby for example) but after that using standard WSL networking.

WSL Networking

There is a detailed section below under the heading “Plumbing” that contains more details on WSL networking and communication. The short version is that all WSL instances run in non-standard containers under a single Microsoft Linux Kernel VM. Windows communicates to WSL to share files and transfer commands using an old Bell Labs protocol called 9P. There is a virtual LAN between Windows and the Linux Kernel shared by all the WSL instances. However, any Container runtime has to create its own VPN that is separate from the host network.

There is one extra trick that needs to be understood. All virtual machines run under the Hyper-V system and are connected to the Windows Kernel by a low-level high performance virtual bus. The Microsoft Linux Kernel has a special driver for its “localhost” or “loopback” virtual device with IP address 127.0.0.1. The loopback “adapter” on all the different Linux distributions that you install in WSL share the same loopback adapter, and the Microsoft Linux Kernel loopback communicates directly to the corresponding loopback driver in the Windows Kernel.

As a result, any program running in Windows or any WSL Linux environment can reserve a loopback port number by being the first program in any environment to list to, for example, port 9876. Then any client program on Windows or any WSL system can connect to 127.0.0.1:9876 and communicate to that program. This communication is done by a direct Windows Kernel to Linux Kernel driver level communication over the Hyper-V service bus and has nothing to do with any of the virtual LAN networks that Hyper-V creates. It also only works for WSL and does not apply to any other virtual machine.

This trick is important because while Docker has decided to convert its standard internal VPN from Linux-Linux to Linux-Windows, most of the alternative solutions only offer “localhost” networking. A test client program in Windows (or any other WSL distribution) can connect to a containerized application listening to a port number redirected by the equivalent to “docker run -p 9876:9876” which in turn ends up as “localhost:9876” on Windows and throughout the WSL environment.

WSL Ubuntu 24.04 Distro

Containers run under a runtime engine. The Docker engine runs as a Linux background service under the “systemd” supervisor. After WSL 2 introduced the Microsoft Linux Kernel, it was possible to run systemd in a WSL Linux instance, but none of the standard WSL systems you could get from Microsoft supported this. Then WSL Linux distributions began to support an option to turn on systemd using a configuration file /etc/wsl.conf in the image file system. Starting with later versions of Ubuntu 23.04.3 or 23.10, systemd was turned on by default. Once that happened, the Ubuntu package libraries were changed so you could install the docker.io package into WSL Ubuntu in the same way you install it on an ordinary computer or VM. Just run “sudo apt install docker.io” (or type “docker” into the command line of an Ubuntu that doesn’t have it and get a list of installation options).

This creates a WSL that has a “docker” command and a docker engine running in the background. However, you probably want to edit in Windows and only use this command to build and run the containers. There are two easy to use solutions.

In any Windows command line (cmd.exe, terminal, or PowerShell) you can run a Linux command in the default WSL environment by just typing “wsl“ in front of the command (or “wsl. exe” if you want to be more explicit). The Linux command inherits the current Windows directory and many environment variables. So to build an image type “wsl docker build …” in the directory containing the Dockerfile project, and to run a container type “wsl docker run …”.

Alternately, you can use Visual Studio Code Remote (WSL). The GUI editor in VS Code connects to a directory in the WSL Linux image, edits files, and with the Docker plugin can build images and run containers.

The one restriction is that client programs can only connect to the container from Windows or a WSL environment, and they must connect through the localhost trick. So if you start the container with

wsl docker run -p 8080:8080

then the client program, which can be a Windows browser, is going to connect to it as “http://localhost:8080/…”.

You can create a docker.bat file with contents

If you need it, you can add docker-compose the same way with “wsl sudo apt install docker-compose” and create another bat file for that command.

Podman Desktop for Windows

Podman is the Red Hat alternative to the standard Docker system. It is a redesign and reimagining of containers that is nearly 100% compatible with Docker syntax, but not with the API for third party utility programs. It is perfectly fine for building and testing applications.

Red Hat makes a free Podman Desktop for Windows available to anyone who wants to download it. It basically duplicates a lot of the ease-of-use features of Docker Desktop for Windows in terms of a GUI interface that will lead you step by step through the install and configuration process and provides a Windows based tool to display the images, containers, and volumes.

You can install Podman using “winget”. If you don’t want to read any documentation, do a

winget install redhat.podman-desktop

Then run it and it will lead you through the process of installing the podman programs in windows and creating a WSL distro with Fedora 40 and the podman service running in the background.

If you regard the Podman Desktop GUI as unnecessary fluff, you can just “winget install redhat.podman” (without the -desktop suffix) to get all the command programs. Then you have to manually create and start the WSL engine with the commands:

podman.exe machine init
podman.exe machine start

Because there is a podman.exe program running in Windows duplicating the docker.exe program of Docker (and with all the same commands and options), you can either change your existing programs to use “podman” instead, or you can create an alias by going to a directory early in your Path and creating a symlink named “docker.exe” that points to the “podman.exe” program:

mklink docker.exe "C:\Program Files\RedHat\Podman\podman.exe"

Then whenever you enter the “docker” command, path search will find the symlink and run podman.exe instead.

Podman for Windows relies on the same WSL “localhost” trick. A container that has been mapped to use port 8080 is available to client test programs in Windows or WSL as “localhost:8080”, but no client can run from another computer or even another VM on the same Hyper-V system.

Podman for Windows (History and Design)

Docker provided open-source code that could be distribute for any Linux system. Red Hat could and did distribute the docker package if you wanted to install it on RHEL.

However, none of the other vendors was stuck with the original decisions made by the Docker group when they built the first system and then decided to maintain its limitations and design problems in all future versions.

Red Hat is a Linux company, so they were mostly interested in solving the problems of Linux developers. The most serious problem with Docker on Linux is that it runs a single Engine in the background under root and all of the image files that it builds and runs are stored in a single repository with no individual ownership or permissions. Anyone who can run docker commands on a machine can change any image file created by any other user.

No programming language works this way. Developers login under their own userid and store files in their home directory. Each user owns his own files and permits other users to access them selectively. Podman is a complete replacement of the “docker” command that builds and runs containers under the developer’s userid in the developer’s login and home directory. To make this work, the “podman” executable program includes its own container engine that runs in the login rather than in the background.

There are many advantages to Podman if you have Linux developers sharing a large development server machine, but that doesn’t happen anymore. Today each developer has his own personal machine, and if you have one person who is the only user and administrator of a machine, then the major design different between Docker and Podman goes away. More importantly, if you have to run a command in Windows (docker.exe or podman.exe) and then run the containers in a WSL environment, then you end up configuring Podman Desktop to run exactly the same way that Docker Desktop runs.

So, the only really important differences are that Podman Desktop is absolutely free and Podman has a few extra command options that Docker doesn’t make available in the same way.

Podman builds Dockerfiles with the same syntax. A Podman Image can be pushed to a network image repository and then run under Docker on another system. However, at Yale production images have to be rebuilt by Jenkins from the Dockerfile and developers are not allowed to push images from a development machine.

The Podman engine does not speak the same protocol as the docker engine, so the VS Code Docker Plugin does not recognize it and cannot display useful information.

A Yale developer will see some extra useful commands and options in the following bat/powershell script that runs a container for testing:

The “docker secret create” command only works with Swarm, but Podman makes it a normal command for end users. Then they add --secret to the run command so you don’t have to use compose. They add --replace to both commands to delete and replace an existing secret or container with the same name before you create a new one. These are all options you wish Docker made available, but Docker will not change decisions it made long ago or add convenience features to new releases.

Podman build supports Dockerfiles, although it also allows you to use the alternate name “Containerfile” to avoid brand name confusion.

WSL is a different Container Runtime Tightly Integrated in Windows

Windows Subsystem for Linux is a tightly coupled Linux environment that can run Linux commands as if they were a natural part of Windows. Someone familiar with a Linux command like “grep” can use it on the Windows command line by prefixing the command name with “wsl”:

The normal processing of cmd.exe sees “tasklist | wsl" and pipes the standard output of tasklist.exe to the standard input of wsl.exe. Then wsl.exe connects to the default WSL Linux environment using a P9 substitute for SSH and sends the standard input it is getting through to a WSL Linux command of “grep sql”. Grep runs in Linux and generates standard output, which WSL then grabs and like SSH sends back to Windows where the output is typed on the screen by cmd.exe.

With WSL a Windows user can mix in Linux commands with Windows commands whenever it is convenient.

WSL is implemented as a single Linux VM running the Microsoft Linux kernel, on top of which a Microsoft container runtime supports multiple Linux “distribution” containers. You can install these WSL containers using Windows Store or wsl.exe. A subset of the available containers is:

Microsoft tries to keep things simple, so they use the term “distro” to mean two different things that they assume the reader will not have to distinguish. In a list of install options, “OracleLinux_9_1” refers to an image tar file prepared by Oracle and submitted to Microsoft for distribution. When you do a “wsl --install”, this image file is used to create a WSL container, and the container is also given the name “OracleLinux_9_1”. After that, when the wsl documentation refers to running a program or an interactive shell in a “distro” they are talking about the container and not the image used to build it.

Over time the vendor may provide an updated image file under the same name. The “Ubuntu-22.04” name has been used for two years, but if you install it today and then display the /etc/lsb-release file you find it has “DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS”, so this is probably the fourth refresh update of the image.

A WSL container is different from a Docker container. Docker containers are intended to be short term and lose all their disk data when the system shuts down. WSL containers have a Microsoft virtual hard drive file (a VHDX, like a VM disk), so changes are permanent until you discard the container.

WSL containers shut down and restart. Traditionally they run in single-user mode, but Microsoft changed the rules and today many of them, notably Ubuntu, will run in “systemd” mode if you add “[boot] systemd=true” to the /etc/wsl.conf file in the image file system disk and restart the container.

Special Purpose WSL “distro” Images

The “wsl --import” command will create a WSL container from any correctly formatted tar file on your disk. Software packages can include such a tar file and use it to create WSL containers directly, bypassing the Microsoft distribution system.

Docker Desktop for Windows includes a tar file for their Moby Linux and creates a WSL container named “docker-desktop-data”.

Podman includes a Fedora 40 tar file and creates a container named “podman-machine-default”. You can then create additional WSL Fedora containers with different names using the “podman machine init” command.

Plumbing

This is a very complicated topic that must be explained precisely if you are to understand how things work. Otherwise, it will just appear as if testing containers works by magic.

When the WSL feature is activated, it creates a VM to run the Microsoft Kernel and the containers. That VM has a virtual LAN adapter, connected to a virtual network called “WSL”. A matching virtual LAN adapter is created in Windows connected to the same virtual network.

Windows sees its adapter as:

While all WSL Linux containers share a common eth0 adapter with its randomly chosen IP address:

The eth0 adapter is configured to treat the Windows 172.21.16.1 address as the gateway. Therefore, any client connect from the WSL Linux containers to the Yale network or the Internet flows thorugh the Windows system and uses the networks to which the Windows host is connected at the time.

However, the connections work only one way. The Windows host system can connect to the WSL VM using the 172.21.21.137 address, but this is not visible to any computer outside of the Windows host. Therefore, WSL applications and containers can only be tested from the Windows host (and other WSL client programs).

The WSL containers all share the one VM LAN adapter. Port numbers on that adapter are owned by the first program in any WSL container to bind to them. A program running in WSL Ubuntu can get back a “port is being used by another program” error because a program in WSL Fedora is already using it.

Every system also has a Loopback virtual adapter, also known by the name “localhost” and the IP address 127.0.0.1. The loopback is entirely synthetic and in every other case is local to the one system. Loopback is handled by a device driver in the kernel. Here the Linux Kernel is shared by the WSL containers, so port numbers are owned by the first program in any WSL container to bind to it.

However, to allow Windows to easily communicate to WSL programs, the loopback driver in the Linux Kernel communicates to the loopback driver in the Windows kernel to extend the impression of a single shared device. A Windows Browser can communicate to a WSL Linux Web server by using a URL like “http://localhost:8080”.

Of course, using Internet protocol it is also possible to connect from Windows to WSL Web servers using the URL “http://172.21.21.137:8080”, but you have to discover this IP address and there is no nice hostname you can use instead of the address.

Suppose you run Tomcat as an ordinary Linux program (not in a container) in any WSL distro. It will bind to 0.0.0.0:8080, which means port 8080 on any LAN adapter it can see. WSL containers see two LAN adapters, so it will bind to loopback (127.0.0.1:8080) and eth0 (172.21.21.137:8080). A Widows Browser talks to Tomcat through a virtual network using the eth0 IP address, and through the internal Kernel to Kernel connection between the two loopback device drivers using “localhost”.

Now run Tomcat in a Docker or Podman container using the -p 8080:8080 parameter on the run command. Inside the container, Tomcat binds to 0.0.0.0:8080, but the container has a restricted namespace where the only network it sees is a VPN generated by Docker/Podman. A VPN is necessary because, unlike WSL containers, Docker containers are completely independent of each other. In theory, dozens of Docker containers can all bind to port 8080 on what they see as their LAN adapter and never get a “port is being used by another program” error. The containers use the VPN to talk first to the container Engine.

The Docker Engine in the WSL container named “docker-desktop-data” or Podman in the WSL container named “podman-machine-default” bind to 0.0.0.0:8080 and like Tomcat in the previous example, this becomes a bind to loopback (127.0.0.1:8080) and eth0 (172.21.21.137:8080).

A Windows Web Browser can connect to Tomcat in the container with the same URLs. The only difference is that the Browser communicates to the Container Engine, which forwards the data to the Docker container using the VPN.

The connection between the Windows Browser and any program running in WSL is a standard feature of WSL itself and did not depend on any additional code added by Docker or Podman, nor was it helped along by the docker.exe or podman.exe program. This means that if you decide to use the “wsl docker” or “wsl podman” approach and install nothing in Windows itself, you can still run containers with “wsl docker run -p …” or “wsl podman run -p …” and test then with a Windows Browser and a “http://localhost” URL.

Exposing the Docker API to Windows Programs

The Docker Engine exposes an API through a Unix socket to other programs running on the same Linux system. Podman exposes an identical API to allow third party Docker tools to work with it.

However, Unix sockets do not exist on Windows. Docker created a proxy program that runs in Windows and exposes the API using a Windows “named pipe”, and again Podman created its own proxy that works the same way. There are not a lot of programs that know how to talk to the named pipe, but the Docker plugin for VS Code is at least one example.

If you run Docker on a real Hyper-V Linux VM instead of WSL, you can configure the Docker Engine to listen for requests on port 2375 of a virtual LAN adapter. However, Docker does not provide authentication or access control, so the virtual network has to be entirely private inside your laptop.

This is one case where the Docker approach is so bad that Podman refused to duplicate it. You can use Podman on one machine to connect to Podman acting as an Engine on another machine, but it uses SSH. Security is provided by the SSH login, normally using public keys. Once the client Podman has done an SSH login, it can run a podman command as that user on the other system, which provides access to the images and files in that user’s home directory.

Configure the Docker Engine in a Hyper-V Linux VM

Create a Linux VM with Hyper-V Quick Create, Ubuntu Multipass, or download the ISO and create the VM from scratch. When it comes up, you can use sudo apt or sudo yum to install the normal docker.io or podman packages.

This is not the place to discuss Hyper-V networking in detail. There is a Default network that connects VMs to the host Windows. You can also create other networks, including ones connected to a real LAN adapter. You can always configure your VM to connect to more than one network, but then you have to configure a static IP address because it will be a server of sorts. Assume the address on the network you want to use is 192.168.2.204.

On the VM, edit the file /lib/systemd/system/docker.service. Find the line beginning “ExecStart=/usr/bin/dockerd”, and add a “-H” parameter followed by that IP address:

ExecStart=/usr/bin/dockerd -H fd:// -H 192.168.2.204 --containerd=/run/containerd/containerd.sock

save the file and then restart the docker.service or reboot the VM.

DOCKER_HOST

All versions of docker.exe (but specially the open-source one called “docker-cli” on GitHub and Chocolatey) will look for an Environment variable named DOCKER_HOST that points to a network address of a Linux machine running Docker (or Podman). To connect to the Engine configured above, the value of DOCKER_HOST should be ”tcp://192.168.2.204:2375”.

You can install both podman.exe and docker.exe on the same Windows system. However, you do not want DOCKER_HOST to be set when the podman.exe command or Podman Desktop Dashboard is running. In that case, it is better to set the environment variable locally where the docker.exe program is running. A solution may be a docker.bat file that looks like this: