I’ve been using a Mac mini for personal projects lately, but I prefer developing on Linux. It’s closer to production, and I happen to know Linux internals better than MacOS internals. With some quality time on eBay, I recently built an Unraid homelab/media serverwith 24 cores, 32g of memory, and 20tb usable storage (i.e. not including parity disks). It seemed a perfect opportunity to set up a new devbox.

I knew I wanted some sort of remotely accessible containerized environment. I’ve started using VSCode for Rust development, so I would need that to work too. Finally, static IPs are tedious, so mDNS support would be ideal.

Setting up Unraid

Unraid was the easy part, minus some fiddling with file permissions. I created a share named development and set it to prefer using the cache, keeping my development work on the SSD instead of spinning rust:

Easy peasy and accessible at /mnt/user/development.

I then created a new Unraid user, since I didn’t want to run as root inside the container. Later, I used the same uid/gid in the container to avoid annoying permissions mismatches.

I also created a persistent home directory for the container, preloaded with my ssh public keys:

$ cd /mnt/user/development/
$ mkdir -p home_dir/.ssh
$ echo "ssh-ed25519 BLAH_BLAH_MY_PUBLICKEY me@somewhere" > home_dir/.ssh/authorized_keys
$ chown -R kfb:users home_dir
$ chmod -R 0755 home_dir

Setting Up the Container

I opted for a container instead of a VM for this devbox. Unraid supports both, but I work with containers more and find them a bit more lightweight for this use case. VMs have a lot of appeal, and Firecracker is super cool, so I might do a follow-up post about that someday.

My first version installed the system, created a user, and started sshd:

FROM ubuntu:latest

RUN yes | unminimize
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y man coreutils sudo apt-utils language-pack-en \
                       openssh-server build-essential net-tools pkg-config \
                       bzip2 rsync vim curl htop git tmux zsh

# Configure locales and ssh
RUN locale-gen en_US.UTF-8
RUN dpkg-reconfigure locales
RUN echo "%sudo   ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN mkdir /var/run/sshd

# Create user using same UID/GID as on Unraid host.
RUN useradd kfb -d /home/kfb \
                --groups users,sudo \
                --create-home \
                --shell /bin/zsh \
                --gid 100 \
                --uid 1000

CMD ["/usr/sbin/sshd" "-D"]

Then this to start it up:

docker run \
    -d \
    --name devbox \
    --hostname devbox \
    -p 2000:22 \
    -p 2080:2080 \
    -p 2088:2088 \
    -v /mnt/user/development/devbox/home_dir:/home/kfb/ \
    devbox

This was a strong start, and the version I used for a good while. The port mappings were tedious, but usable. I configured SSH to use a different port for the host, so it was mostly hidden. The other ports were used to expose whatever else I was working on.

Unsatisfied with port shenanigans, I dug into Docker networking and learned about macvlan networks. These networks make a container’s mac address appear as a physical network interface to the rest of the network. Happily, Unraid comes with a macvlan network already configured, named br0. Also, macvlan doesn’t rely on docker for port mappings, so I could drop all that cruft as well.

The only hangup was that now I needed mDNS working inside the container unless I wanted to deal with static IPs. The old version relied on the Unraid hostname, which was no longer tenable. Instead, I needed to get Avahi running inside the container. There are two problems: containers don’t have systemd to run Avahi or dbus for it to mount. Both of these can be solved in the Dockerfile

... snip ...

# Disable dbus in Avahi
# source: https://stackoverflow.com/questions/44078097/how-do-i-advertise-and-browse-mdns-from-within-docker-container
RUN sed -i 's/#enable-dbus=yes/enable-dbus=no/g' /etc/avahi/avahi-daemon.conf

... snip ...

# Run avahi-daemon and sshd
CMD ["/bin/sh", "-c", "service avahi-daemon start && /usr/sbin/sshd -eD"]

With Avahi running, adding the macvlan network to the docker run call was simple, add --network br0 .

Phew! That was a trip, but I got there. I can SSH into devbox.local, do whatever work I want, and open whatever ports I want. If I need something running in the background, tmux is sufficient. This is just a devbox after all.

The full Dockerfile and script to start it:

FROM ubuntu:latest

# Install the base system
#
# This is split out beacuse it changes less frequently. If I'm adding
# a library then I don't need to rebuild the whole container.

RUN yes | unminimize
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y man coreutils sudo apt-utils language-pack-en \
                       openssh-server build-essential net-tools pkg-config \
                       bzip2 rsync vim curl htop git tmux zsh avahi-daemon

# Generate locales and do some configuration
RUN locale-gen en_US.UTF-8
RUN dpkg-reconfigure locales

# Passwordless sudo
RUN echo "%sudo   ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

# Configure avahi-daemon
RUN sed -i 's/#enable-dbus=yes/enable-dbus=no/g' /etc/avahi/avahi-daemon.conf

# Configure ssh and set host keys
RUN mkdir /var/run/sshd
COPY ssh_host_* /etc/ssh/
COPY disable_root_and_passwords.conf /etc/ssh/sshd_config.d/

# Create user using same UID/GID as on Unraid host.
RUN useradd kfb -d /home/kfb \
                --groups users,sudo \
                --create-home \
                --shell /bin/zsh \
                --gid 100 \
                --uid 1000

# Install dev tools and libraries
# N.B.: Update apt again in case docker is doing a partial build and the apt
#       cache is out of date
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y libssl-dev libncurses-dev libffi-dev libreadline-dev \
                       lzma-dev libreadline-dev libsqlite3-dev zlib1g-dev \
                       libbz2-dev liblzma-dev hugo sqlite3

# Start avahi-daemon and sshd here since there is no systemd to do it for us
CMD ["/bin/sh", "-c", "service avahi-daemon start && /usr/sbin/sshd -eD"]

#!/usr/bin/env bash

set -ex

docker kill devbox || true
docker rm devbox || true

docker build -t devbox .

docker run \
    -d \
    --name devbox \
    --hostname devbox \
    --network br0 \
    -v /mnt/user/development/devbox/home_dir:/home/kfb \
    devbox

NB.: If you’re trying to run this, put the Dockerfile in /mnt/user/development/devbox/docker. Otherwise, your docker build context will include home_dir and will be super slow to start.

Setting up VSCode

VSCode was significantly easier to set up, since remote deelopment is supported out of the box.

I don’t need to rewrite the instructions on that page, since it all worked as advertised. I spent a little bit of time making sure the integrated terminal looked right, but that’s because of my zsh configuration. I’m able use any VSCode plugins, including remote debugging of Rust programs. I worked through most of this year’s AoC that way. Having 24 cores came in especially handy for the days I couldn’t figure out the efficient implementations.

Side Quest: Locking Down SSH and Setting a Static Host Key

The final annoyance was that the devbox’s SSH host key would sometimes get regenerated when the image was rebuilt. Not a big deal, but I had come too far to leave this one bit hanging. I created a set of static ssh keys that are added to the container so that it remains constant. It wasn’t hard to generate the keys:

$ mkdir -p ./etc/ssh
$ ssh-keygen -A -f `pwd`

I also locked down SSH, disabling password logins and root access. It was 3 lines in a config file:

# Change some defaults like disabling root login and passwords
ChallengeResponseAuthentication no
PasswordAuthentication no
PermitRootLogin no

The COPY to handle these files is in the Dockerfile above.

Wrapping Up

All in all, this was a fun setup to put together. My next project will be to learn nix, which will probably mean redoing a bit of this setup. I’ll see if that’s interesting enough to write about here, though.