Spicule - Data Processing Experts

Using LXD Containers!


I don’t know about you, but I’m a Linux user and in my home directory I have a Projects directory. In this directory is the culmination of all the work I’ve done on this laptop and probably some projects from before then as well. In that directory, I have Java projects, Scala projects, ETL projects, Big Data, Small Data, pretty much everything. This isn’t necessarily a bad thing, I have everything I need in front of me and the tooling I require to interact with these projects, but over time the laptop gets congested. I have a miriad of old MySQL databases kicking around, old builds of Apache OODT which I have no idea why I built them and so on. Also, I develop on Ubuntu, but what if my target server is another Linux flavour? Wouldn’t it be best to test on that so I know what will happen when its deployed? Back in the day I’d have a bunch of chunky Virtual Box images taking up disk space for the random time I needed them, or I’d spin up an EC2 instance, but these days are a thing of the past…. Docker has been making a lot of waves recently with the various pre packaged apps you can download and run. Docker builds off of the container support that LXC provides, but no one has heard of LXC right? Maybe Canonical didn’t do enough marketing when it was release, it’s certainly been overshadowed by Docker, or perhaps they knew LXD was in the pipeline and were just holding off for LXD. Who knows, but one thing I do know, is, LXD absolutely rocks.


Okay, LXD. I’m running Xenial where LXD is a first class citizen, I know backports are on their way or available for other platforms, but I like bleeding edge so hey, why not go Xenial. To get LXD app I do is run apt-get install lxd. Unsurprisingly once that is finished I have a running LXD installation. I’m running on Ubuntu so lets get a few different Ubuntu images to get started. lxd-images import ubuntu trusty amd64 –alias ubuntu-trusty lxd-images import ubuntu xenial amd64 –sync –alias ubuntu-xenial By running these 2 commands we’re instructing LXD to go to the image server and download a stable trusty and xenial image. I can see them by running lxc image list
ubuntu-trusty 510c27eb5e30 no Ubuntu 14.04 LTS server (20160222) x86_64 118.51MB Feb 29, 2016 at 10:54pm (UTC)
ubuntu-xenial 618ab2ddf554 no Ubuntu 16.04 LTS server (20160223.1) x86_64 139.53MB Feb 27, 2016 at 3:15pm (UTC)
You can see the size listed, 140MB for the latest Xenial. Taking up lots of space on your hard disk, it is not. The other great thing about images is they are cached locally and I can sync them. So I can do lxd-images sync and it will check my containers and download any newer releases. New releases are shipped for security updates and container fixes so they are worth keeping in sync. Of course the burning question now is, how do I get started? Easy, lxc launch ubuntu-trusty mydemocontainer because the image has been downloaded and sync’d startup takes seconds, not bad eh? Certainly a lot quicker than cracking out your virtual box image. You’ll also notice I didn’t prepend that command with sudo. One of the great things about LXD is there is no requirement for super user privilveges and as such anyone with the correct group membership can spin up lightweight containers on the server. To see the status of your images you can run lxc list and it will return something like:
juju-87b2a434-8e29-4a84-87fb-637628ca86c6-machine-0 RUNNING (eth0) PERSISTENT 0
mydemocontainer RUNNING (eth0) PERSISTENT 0
Here you can see I have 3 running containers. Each has a RUNNING state, an IP address, a type and the number of snapshots taken. If you are running a remote development server like me, you can add a VPN to the server and make those IP addresses accessible if you need to see things running in a browser or similar. OpenVPN is reasonably quick and painless to setup and I can securely visit from my laptop and see what that container is doing. Normal operations are pretty self explanatory
lxc restart mydemocontainer
lxc stop mydemocontainer
lxc start mydemocontainer



The easiest way to access your new image is by “executing” Bash on the instance. To do this just run lxc exec mydemocontainer /bin/bash and you will be dropped into a bash prompt. If you’d rather use SSH for remote access over VPN etc, then you can put your public key in /home/ubuntu/authorized_keys and you’ll be able to SSH in after that. Once you have a bash prompt the container is a fully functioning Linux box, so you can start to do whatever you require to get your Linux box up and running, even install X11 if you require it!


To get files on and off the platform you can setup SSH and use SCP/Rsync or for simple workloads you can do
lxc file pull mydemocontainer/etc/hosts .
lxc file push hosts mydemocontainer/etc/hosts


Of course you may not want all your resources being hogged by a single image so there are a number of configuration options you can set on your container, for example


lxc config set mydemocontainer limits.memory 256M


lxc config set mydemocontainer limits.cpus 1


On my server I’m running a ZFS pool. ZFS is fast and supports great snapshotting and it makes LXC/LXD snapshots super quick. For example I can create a snapshot on a running container with
 lxc snapshot mydemocontainer snapshot-name
and within a second or two the snapshot is complete. I can then restore it with
 lxc restore mydemocontainer snapshot-name
and again within a couple of seconds everything is back as it was. I don’t even shut down the container!



To run anything in LXD you need an image, but if you have a setup you like for ETL work, or a setup you like for Java Dev work? It’s a pain recreating that same setup everytime. Using the snapshots though you just need to do it once. Create your image in the way you like it then create a snapshot like we did above:
 lxc snapshot mydemocontainer my-dev-env
Then we publish it with
 lxc publish mydemocontainer/my-dev-env --alias development-environment
Again, in a short space of time we’ve created our environment and then published that snapshot to the image repository, you can see your new image with the lxc image list command.


Okay, so we’ve dealt a lot with Trusty and Xenial. What if I want a different flavour Linux? Well LXD supports the concept of remote image servers. This allows us to add a new remote and get different images. Here we’re going to add the linuxcontainers.org repository:
lxc remote add images https://images.linuxcontainers.org/ --public
Then you can see it listed in lxc remote list. Next you can list the images now available lxc image list images: |less, you can see we instantly have access to Centos, Debian, Gentoo, Oracle and Plamo images and can spin them up instantly.
lxc launch images:centos/7/amd64 mycentosbox
lxc launch images:gentoo/current/amd64 mygentoobox
There are also privileged container images available for Fedora and OpenSuse. Thats a pretty comprehensive image list.


So this is a very quick overview of LXC/LXD containers and how to manipulate them. Docker lovers, we don’t dislike Docker, but LXD currently rocks for userspace containers that aren’t prepackaged for a specific purpose. Be it, development work, production systems that need separating, or something else, there are a miriad of use cases for LXD and as you can see, with the speed of the platform, there is no longer a reason not to use containers.


Containers Lxd