Using vagrant with lxc

One command; vagrant up

Now that I successfully managed to get vagrant-lxc plugin to work (see previous post), I decided to migrate my virtualbox Vagrantfile to vagrant lxc. I have used vagrant and virtualbox to create and bring up 3 vm guest nodes on my laptop and then use node1 as a dev machine, node2 as a staging machine, and node3 as a production machine. That way I can test easily with greater confidence with each promotion of new code.

So why use lxc when virtualbox is working fine? The answer is obvious for anyone who is familiar with these platforms… lxc is much much faster to bring up, halt, tear down.. and you can do quich-n-dirty temporary testing with lxc-start-ephemeral. Just too many advantages for developing and testing code in a small network.

The drawback is that vagrant-lxc is not as mature as vagrant with virtualbox. But the lxc tools easily get you around all this. I’ll discuss some tips on this too.

Getting started with lxc

This all assume ubuntu (translation for using another Linux platform is an exercise for the reader)

sudo apt-get install lxc
mkdir -p $HOME/tmp/vagrant-lxc
cd $HOME/tmp/vagrant-lxc
# keep it simple
# NOTE: all lxc-xxxx commands will need sudo in this article
# user space lxc is possible but we are not going there today.
sudo lxc-create -n node1 -t ubuntu
sudo lxc-start -n node1
sudo lxc-ls --fancy # Note the IP is assigned by dhcp here (10.0.3.xxx)
sudo lxc-console -n node1 # login with vagrant/vagrant
# To EXIT use CTRL-a q ... if you are using screen then CTRL-a a q
sudo lxc-stop -n node1

This gives just enough info to play with a single container. By the way, while this container is running, it is NAT’ed to the outside world and therefore you can reach the internet from within the guest container vm.

But I want 3 containers minimum. For this next exercise we will use vagrant along with vagrant-lxc plugin to build three nodes. However vagrant requires each of its nodes to have a user named vagrant with sudo priviledges. The node build above does not have that… so we will destroy the one above and build 3 nodes from the ground up with vagrant. Beside this exposes us to new commands and features along the way.

To destroy the container already built we run lxc-destroy. Before we do, it is worth taking a look at what lies just under the covers and controls lxc containers. Run this to view some of the improtant files created for the node1 container built previously.

sudo ls -l /var/lib/lxc/
sudo ls -l /var/lib/lxc/node1
sudo cat /var/lib/lxc/node1/config

There is much you can do by editing the config file but save that research for later. To destroy this container run

sudo lxc-destroy -n node1
sudo ls -l /var/lib/lxc # just to see what happened

We are going to use vagrant now to build and start three nodes with one command and one config file. But you need to install vagrant and vagrant-lxc. The vagrant application is available in you standard ubuntu repo but it maybe a slightly older version and the plugin we are going to install (vagrant-lxc) relies on a newer version of vagrant. Go to this link and download the deb pkg: debian vagrant pkg Install it with sudo dpkg -i vagrant-xxxx.deb If you previously had vagrant installed you may have to rm $HOME/.vagrant.d/plugins.json Now install the needed lxc plugin:

vagrant plugin install vagrant-lxc

That may seem like a lot to do to get 3 simple lxc container nodes up and running but trust me, this will save a lot of time and steps in the future. One more thing is needed here. The vagrant app requires a file to control it and it by default is called Vagrantfile. I am going to provide a simple one here. It has some comments within; understanding the rest is again for later. By the way vagrant is a wrapper bash script for setting up and running ruby commands so you will need ruby installed.

Sideline: I respect and demand ruby on my development systems. It has incredible power and forces you to understand object oriented programming (that is a good thing). It’s weakness is version mixing which can create chaos - there are tools to help like rvm but it is still ugly. Much of this comes from the rapid growth and changes ruby has gone through. Because vagrant is ruby based, the config file (Vagrantfile) is “rubyesque” and therefor has strong object oriented syntax. In the end - learning ruby is well worth and discomfort along the way.

cd $HOME/tmp/vagrant-lxc # created earlier above
vi Vagrantfile
# enter these lines 

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"
# make domain whatever you want - it really is not that critical normally
domain="companionway.lab"
nodes=3
# Do NOT make the next network segment the same as lxc default (10.0.3.x)
# the "2" at the end is to just allow us to easily set ...21 ...22 ...23 etc
ipAddrPrefix = "10.0.30.2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  #config.ssh.username = "myuser"
  #config.ssh.password = "passwd4myuser"
  #config.vm.provision :shell, :inline => "apt-get update --fix-missing"
  config.vm.box = "fgrehm/precise64-lxc"
  # This next block is if you want to run an ansible playbook
  # Ansible can be used to complete provision your virtual machines
  # and keep them up to date
  #config.vm.provision :ansible do |ansible|
  #  ansible.playbook = "playbook.yml"
  #end
  (1..3).each do |i|
    nodeName = "node#{i}"
    config.vm.define nodeName do |node|
      # Do NOT make lxc__bridge_name = your default lxc bridge (lxcbr0) 
      # and yes there are 2 underscores between lxc__bridge.... got me as to why.
      node.vm.network :private_network, ip: "#{ipAddrPrefix}#{i}",lxc__bridge_name: "lxcbr1"
      node.vm.host_name = "#{nodeName}.#{domain}"
      node.vm.provider :lxc do |lxc|
        lxc.container_name = "#{nodeName}"
        lxc.customize 'cgroup.memory.limit_in_bytes', '512M'
      end
    end
  end
end

To run this:

vagrant up --provider=lxc

The --provider=lxc argument is to force vagrant to loadup the virtual machines using lxc containers as opposed virtualbox or other “providers”. That is one of the nice features of vagrant, it is virtual machine guest provider agnostic. You might be able to get away with dropping the argument and just run:

vagrant up

Here a few quick-n-easy commands using vagrant:

vagrant halt node1 # obvious
vagrant halt       # shuts down all the nodes listed in Vagrantfile
vagrant status
vagrant connect node1
vagrant ssh node1
vagrant destroy node1 
vagrant help

Simple … but wait, there’s more. Lots of things can be configured through vagrant Vagrantfile and one of them is the provisioning tool called ansible.

You will see a block of code in this Vagrantfile that calls ansible to run a playbook.yml file…. I will save that for an upcoming post.

Enjoy

-g-