Setting up a virtual machine using User-Mode Linux

Thursday 22 April 2004

I want to be able to set up backup servers running in virtual machines. The idea being that if a server fails, I will have a ready-made backup machine to bring on-line at a moment's notice.

Source Material

My main source of information and downloads is the User-mode Linux Kernel Home Page on SourceForge.

What To Do

The main steps in setting up a virtual machine appear to be:
  1. Download and compile a UML kernel.
  2. Create or download a filesystem to run the UML kernel in.
  3. Start the virtual machine and configure it to do something useful.
My host machine is a RedHat 9 machine installed using the "everything" option and running an updated 2.4.24 kernel. It has two Athlon 1800 processors and 1GB of RAM. I have created the directory /local/mckenzie/UML which is on a 12 GB filesystem and will try to use the usual subdirectory structure bin, src, var etc.

Downloading and Compiling a UML Kernel

I have downloaded a UML patch uml-patch-2.4.24-2.bz2 from the UML home page. I am slightly concerned that the revision level is different, but I'll go ahead regardless.
I unpacked the source for kernel 2.4.24, then unzipped the uml patch and copied it to the root of the kernel source tree and ran the command: 
% patch -p1 < uml-patch-2.4.24-2
which created/patched a lot of files in arch/um/, include/ and some others.
Next I ran make xconfig ARCH=um and configured a kernel with the default parameters.
I then did: 
% make linux ARCH=um
The compile failed with the error: 
aio.c:12: linux/aio_abi.h: No such file or directory
I found this file in the source for kernel 2.6.5, and put it into arch/um/include, then edited arch/um/os-Linux/aio.c to look for aio_abi.h rather than linux/aio_abi.h, but this just produced more errors, so I undid these changes.

I went back into the general setup section of the kernel configuration and removed support for Separate kernel address space and /proc/mm, then ran: 

% make clean ARCH=um
% make dep ARCH=um
% make linux ARCH=um
But it still didn't work.

I unpacked a fresh copy of the 2.4.24 kernel source, then applied the other uml patch - uml-patch-2.4.24-1, and tried again with the same configuration options. This time it worked and an executable called linux was created.

Providing a root filesystem for my kernel to run in

It is possible to download ready-made UML filesystems from the User-mode linux home page but I would prefer to do it myself. I will try to use UML Builder, which can be downloaded from the UML Builder home page on Sourceforge. I downloaded the rpm umlbuilder-1.40-5.i386.rpm and installed it and ran umlbuilder_gui. This version of UML Builder only supports the use of Redhat 8, but I have a set of RPMs for Redhat 8 sitting about, so I pointed it at them. I also gave it the hostname redhat-uml and IP address 192.168.1.2, and told it to live in /local/mckenzie/UML/redhat-uml/. The process failed, complaining that it couldn't see the file /usr/lib/uml/modules-2.4.tar. I created this file as an empty archive, which got it past that problem (I have compiled a monolithic kernel, so this shouldn't become an issue). It then failed because it couldn't start the UML virtual machine, which is not surprising as it didn't know where I had put the linux executable. However, by this time it had created a filesystem (in a file) and unpacked a load of rpms into it, so I am hopeful that this is OK. Looking in /local/mckenzie/UML/redhat-uml/ I now see: 
>% ls -l
total 10572
-r-xr-xr-x    1 mckenzie users        4982 Apr 23 17:57 control*
-rwxr-xr-x    1 mckenzie users    10766166 Apr 23 18:03 linux*
drwxr-xr-x    2 mckenzie users        4096 Apr 23 17:57 mnt/
-rw-r--r--    1 mckenzie users    1073741824 Apr 23 17:57 rootfs
-rw-r--r--    1 mckenzie users    134217728 Apr 23 17:57 swapfs
%
so I tried to start the virtual machine with: 
% ./linux ubd0=rootfs
The kernel loaded OK, but panicked and stopped when it tried and failed to mount the root filesystem. There was no kernel message confirming support for ext2 or ext3 filesystems (although both are compiled in) so I tried running umlbuilder_gui to create a reiserfs filesystem, but although it claimed to have the support, it still couldn't mount the filesystem. I re-made the filesystem as ext2, and tried to mount it as a loopback device: 
# mount -o loop -t ext2 rootfs /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       or too many mounted file systems
#
So I now suspect that umlbuilder is not creating a good filesystem.

I tried downloading one of the ready-made filesystems from sourceforge. I used an ext2 filesystem that contains Debian-3.0r0. I could mount the filesystem using: 

# mount -o loop -t ext2 Debian-3.0r0.ext2 /mnt/tmp
and boot my UML kernel into it using: 
./linux ubd0=Debian-3.0r0.ext2
Sadly the system seemed to be broken, but at least I have verified that my kernel is good, umlbuilder is pants and all I need is to make a good root filesystem. I have kept a copy of the control script created by uml builder as it might contain some useful information.

Making a root filesystem using mkrootfs

The script mkrootfs is available from http://www.stearns.org/mkrootfs/ along with mkemptyfs, mkswapfs, functions, samlib and resize-fs-filesystem. samlib was to be installed in the directory /usr/lib/samlib/. mkrootfs is a long script, and almost certainly needs some customisation. It appears to want to run as root, and to only make files in the current directory, so I think it should be copied (along with its companion scripts) to the proposed working directory and run there.

As a first attempt, I changed some lines to be the following: 

DIST=${1:-rh-9}
RPMDIR=/d/redhat/RH-9-i386/RedHat/RPMS
I then ran the script. After a very long time when it appeared to be checking all of the available RPMS and assembling a dependencies tree or something, it appeared to try to install some of them. However it immediately started spitting out a whole load of failure messages, starting with: 
**** warning: /d/redhat/RH-9-i386/RedHat/RPMS/glibc-2.3.2-11.9.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
In the absence of glibc, I doubt I will obtain a usable filesystem. However, I can use information gleaned from mkrootfs to build a filesystem manually, then incorporate this into a script of my own (that will probably be very specific to this system). Despite my scepticism, my kernel could boot and mount the resulting filesystem, although it didn't get as far as offering a login prompt. The last messages I got were: 
Starting crond: [  OK  ]
INIT: Id "0" respawning too fast: disabled for 5 minutes
INIT: Id "2" respawning too fast: disabled for 5 minutes
INIT: Id "c" respawning too fast: disabled for 5 minutes
INIT: Id "1" respawning too fast: disabled for 5 minutes
INIT: no more processes left in this runlevel

Making a Filesystem "By Hand"

I did the following: 
% dd if=/dev/zero of=root_fs bs=1048576 count=512
512+0 records in
512+0 records out
% ls -l
total 524804
-rw-r--r--    1 mckenzie users    536870912 Apr 28 10:32 root_fs
% /sbin/mkfs -t ext3 root_fs
mke2fs 1.32 (09-Nov-2002)
root_fs is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=1
64 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
%
and as root: 
# mount -o loop -t ext3 root_fs /mnt/tmp
# ls -al /mnt/tmp
total 17
drwxr-xr-x    3 mckenzie users        1024 Apr 28 10:32 .
drwxr-xr-x   10 root     root         4096 Mar  8 11:18 ..
drwx------    2 root     root        12288 Apr 28 10:32 lost+found
#
so I have made a 512MB filesystem which I can mount.

Installing Some RPMs

The package manager rpm has some potentially useful switches:

After a few iterations of running rpm, then adding more rpms to satisfy the failed dependencies (the --aid switch didn't work as I had expected), I did: 

# rpm --root /mnt/tmp -i --aid filesystem-2.2.1-3.i386.rpm dev-3.3.2-5.i386.rpm basesystem-8.0-2.noarch.rpm bash-2.05b-20.i386.rpm rpm-4.2-0.69.i386.rpm setup-2.5.25-1.noarch.rpm coreutils-4.5.3-19.i386.rpm mktemp-1.5-18.i386.rpm glibc-2.3.2-11.9.i386.rpm elfutils-0.76-3.i386.rpm popt-1.8-0.69.i386.rpm shadow-utils-4.0.3-6.i386.rpm sed-4.0.5-1.i386.rpm libtermcap-2.0.8-35.i386.rpm gawk-3.1.1-9.i386.rpm bzip2-libs-1.0.2-8.i386.rpm elfutils-libelf-0.76-3.i386.rpm info-4.3-5.i386.rpm findutils-4.1.7-9.i386.rpm grep-2.5.1-7.i386.rpm pam-0.75-48.i386.rpm glibc-common-2.3.2-11.9.i386.rpm termcap-11.0.1-16.noarch.rpm initscripts-7.14-1.i386.rpm ncurses-5.3-4.i386.rpm zlib-1.1.4-8.i386.rpm pcre-3.9-10.i386.rpm cracklib-2.7-21.i386.rpm cracklib-dicts-2.7-21.i386.rpm db4-4.0.14-20.i386.rpm glib-1.2.10-10.i386.rpm redhat-release-9-3.i386.rpm iputils-20020927-2.i386.rpm chkconfig-1.3.8-1.i386.rpm psmisc-21.2-4.i386.rpm iproute-2.4.7-7.i386.rpm procps-2.0.11-6.i386.rpm SysVinit-2.84-13.i386.rpm e2fsprogs-1.32-6.i386.rpm mingetty-1.01-1.i386.rpm modutils-2.4.22-8.i386.rpm mount-2.11y-9.i386.rpm net-tools-1.60-12.i386.rpm sysklogd-1.4.1-12.i386.rpm util-linux-2.11y-9.i386.rpm which-2.14-5.i386.rpm words-2-21.noarch.rpm 
warning: filesystem-2.2.1-3.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
/var/tmp/rpm-tmp.36112: line 1: /dev/null: No such file or directory
This is a good start, and the filsystem appears to have been populated with an appropriate amount of stuff. However, a closer look shows that it is missing /etc/fstab and /etc/hosts, so I created these two: 
# cat > /mnt/tmp/etc/fstab
/dev/ubd0       /               ext3    defaults        1 1
none            /proc           proc    defaults        0 0
none            /dev/pts        devpts  gid=5,mode=620  0 0
/dev/ubd1       swap            swap    defaults        0 0
# cat > /mnt/tmp/etc/hosts
127.0.0.1       localhost
#
I tried booting my kernel with this filesystem, but it failed, the probable cause being a lack of entries in /dev for the virtual disk /dev/ubd0, so I tried to remedy this: 
# mount -o loop -t ext3 root_fs /mnt/tmp
# cd /mnt/tmp/dev
# mknod --mode=660 ubd0 b 98 0
# chown root:disk ubd0
# mknod --mode=660 ubd1 b 98 1
# chown root:disk ubd1
# cd
# umount /mnt/tmp
I tried to run my kernel in this filesystem using: 
% ./linux ubd0=root_fs
and it worked. The kernel booted with a few errors (more of this later) and opened 7 xterm windows each containing a virtual terminal. I logged into the first as root, and was given a shell prompt without being asked for a password. I would like to set a password for root, so I decided to install the passwd package. I stopped the virtual machine with halt then I mounted its root filesystem on /mnt/tmp and ran the following command: 
# rpm --root /mnt/tmp -i passwd-0.68-3.i386.rpm glib2-2.2.1-1.i386.rpm libuser-0.51.7-1.i386.rpm openldap-2.0.27-8.i386.rpm cyrus-sasl-2.1.10-4.i386.rpm cyrus-sasl-md5-2.1.10-4.i386.rpm openssl-0.9.7a-2.i386.rpm gdbm-1.8.0-20.i386.rpm krb5-libs-1.2.7-10.i386.rpm 
warning: passwd-0.68-3.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
#
When I restarted the virtual machine I could set a root password, and verify that the passwd package was installed.

Kernel error messages

I see the error: 
Checking for /dev/anon on the host...Not available (open failed with errno 2)
when I boot my UML kernel. There is an explanation of /dev/anon here. Having read it I decided I could safely ignore the error.
I also saw errors of the sort: 
modprobe: Can't open dependencies file /lib/modules/2.4.24-1um/modules.dep (No such file or directory)
I tried creating the directory /lib/modules/2.4.24-1um/ (which did not exist), but this gave rise to more errors of the sort: 
depmod: /lib/modules/2.4.24-1um/modules.ieee1394map is not an ELF file
which I shall ignore, until I can be bothered to create the necessary files (with a make modules_install probably).
Finally I see errors relating to the lack of a swap device: 
swapon: /dev/ubd1: No such device
First I need to make a swap file: 
% dd if=/dev/zero of=swapfs bs=1048576 count=128
128+0 records in
128+0 records out
% /sbin/mkswap swapfs
Setting up swapspace version 1, size = 134213 kB
%
I then tried to start the virtual machine with the following command: 
% ./linux ubd0=root_fs ubd1=swapfs
but I still get the "no such device" error. After much fiddling about, with no success, I decided to try reformatting the file swapfs as an ext3 filesystem that I could mount (eg. under /swap) and create a swapfile in that mounted partition. When I tried mounting /dev/ubd1, I found that the virtual machine simply mounted another copy of /dev/ubd0 instead. I don't understand exactly what's going on, but for the time being I shall ignore swap, and if it is needed I will simply use a bigger root filesystem and use a swapfile.

Setting the Memory Size

I can set the memory allocated to the virtual machine by adding eg. mem=64m to the command line. So I now have: 
./linux ubd0=root_fs mem=64m
which appears to work, given the kernel message: 
Memory: 60996k available

Networking

Networking appears to depend on the host supporting the TUN/TAP device. I recompiled the host kernel to include this driver. I rebooted the host with this new kernel, and saw the following message: 
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
so I think it is working. However, this hasn't created the required device file /dev/tap0. To get networking for my UML virtual machine, it appears that I need some of the utilities in the uml_utilities package.

Installing uml_utilities

I downloaded uml_utilities_20040406.tar.bz2 from sourceforge, and unpacked it. The instructions were simply to do:
make all
make install DESTDIR=/local/mckenzie/UML
but this failed when trying to compile the mconsole utility, complaining that it couldn't find the header files readline/readline.h and readline/history.h. I installed the readline-devel and ncurses-devel packages, and all compiled fine.

Configuring the tap device

I used the tunctl utility, which I had just compiled, to create a tap0 device: 
# /local/mckenzie/UML/bin/tunctl -u 100
Set 'tap0' persistent and owned by uid 100
#
This created a tap device that was owned by uid 100, but did not appear to create any entries in /dev (or anywhere else, if find is to be believed). As far as I can gather this device will be lost when the system is rebooted. I can configure the tap device as a network interface: 
# ifconfig tap0 192.168.1.2 up
# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:02:B3:8B:7C:BD  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5968695 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3852917 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3290657508 (3138.2 Mb)  TX bytes:2886097475 (2752.3 Mb)
          Interrupt:18 Base address:0x2480 Memory:e9022000-e9022038 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3898 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3898 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:208884 (203.9 Kb)  TX bytes:208884 (203.9 Kb)

tap0      Link encap:Ethernet  HWaddr 00:FF:91:38:D1:1D  
          inet addr:192.168.1.2  Bcast:192.168.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
	  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

#
I then configured a bridge device to allow the UML machine direct access to our network: 
# brctl addbr br0
# ifconfig eth0 0.0.0.0 promisc up
# ifconfig tap0 0.0.0.0 promisc up
# ifconfig br0 192.168.1.1 netmask 255.255.255.0 up
# brctl stp br0 off
# brctl setfd br0 1
# brctl sethello br0 1
# brctl addif br0 eth0
# brctl addif br0 tap0
# chmod 666 /dev/net/tun
#
To see my attempt to understand what was going on with the bridging setup, see my notes on the subject.

I started mt UML virtual machine with the command: 

% ./linux ubd0=root_fs mem=64m eth0=tuntap,tap0
and it came up with a device eth0 which I could configure using ifconfig and access the network.

A consequence of configuring the bridge device br0 was that the host machine lost its default root. I restored it with : 

# route -n add default gw 192.168.1.254
I edited /etc/resolv.conf, /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0 on the UML machine so that it would configure its networking automatically at boot time. I now had a machine that appeared to be able to use the network correctly.

A copy of my kernel and filesystem can be found here.

Please email comments or suggestions to me at mckenzie@biochem.ucl.ac.uk
Duncan.