Understanding bhyve: How-To run Ubuntu & CentOS under FreeBSD

Years ago, I resolved to only master FreeBSD and to leave Linux and Windows to other people. I've waited years for a Hypervisor -- and I have had zero understanding of how they work until now. There are a bunch of scripts to 'help' you set up bhyve instances; however, they don't teach you want is going on and make debugging more difficult.

So, here are some notes from my learning bhyve (took about 3 or 4 hours, and I hope you get things rolling faster from the HOW-TO, dear reader). This was all on FreeBSD 10.0. I'll be upgrading to 10-STABLE to get the power button working.

Overview

bhyveloader or grub-bhyve (which I'll focus on here) is used to do the boot loading of your guest kernel. Loading the guest OS creates a device in /dev/vmm/. The next step is to launch loaded OS, you use the bhyve command. Once the OS is launched, you can connect to the guest's console and setup accounts.

Really there is a Step 0 if you are doing this over and over... you need to clear (--destroy) the old vm if you want to reuse the same vm1 name... for instance, when you reboot a vm.

Grub

In Linux, the usual root device is /dev/sda1. When using virtio, the device becomes /dev/vda1 (don't worry mondern kernels have virtio support built in so you don't have to mess with modprobe or anything from a initfsram prompt). Below, a ZFS volume is used for vm1 and a regular file for vm2.

The Ubuntu install makes for a clean booting experience using grub-bhyve, but the CentOS booting needs some massaging.

The Goal

The plan is to install Ubuntu and CentOS from the CD-ROM install iso's from the Ubuntu and CentOS websites. Here is what it looks like to boot your VM once it is installed... I put this at the beginning of the HOW-TO so you know what you are building up to.

Launching an Ubuntu VM

# cd /data/images/vm1

# cat device.map
(hd0) /dev/zvol/data/vm1
(cd0) /data/images/vm1/vm1.iso

# cat grub.in 
set root=(hd0,msdos1)
linux /vmlinuz root=/dev/vda1
initrd /initrd.img
boot

# bhyvectl --vm=vm1 --destroy
# grub-bhyve -r hd0 -m /data/images/vm1/device.map -M 2048 vm1 < grub.in > /dev/null
# bhyve -c 2 -m 2048M -H -P -A \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,vm1.iso -l com1,stdio \
    -s 4,virtio-blk,/dev/zvol/data/vm1 vm1

Prep your host FreeBSD system

Before you can even mess with the bhyve command, you need to load some kernel modules and setup your host level OS. The tap is a virtual ethernet device. The guest OS will have an 'eth0' that is actually the tapX on the host. The IP assignment is done within the guest OS. To connect the virtual ethernet device to the Internet, the tapX is added to a bridge which also has the host's uplink interface (in this example, em0). If you want to isolate guest VMs from one another, you can flag the added interfaces as private. Search for the word 'private' in the ifconfig man page.

How to get there: prep host

# kldload if_tap  
# kldload if_bridge
# kldload vmm
# kldload nmdm
# sysctl net.link.tap.up_on_open=1
# sysctl net.inet.ip.forwarding=1
# echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
# echo "vm1:dv=/dev/nmdm1B:br#9600:pa=none:" >> /etc/remote
# ifconfig tap1 create
# ifconfig bridge0 create
# ifconfig bridge0 addm tap1 addm em0 up
# fetch -o /data/distributions/ubuntu-13.10-server-amd64.iso http://releases.ubuntu.com/13.10/ubuntu-13.10-server-amd64.iso 

Install the OS into a new VM

Now that the host system is ready, you need to install the OS into your VM. First, create either a zvol or .img file to be the guest filesystem, next fetch an install ISO, and finally boot the install iso so you can install the OS onto the guest file system.

How to get there: install Ubuntu ISO onto a zvol or in a file backed .img

# cd /data/images

ZFS backed:  # zfs create -V 100G data/vm1
File backed: # truncate -s 100g vm1.img

ZFS backed:  # cat vm1-device.map
(hd0) /dev/zvol/data/vm1
(cd0) /data/distributions/ubuntu-13.10-server-amd64.iso
File backed: # cat vm1-device.map
(hd0) /data/images/vm1.img
(cd0) /data/distributions/ubuntu-13.10-server-amd64.iso

# bhyvectl --vm=vm1 --destroy
# grub-bhyve -r cd0 -m /data/images/vm1-device.map -M 2048 vm1
ZFS backed:  # bhyve -c 2 -m 2048M -H -P -A \
    -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,/data/distributions/ubuntu-13.10-server-amd64.iso \
    -s 4,virtio-blk,/dev/zvol/data/vm1 vm1
File backed: # bhyve -c 2 -m 2048M -H -P -A \
    -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,/data/distributions/ubuntu-13.10-server-amd64.iso \
    -s 4,virtio-blk,/data/images/vm1.img vm1
Similar, but different is another distribution. Here is how to get CentOS working. I used an interactive grub menu to find the kernel (type linux /[TAB] and the [TAB] filename completion shows you options... you can sniff out the kernel that way). To mix it up, I put CentOS in a .img file instead of a zvol.

How to get there: install CentOS ISO

# cd /data/images/
# truncate -s 200G vm2.img
# cat vm2-device.map
(hd0) /data/images/vm2.img
(cd0) /data/distributions/CentOS-6.5-x86_64-minimal.iso

# bhyvectl --vm=vm2 --destroy
# grub-bhyve -r cd0 -m /data/images/vm2-device.map -M 2048 vm2
grub> linux /isolinux/vmlinuz
grub> initrd /isolinux/initrd.img
grub> boot

# bhyve -c 2 -m 2048M -H -P -A \
    -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,/data/distributions/CentOS-6.5-x86_64-minimal.iso \
    -s 4,virtio-blk,/data/images/vm2.img vm2

Load Ubuntu!

If you want the Ubuntu grub menu (which I don't like as it has errors that requre hitting the enter key), you can specify the partition to boot.

Replace '-r hd0' with '-r hd0,msdos1'

# bhyvectl --vm=vm1 --destroy
# grub-bhyve -r hd0,msdos1 -m /data/images/vm1/device.map -M 2048 vm1
                   GNU GRUB  version 2.00

 +----------------------------------------------------------- +
 |Ubuntu                                                      |
 |Advanced options for Ubuntu                                 |
 |Memory test (memtest86+)                                    |
 |Memory test (memtest86+, serial console 115200)             |
 |                                                            |
 |                                                            |
 |                                                            |
 |                                                            |
 |                                                            |
 +------------------------------------------------------------+

      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a
      command-line.
   The highlighted entry will be executed automatically in 0s.

Hit Enter

  Booting `Ubuntu'

error: file `/boot/grub/x86_64-emu/all_video.mod' not found.
error: file `/boot/grub/x86_64-emu/gzio.mod' not found.
error: file `/boot/grub/x86_64-emu/part_msdos.mod' not found.
error: file `/boot/grub/x86_64-emu/ext2.mod' not found.

Press any key to continue...

At this point, you have a VM instance that has run the boot loader, but hasn't started yet. The next step is to run the loaded kernel.

Boot the kernel

  bhyve -c 2 -m 2048M -H -P -A \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,vm1.iso -l com1,stdio \
    -s 4,virtio-blk,/dev/zvol/data/vm1 vm1
Besure to read the man page for bhyve. :) One caveat: don't have the -W flag in your bhyve command when launching Linux, or your drives will not be found.

Console

Now, how do you connect to the console once you've logged out? The above example bound com1 to STDIO -- not so good unless you want to manage a pile of screen instances. Let's check the internet for the answer:
  One way is to use the cloneable null-modem driver, nmdm(4). Others 
have used tmux for this, but I'll give a quick overview of the former.

  kldload nmdm.ko before starting VMs, e.g. at boottime or in rc.conf's 
kld_list variable.

  Use the '-c' option for bhyveload to point it at one end of an nmdm 
instance, and use it in place of "stdio" in the bhyve commandline.

  bhyveload .... -c /dev/nmdm0A ...
  bhyve  ... -l com1,/dev/nmdm0A ...
- Peter on freebsd-virtualization list
Hmmm.... do I need to create this null modem? Nope! man nmdm
     /dev/nmdmN[AB]  nullmodem device nodes.  Where the A node has a matching
                     B node.

     The nmdm driver implements ``on-demand device creation'' so simply
     accessing a given instance in /dev will create it.
Node A and Node B? OK, I'll assign A to bhyve and try using tip to get attach to the B side.
# grub-bhyve -r hd0,msdos1 -m /data/images/vm1/device.map -M 2048 vm1
# bhyve -c 2 -m 2048M -H -P -A \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap1 \
    -s 3,ahci-cd,vm1.iso -l com1,/dev/nmdm1A \
    -s 4,virtio-blk,/dev/zvol/data/vm1 vm1 &
Tip is most handy with shortcuts in /etc/remote
# cat >> /etc/remote
   vm0:dv=/dev/nmdb0B:br#9600:pa=none:
   vm1:dv=/dev/nmdb1B:br#9600:pa=none:
   vm2:dv=/dev/nmdb2B:br#9600:pa=none:
   ^D
OK, now run tip! I am excited to connect, do I need an A or a B in my remote, let's find out.
tip vm1
Could not open backing file: No such file or directory
connected

[1]    Segmentation fault            bhyve -c 2 -m 2048M -H -P -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap1 -s 3,ahci-cd,vm1.iso -l com1,/dev/nmdm1A -s 4,virtio-blk,/dev/zvol/data/vm1 vm1 (core dumped)
I wasn't expecting to dump the core from running tip... this only happened once. Try again!

Clear, Load, Run, Connect!

# bhyvectl --vm=vm1 --destroy
# grub-bhyve -r hd0,msdos1 -m /data/images/vm1/device.map -M 2048
# bhyve -c 2 -m 2048M -H -P -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap1 -s 3,ahci-cd,vm1.iso -l com1,/dev/nmdm1A -s 4,virtio-blk,/dev/zvol/data/vm1 vm1 &
# tip vm1
connected

Ubuntu 13.10 ubuntu ttyS0

ubuntu login:
It works! Now you can log in to your ubuntu and setup user accounts, add an IP and gateway, add packages, etc. Then you can SSH to it remotely, yeah!

Shutdown

How do you stop a VM from within?
root@ubuntu:/etc# shutdown -P now

Broadcast message from monkey@ubuntu
	(/dev/ttyS0) at 23:26 ...

The system is going down for power off NOW!
root@ubuntu:/etc# acpid: exiting
 * Asking all remaining processes to terminate...                        [ OK ]
 * All processes ended within 6 seconds...                               [ OK ]
 * Deactivating swap...                                                  [ OK ]
 * Will now halt
[  703.405266] reboot: System halted
Running top on the host, I see bhyve still running...
53570 root          5  23    0  2077M   182M vmidle  9   0:19   0.00% bhyve

At least cpu is at 0%. Let's try reboot! Yepr, sudo reboot from within the Ubuntu guest causes bhyve to exit.
root@ubuntu:/etc# reboot
Now, you can set up shell script wrapper that does something like this:
while true
 bhyvectl --vm=vm1 --destroy
 grub-bhyve ...
 bhyve ...
 sleep 10
end
A first stab at: /etc/rc.d/bhyve

Grow the image size

Cloning images and growing them.
We have a 'standard install' image called, 'ubuntu14-clone-me.img'.
To copy, enlarge, and then grow the guest image size, do this:
  • FreeBSD# gcp --sparce=always ubuntu14-clone-me.img vm12.img
  • FreeBSD# truncate -s 300g vm12.img
  • Guest-VM# swapoff -a
  • Guest-VM# fdisk /dev/vda (delete 2 and 5, and expand 1 and recreate 2 and 5 at end)
  • Guest-VM# resize2fs /dev/vda1 Pro tip: research sparse files, install gcp (pkg install coreutils).
  • Mistakes you can make

    1. If you use the -W flag of bhyve when booting Linux, you don't get your /dev/vda1.
    2. Using the -W with 'ahci-hd', you get bhyve to crash:
      Assertion failed: (aior != NULL), function ahci_handle_dma, file 
      /usr/src/usr.sbin/bhyve/pci_ahci.c, line 445.
                              Abort (core dumped)
      
      Details

    More documentation to come... also, the '/data/images/vm1' directory layout is messy... but now I know how to set up bhyve!