Sunday, April 19, 2015

Nested KVM set up on Fedora 22 && Running devstack on Ubuntu 14.04 guests

Following bellow are brief instructions how to achieve extremely high performance of VMs created via devstack ( ) inside another virtual machine created with Fedora 22 KVM Hypervisor and having Nested KVM feature enabled, working with sufficiently advanced Intel CPUs (Haswell kernel or above which have newer hardware virt extensions ) and 16 GB or more RAM.

Create non-default libvirt subnet

1. Create a new libvirt network (other than your default 198.162.x.x) file:

$ cat devstackvms.xml

   <forward mode='nat'>
       <port start='1024' end='65535'/>
   <bridge name='virbr1' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6e'/>
   <ip address='' netmask=''>
       <range start='' end='' />

 $ virsh net-define devstackvms.xml

 Then start the network and enable "autostart"

 $ virsh net-start devstackvms
 $ virsh net-autostart devstackvms

4. List your libvirt networks to see if it reflects:

$ virsh net-list

  Name              State      Autostart     Persistent
  default              active     yes           yes
 devstackvms      active     yes           yes

Launch VM Ubuntu1404 attached to subnet created. Set Disk && Network to "Virtio" mode before start installation 

 Procedure to enable nested virtualization (on Intel-based machines) [ 1 ]

1. List modules and ensure KVM Kernel modules are enabled on L0:

    $ lsmod | grep -i kvm
    kvm_intel             133627  0
    kvm                   435079  1 kvm_intel

2. Show information for `kvm_intel` module:

    $ modinfo kvm_intel | grep -i nested
    parm:           nested:boolkvm                   435079  1 kvm_intel

3. Ensure nested virt is persistent across reboots by adding it as a
   config directive:

    $ cat /etc/modprobe.d/dist.conf
    options kvm-intel nested=y

4. Reboot the host.

5. Check if the Nested KVM Kernel module option is enabled:

    $ cat /sys/module/kvm_intel/parameters/nested

6. Before you boot your L1 guest (i.e. the guest hypervisor that runs
   the nested guest), expose virtualization extensions to it. The
   following exposes all the CPU features of host to your guest

    $ virt-xml Ubuntu1404 --edit  --cpu host-passthrough,clearxml=yes

7. Start your L1 guest (i.e. guest hypervisor):

    $ virsh start Ubuntu1404  --console

8. Ensure KVM extensions are enabled in L1 guest by running the below

$ file /dev/kvm      
    /dev/kvm: character special

You might enable Shadow VMCS, APIC Virtualization on the physical host (L0):
    $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs

    $ cat /sys/module/kvm_intel/parameters/enable_apicv

    $ cat /sys/module/kvm_intel/parameters/ept


Devstack installation procedure on Ubuntu 14.04.2 VM

$ git clone
$ cd devstack

Create local.conf



# Useful logging options for debugging:

# The default fixed range ( conflicted with an address
# range I was using locally.

# Services
disable_service n-net
enable_service  n-cauth
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

Then run ./





To provide outbound  connectivity  run from within VM running stack instance

 # iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

To provide inbound  connectivity (from host running KVM Hypervisor)
to VMs (L2) created  run from within VM (L1)

# route add -net  gw 

where is KVM's IP on non-standard libvirt subnet - devstackvms is subnet hosting machine running KVM Hypervisor

On machine (L0) ,which is Fedora 22 box plus KVM/QEMU/LIBVIRT
run :-

# route add -net gw

where is devstack public network ( view local.conf).