Nested KVM set up on Fedora 22 && Running devstack on Ubuntu 14.04 guests

Following bellow are brief instructions how to achieve extremely high performance of VMs created via devstack ( stack.sh ) inside another virtual machine created with Fedora 22 KVM Hypervisor and having Nested KVM feature enabled, working with sufficiently advanced Intel CPUs (Haswell kernel or above which have newer hardware virt extensions ) and 16 GB or more RAM.

****************************************
Create non-default libvirt subnet
****************************************

1. Create a new libvirt network (other than your default 198.162.x.x) file:

$ cat devstackvms.xml

<network>
   <name>devstackvms</name>
   <uuid>d0e9964a-f91a-40c0-b769-a609aee41bf2</uuid>
   <forward mode='nat'>
     <nat>
       <port start='1024' end='65535'/>
     </nat>
   </forward>
   <bridge name='virbr1' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6e'/>
   <ip address='192.157.141.1' netmask='255.255.255.0'>
     <dhcp>
       <range start='192.157.141.2' end='192.157.141.254' />
     </dhcp>
   </ip>
 </network>


 $ virsh net-define devstackvms.xml

 Then start the network and enable "autostart"

 $ virsh net-start devstackvms
 $ virsh net-autostart devstackvms


4. List your libvirt networks to see if it reflects:

$ virsh net-list

  Name              State      Autostart     Persistent
  ----------------------------------------------------------
  default              active     yes           yes
 devstackvms      active     yes           yes



Launch VM Ubuntu1404 attached to subnet created. Set Disk && Network to "Virtio" mode before start installation 

**********************************************************************************
 Procedure to enable nested virtualization (on Intel-based machines) [ 1 ]
**********************************************************************************

1. List modules and ensure KVM Kernel modules are enabled on L0:

    $ lsmod | grep -i kvm
    kvm_intel             133627  0
    kvm                   435079  1 kvm_intel


2. Show information for `kvm_intel` module:

    $ modinfo kvm_intel | grep -i nested
    parm:           nested:boolkvm                   435079  1 kvm_intel


3. Ensure nested virt is persistent across reboots by adding it as a
   config directive:

    $ cat /etc/modprobe.d/dist.conf
    options kvm-intel nested=y


4. Reboot the host.


5. Check if the Nested KVM Kernel module option is enabled:

    $ cat /sys/module/kvm_intel/parameters/nested
    Y


6. Before you boot your L1 guest (i.e. the guest hypervisor that runs
   the nested guest), expose virtualization extensions to it. The
   following exposes all the CPU features of host to your guest
   unconditionally:

    $ virt-xml Ubuntu1404 --edit  --cpu host-passthrough,clearxml=yes


7. Start your L1 guest (i.e. guest hypervisor):

    $ virsh start Ubuntu1404  --console


8. Ensure KVM extensions are enabled in L1 guest by running the below
   command:

$ file /dev/kvm      
    /dev/kvm: character special


You might enable Shadow VMCS, APIC Virtualization on the physical host (L0):
    $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
    Y

    $ cat /sys/module/kvm_intel/parameters/enable_apicv
    N

    $ cat /sys/module/kvm_intel/parameters/ept
    Y

 

   

***************************************************************
Devstack installation procedure on Ubuntu 14.04.2 VM
***************************************************************


$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack

********************************************
Create local.conf
********************************************

[[local|localrc]]
HOST_IP= 192.157.141.57
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

FLOATING_RANGE=192.168.12.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.12.150,end=192.168.12.254
PUBLIC_NETWORK_GATEWAY=192.168.12.15

# Useful logging options for debugging:
DEST=/opt/stack
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen

# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service  n-cauth
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest


Then run ./stack.sh

 

 

  

  

****************************************************************************
To provide outbound  connectivity  run from within VM running stack instance
****************************************************************************

 # iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE



****************************************************************************
To provide inbound  connectivity (from host running KVM Hypervisor)
to VMs (L2) created  run from within VM (L1)
****************************************************************************

# route add -net 192.168.1.0/24  gw 192.157.141.57 

where 192.157.141.57 is KVM's IP on non-standard libvirt subnet - devstackvms  192.168.1.0/24 is subnet hosting machine 192.168.1.47 running KVM Hypervisor


On machine 192.168.1.47 (L0) ,which is Fedora 22 box plus KVM/QEMU/LIBVIRT
run :-

# route add -net 192.168.12.0/24 gw 192.157.141.57


where 192.168.12.0/24 is devstack public network ( view local.conf).