Xen Virtualization on Linux and Solaris: CPU Pinning and NUMA Topology on RDO Kilo on Fedora Server 22

Posting bellow follows up http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
on RDO Kilo installed on Fedora 22 . After upgrade to upstream version
of openstack-puppet-modules-2015.1.9 procedure of RDO Kilo install on F22
significantly changed. Details follow bellow :-

*****************************************************************************************

RDO Kilo set up on Fedora ( openstack-puppet-modules-2015.1.9-4.fc23.noarch)

*****************************************************************************************

# dnf install -y https://rdoproject.org/repos/rdo-release.rpm
# dnf install -y openstack-packstack

Generate answer-file and make update :-

# packstack --gen-answer-file answer-file-aio.txt
and set CONFIG_KEYSTONE_SERVICE_NAME=httpd

****************************************************************************
I also commented out second line in /etc/httpd/conf.d/mod_dnssd.conf
****************************************************************************

You might be hit by bug https://bugzilla.redhat.com/show_bug.cgi?id=1249482

As pre-install step apply patch https://review.openstack.org/#/c/209032/
to fix neutron_api.pp. Location of puppet templates
/usr/lib/python2.7/site-packages/packstack/puppet/templates.

Another option rebuild openstack-packstack-2015.1-0.10.dev1608.g6447ff7.fc23.src.rpm on Fedora 22
with patch 0002-Avoid-running-neutron-db-manage-twice
Place patch in SOURCES and update correspondingly spec file.

$ rpmbuild -bb openstack-packstack.spec
$ cd ../RPMS/noarch
$ dnf install openstack-packstack-2015.1-0.10.dev1608.g6447ff7.fc22.noarch.rpm
openstack-packstack-doc-2015.1-0.10.dev1608.g6447ff7.fc22.noarch.rpm
openstack-packstack-puppet-2015.1-0.10.dev1608.g6447ff7.fc22.noarch.rpm

You might be also hit by https://bugzilla.redhat.com/show_bug.cgi?id=1234042
Workaround is in comments 6,11
****************
Then run :-
****************

# packstack --answer-file=./answer-file-aio.txt

If swift puppet generate error :-

192.168.1.57_swift.pp:                            [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.1.57_swift.pp
Error: Could not get latest version: undefined method `[]' for nil:NilClass

Then run : `dnf check-update` and replace obsoleted packages.
For instance :-

[root@fedora22wks ~]# yum check-update
Yum command has been deprecated, redirecting to '/usr/bin/dnf check-update'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Last metadata expiration check performed 0:21:19 ago on Fri Aug 7 12:25:41 2015.
Obsoleting Packages
python-mysql.x86_64                      1.3.6-4.fc22                              updates
    MySQL-python.x86_64                  1.3.6-3.fc22                              @System
[root@fedora22wks ~]# yum install python-mysql
Yum command has been deprecated, redirecting to '/usr/bin/dnf install python-mysql'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Last metadata expiration check performed 0:21:47 ago on Fri Aug 7 12:25:41 2015.
Dependencies resolved.
=================================================================================================
Package                  Arch               Version                   Repository           Size
=================================================================================================
Installing:
python-mysql             x86_64             1.3.6-4.fc22              updates              98 k
     replacing MySQL-python.x86_64 1.3.6-3.fc22

Transaction Summary
=================================================================================================
Install 1 Package

Total download size: 98 k
Installed size: 265 k
Is this ok [y/N]: y
Downloading Packages:
python-mysql-1.3.6-4.fc22.x86_64.rpm                              62 kB/s | 98 kB     00:01
-------------------------------------------------------------------------------------------------
Total                                                             35 kB/s | 98 kB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Installing : python-mysql-1.3.6-4.fc22.x86_64                                             1/2
Obsoleting : MySQL-python-1.3.6-3.fc22.x86_64                                             2/2
Verifying   : python-mysql-1.3.6-4.fc22.x86_64                                             1/2
Verifying   : MySQL-python-1.3.6-3.fc22.x86_64                                             2/2

Installed:
python-mysql.x86_64 1.3.6-4.fc22

Complete!
***************************
Rerun packstack.
***************************

Final target is to reproduce mentioned article on i7 4790 Haswell CPU box, perform launching nova instance with CPU pinning.

[root@fedora22server ~(keystone_admin)]# uname -a
Linux fedora22server.localdomain 4.1.3-200.fc22.x86_64 #1 SMP Wed Jul 22 19:51:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@fedora22server ~(keystone_admin)]# rpm -qa \*qemu\*
qemu-system-x86-2.3.0-6.fc22.x86_64
qemu-img-2.3.0-6.fc22.x86_64
qemu-guest-agent-2.3.0-6.fc22.x86_64
qemu-kvm-2.3.0-6.fc22.x86_64
ipxe-roms-qemu-20150407-1.gitdc795b9f.fc22.noarch
qemu-common-2.3.0-6.fc22.x86_64
libvirt-daemon-driver-qemu-1.2.13.1-2.fc22.x86_64

[root@fedora22server ~(keystone_admin)]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 15991 MB
node 0 free: 4399 MB
node distances:
node   0
0: 10

[root@fedora22server ~(keystone_admin)]# virsh capabilities
<capabilities>
<host>
    <uuid>00fd5d2c-dad7-dd11-ad7e-7824af431b53</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Haswell-noTSX</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='invtsc'/>
      <feature name='abm'/>
      <feature name='pdpe1gb'/>
      <feature name='rdrand'/>
      <feature name='f16c'/>
      <feature name='osxsave'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>16374824</memory>
          <pages unit='KiB' size='4'>4093706</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,7'/>
            <cpu id='4' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='5' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='6' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='7' socket_id='0' core_id='3' siblings='3,7'/>
          </cpus>
        </cell>
      </cells>
    </topology>

On each Compute node that pinning of virtual machines will be permitted on open the /etc/nova/nova.conf file and make the following modifications:

Set the vcpu_pin_set value to a list or range of logical CPU cores to reserve for virtual machine processes. OpenStack Compute will ensure guest virtual machine instances are pinned to these virtual CPU cores.
vcpu_pin_set=2,3,6,7

Set the reserved_host_memory_mb to reserve RAM for host processes. For the purposes of testing used the default of 512 MB:
reserved_host_memory_mb=512

# systemctl restart openstack-nova-compute.service

************************************
SCHEDULER CONFIGURATION
************************************
Update /etc/nova/nova.conf

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,
ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,
NUMATopologyFilter,AggregateInstanceExtraSpecsFilter

# systemctl restart openstack-nova-scheduler.service

At this point if creating a guest you may see some changes to appear in the XML, pinning the guest vCPU(s) to the cores listed in vcpu_pin_set:

<vcpu placement='static' cpuset='2-3,6-7'>1</vcpu>

Add to vmlinuz grub2 command line at the end

isolcpus=2,3,6,7

[root@fedora22server ~(keystone_demo)]# nova boot --image 7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52 --key-name oskeydev --flavor m1.small.performance --nic net-id=c85a4215-1558-4a95-886d-a2f75500e052 vf22-instance

+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | nova                                             |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | XsGr87ZLGX8P                                     |
| config_drive                         |                                                  |
| created                              | 2015-07-31T08:03:49Z                             |
| flavor                               | m1.small.performance (6)                         |
| hostId                               |                                                  |
| id                                   | 4b99f3cf-3126-48f3-9e00-94787f040e43             |
| image                                | VF22Image (7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52) |
| key_name                             | oskeydev                                         |
| metadata                             | {}                                               |
| name                                 | vf22-instance                                    |
| os-extended-volumes:volumes_attached | []                                               |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | 14f736e6952644b584b2006353ca51be                 |
| updated                              | 2015-07-31T08:03:50Z                             |
| user_id                              | 4ece2385b17a4490b6fc5a01ff53350c                 |
+--------------------------------------+--------------------------------------------------+
[root@fedora22server ~(keystone_demo)]#nova list
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
| ID                                   | Name          | Status | Task State | Power State | Networks                          |
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
| 93906a61-ec0b-481d-b964-2bb99d095646 | CentOS71RLX   | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.21, 192.168.1.159 |
| ac7e9be5-d2dc-4ec0-b0a1-4096b552e578 | VF22Devpin    | ACTIVE | -          | Running     | demo_net=50.0.0.22                |
| b93c9526-ded5-4b7a-ae3a-106b34317744 | VF22Devs      | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.19, 192.168.1.157 |
| bef20a1e-3faa-4726-a301-73ca49666fa6 | WinSrv2012    | SHUTOFF | -          | Shutdown    | demo_net=50.0.0.16                |
| 4b99f3cf-3126-48f3-9e00-94787f040e43 | vf22-instance | ACTIVE | -          | Running     | demo_net=50.0.0.23, 192.168.1.160                |
+--------------------------------------+---------------+---------+------------+-------------+-----------------------------------+
[root@fedora22server ~(keystone_demo)]#virsh list
Id    Name                           State
----------------------------------------------------
2     instance-0000000c              running
3     instance-0000000d              running

Please, see http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
regarding detailed explanation of highlighted blocks, keeping in mind that pinning is done to logical CPU cores ( not physical due to 4 Core CPU with HT enabled ). Multiple cells are also absent, due limitations of i7 47XX Haswell CPU architecture

[root@fedora22server ~(keystone_demo)]#virsh dumpxml instance-0000000d > vf22-instance.xml
<domain type='kvm' id='3'>
<name>instance-0000000d</name>
<uuid>4b99f3cf-3126-48f3-9e00-94787f040e43</uuid>
<metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.0-3.fc23"/>
      <nova:name>vf22-instance</nova:name>
      <nova:creationTime>2015-07-31 08:03:54</nova:creationTime>
      <nova:flavor name="m1.small.performance">
        <nova:memory>4096</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="4ece2385b17a4490b6fc5a01ff53350c">demo</nova:user>
        <nova:project uuid="14f736e6952644b584b2006353ca51be">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="7b2085b8-4fe7-4d32-a5c9-5eadaf8bfc52"/>
    </nova:instance>
</metadata>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>4</vcpu>
<cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='2-3,6-7'/>
</cputune>
<numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
<resource>
    <partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2015.1.0-3.fc23</entry>
      <entry name='serial'>f1b336b1-6abf-4180-865a-b6be5670352e</entry>
      <entry name='uuid'>4b99f3cf-3126-48f3-9e00-94787f040e43</entry>
    </system>
</sysinfo>
<os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
</os>
<features>
    <acpi/>
    <apic/>
</features>
<cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-3' memory='4194304' unit='KiB'/>
    </numa>
</cpu>
<clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/6c60a5ed1b3037bbdb2bed198dac944f4c0d09cb'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:4f:25:03'/>
      <source bridge='qbr567b21fe-52'/>
      <target dev='tap567b21fe-52'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/4b99f3cf-3126-48f3-9e00-94787f040e43/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5901' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      <stats period='10'/>
    </memballoon>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c359,c706</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c359,c706</imagelabel>
</seclabel>
</domain>

Xen Virtualization on Linux and Solaris

Friday, July 31, 2015

CPU Pinning and NUMA Topology on RDO Kilo on Fedora Server 22

About Me

Blog Archive