Xen Virtualization on Linux and Solaris

Thursday, March 27, 2014

Launching Instance via image and creating simultaneously bootable cinder volume (glusterfs) on Two Node GRE+OVS+Gluster F20 Cluster

Actually, post is about attempt to make working following kind of `nova boot .. ` command via CLI or Dashboard, launching Instance via image and creating simultaneously boot able cinder volume (on glusterfs). It follows up
http://lxer.com/module/newswire/view/199524/index.html
http://lxer.com/module/newswire/view/199764/index.html

[root@dfw02 ~(keystone_admin)]$ nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

where

[root@dfw02 ~(keystone_boris)]$ glance image-list
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                     | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image         | qcow2       | bare             | 344457216 | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                 | qcow2       | bare             | 13147648 | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64         | qcow2       | bare             | 237371392 | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image          | qcow2       | bare             | 214106112 | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10      | qcow2       | bare             | 244514816 | active |
| 81734fc7-7300-48db-9b91-95e653e5ed5b | Ubuntu Trusty 03/25/2014 | qcow2       | bare             | 251920896 | active |
| 23ac0b82-9579-4491-8a12-9dfb566361a1 | Ubuntu Trusty 03/26/2014 | qcow2       | bare             | 251986432 | active |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+

Controller is dfw02.localdomain (192.168.1.127)
Compute is dfw01.localdomain (192.168.1.137)

Cluster Config

[root@dfw02 ~(keystone_admin)]$ openstack-status
== Nova services ==
openstack-nova-api:                     active
openstack-nova-cert:                    inactive (disabled on boot)
openstack-nova-compute:                 inactive (disabled on boot)
openstack-nova-network:                 inactive (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-volume:                  inactive (disabled on boot)
openstack-nova-conductor:               active

== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active

== Keystone service ==
openstack-keystone:                     active

== Horizon service ==
openstack-dashboard:                    active

== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    inactive (disabled on boot)
neutron-openvswitch-agent:              active
neutron-linuxbridge-agent:              inactive (disabled on boot)
neutron-ryu-agent:                      inactive (disabled on boot)
neutron-nec-agent:                      inactive (disabled on boot)
neutron-mlnx-agent:                     inactive (disabled on boot)

== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active

== Ceilometer services ==
openstack-ceilometer-api:               inactive (disabled on boot)
openstack-ceilometer-central:           inactive (disabled on boot)
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         inactive (disabled on boot)
openstack-ceilometer-alarm-notifier:    inactive (disabled on boot)
openstack-ceilometer-alarm-evaluator:   inactive (disabled on boot)
== Support services ==
mysqld:                                 inactive (disabled on boot)
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
tgtd:                                   active
qpidd:                                  active
memcached:                              active

== Keystone users ==
+----------------------------------+---------+---------+-------+
|                id                |   name | enabled | email |
+----------------------------------+---------+---------+-------+
| 970ed56ef7bc41d59c54f5ed8a1690dc | admin |   True |       |
| 162021e787c54cac906ab3296a386006 | boris |   True |       |
| 1beeaa4b20454048bf23f7d63a065137 | cinder |   True |       |
| 006c2728df9146bd82fab04232444abf | glance |   True |       |
| 5922aa93016344d5a5d49c0a2dab458c | neutron |   True |       |
| af2f251586564b46a4f60cdf5ff6cf4f |   nova |   True |       |
+----------------------------------+---------+---------+-------+

== Glance images ==
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image                | qcow2       | bare             | 344457216   | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                        | qcow2       | bare             | 13147648    | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64                | qcow2       | bare             | 237371392   | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image                 | qcow2       | bare             | 214106112   | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10             | qcow2       | bare             | 244514816   | active |
| 2a95cb39-6e5f-422f-a113-49c3db8dbade | Ubuntu Trusty 03/27/2014        | qcow2       | bare             | 251986432   | active |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+

== Nova managed services ==
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| Binary           | Host              | Zone     | Status | State | Updated_at                 | Disabled Reason |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| nova-scheduler   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:22.000000 | None            |
| nova-conductor   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
| nova-compute     | dfw01.localdomain | nova     | enabled | up    | 2014-03-27T12:03:18.000000 | None            |
| nova-consoleauth | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+

== Nova networks ==
+--------------------------------------+-------+------+
| ID                                   | Label | Cidr |
+--------------------------------------+-------+------+
| 1eea88bb-4952-4aa4-9148-18b61c22d5b7 | int   | None |
| 426bb226-0ab9-440d-ba14-05634a17fb2b | int1 | None |
| 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | ext   | None |
+--------------------------------------+-------+------+

== Nova instance flavors ==
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2 | m1.small | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3 | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4 | m1.large | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5 | m1.xlarge | 16384     | 160 | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
== Nova instances ==
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

[root@dfw02 ~(keystone_admin)]$ nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:12
nova-conductor   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10
nova-compute     dfw01.localdomain                    nova             enabled    :-)   2014-03-27 12:13:08
nova-consoleauth dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10

[root@dfw02 ~(keystone_admin)]$ neutron agent-list
+--------------------------------------+--------------------+-------------------+-------+----------------+
| id                                   | agent_type         | host              | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------+-------+----------------+
| 037b985d-7a7d-455b-8536-76eed40b0722 | L3 agent           | dfw02.localdomain | :-)   | True           |
| 22438ee9-b4ea-4316-9492-eb295288f61a | Open vSwitch agent | dfw02.localdomain | :-)   | True           |
| 76ed02e2-978f-40d0-879e-1a2c6d1f7915 | DHCP agent         | dfw02.localdomain | :-)   | True           |
| 951632a3-9744-4ff4-a835-c9f53957c617 | Open vSwitch agent | dfw01.localdomain | :-)   | True           |
+--------------------------------------+--------------------+-------------------+-------+----------------+

[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   52G 44% /
devtmpfs                        3.9G     0 3.9G   0% /dev
tmpfs                           3.9G   92K 3.9G   1% /dev/shm
tmpfs                           3.9G 9.1M 3.9G   1% /run
tmpfs                           3.9G     0 3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G   76K 3.9G   1% /tmp
/dev/sda5                       477M 122M 326M 28% /boot
/dev/mapper/fedora00-data1       77G 7.1G   66G 10% /data1
192.168.1.127:cinder-volumes05   77G 7.1G   66G 10% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns

Tenants environment

[root@dfw02 ~(keystone_boris)]$ nova list
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                     |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | SUSPENDED | None       | Shutdown    | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | SUSPENDED | None       | Shutdown    | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | SUSPENDED | None       | Shutdown    | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | SUSPENDED | None       | Shutdown    | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+

[root@dfw02 ~(keystone_boris)]$ neutron subnet-list
+--------------------------------------+------+-------------+--------------------------------------------+
| id                                   | name | cidr        | allocation_pools                           |
+--------------------------------------+------+-------------+--------------------------------------------+
| 9e0d457b-c4c4-45cf-84e2-4ac7550f3b06 |      | 40.0.0.0/24 | {"start": "40.0.0.2", "end": "40.0.0.254"} |
+--------------------------------------+------+-------------+--------------------------------------------+

Check file /usr/share/cinder/cinder-dist.conf

[root@dfw02 ~(keystone_admin)]$ cat /usr/share/cinder/cinder-dist.conf
[DEFAULT]
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
iscsi_helper = tgtadm
sql_connection = mysql://cinder:cinder@localhost/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone

[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

Hack bellow is similar https://ask.openstack.org/en/question/11672/ongoing-problem-with-starting-neutron-server-on-fedora-20-rdo-havana/

Log into MariaDB:-

MariaDB [(none)]> INSERT INTO mysql.user (User,Host,Password) VALUES('cinder','dfw02.localdomain',' ');

MariaDB [(none)]> UPDATE mysql.user SET Password = PASSWORD('cinder') WHERE User = 'cinder';

Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings: 0

MariaDB [(none)]> FLUSH PRIVILEGES ;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user;
+----------+-------------------+-------------------------------------------+
| User     | Host              | Password                                  |
+----------+-------------------+-------------------------------------------+
| root     | localhost         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | dfw02.localdomain | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | 127.0.0.1         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | ::1               | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| keystone | localhost         | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| keystone | %                 | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| glance   | localhost         | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| glance   | %                 | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| cinder   | localhost         | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
| cinder   | %                 | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
| neutron | localhost         | *4DF421833991170108648F1103CD74FCB66BBE9E |
| neutron | %                 | *03A31004769F9E4F94ECEEA61AA28D9649084839 |
| nova     | localhost         | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | %                 | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | dfw02.localdomain | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| dash     | %                 | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| dash     | localhost         | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| cinder   | dfw02.localdomain | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
+----------+-------------------+-------------------------------------------+
18 rows in set (0.00 sec)

[root@dfw02 ~(keystone_admin)]$ systemctl restart openstack-cinder-scheduler

[root@dfw02 ~(keystone_admin)]$ systemctl status openstack-cinder-scheduler -l

openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-scheduler.service; enabled)
   Active: active (running) since Thu 2014-03-27 11:11:11 MSK; 25min ago
Main PID: 4377 (cinder-schedule)
   CGroup: /system.slice/openstack-cinder-scheduler.service
           └─4377 /usr/bin/python /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/scheduler.log

[root@dfw02 ~(keystone_admin)]$ systemctl restart openstack-cinder-volume

[root@dfw02 ~(keystone_admin)]$ cinder service-list
+------------------+-------------------+------+---------+-------+----------------------------+
|      Binary      |        Host       | Zone | Status | State |         Updated_at         |
+------------------+-------------------+------+---------+-------+----------------------------+
| cinder-scheduler | dfw02.localdomain | nova | enabled |   up | 2014-03-28T09:42:51.000000 |
| cinder-volume   | dfw02.localdomain | nova | enabled |   up | 2014-03-28T09:42:48.000000 |
+------------------+-------------------+------+---------+-------+----------------------------+

[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   53G 43% /
devtmpfs                        3.9G     0 3.9G   0% /dev
tmpfs                           3.9G 152K 3.9G   1% /dev/shm
tmpfs                           3.9G 9.1M 3.9G   1% /run
tmpfs                           3.9G     0 3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G 140K 3.9G   1% /tmp
/dev/sda5                       477M 122M 326M 28% /boot
/dev/mapper/fedora00-data1       77G 4.7G   69G   7% /data1
192.168.1.127:cinder-volumes05   77G 4.7G   69G   7% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns

Afterwards create F20 instance via image loadable from volume via dashboard

[root@dfw02 ~(keystone_boris)]$ nova list

+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks                     |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | ACTIVE | None       | Running     | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | ACTIVE | None       | Running     | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | ACTIVE | None       | Running     | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | ACTIVE | None       | Running     | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+

[root@dfw02 ~(keystone_boris)]$ nova show VF20RS047
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| status                               | ACTIVE                                                   |
| updated                              | 2014-03-27T07:30:08Z                                     |
| OS-EXT-STS:task_state                | None                                                     |
| key_name                             | oskey25                                                  |
| image                                | Attempt to boot from volume - no image supplied          |
| hostId                               | 73ee4f5bd4da8ad7b39d768d0b167a03ac0471ea50d9ded6c6190fb1 |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-03-27T07:30:08.000000                               |
| flavor                               | m1.small (2)                                             |
| id                                   | 97825e96-b4d1-4638-9212-480a37c61dd5                     |
| security_groups                      | [{u'name': u'default'}]                                  |
| OS-SRV-USG:terminated_at             | None                                                     |
| user_id                              | 162021e787c54cac906ab3296a386006                         |
| name                                 | VF20RS047                                                |
| created                              | 2014-03-27T07:28:57Z                                     |
| tenant_id                            | 4dacfff9e72c4245a48d648ee23468d5                         |
| OS-DCF:diskConfig                    | MANUAL                                                   |
| metadata                             | {}                                                       |
| os-extended-volumes:volumes_attached | [{u'id': u'd027a1bb-0739-4443-ab60-b00720bc3ba6'}]       |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| progress                             | 0                                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| int1 network                         | 40.0.0.6, 192.168.1.107                                  |
| config_drive                         |                                                          |
+--------------------------------------+----------------------------------------------------------+

[root@dfw02 ~(keystone_boris)]$ cinder list
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
| 24f96221-c535-436c-832c-fe0130f43604 | in-use |               | 5   |     None    |   true   | e237b214-d170-4c79-8fcb-a32a11863733 |
| 2adb799d-079c-49ef-bcb3-ed13d1cb11a6 | in-use |   VF20VLG050 | 5   | glusterfs |   true   | e896bf9f-e48b-4863-b30f-7bacde1abf4f |
| 2eab2019-e80f-46f4-96fc-5afad10cfa27 | in-use | UbuntuTLVG027 | 5   | glusterfs |   true   | 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa |
| d027a1bb-0739-4443-ab60-b00720bc3ba6 | in-use |               | 5   |     None    |   true   | 97825e96-b4d1-4638-9212-480a37c61dd5 |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+

However command like :

[root@dfw02 ~(keystone_admin)]$ nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

even ending up with "Error" status still creates glusterfs cinder volume ( with system_id ) , which is quite healthy and may be utilized for building new instance of F20 or Ubuntu 14.04, whatever original image was, via CLI or via Dashboard. It looks like a kind of bug in Nova&Neutron inter process communications. I would say synchronization at boot up.

I have to notice that procedure above is actually a hack , bug is serious and well known . View :-
"Provide an API for external services to send defined events to the compute service for synchronization. This includes immediate needs for nova-neutron interaction around boot timing and network info updates"

https://blueprints.launchpad.net/nova/+spec/admin-event-callback-api

      and bug report :-

https://bugs.launchpad.net/nova/+bug/1280357

As far as I can see target milestone is "Icehouse-rc1"

Sunday, March 16, 2014

Setting up Neutron-dhcp-agent & Dnsmasq on Two Node Neutron GRE+OVS F20 Cluster to launch VMs with MTU 1454 automatically

When you first time boot cloud instance (RDO Havana) via `nova boot ..` or via Dashboard associated with ssh keypair , the default MTU value for eth0 of instance has been built would be 1500. The last one in case of GRE tunnelling makes useless attempt to connect to instance via ssh ( $ ssh -l fedora -i key2.pem Instance-IP ). What actually forces you to write customization script to log into instance via VNC console with account like "fedora","ubuntu","cloud-user" and set MTU manually to 1454 inside instance.    However , it exists known approach to tune dhcp_agent.ini and dnsmasq.conf to assign internal IP for instance with MTU 1454 at the first boot up updating neutron-dhcp-agent.ini & dnsmasq.conf files as follows [ 1 ] :-

[root@dfw02 neutron(keystone_admin)]$ cat dhcp_agent.ini | grep -v ^# | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
handle_internal_only_routers = TRUE
external_network_bridge = br-ex
ovs_use_veth = True
use_namespaces = True
dnsmasq_config_file = /etc/neutron/dnsmasq.conf

[root@dfw02 neutron(keystone_admin)]$ cat dnsmasq.conf
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp
dhcp-option=26,1454

Then restart dnsmasq and neutron-dhcp-agent service.

# ps -ef | grep dnsmasq

nobody    1953     1 0 09:28 ?        00:00:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

nobody    5192     1 0 09:35 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-343b0090-24 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/host --dhcp-optsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/opts --leasefile-ro --dhcp-range=set:tag0,40.0.0.0,static,120s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

nobody    5197     1 0 09:35 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-40dd712c-e4 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/host --dhcp-optsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

# killall dnsmasq

Restart services :-

# /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

# [root@dfw02 neutron(keystone_admin)]$ systemctl restart neutron-dhcp-agent.service

[root@dfw02 neutron(keystone_admin)]$ systemctl status neutron-dhcp-agent.service

neutron-dhcp-agent.service - OpenStack Quantum DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: active (running) since Sun 2014-03-16 13:03:47 MSK; 9s ago
Main PID: 20767 (neutron-dhcp-ag)
   CGroup: /system.slice/neutron-dhcp-agent.service
           ├─20767 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.con...
           ├─20814 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-343b0090-24 --...
           └─20816 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-40dd712c-e4 --...

Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: started, version 2.68 cachesize 150
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHC...auth
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: warning: no upstream servers configured
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: DHCP, static leases only on 10.0.0.0, lease time 2m
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: DHCP, sockets bound exclusively to interface ns-40dd712c-e4
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: cleared cache
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: read /var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b...ost
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: read /var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b...pts
Mar 16 13:03:58 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPREQUEST(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:03:58 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPNAK(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35 lea...und
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPDISCOVER(ns-343b0090-24) fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPOFFER(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPREQUEST(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPACK(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35 hos...0-6
Hint: Some lines were ellipsized, use -l to show in full.

Creating instance via dashboard manage as usually ( just submit ssh key-pair when launching instance)

It will now result :-

$ ssh -l fedora -i key2.pem 192.168.1.106
The authenticity of host '192.168.1.106 (192.168.1.106)' can't be established.
RSA key fingerprint is 68:1a:79:33:c3:f8:2f:61:6a:a3:e1:40:9f:1c:76:33.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.106' (RSA) to the list of known hosts.

[fedora@vf20rs001 ~]$ ifconfig
eth0: flags=4163 mtu 1454
        inet 40.0.0.5 netmask 255.255.255.0 broadcast 40.0.0.255
        inet6 fe80::f816:3eff:feb9:83e9 prefixlen 64 scopeid 0x20
        ether fa:16:3e:b9:83:e9 txqueuelen 1000 (Ethernet)
        RX packets 252 bytes 30901 (30.1 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 230 bytes 28014 (27.3 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10
        loop txqueuelen 0 (Local Loopback)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

and at the same time

# tail -f /var/log/neutron/dnsmasq.log will report

Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 next server: 40.0.0.3
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 1 option: 53 message-type 5
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 51 lease-time 2m
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 58 T1 56s
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 59 T2 1m41s
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 13 option: 12 hostname host-40-0-0-7
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 3 router 40.0.0.1
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 2 option: 26 mtu 1454

[boris@dfw02 Downloads]$ ssh -l ubuntu -i key2.pem 192.168.1.109
The authenticity of host '192.168.1.109 (192.168.1.109)' can't be established.
ECDSA key fingerprint is 9f:c6:eb:23:ee:87:6b:47:b1:5f:f0:e6:e8:1c:80:45.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.109' (ECDSA) to the list of known hosts.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubunturs001:~$ ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:97:b1:6a
          inet addr:40.0.0.2 Bcast:40.0.0.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe97:b16a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1
          RX packets:337 errors:0 dropped:0 overruns:0 frame:0
          TX packets:272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48072 (48.0 KB) TX bytes:31092 (31.0 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Three active instances running simultaneously :-

[root@dfw02 ~(keystone_boris)]$ nova list | grep ACTIVE

| 878cb009-8afa-4fb2-ab81-b59d7b94a040 | UbuntuRS001 | ACTIVE    | None       | Running     | int1=40.0.0.2, 192.168.1.109 |

| 47bae1d0-1be7-4b84-8392-b36270410934 | VF20RS001   | ACTIVE    | None       | Running     | int1=40.0.0.5, 192.168.1.106 |

| 6e88e18c-c2f9-48c4-b19c-b757b104a194 | VF20RS003   | ACTIVE    | None       | Running     | int1=40.0.0.4, 192.168.1.107 |

Log file /var/log/neutron/dnsmasq.log (renewing lease in loop all the time) :-

   Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 1 option: 53 message-type 5
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 58 T1 56s
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 59 T2 1m41s
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 13 option: 12 hostname host-40-0-0-4
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 client provides name: ubunturs001
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 DHCPREQUEST(ns-343b0090-24) 40.0.0.2 fa:16:3e:97:b1:6a
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 DHCPACK(ns-343b0090-24) 40.0.0.2 fa:16:3e:97:b1:6a host-40-0-0-2
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 next server: 40.0.0.3
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 1 option: 53 message-type 5
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 58 T1 55s
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 59 T2 1m40s
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 13 option: 12 hostname host-40-0-0-2
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 client provides name: vf20rs001
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 DHCPREQUEST(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 DHCPACK(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9 host-40-0-0-5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 next server: 40.0.0.3
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 1 option: 53 message-type 5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 58 T1 55s
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 59 T2 1m40s
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 13 option: 12 hostname host-40-0-0-5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 client provides name: vf20rs003
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 DHCPREQUEST(ns-343b0090-24) 40.0.0.4 fa:16:3e:a4:21:bb
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 DHCPACK(ns-343b0090-24) 40.0.0.4 fa:16:3e:a4:21:bb host-40-0-0-4
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 next server: 40.0.0.3
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 1 option: 53 message-type 5
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 58 T1 53s
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 59 T2 1m38s
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 13 option: 12 hostname host-40-0-0-4
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 2 option: 26 mtu 1454
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 client provides name: vf20rs001
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 DHCPREQUEST(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 tags: tag0, known, ns-343b0090-24
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 DHCPACK(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9 host-40-0-0-5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 next server: 40.0.0.3
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 1 option: 53 message-type 5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 51 lease-time 2m
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 58 T1 53s
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 59 T2 1m38s
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 13 option: 12 hostname host-40-0-0-5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 2 option: 26 mtu 1454
Mar 16 23:14:41 dnsmasq-dhcp[26111]: 933653313 available DHCP subnet: 40.0.0.0/255.255.255.0

[root@dallas2 ~]# ps -ef | grep nova
nova      1702     1 0 10:01 ?        00:03:50 /usr/bin/python /usr/bin/nova-compute --logfile /var/log/nova/compute.log
root      3531     1 0 10:06 ?        00:01:39 /usr/sbin/glusterfs --volfile-id=cinder-volumes012 --volfile-server=192.168.1.130 /var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34
qemu     12054     1 7 18:20 ?        00:05:32 /usr/bin/qemu-system-x86_64 -name instance-0000007f -S -machine pc-i440fx-1.6,accel=tcg,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 773e454d-675a-4f93-98db-12a1a2812e58 -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2013.2.2-1.fc20,serial=6050001e-8c00-00ac-818a-90e6ba2d11eb,uuid=773e454d-675a-4f93-98db-12a1a2812e58 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000007f.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34/volume-8d696bc5-4b12-4d44-92bb-28a00a3700fe,if=none,id=drive-virtio-disk0,format=raw,serial=8d696bc5-4b12-4d44-92bb-28a00a3700fe,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:8a:cd:db,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/773e454d-675a-4f93-98db-12a1a2812e58/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming fd:23 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
qemu     27517     1 45 16:57 ?        01:11:56 /usr/bin/qemu-system-x86_64 -name instance-00000080 -S -machine pc-i440fx-1.6,accel=tcg,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9c5333f1-b5eb-4de7-90f8-6cbd1c72199d -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2013.2.2-1.fc20,serial=6050001e-8c00-00ac-818a-90e6ba2d11eb,uuid=9c5333f1-b5eb-4de7-90f8-6cbd1c72199d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000080.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34/volume-6cbf9ac4-0ebd-4f30-9c4d-2354343975c0,if=none,id=drive-virtio-disk0,format=raw,serial=6cbf9ac4-0ebd-4f30-9c4d-2354343975c0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:65:d7:64,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/9c5333f1-b5eb-4de7-90f8-6cbd1c72199d/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
root     28361 28057 0 19:34 pts/3    00:00:00 grep --color=auto nova
[root@dallas2 ~]# cd /FDR/Replicate
[root@dallas2 Replicate]# ls -la
total 8712708
drwxr-xr-x. 3 165 165       4096 Mar 21 16:56 .
drwxr-xr-x. 3 root root       4096 Mar 6 00:50 ..
drw-------. 39 root root       4096 Mar 21 16:54 .glusterfs
-rw-rw-rw-. 2 root root 5368709120 Mar 17 21:58 volume-4b807fe8-dcd2-46eb-b7dd-6ab10641c32a
-rw-rw-rw-. 2 qemu qemu 5368709120 Mar 21 19:44 volume-6cbf9ac4-0ebd-4f30-9c4d-2354343975c0
-rw-rw-rw-. 2 qemu qemu 5368709120 Mar 21 19:44 volume-8d696bc5-4b12-4d44-92bb-28a00a3700fe
-rw-rw-rw-. 2 root root 5368709120 Mar 21 14:10 volume-abc0f5b8-5144-42b7-b49f-a42a20ddd88f
-rw-rw-rw-. 2 root root 5368709120 Mar 20 23:41 volume-ec9670b8-fa64-46e9-9695-641f51bf1421

SSH connect to newly built Ubuntu 14.04 instance :-

[boris@dallas1 Downloads]$ ssh -l ubuntu -i key2.pem 192.168.1.106
The authenticity of host '192.168.1.106 (192.168.1.106)' can't be established.
ECDSA key fingerprint is cf:6b:20:a9:6b:30:9b:30:c4:a0:13:7e:82:98:fc:3d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.106' (ECDSA) to the list of known hosts.
Welcome to Ubuntu Trusty Tahr (development branch) (GNU/Linux 3.13.0-19-generic x86_64)

* Documentation: https://help.ubuntu.com/

System information as of Mon Mar 24 06:34:23 UTC 2014

System load: 3.1               Processes:           55
Usage of /:   55.1% of 1.32GB   Users logged in:     0
Memory usage: 3%                IP address for eth0: 10.0.0.4
Swap usage:   0%

Graph this data and manage this system at:
    https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntuts024:~$ ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:14:69:59
          inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe14:6959/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1
          RX packets:339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:342 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48048 (48.0 KB) TX bytes:39526 (39.5 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ubuntu@ubuntuts024:~$ sudo su -
sudo: unable to resolve host ubuntuts024

root@ubuntuts024:~# vi /etc/hosts

root@ubuntuts024:~# apt-get update
Ign http://security.ubuntu.com trusty-security InRelease
Hit http://security.ubuntu.com trusty-security Release.gpg
Hit http://security.ubuntu.com trusty-security Release
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:1 http://nova.clouds.archive.ubuntu.com trusty Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg
Get:2 http://nova.clouds.archive.ubuntu.com trusty Release [58.5 kB]
Get:3 http://security.ubuntu.com trusty-security/main Sources [14 B]
Get:4 http://security.ubuntu.com trusty-security/universe Sources [14 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Get:5 http://nova.clouds.archive.ubuntu.com trusty/main Sources [1,074 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://security.ubuntu.com trusty-security/main Translation-en_US
Ign http://security.ubuntu.com trusty-security/universe Translation-en_US
Get:6 http://nova.clouds.archive.ubuntu.com trusty/universe Sources [6,403 kB]
Get:7 http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages [1,362 kB]
Get:8 http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages [5,867 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Get:9 http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en [4,048 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [14 B]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [14 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en_US
Fetched 18.8 MB in 1min 17s (243 kB/s)
Reading package lists... Done

root@ubuntuts024:~# apt-get upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@ubuntuts024:~# passwd ubuntu
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

References

1. https://ask.openstack.org/en/question/12499/forcing-mtu-to-1400-via-etcneutrondnsmasq-neutronconf-per-daniels/?comment=25356#comment-25356

Thursday, March 13, 2014

VNC Console in Dashboard on Two Node Controller&Compute Neutron GRE+OVS+Gluster Fedora 20 Cluster

*********************************************************************************

UPDATE on 03/16/2014. I am done with "dhcp-option=26,1454" .View [ 3 ]

*********************************************************************************
It was just a typo in dhcp_agent.ini in last line :-

# cat dhcp_agent.ini | grep -v ^# | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
handle_internal_only_routers = TRUE
external_network_bridge = br-ex
ovs_use_veth = True
use_namespaces = True
dnsmasq_config_file = /etc/neutron/dnsmasq.conf

# cat dnsmasq.conf
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp
dhcp-option=26,1454

# tail -f /var/log/neutron/dnsmasq.log
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 51 lease-time 2m
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 58 T1 53s
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 59 T2 1m38s
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 13 option: 12 hostname host-40-0-0-7
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 3 router 40.0.0.1
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 13:57:52 dnsmasq-dhcp[26111]: 1830014980 sent size: 2 option: 26 mtu 1454
********************************************************************************

This post follows up http://lxer.com/module/newswire/view/197613/index.html In particular, it could be performed after Basic Setup to make system management more comfortable then only CLI. For instance assigning floating IP is just a mouse click instead of shell script :-

neuutron floatingip-create ext > newip.log
NewIpId=`cat newip.log | grep ^"| id" | awk '{print $4}' ` ;
NewIp=`cat newip.log | grep "192.168.1"| awk '{print $4}' ` ;
NewVmId=`nova list | grep $1 | awk '{print $2}' ` ;
NewDevId=`neutron port-list --device-id $NewVmId | grep "subnet_id" | awk '{print $2}' ` ;
neutron floatingip-associate $NewIpId $NewDevId ;
echo "New floating Ip assigned : "$NewIp

It's also easy to create instance via Dashboard :

   Placing in post creating panel customization script ( analog --user-data)

#cloud-config
    password: mysecret
    chpasswd: { expire: False }
    ssh_pwauth: True

   To be able log in as "fedora" and set MTU=1454 inside VM (GRE tunneling)

   Key-pair submitted upon creation works like this :

[root@dfw02 Downloads(keystone_boris)]$ ssh -l fedora -i key2.pem 192.168.1.109
Last login: Sat Mar 15 07:47:45 2014

[fedora@vf20rs015 ~]$ uname -a
Linux vf20rs015.novalocal 3.13.6-200.fc20.x86_64 #1 SMP Fri Mar 7 17:02:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[fedora@vf20rs015 ~]$ ifconfig
eth0: flags=4163 mtu 1454
        inet 40.0.0.7 netmask 255.255.255.0 broadcast 40.0.0.255
        inet6 fe80::f816:3eff:fe1e:1de6 prefixlen 64 scopeid 0x20
        ether fa:16:3e:1e:1d:e6 txqueuelen 1000 (Ethernet)
        RX packets 225 bytes 25426 (24.8 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 221 bytes 23674 (23.1 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10
        loop txqueuelen 0 (Local Loopback)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Setup described at link mentioned above was originally suggested by Kashyap Chamarthy for VMs running on non-default Libvirt's subnet . From my side came attempt to reproduce this setup on physical F20 boxes and arbitrary network , not connected to Libvirt, preventive updates for mysql.user table ,which allowed remote connections for nova-compute and neutron-openvswitch-agent from Compute to Controller, changes to /etc/sysconfig/iptables to enable Gluster 3.4.2 setup on F20 systems ( view http://bderzhavets.blogspot.com/2014/03/setup-gluster-342-on-two-node-neutron.html ) . I have also fixed typo in dhcp_agent.ini :- the reference for "dnsmasq.conf" and added to dnsmasq.conf line "dhcp-option=26,1454". Updated configuration files are critical for launching instance without "Customization script" and allow to work with usual ssh keypair. Actually , when updates are done instance gets created with MTU 1454. View [2]. This setup is pretty much focused on ability to transfer neutron metadata from Controller to Compute F20 nodes and is done manually with no answer-files. It stops exactly at the point when `nova boot ..` loads instance on Compute, which obtains internal IP via DHCP running on Controller and may be assigned floating IP to be able communicate with Internet. No attempts to setup dashboard has been done due to core target was neutron GRE+OVS functionality (just a proof of concept).

Basic updated setup procedure

Final draft for Nova & Neutron Configuration files

I also have to notice that http://kashyapc.fedorapeople.org/virt/openstack/neutron-configs-GRE-OVS-two-node.txt in meantime (03/26/2014) has dhcp_agent.ini and l3_agent.ini identical (i.e. the last one is missing rows

metadata_ip = 192.168.1.127
metadata_port = 8700

# and contains instead of them reference:-

dnsmasq_config_file = /etc/neutron/dnsmasq.conf

)

Setup

- Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open vSwitch plugin and GRE tunneling ), Dashboard

- Compute node: Nova (nova-compute), Neutron (openvswitch-agent)

dwf02.localdomain - Controller (192.168.1.127)
dwf01.localdomain - Compute   (192.168.1.137)

1. First step follows  http://docs.openstack.org/havana/install-guide/install/yum/content/install_dashboard.html and http://docs.openstack.org/havana/install-guide/install/yum/content/dashboard-session-database.html

Sequence of actions per manuals above :-

# yum install memcached python-memcached mod_wsgi openstack-dashboard

Modify the value of CACHES['default']['LOCATION'] in /etc/openstack-dashboard/local_settings to match the ones set in /etc/sysconfig/memcached.
Open /etc/openstack-dashboard/local_settings and look for this line:

CACHES = {
'default': {
'BACKEND' : 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION' : '127.0.0.1:11211'
}
}

Update the ALLOWED_HOSTS in local_settings.py to include the addresses you wish to access the dashboard from.

Edit /etc/openstack-dashboard/local_settings:


ALLOWED_HOSTS = ['Controller-IP', 'my-desktop']

This guide assumes that you are running the Dashboard on the controller node. You can easily run the dashboard on a separate server, by changing the appropriate settings in local_settings.py.

Edit /etc/openstack-dashboard/local_settings and change OPENSTACK_HOST to the hostname of your Identity Service:

OPENSTACK_HOST = "Controller-IP"

Start the Apache web server and memcached:

# service httpd restart
# systemctl start memcached
# systemctl enable memcached

To configure the MySQL database, create the dash database:

mysql> CREATE DATABASE dash;

Create a MySQL user for the newly-created dash database that has full control of the database. Replace DASH_DBPASS with a password for the new user:

mysql> GRANT ALL ON dash.* TO 'dash'@'%' IDENTIFIED BY 'fedora';
mysql> GRANT ALL ON dash.* TO 'dash'@'localhost' IDENTIFIED BY 'fedora';

In the local_settings file /etc/openstack-dashboard/local_settings

SESSION_ENGINE = 'django.contrib.sessions.backends.db'
DATABASES = {
'default': {
# Database configuration here
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dash',
'USER': 'dash',
'PASSWORD': 'fedora',
'HOST': 'Controller-IP',
'default-character-set': 'utf8'
}
}

After configuring the local_settings as shown, you can run the manage.py syncdb command to populate this newly-created database.

# /usr/share/openstack-dashboard/manage.py syncdb

Attempting to run syncdb you might get an error like 'dash'@'yourhost' is not authorized to do it with password 'YES'. Then ( for instance in my case)

# mysql -u root -p

MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user;

MariaDB [(none)]> insert into mysql.user(User,Host,Password) values ('dash','dallas1.localdomain',' ');
Query OK, 1 row affected, 4 warnings (0.00 sec)

MariaDB [(none)]> UPDATE mysql.user SET Password = PASSWORD('fedora')
    -> WHERE User = 'dash' ;
Query OK, 1 row affected (0.00 sec)
Rows matched: 3 Changed: 1 Warnings: 0

MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user;

. . . . .

| dash     | %                 | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| dash     | localhost       | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| dash     | dallas1.localdomain | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
+----------+---------------------+-------------------------------------------+
20 rows in set (0.00 sec)

That is exactly the same issue which comes up when starting openstack-nova-scheduler & openstcak-nova-conductor services during basic installation of Controller on Fedora 20. View Basic setup
in particular :-

Set table mysql.user in proper status

shell> mysql -u root -p
mysql> insert into mysql.user (User,Host,Password) values ('nova','dfw02.localdomain',' ');
mysql> UPDATE mysql.user SET Password = PASSWORD('nova')
    ->     WHERE User = 'nova';
mysql> FLUSH PRIVILEGES;
  
Start, enable nova-{api,scheduler,conductor} services

  $ for i in start enable status; \
    do systemctl $i openstack-nova-api; done

  $ for i in start enable status; \
    do systemctl $i openstack-nova-scheduler; done

  $ for i in start enable status; \
    do systemctl $i openstack-nova-conductor; done

# service httpd restart

Finally on Controller (dfw02 - 192.168.1.127) file /etc/openstack-dashboard/local_settings looks like http://bderzhavets.wordpress.com/2014/03/14/sample-of-etcopenstack-dashboardlocal_settings/

At this point dashboard is functional, but instances sessions outputs are unavailable via dashboard. I didn't get any error code, just

Instance Detail: VF20RS03

Overview
Log
Console

Loading…

2. Second step (view [1]) . Make sure you ran on Controller (I apologize for skipping mentioning this step for about 24 hr ) :-

# ssh-keygen (Hit Enter to accept all of the defaults)
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@dfw01

[root@dfw02 ~(keystone_boris)]$ ssh -L 5900:127.0.0.1:5900 -N -f -l root 192.168.1.137
[root@dfw02 ~(keystone_boris)]$ ssh -L 5901:127.0.0.1:5901 -N -f -l root 192.168.1.137
[root@dfw02 ~(keystone_boris)]$ ssh -L 5902:127.0.0.1:5902 -N -f -l root 192.168.1.137
[root@dfw02 ~(keystone_boris)]$ ssh -L 5903:127.0.0.1:5903 -N -f -l root 192.168.1.137
[root@dfw02 ~(keystone_boris)]$ ssh -L 5904:127.0.0.1:5904 -N -f -l root 192.168.1.137

Compute's IP is 192.168.1.137

**************************************
Controller dfw02 - 192.168.1.127
**************************************
Update /etc/nova/nova.conf:

novncproxy_host=0.0.0.0
novncproxy_port=6080
novncproxy_base_url=http://192.168.1.127:6080/vnc_auto.html

[root@dfw02 ~(keystone_admin)]$ systemctl enable openstack-nova-consoleauth.service
ln -s '/usr/lib/systemd/system/openstack-nova-consoleauth.service' '/etc/systemd/system/multi-user.target.wants/openstack-nova-consoleauth.service'
[root@dfw02 ~(keystone_admin)]$ systemctl enable openstack-nova-novncproxy.service
ln -s '/usr/lib/systemd/system/openstack-nova-novncproxy.service' '/etc/systemd/system/multi-user.target.wants/openstack-nova-novncproxy.service'

[root@dfw02 ~(keystone_admin)]$ systemctl start openstack-nova-consoleauth.service
[root@dfw02 ~(keystone_admin)]$ systemctl start openstack-nova-novncproxy.service

[root@dfw02 ~(keystone_admin)]$ systemctl status openstack-nova-consoleauth.service
openstack-nova-consoleauth.service - OpenStack Nova VNC console auth Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-consoleauth.service; enabled)
   Active: active (running) since Thu 2014-03-13 19:14:45 MSK; 20min ago
Main PID: 14679 (nova-consoleaut)
   CGroup: /system.slice/openstack-nova-consoleauth.service
           └─14679 /usr/bin/python /usr/bin/nova-consoleauth --logfile /var/log/nova/consoleauth.log

Mar 13 19:14:45 dfw02.localdomain systemd[1]: Started OpenStack Nova VNC console auth Server.
[root@dfw02 ~(keystone_admin)]$ systemctl status openstack-nova-novncproxy.service
openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled)
   Active: active (running) since Thu 2014-03-13 19:14:58 MSK; 20min ago
Main PID: 14762 (nova-novncproxy)
   CGroup: /system.slice/openstack-nova-novncproxy.service
           ├─14762 /usr/bin/python /usr/bin/nova-novncproxy --web /usr/share/novnc/
           └─17166 /usr/bin/python /usr/bin/nova-novncproxy --web /usr/share/novnc/

Mar 13 19:23:54 dfw02.localdomain nova-novncproxy[14762]: 20: 127.0.0.1: Path: '/websockify'
Mar 13 19:23:54 dfw02.localdomain nova-novncproxy[14762]: 20: connecting to: 127.0.0.1:5900
Mar 13 19:23:55 dfw02.localdomain nova-novncproxy[14762]: 19: 127.0.0.1: ignoring empty handshake
Mar 13 19:24:31 dfw02.localdomain nova-novncproxy[14762]: 22: 127.0.0.1: ignoring socket not ready
Mar 13 19:24:32 dfw02.localdomain nova-novncproxy[14762]: 23: 127.0.0.1: Plain non-SSL (ws://) WebSocket connection
Mar 13 19:24:32 dfw02.localdomain nova-novncproxy[14762]: 23: 127.0.0.1: Version hybi-13, base64: 'True'
Mar 13 19:24:32 dfw02.localdomain nova-novncproxy[14762]: 23: 127.0.0.1: Path: '/websockify'
Mar 13 19:24:32 dfw02.localdomain nova-novncproxy[14762]: 23: connecting to: 127.0.0.1:5901
Mar 13 19:24:37 dfw02.localdomain nova-novncproxy[14762]: 26: 127.0.0.1: ignoring empty handshake
Mar 13 19:24:37 dfw02.localdomain nova-novncproxy[14762]: 25: 127.0.0.1: ignoring empty handshake
Hint: Some lines were ellipsized, use -l to show in full.

[root@dfw02 ~(keystone_admin)]$ netstat -lntp | grep 6080
tcp        0      0 0.0.0.0:6080            0.0.0.0:*               LISTEN      14762/python

*********************************
Compute dfw01 - 192.168.1.137
*********************************

Update /etc/nova/nova.conf:

vnc_enabled=True
novncproxy_base_url=http://192.168.1.127:6080/vnc_auto.html
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=192.168.1.137

# systemctl restart openstack-nova-compute

Finally :-

[root@dfw02 ~(keystone_admin)]$ systemctl list-units | grep nova
openstack-nova-api.service                loaded active running   OpenStack Nova API Server
openstack-nova-conductor.service           loaded active running   OpenStack Nova Conductor Server
openstack-nova-consoleauth.service       loaded active running   OpenStack Nova VNC console auth Server
openstack-nova-novncproxy.service         loaded active running   OpenStack Nova NoVNC Proxy Server
openstack-nova-scheduler.service          loaded active running   OpenStack Nova Scheduler Server

[root@dfw02 ~(keystone_admin)]$ nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   dfw02.localdomain                    internal         enabled    :-)   2014-03-13 16:56:54
nova-conductor   dfw02.localdomain                    internal         enabled    :-)   2014-03-13 16:56:54
nova-compute     dfw01.localdomain                     nova             enabled    :-)   2014-03-13 16:56:45
nova-consoleauth dfw02.localdomain                   internal         enabled    :-)   2014-03-13 16:56:47

[root@dfw02 ~(keystone_admin)]$ neutron agent-list
+--------------------------------------+--------------------+-------------------+-------+----------------+
| id                                   | agent_type         | host              | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------+-------+----------------+
| 037b985d-7a7d-455b-8536-76eed40b0722 | L3 agent           | dfw02.localdomain | :-)   | True           |
| 22438ee9-b4ea-4316-9492-eb295288f61a | Open vSwitch agent | dfw02.localdomain | :-)   | True           |
| 76ed02e2-978f-40d0-879e-1a2c6d1f7915 | DHCP agent         | dfw02.localdomain | :-)   | True           |
| 951632a3-9744-4ff4-a835-c9f53957c617 | Open vSwitch agent | dfw01.localdomain | :-)   | True           |
+--------------------------------------+--------------------+-------------------+-------+----------------+

To manage remotely :-

# ssh -L 6080:127.0.0.1:6080 -N -f -l root 192.168.1.127

from remote box.

Users console views :-

Admin Console views :-

Starting new instance on Compute Node

[root@dallas2 ~]# service openstack-nova-compute status -l
Redirecting to /bin/systemctl status -l openstack-nova-compute.service
openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled)
   Active: active (running) since Fri 2014-03-21 10:01:30 MSK; 31min ago
Main PID: 1702 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           ├─1702 /usr/bin/python /usr/bin/nova-compute --logfile /var/log/nova/compute.log
           └─3531 /usr/sbin/glusterfs --volfile-id=cinder-volumes012 --volfile-server=192.168.1.130 /var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34

Mar 21 10:28:15 dallas2.localdomain sudo[8180]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link add qvbf6f65fcd-23 type veth peer name qvof6f65fcd-23
Mar 21 10:28:15 dallas2.localdomain sudo[8186]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qvbf6f65fcd-23 up
Mar 21 10:28:15 dallas2.localdomain sudo[8189]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qvbf6f65fcd-23 promisc on
Mar 21 10:28:15 dallas2.localdomain sudo[8192]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qvof6f65fcd-23 up
Mar 21 10:28:15 dallas2.localdomain sudo[8195]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qvof6f65fcd-23 promisc on
Mar 21 10:28:15 dallas2.localdomain sudo[8198]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link set qbrf6f65fcd-23 up
Mar 21 10:28:15 dallas2.localdomain sudo[8201]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf brctl addif qbrf6f65fcd-23 qvbf6f65fcd-23
Mar 21 10:28:15 dallas2.localdomain sudo[8204]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl -- --may-exist add-port br-int qvof6f65fcd-23 -- set Interface qvof6f65fcd-23 external-ids:iface-id=f6f65fcd-2316-4d34-b478-436d6c51d3aa external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:8a:cd:db external-ids:vm-uuid=773e454d-675a-4f93-98db-12a1a2812e58
Mar 21 10:28:15 dallas2.localdomain ovs-vsctl[8206]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl -- --may-exist add-port br-int qvof6f65fcd-23 -- set Interface qvof6f65fcd-23 external-ids:iface-id=f6f65fcd-2316-4d34-b478-436d6c51d3aa external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:8a:cd:db external-ids:vm-uuid=773e454d-675a-4f93-98db-12a1a2812e58
Mar 21 10:28:15 dallas2.localdomain sudo[8220]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/nova-rootwrap /etc/nova/rootwrap.conf tee /sys/class/net/tapf6f65fcd-23/brport/hairpin_mode

[root@dallas2 ~]# ovs-vsctl show
3e7422a7-8828-4e7c-b595-8a5b6504bc08
    Bridge br-int
        Port "qvo372fd13e-d2"
            tag: 1
            Interface "qvo372fd13e-d2"
        Port br-int
            Interface br-int
                type: internal
        Port "qvof6f65fcd-23"
            tag: 1
            Interface "qvof6f65fcd-23"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvod0e086e7-32"
            tag: 1
            Interface "qvod0e086e7-32"
        Port "qvob49ecf5e-8e"
            tag: 1
            Interface "qvob49ecf5e-8e"
        Port "qvo756757a8-40"
            tag: 1
            Interface "qvo756757a8-40"
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-1"
            Interface "gre-1"
                type: gre
                options: {in_key=flow, local_ip="192.168.1.140", out_key=flow, remote_ip="192.168.1.130"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.0"

On Controller :

[root@dallas1 ~(keystone_boris)]$ nova list
+--------------------------------------+--------------+-----------+------------+-------------+-----------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                    |
+--------------------------------------+--------------+-----------+------------+-------------+-----------------------------+
| 690d29ae-4c3c-4b2e-b2df-e4d654668336 | UbuntuSRS007 | SUSPENDED | None       | Shutdown    | int=10.0.0.6, 192.168.1.103 |
| 9c791573-1238-44c4-a103-6873fddc17d1 | UbuntuTS019 | SUSPENDED | None       | Shutdown    | int=10.0.0.5, 192.168.1.107 |
| 3c888e6a-dd4f-489a-82bb-1f1f9ce6a696 | VF20RS017    | SUSPENDED | None       | Shutdown    | int=10.0.0.4, 192.168.1.102 |
| 9679d849-7e4b-4cb5-b644-43279d53f01b | VF20RS024    | ACTIVE    | None       | Running     | int=10.0.0.2, 192.168.1.105 |
| 773e454d-675a-4f93-98db-12a1a2812e58 | VF20RS027    | ACTIVE    | None       | Running     | int=10.0.0.7, 192.168.1.106 |
+--------------------------------------+--------------+-----------+------------+-------------+---------------------

References

1. https://ask.openstack.org/en/question/520/vnc-console-in-dashboard-fails-to-connect-ot-server-code-1006/
2. http://bderzhavets.blogspot.com/2014/03/tuning-neutron-dhcp-agent-dnsmasq-on.html