Xen Virtualization on Linux and Solaris: March 2014

Monday, March 31, 2014

Attempt to reproduce "Direct access to Nova metadata per Lars Kellogg-Stedman"

Quoting http://blog.oddbit.com/2014/01/14/direct-access-to-nova-metadata/

In an environment running Neutron, a request from your instance must traverse a number of steps:

    From the instance to a router,
    Through a NAT rule in the router namespace,
    To an instance of the neutron-ns-metadata-proxy,
    To the actual Nova metadata service

When there are problem accessing the metadata, it can be helpful to verify that the metadata service itself is configured correctly and returning meaningful information.

end quoting and start reproducing on Controller of Two Node Neutron GRE+OVS+Gluster Fedora 20 Cluster

[root@dallas1 ~(keystone_admin)]$ ip netns list
qrouter-cb80b040-f13f-4a67-a39e-353b1c873a0d
qdhcp-166d9651-d299-47df-a5a1-b368e87b612f

Check on the Routing on Cloud controller's router namespace, it should show
port 80 for 169.254.169.254 routes to the host at port 8700

[root@dallas1 ~(keystone_admin)]$ ip netns exec qrouter-cb80b040-f13f-4a67-a39e-353b1c873a0d iptables -L -t nat | grep 169
REDIRECT   tcp -- anywhere             169.254.169.254      tcp dpt:http redir ports 8700

Check routing table inside the router namespace:

[root@dallas1 ~(keystone_admin)]$ ip netns exec qrouter-cb80b040-f13f-4a67-a39e-353b1c873a0d ip r
default via 192.168.1.1 dev qg-8fbb6202-3d
10.0.0.0/24 dev qr-2dd1ba70-34 proto kernel scope link src 10.0.0.1
192.168.1.0/24 dev qg-8fbb6202-3d proto kernel scope link src 192.168.1.100

[root@dallas1 ~(keystone_admin)]$ ip netns exec qrouter-cb80b040-f13f-4a67-a39e-353b1c873a0d netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path

[root@dallas1 ~(keystone_admin)]$ ip netns exec qdhcp-166d9651-d299-47df-a5a1-b368e87b612f netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 10.0.0.3:53             0.0.0.0:*               LISTEN
tcp6       0      0 fe80::f816:3eff:feef:53 :::*                    LISTEN
udp        0      0 10.0.0.3:53             0.0.0.0:*
udp        0      0 0.0.0.0:67              0.0.0.0:*
udp6       0      0 fe80::f816:3eff:feef:53 :::*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path

[root@dallas1 ~(keystone_admin)]$ iptables-save | grep 8700
-A INPUT -p tcp -m multiport --dports 8700 -m comment --comment "001 metadata incoming" -j ACCEPT

[root@dallas1 ~(keystone_admin)]$ netstat -lntp | grep 8700
tcp        0      0 0.0.0.0:8700            0.0.0.0:*               LISTEN      2830/python

[root@dallas1 ~(keystone_admin)]$ ps -ef | grep 2830
nova      2830     1 0 09:41 ?        00:00:57 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova      2856 2830 0 09:41 ?        00:00:00 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova      2874 2830 0 09:41 ?        00:00:09 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova      2875 2830 0 09:41 ?        00:00:01 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log

Checks are done then follow http://blog.oddbit.com/2014/01/14/direct-access-to-nova-metadata/

[root@dallas1 ~]# grep shared_secret /etc/nova/nova.conf
neutron_metadata_proxy_shared_secret = fedora

[root@dallas1 ~]# . keystonerc_boris

[root@dallas1 ~(keystone_boris)]$ nova list
+--------------------------------------+--------------+-----------+------------+-------------+-----------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                    |
+--------------------------------------+--------------+-----------+------------+-------------+-----------------------------+
| 8543e339-724c-438e-80be-8259906ccf6d | UbuntuTRS005 | ACTIVE    | None       | Running     | int=10.0.0.6, 192.168.1.116 |
| 8bb32603-c27b-4665-a025-859f1a5bc04e | UbuntuTRS031 | SUSPENDED | None       | Shutdown    | int=10.0.0.5, 192.168.1.113 |
| 177ab5b8-c86b-44d8-aa50-b4b09cc46274 | VF20RS007    | SUSPENDED | None       | Shutdown    | int=10.0.0.4, 192.168.1.112 |
| a34ece35-afd2-466e-b591-93b269c8e41a | VF20RS017    | ACTIVE    | None       | Running     | int=10.0.0.7, 192.168.1.114 |
| 8775924c-dbbd-4fbb-afb8-7e38d9ac7615 | VF20RS037    | ACTIVE    | None       | Running     | int=10.0.0.2, 192.168.1.115 |
+--------------------------------------+--------------+-----------+------------+-------------+-----------------------------+

[root@dallas1 ~(keystone_boris)]$ python
Python 2.7.5 (default, Feb 19 2014, 13:47:28)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>timport hmac
>>> import hashlib
>>> hmac.new('fedora','8543e339-724c-438e-80be-8259906ccf6d',hashlib.sha256).hexdigest()
'c31469feb2b865d76285612331d009bf2b1109674bf4cb745954f1e482c62e7f'
>>>

# exit
#. keystonerc_admin

[root@dallas1 ~(keystone_admin)]$ keystone tenant-list
+----------------------------------+----------+---------+
|                id                |   name   | enabled |
+----------------------------------+----------+---------+
| 28d7e48acf74466e84fbb3cbd53c1ccb | admin   |   True |
| e896be65e94a4893b870bc29ba86d7eb | ostenant |   True |
| 2c28cccb99fd4939a5af03548089ab07 | services |   True |
+----------------------------------+----------+---------+
exit

# sudo su -
[root@dallas1 ~]# . keystonerc_boris

[root@dallas1 ~(keystone_boris)]$ curl \
   -H 'x-instance-id: 8543e339-724c-438e-80be-8259906ccf6d' \
   -H 'x-tenant-id: e896be65e94a4893b870bc29ba86d7eb' \
   -H 'x-instance-id-signature: c31469feb2b865d76285612331d009bf2b1109674bf4cb745954f1e482c62e7f' \
    http://localhost:8700/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id

Snapshots with different VMs involved :-

Thursday, March 27, 2014

Launching Instance via image and creating simultaneously bootable cinder volume (glusterfs) on Two Node GRE+OVS+Gluster F20 Cluster

Actually, post is about attempt to make working following kind of `nova boot .. ` command via CLI or Dashboard, launching Instance via image and creating simultaneously boot able cinder volume (on glusterfs). It follows up
http://lxer.com/module/newswire/view/199524/index.html
http://lxer.com/module/newswire/view/199764/index.html

[root@dfw02 ~(keystone_admin)]$ nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

where

[root@dfw02 ~(keystone_boris)]$ glance image-list
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                     | Disk Format | Container Format | Size      | Status |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image         | qcow2       | bare             | 344457216 | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                 | qcow2       | bare             | 13147648 | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64         | qcow2       | bare             | 237371392 | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image          | qcow2       | bare             | 214106112 | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10      | qcow2       | bare             | 244514816 | active |
| 81734fc7-7300-48db-9b91-95e653e5ed5b | Ubuntu Trusty 03/25/2014 | qcow2       | bare             | 251920896 | active |
| 23ac0b82-9579-4491-8a12-9dfb566361a1 | Ubuntu Trusty 03/26/2014 | qcow2       | bare             | 251986432 | active |
+--------------------------------------+--------------------------+-------------+------------------+-----------+--------+

Controller is dfw02.localdomain (192.168.1.127)
Compute is dfw01.localdomain (192.168.1.137)

Cluster Config

[root@dfw02 ~(keystone_admin)]$ openstack-status
== Nova services ==
openstack-nova-api:                     active
openstack-nova-cert:                    inactive (disabled on boot)
openstack-nova-compute:                 inactive (disabled on boot)
openstack-nova-network:                 inactive (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-volume:                  inactive (disabled on boot)
openstack-nova-conductor:               active

== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active

== Keystone service ==
openstack-keystone:                     active

== Horizon service ==
openstack-dashboard:                    active

== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    inactive (disabled on boot)
neutron-openvswitch-agent:              active
neutron-linuxbridge-agent:              inactive (disabled on boot)
neutron-ryu-agent:                      inactive (disabled on boot)
neutron-nec-agent:                      inactive (disabled on boot)
neutron-mlnx-agent:                     inactive (disabled on boot)

== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                active

== Ceilometer services ==
openstack-ceilometer-api:               inactive (disabled on boot)
openstack-ceilometer-central:           inactive (disabled on boot)
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         inactive (disabled on boot)
openstack-ceilometer-alarm-notifier:    inactive (disabled on boot)
openstack-ceilometer-alarm-evaluator:   inactive (disabled on boot)
== Support services ==
mysqld:                                 inactive (disabled on boot)
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
tgtd:                                   active
qpidd:                                  active
memcached:                              active

== Keystone users ==
+----------------------------------+---------+---------+-------+
|                id                |   name | enabled | email |
+----------------------------------+---------+---------+-------+
| 970ed56ef7bc41d59c54f5ed8a1690dc | admin |   True |       |
| 162021e787c54cac906ab3296a386006 | boris |   True |       |
| 1beeaa4b20454048bf23f7d63a065137 | cinder |   True |       |
| 006c2728df9146bd82fab04232444abf | glance |   True |       |
| 5922aa93016344d5a5d49c0a2dab458c | neutron |   True |       |
| af2f251586564b46a4f60cdf5ff6cf4f |   nova |   True |       |
+----------------------------------+---------+---------+-------+

== Glance images ==
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size        | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+
| a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | CentOS 6.5 image                | qcow2       | bare             | 344457216   | active |
| dc992799-7831-4933-b6ee-7b81868f808b | CirrOS31                        | qcow2       | bare             | 13147648    | active |
| 03c9ad20-b0a3-4b71-aa08-2728ecb66210 | Fedora 19 x86_64                | qcow2       | bare             | 237371392   | active |
| de93ee44-4085-4111-b022-a7437da8feac | Fedora 20 image                 | qcow2       | bare             | 214106112   | active |
| e70591fc-6905-4e57-84b7-4ffa7c001864 | Ubuntu Server 13.10             | qcow2       | bare             | 244514816   | active |
| 2a95cb39-6e5f-422f-a113-49c3db8dbade | Ubuntu Trusty 03/27/2014        | qcow2       | bare             | 251986432   | active |
+--------------------------------------+---------------------------------+-------------+------------------+-------------+--------+

== Nova managed services ==
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| Binary           | Host              | Zone     | Status | State | Updated_at                 | Disabled Reason |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
| nova-scheduler   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:22.000000 | None            |
| nova-conductor   | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
| nova-compute     | dfw01.localdomain | nova     | enabled | up    | 2014-03-27T12:03:18.000000 | None            |
| nova-consoleauth | dfw02.localdomain | internal | enabled | up    | 2014-03-27T12:03:20.000000 | None            |
+------------------+-------------------+----------+---------+-------+----------------------------+-----------------+

== Nova networks ==
+--------------------------------------+-------+------+
| ID                                   | Label | Cidr |
+--------------------------------------+-------+------+
| 1eea88bb-4952-4aa4-9148-18b61c22d5b7 | int   | None |
| 426bb226-0ab9-440d-ba14-05634a17fb2b | int1 | None |
| 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | ext   | None |
+--------------------------------------+-------+------+

== Nova instance flavors ==
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1 | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2 | m1.small | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3 | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4 | m1.large | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5 | m1.xlarge | 16384     | 160 | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
== Nova instances ==
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

[root@dfw02 ~(keystone_admin)]$ nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:12
nova-conductor   dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10
nova-compute     dfw01.localdomain                    nova             enabled    :-)   2014-03-27 12:13:08
nova-consoleauth dfw02.localdomain                    internal         enabled    :-)   2014-03-27 12:13:10

[root@dfw02 ~(keystone_admin)]$ neutron agent-list
+--------------------------------------+--------------------+-------------------+-------+----------------+
| id                                   | agent_type         | host              | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------+-------+----------------+
| 037b985d-7a7d-455b-8536-76eed40b0722 | L3 agent           | dfw02.localdomain | :-)   | True           |
| 22438ee9-b4ea-4316-9492-eb295288f61a | Open vSwitch agent | dfw02.localdomain | :-)   | True           |
| 76ed02e2-978f-40d0-879e-1a2c6d1f7915 | DHCP agent         | dfw02.localdomain | :-)   | True           |
| 951632a3-9744-4ff4-a835-c9f53957c617 | Open vSwitch agent | dfw01.localdomain | :-)   | True           |
+--------------------------------------+--------------------+-------------------+-------+----------------+

[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   52G 44% /
devtmpfs                        3.9G     0 3.9G   0% /dev
tmpfs                           3.9G   92K 3.9G   1% /dev/shm
tmpfs                           3.9G 9.1M 3.9G   1% /run
tmpfs                           3.9G     0 3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G   76K 3.9G   1% /tmp
/dev/sda5                       477M 122M 326M 28% /boot
/dev/mapper/fedora00-data1       77G 7.1G   66G 10% /data1
192.168.1.127:cinder-volumes05   77G 7.1G   66G 10% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns

Tenants environment

[root@dfw02 ~(keystone_boris)]$ nova list
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| ID                                   | Name         | Status    | Task State | Power State | Networks                     |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | SUSPENDED | None       | Shutdown    | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | SUSPENDED | None       | Shutdown    | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | SUSPENDED | None       | Shutdown    | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | SUSPENDED | None       | Shutdown    | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+-----------+------------+-------------+------------------------------+

[root@dfw02 ~(keystone_boris)]$ neutron subnet-list
+--------------------------------------+------+-------------+--------------------------------------------+
| id                                   | name | cidr        | allocation_pools                           |
+--------------------------------------+------+-------------+--------------------------------------------+
| 9e0d457b-c4c4-45cf-84e2-4ac7550f3b06 |      | 40.0.0.0/24 | {"start": "40.0.0.2", "end": "40.0.0.254"} |
+--------------------------------------+------+-------------+--------------------------------------------+

Check file /usr/share/cinder/cinder-dist.conf

[root@dfw02 ~(keystone_admin)]$ cat /usr/share/cinder/cinder-dist.conf
[DEFAULT]
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
iscsi_helper = tgtadm
sql_connection = mysql://cinder:cinder@localhost/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone

[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

Hack bellow is similar https://ask.openstack.org/en/question/11672/ongoing-problem-with-starting-neutron-server-on-fedora-20-rdo-havana/

Log into MariaDB:-

MariaDB [(none)]> INSERT INTO mysql.user (User,Host,Password) VALUES('cinder','dfw02.localdomain',' ');

MariaDB [(none)]> UPDATE mysql.user SET Password = PASSWORD('cinder') WHERE User = 'cinder';

Query OK, 3 rows affected (0.00 sec)
Rows matched: 3 Changed: 3 Warnings: 0

MariaDB [(none)]> FLUSH PRIVILEGES ;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user;
+----------+-------------------+-------------------------------------------+
| User     | Host              | Password                                  |
+----------+-------------------+-------------------------------------------+
| root     | localhost         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | dfw02.localdomain | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | 127.0.0.1         | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| root     | ::1               | *E0DC09146F1310B49A34199B04274A9EED6F9EC7 |
| keystone | localhost         | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| keystone | %                 | *936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A |
| glance   | localhost         | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| glance   | %                 | *CC67CAF178CB9A07D756302E0BBFA3B0165DFD49 |
| cinder   | localhost         | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
| cinder   | %                 | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
| neutron | localhost         | *4DF421833991170108648F1103CD74FCB66BBE9E |
| neutron | %                 | *03A31004769F9E4F94ECEEA61AA28D9649084839 |
| nova     | localhost         | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | %                 | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova     | dfw02.localdomain | *0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| dash     | %                 | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| dash     | localhost         | *C9E492EC67084E4255B200FD34BDF396E3CE1A36 |
| cinder   | dfw02.localdomain | *028F8298C041368BA08A280AA8D1EF895CB68D5C |
+----------+-------------------+-------------------------------------------+
18 rows in set (0.00 sec)

[root@dfw02 ~(keystone_admin)]$ systemctl restart openstack-cinder-scheduler

[root@dfw02 ~(keystone_admin)]$ systemctl status openstack-cinder-scheduler -l

openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-scheduler.service; enabled)
   Active: active (running) since Thu 2014-03-27 11:11:11 MSK; 25min ago
Main PID: 4377 (cinder-schedule)
   CGroup: /system.slice/openstack-cinder-scheduler.service
           └─4377 /usr/bin/python /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/scheduler.log

[root@dfw02 ~(keystone_admin)]$ systemctl restart openstack-cinder-volume

[root@dfw02 ~(keystone_admin)]$ cinder service-list
+------------------+-------------------+------+---------+-------+----------------------------+
|      Binary      |        Host       | Zone | Status | State |         Updated_at         |
+------------------+-------------------+------+---------+-------+----------------------------+
| cinder-scheduler | dfw02.localdomain | nova | enabled |   up | 2014-03-28T09:42:51.000000 |
| cinder-volume   | dfw02.localdomain | nova | enabled |   up | 2014-03-28T09:42:48.000000 |
+------------------+-------------------+------+---------+-------+----------------------------+

[root@dfw02 ~(keystone_admin)]$ df -h
Filesystem                      Size Used Avail Use% Mounted on
/dev/mapper/fedora00-root        96G   40G   53G 43% /
devtmpfs                        3.9G     0 3.9G   0% /dev
tmpfs                           3.9G 152K 3.9G   1% /dev/shm
tmpfs                           3.9G 9.1M 3.9G   1% /run
tmpfs                           3.9G     0 3.9G   0% /sys/fs/cgroup
tmpfs                           3.9G 140K 3.9G   1% /tmp
/dev/sda5                       477M 122M 326M 28% /boot
/dev/mapper/fedora00-data1       77G 4.7G   69G   7% /data1
192.168.1.127:cinder-volumes05   77G 4.7G   69G   7% /var/lib/cinder/volumes/62f75cf6996a8a6bcc0d343be378c10a
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns
tmpfs                           3.9G 9.1M 3.9G   1% /run/netns

Afterwards create F20 instance via image loadable from volume via dashboard

[root@dfw02 ~(keystone_boris)]$ nova list

+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks                     |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+
| 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa | UbuntuTRS027 | ACTIVE | None       | Running     | int1=40.0.0.7, 192.168.1.113 |
| e237b214-d170-4c79-8fcb-a32a11863733 | VF20RS037    | ACTIVE | None       | Running     | int1=40.0.0.2, 192.168.1.115 |
| 97825e96-b4d1-4638-9212-480a37c61dd5 | VF20RS047    | ACTIVE | None       | Running     | int1=40.0.0.6, 192.168.1.107 |
| e896bf9f-e48b-4863-b30f-7bacde1abf4f | VF20RS050    | ACTIVE | None       | Running     | int1=40.0.0.4, 192.168.1.108 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------------+

[root@dfw02 ~(keystone_boris)]$ nova show VF20RS047
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| status                               | ACTIVE                                                   |
| updated                              | 2014-03-27T07:30:08Z                                     |
| OS-EXT-STS:task_state                | None                                                     |
| key_name                             | oskey25                                                  |
| image                                | Attempt to boot from volume - no image supplied          |
| hostId                               | 73ee4f5bd4da8ad7b39d768d0b167a03ac0471ea50d9ded6c6190fb1 |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-03-27T07:30:08.000000                               |
| flavor                               | m1.small (2)                                             |
| id                                   | 97825e96-b4d1-4638-9212-480a37c61dd5                     |
| security_groups                      | [{u'name': u'default'}]                                  |
| OS-SRV-USG:terminated_at             | None                                                     |
| user_id                              | 162021e787c54cac906ab3296a386006                         |
| name                                 | VF20RS047                                                |
| created                              | 2014-03-27T07:28:57Z                                     |
| tenant_id                            | 4dacfff9e72c4245a48d648ee23468d5                         |
| OS-DCF:diskConfig                    | MANUAL                                                   |
| metadata                             | {}                                                       |
| os-extended-volumes:volumes_attached | [{u'id': u'd027a1bb-0739-4443-ab60-b00720bc3ba6'}]       |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| progress                             | 0                                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| int1 network                         | 40.0.0.6, 192.168.1.107                                  |
| config_drive                         |                                                          |
+--------------------------------------+----------------------------------------------------------+

[root@dfw02 ~(keystone_boris)]$ cinder list
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+
| 24f96221-c535-436c-832c-fe0130f43604 | in-use |               | 5   |     None    |   true   | e237b214-d170-4c79-8fcb-a32a11863733 |
| 2adb799d-079c-49ef-bcb3-ed13d1cb11a6 | in-use |   VF20VLG050 | 5   | glusterfs |   true   | e896bf9f-e48b-4863-b30f-7bacde1abf4f |
| 2eab2019-e80f-46f4-96fc-5afad10cfa27 | in-use | UbuntuTLVG027 | 5   | glusterfs |   true   | 389bd5d2-9dc9-4e73-9ee5-ad336bb3faaa |
| d027a1bb-0739-4443-ab60-b00720bc3ba6 | in-use |               | 5   |     None    |   true   | 97825e96-b4d1-4638-9212-480a37c61dd5 |
+--------------------------------------+--------+---------------+------+-------------+----------+--------------------------------------+

However command like :

[root@dfw02 ~(keystone_admin)]$ nova boot --flavor 2 --key_name oskey25 --block-device source=image,id=de93ee44-4085-4111-b022-a7437da8feac,dest=volume,size=5,shutdown=preserve,bootindex=0 VF20RS025

even ending up with "Error" status still creates glusterfs cinder volume ( with system_id ) , which is quite healthy and may be utilized for building new instance of F20 or Ubuntu 14.04, whatever original image was, via CLI or via Dashboard. It looks like a kind of bug in Nova&Neutron inter process communications. I would say synchronization at boot up.

I have to notice that procedure above is actually a hack , bug is serious and well known . View :-
"Provide an API for external services to send defined events to the compute service for synchronization. This includes immediate needs for nova-neutron interaction around boot timing and network info updates"

https://blueprints.launchpad.net/nova/+spec/admin-event-callback-api

      and bug report :-

https://bugs.launchpad.net/nova/+bug/1280357

As far as I can see target milestone is "Icehouse-rc1"

Sunday, March 16, 2014

Setting up Neutron-dhcp-agent & Dnsmasq on Two Node Neutron GRE+OVS F20 Cluster to launch VMs with MTU 1454 automatically

When you first time boot cloud instance (RDO Havana) via `nova boot ..` or via Dashboard associated with ssh keypair , the default MTU value for eth0 of instance has been built would be 1500. The last one in case of GRE tunnelling makes useless attempt to connect to instance via ssh ( $ ssh -l fedora -i key2.pem Instance-IP ). What actually forces you to write customization script to log into instance via VNC console with account like "fedora","ubuntu","cloud-user" and set MTU manually to 1454 inside instance.    However , it exists known approach to tune dhcp_agent.ini and dnsmasq.conf to assign internal IP for instance with MTU 1454 at the first boot up updating neutron-dhcp-agent.ini & dnsmasq.conf files as follows [ 1 ] :-

[root@dfw02 neutron(keystone_admin)]$ cat dhcp_agent.ini | grep -v ^# | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
handle_internal_only_routers = TRUE
external_network_bridge = br-ex
ovs_use_veth = True
use_namespaces = True
dnsmasq_config_file = /etc/neutron/dnsmasq.conf

[root@dfw02 neutron(keystone_admin)]$ cat dnsmasq.conf
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp
dhcp-option=26,1454

Then restart dnsmasq and neutron-dhcp-agent service.

# ps -ef | grep dnsmasq

nobody    1953     1 0 09:28 ?        00:00:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

nobody    5192     1 0 09:35 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-343b0090-24 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/host --dhcp-optsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/opts --leasefile-ro --dhcp-range=set:tag0,40.0.0.0,static,120s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

nobody    5197     1 0 09:35 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-40dd712c-e4 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/host --dhcp-optsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

# killall dnsmasq

Restart services :-

# /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

# [root@dfw02 neutron(keystone_admin)]$ systemctl restart neutron-dhcp-agent.service

[root@dfw02 neutron(keystone_admin)]$ systemctl status neutron-dhcp-agent.service

neutron-dhcp-agent.service - OpenStack Quantum DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: active (running) since Sun 2014-03-16 13:03:47 MSK; 9s ago
Main PID: 20767 (neutron-dhcp-ag)
   CGroup: /system.slice/neutron-dhcp-agent.service
           ├─20767 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.con...
           ├─20814 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-343b0090-24 --...
           └─20816 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-40dd712c-e4 --...

Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: started, version 2.68 cachesize 150
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHC...auth
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: warning: no upstream servers configured
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: DHCP, static leases only on 10.0.0.0, lease time 2m
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: DHCP, sockets bound exclusively to interface ns-40dd712c-e4
Mar 16 13:03:48 dfw02.localdomain dnsmasq[20816]: cleared cache
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: read /var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b...ost
Mar 16 13:03:48 dfw02.localdomain dnsmasq-dhcp[20816]: read /var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b...pts
Mar 16 13:03:58 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPREQUEST(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:03:58 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPNAK(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35 lea...und
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPDISCOVER(ns-343b0090-24) fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPOFFER(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPREQUEST(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35
Mar 16 13:04:02 dfw02.localdomain dnsmasq-dhcp[20814]: DHCPACK(ns-343b0090-24) 40.0.0.6 fa:16:3e:d7:33:35 hos...0-6
Hint: Some lines were ellipsized, use -l to show in full.

Creating instance via dashboard manage as usually ( just submit ssh key-pair when launching instance)

It will now result :-

$ ssh -l fedora -i key2.pem 192.168.1.106
The authenticity of host '192.168.1.106 (192.168.1.106)' can't be established.
RSA key fingerprint is 68:1a:79:33:c3:f8:2f:61:6a:a3:e1:40:9f:1c:76:33.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.106' (RSA) to the list of known hosts.

[fedora@vf20rs001 ~]$ ifconfig
eth0: flags=4163 mtu 1454
        inet 40.0.0.5 netmask 255.255.255.0 broadcast 40.0.0.255
        inet6 fe80::f816:3eff:feb9:83e9 prefixlen 64 scopeid 0x20
        ether fa:16:3e:b9:83:e9 txqueuelen 1000 (Ethernet)
        RX packets 252 bytes 30901 (30.1 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 230 bytes 28014 (27.3 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73 mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10
        loop txqueuelen 0 (Local Loopback)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

and at the same time

# tail -f /var/log/neutron/dnsmasq.log will report

Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 next server: 40.0.0.3
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 1 option: 53 message-type 5
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 51 lease-time 2m
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 58 T1 56s
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 59 T2 1m41s
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 13 option: 12 hostname host-40-0-0-7
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 3 router 40.0.0.1
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 14:35:42 dnsmasq-dhcp[26111]: 1830014980 sent size: 2 option: 26 mtu 1454

[boris@dfw02 Downloads]$ ssh -l ubuntu -i key2.pem 192.168.1.109
The authenticity of host '192.168.1.109 (192.168.1.109)' can't be established.
ECDSA key fingerprint is 9f:c6:eb:23:ee:87:6b:47:b1:5f:f0:e6:e8:1c:80:45.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.109' (ECDSA) to the list of known hosts.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubunturs001:~$ ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:97:b1:6a
          inet addr:40.0.0.2 Bcast:40.0.0.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe97:b16a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1
          RX packets:337 errors:0 dropped:0 overruns:0 frame:0
          TX packets:272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48072 (48.0 KB) TX bytes:31092 (31.0 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Three active instances running simultaneously :-

[root@dfw02 ~(keystone_boris)]$ nova list | grep ACTIVE

| 878cb009-8afa-4fb2-ab81-b59d7b94a040 | UbuntuRS001 | ACTIVE    | None       | Running     | int1=40.0.0.2, 192.168.1.109 |

| 47bae1d0-1be7-4b84-8392-b36270410934 | VF20RS001   | ACTIVE    | None       | Running     | int1=40.0.0.5, 192.168.1.106 |

| 6e88e18c-c2f9-48c4-b19c-b757b104a194 | VF20RS003   | ACTIVE    | None       | Running     | int1=40.0.0.4, 192.168.1.107 |

Log file /var/log/neutron/dnsmasq.log (renewing lease in loop all the time) :-

   Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 1 option: 53 message-type 5
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 58 T1 56s
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 59 T2 1m41s
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 13 option: 12 hostname host-40-0-0-4
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:05 dnsmasq-dhcp[26111]: 933653313 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 client provides name: ubunturs001
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 DHCPREQUEST(ns-343b0090-24) 40.0.0.2 fa:16:3e:97:b1:6a
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 DHCPACK(ns-343b0090-24) 40.0.0.2 fa:16:3e:97:b1:6a host-40-0-0-2
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 next server: 40.0.0.3
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 1 option: 53 message-type 5
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 58 T1 55s
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 59 T2 1m40s
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 13 option: 12 hostname host-40-0-0-2
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:25 dnsmasq-dhcp[26111]: 1796100447 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 client provides name: vf20rs001
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 DHCPREQUEST(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 DHCPACK(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9 host-40-0-0-5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 next server: 40.0.0.3
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 1 option: 53 message-type 5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 58 T1 55s
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 59 T2 1m40s
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 13 option: 12 hostname host-40-0-0-5
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:36 dnsmasq-dhcp[26111]: 2212500329 sent size: 2 option: 26 mtu 1454
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 client provides name: vf20rs003
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 DHCPREQUEST(ns-343b0090-24) 40.0.0.4 fa:16:3e:a4:21:bb
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 tags: tag0, known, ns-343b0090-24
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 DHCPACK(ns-343b0090-24) 40.0.0.4 fa:16:3e:a4:21:bb host-40-0-0-4
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 next server: 40.0.0.3
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 1 option: 53 message-type 5
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 51 lease-time 2m
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 58 T1 53s
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 59 T2 1m38s
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 13 option: 12 hostname host-40-0-0-4
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:13:58 dnsmasq-dhcp[26111]: 933653313 sent size: 2 option: 26 mtu 1454
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 available DHCP subnet: 40.0.0.0/255.255.255.0
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 client provides name: vf20rs001
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 DHCPREQUEST(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 tags: tag0, known, ns-343b0090-24
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 DHCPACK(ns-343b0090-24) 40.0.0.5 fa:16:3e:b9:83:e9 host-40-0-0-5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 requested options: 26:mtu, 119:domain-search, 3:router
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 next server: 40.0.0.3
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 1 option: 53 message-type 5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 54 server-identifier 40.0.0.3
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 51 lease-time 2m
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 58 T1 53s
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 59 T2 1m38s
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 1 netmask 255.255.255.0
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 28 broadcast 40.0.0.255
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 14 option: 15 domain-name openstacklocal
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 13 option: 12 hostname host-40-0-0-5
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 3 router 40.0.0.1
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 4 option: 6 dns-server 83.221.202.254
Mar 16 23:14:31 dnsmasq-dhcp[26111]: 2212500329 sent size: 2 option: 26 mtu 1454
Mar 16 23:14:41 dnsmasq-dhcp[26111]: 933653313 available DHCP subnet: 40.0.0.0/255.255.255.0

[root@dallas2 ~]# ps -ef | grep nova
nova      1702     1 0 10:01 ?        00:03:50 /usr/bin/python /usr/bin/nova-compute --logfile /var/log/nova/compute.log
root      3531     1 0 10:06 ?        00:01:39 /usr/sbin/glusterfs --volfile-id=cinder-volumes012 --volfile-server=192.168.1.130 /var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34
qemu     12054     1 7 18:20 ?        00:05:32 /usr/bin/qemu-system-x86_64 -name instance-0000007f -S -machine pc-i440fx-1.6,accel=tcg,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 773e454d-675a-4f93-98db-12a1a2812e58 -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2013.2.2-1.fc20,serial=6050001e-8c00-00ac-818a-90e6ba2d11eb,uuid=773e454d-675a-4f93-98db-12a1a2812e58 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000007f.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34/volume-8d696bc5-4b12-4d44-92bb-28a00a3700fe,if=none,id=drive-virtio-disk0,format=raw,serial=8d696bc5-4b12-4d44-92bb-28a00a3700fe,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:8a:cd:db,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/773e454d-675a-4f93-98db-12a1a2812e58/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming fd:23 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
qemu     27517     1 45 16:57 ?        01:11:56 /usr/bin/qemu-system-x86_64 -name instance-00000080 -S -machine pc-i440fx-1.6,accel=tcg,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9c5333f1-b5eb-4de7-90f8-6cbd1c72199d -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2013.2.2-1.fc20,serial=6050001e-8c00-00ac-818a-90e6ba2d11eb,uuid=9c5333f1-b5eb-4de7-90f8-6cbd1c72199d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000080.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/mnt/1c9688348ab38662e3ac8fb121077d34/volume-6cbf9ac4-0ebd-4f30-9c4d-2354343975c0,if=none,id=drive-virtio-disk0,format=raw,serial=6cbf9ac4-0ebd-4f30-9c4d-2354343975c0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:65:d7:64,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/9c5333f1-b5eb-4de7-90f8-6cbd1c72199d/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
root     28361 28057 0 19:34 pts/3    00:00:00 grep --color=auto nova
[root@dallas2 ~]# cd /FDR/Replicate
[root@dallas2 Replicate]# ls -la
total 8712708
drwxr-xr-x. 3 165 165       4096 Mar 21 16:56 .
drwxr-xr-x. 3 root root       4096 Mar 6 00:50 ..
drw-------. 39 root root       4096 Mar 21 16:54 .glusterfs
-rw-rw-rw-. 2 root root 5368709120 Mar 17 21:58 volume-4b807fe8-dcd2-46eb-b7dd-6ab10641c32a
-rw-rw-rw-. 2 qemu qemu 5368709120 Mar 21 19:44 volume-6cbf9ac4-0ebd-4f30-9c4d-2354343975c0
-rw-rw-rw-. 2 qemu qemu 5368709120 Mar 21 19:44 volume-8d696bc5-4b12-4d44-92bb-28a00a3700fe
-rw-rw-rw-. 2 root root 5368709120 Mar 21 14:10 volume-abc0f5b8-5144-42b7-b49f-a42a20ddd88f
-rw-rw-rw-. 2 root root 5368709120 Mar 20 23:41 volume-ec9670b8-fa64-46e9-9695-641f51bf1421

SSH connect to newly built Ubuntu 14.04 instance :-

[boris@dallas1 Downloads]$ ssh -l ubuntu -i key2.pem 192.168.1.106
The authenticity of host '192.168.1.106 (192.168.1.106)' can't be established.
ECDSA key fingerprint is cf:6b:20:a9:6b:30:9b:30:c4:a0:13:7e:82:98:fc:3d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.106' (ECDSA) to the list of known hosts.
Welcome to Ubuntu Trusty Tahr (development branch) (GNU/Linux 3.13.0-19-generic x86_64)

* Documentation: https://help.ubuntu.com/

System information as of Mon Mar 24 06:34:23 UTC 2014

System load: 3.1               Processes:           55
Usage of /:   55.1% of 1.32GB   Users logged in:     0
Memory usage: 3%                IP address for eth0: 10.0.0.4
Swap usage:   0%

Graph this data and manage this system at:
    https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntuts024:~$ ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:14:69:59
          inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe14:6959/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1
          RX packets:339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:342 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48048 (48.0 KB) TX bytes:39526 (39.5 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ubuntu@ubuntuts024:~$ sudo su -
sudo: unable to resolve host ubuntuts024

root@ubuntuts024:~# vi /etc/hosts

root@ubuntuts024:~# apt-get update
Ign http://security.ubuntu.com trusty-security InRelease
Hit http://security.ubuntu.com trusty-security Release.gpg
Hit http://security.ubuntu.com trusty-security Release
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:1 http://nova.clouds.archive.ubuntu.com trusty Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg
Get:2 http://nova.clouds.archive.ubuntu.com trusty Release [58.5 kB]
Get:3 http://security.ubuntu.com trusty-security/main Sources [14 B]
Get:4 http://security.ubuntu.com trusty-security/universe Sources [14 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Get:5 http://nova.clouds.archive.ubuntu.com trusty/main Sources [1,074 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://security.ubuntu.com trusty-security/main Translation-en_US
Ign http://security.ubuntu.com trusty-security/universe Translation-en_US
Get:6 http://nova.clouds.archive.ubuntu.com trusty/universe Sources [6,403 kB]
Get:7 http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages [1,362 kB]
Get:8 http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages [5,867 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Get:9 http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en [4,048 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [14 B]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [14 B]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en_US
Fetched 18.8 MB in 1min 17s (243 kB/s)
Reading package lists... Done

root@ubuntuts024:~# apt-get upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@ubuntuts024:~# passwd ubuntu
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

References

1. https://ask.openstack.org/en/question/12499/forcing-mtu-to-1400-via-etcneutrondnsmasq-neutronconf-per-daniels/?comment=25356#comment-25356

Xen Virtualization on Linux and Solaris