Xen Virtualization on Linux and Solaris: February 2015

Sunday, February 15, 2015

Testing the most recent Nova-Docker driver on Ubuntu 14.04 in devstack environment recoverable between reboots

*******************************************************************************
UPDATE : As of 03/11/2015 Patch bellow merged upstream
In meantime instructions in UPDATE : As of 03/09/2015 are already a history

View : https://review.openstack.org/#/c/163022/
View : https://git.openstack.org/cgit/stackforge/nova-docker/

*******************************************************************************
UPDATE : As of 03/09/2015

View What is missing commit 9d06520645f28d96ef905a709f8ff0c27842b58b in nova-docker master branch ?

for details and explanation what is wrong with commit mentioned above.
To succeed with Nova Docker driver build on Ubuntu 14.04.2 proceed as
follows, otherwise you will be able load driver via stack.sh run, but network
- floating and private IPs wouldn't work . Nova will just boot container and nothing else. Patch bellow is easy to apply manually . It will result bringing container's interface up, and network alive and ready to work for you.

$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker

Apply patch

diff --git a/novadocker/virt/docker/vifs.py b/novadocker/virt/docker/vifs.py
index a2e7b23..1d159f7 100644
--- a/novadocker/virt/docker/vifs.py
+++ b/novadocker/virt/docker/vifs.py
@@ -248,6 +248,8 @@ class DockerGenericVIFDriver(object):
                           run_as_root=True)
             utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'addr',
                           'add', ip, 'dev', if_remote_name, run_as_root=True)
+            utils.execute('ip', 'netns', 'exec', container_id, 'ip', 'link',
+                          'set', if_remote_name,'up',run_as_root=True)
             if gateway is not None:
                 utils.execute('ip', 'netns', 'exec', container_id,
                               'ip', 'route', 'replace', 'default', 'via',

Then build driver
$ sudo pip install .
********************************************************************************
Recently new patch https://review.openstack.org/#/c/154750/ merged
https://github.com/stackforge/nova-docker.git what made possible
to test Nova-docker driver built via current git tree status with the most
recent openstack code obtained by devstack by cloning https://git.openstack.org/openstack-dev/devstack However,nova-docker containers have been lost after every reboot due to bridge br-ex came up with no IP and running ./rejoin-stack.sh didn't help much. This post describes workaround for this issue.

   First part of article actually follows http://blog.oddbit.com/2015/02/11/installing-novadocker-with-devstack/
written by Lars Kellogg-Stedman with non-critical changes in local.conf file.

   Second part of article provides workaround making created nova-docker
instances and all devstack environment recoverable between reboots.

Reproducing the first part I also installed horizon launching nova-docker containers and assigning floating IPs clicking by mouse ( via admin login working with preinstalled Demo project )
Run as root ( post install ) to open way out for VMs
*************************************************************************
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
*************************************************************************
$ sudo apt-get update
$ sudo apt-get -y install git git-review python-pip python-dev
$ sudo apt-get -y upgrade

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main \
   > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

*********************************************
Update /etc/default/docker and setting:
*********************************************
DOCKER_OPTS='-G ubuntu'

#service docker restart

*******************************
Installing nova-docker
*******************************
This block is a subject to change as far as commits done
after e9dcf7e790e4df2f9025b19896173995a32692fc
in particular 85071220cbc3c1edb4a4c67db3e7060284f35c6b
will be tested as not disabling floating IPs.

$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ git checkout e9dcf7e790e4df2f9025b19896173995a32692fc
$ sudo pip install .
***************************************************************************
UPDATE 03/12/2015 To get floating IPs working in meantime I have
***************************************************************************
$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ git revert -m 1 661998214962d3e86063196bda0b3a619b7f4e26
$ sudo pip install .

************************************
UPDATE 03/13/2015
************************************
$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ sudo pip install .

Seems to be working , however I've noticed strange issue with

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

This directive has a potential danger to lock your floating IPs, if you MASQUERADE a concrete sub-net which is providing floating IPs.

iptables -t nat -A POSTROUTING -o eth0 -j

*****************************
Configuring devstack
*****************************

Now we're ready to get devstack up and running. Start by cloning the repository:

$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack
1. Create local.conf under devstack ( original version )
***************
local.conf
***************
[[local|localrc]]
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=super-secret-admin-token
VIRT_DRIVER=novadocker.virt.docker.DockerDriver

DEST=$HOME/stack
SERVICE_DIR=$DEST/status
DATA_DIR=$DEST/data
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs

# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

# Introduce glance to docker images
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker

# Configure nova to use the nova-docker driver
[[post-config|$NOVA_CONF]]
[DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver

*****************************************************************************
My version of local.conf which allows define floating pool as you need,
a bit more flexible then original
*****************************************************************************
[[local|localrc]]
HOST_IP=192.168.1.57
ADMIN_PASSWORD=secret
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
SERVICE_PASSWORD=secret
FLOATING_RANGE=192.168.10.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.10.150,end=192.168.10.254
PUBLIC_NETWORK_GATEWAY=192.168.10.15
SERVICE_TOKEN=super-secret-admin-token
VIRT_DRIVER=novadocker.virt.docker.DockerDriver

DEST=$HOME/stack
SERVICE_DIR=$DEST/status
DATA_DIR=$DEST/data
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs

# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

# Introduce glance to docker images
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker

# Configure nova to use the nova-docker driver
[[post-config|$NOVA_CONF]]
[DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver

**************************************
Corresponding iptables entry
**************************************
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

At this point you are ready to run :-

$ ./stack.sh

*****************************************************************************
Attention skipping this step causes message "No hosts available"
when launching, either causes failure to launch nova-docker instances
in case of stack.sh rerun after ./unstack.sh
******************************************************************************

$ sudo cp nova-docker/etc/nova/rootwrap.d/docker.filters \
/etc/nova/rootwrap.d/

$ .   openrc admin

For docker pull && docker save

$ .   openrc demo

To launch instances

*********************************************************************************
Next issue , you have run `sudo ./unstack.sh` , rebooted box hosting devstack instance and OVS bridge "br-ex" came up with no IP no matter which one of local.conf has been used for ./stack.sh deployment.
Before running ./rejoin-stack.sh following actions have to be undertaken
*********************************************************************************
This version is supposed to work with second version of local.conf
PUBLIC_NETWORK_GATEWAY=192.168.10.15

sudo ip addr flush dev br-ex
    sudo ip addr add 192.168.10.15/24 dev br-ex
    sudo ip link set br-ex up
    sudo route add -net 10.254.1.0/24 gw 192.168.10.15

******************************************************
Verify correct environment installed:-
******************************************************

ubuntu@ubuntu-System-Product-Name:~$ ifconfig
br-ex     Link encap:Ethernet HWaddr de:64:4b:ba:a7:48
          inet addr:192.168.10.15 Bcast:0.0.0.0 Mask:255.255.255.0
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:2186 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2649 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1801780 (1.8 MB) TX bytes:2194422 (2.1 MB)

br-int    Link encap:Ethernet HWaddr b2:cf:54:c5:a0:49
          inet6 addr: fe80::b007:79ff:fe87:4260/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:648 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:120474 (120.4 KB) TX bytes:648 (648.0 B)

br-tun    Link encap:Ethernet HWaddr 3a:fb:71:08:1a:45
          inet6 addr: fe80::899:bcff:fed6:8d8d/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

docker0   Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet HWaddr 90:e6:ba:2d:11:eb
          inet addr:192.168.1.37 Bcast:192.168.1.255 Mask:255.255.255.0
          inet6 addr: fe80::92e6:baff:fe2d:11eb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:64604 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37999 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:88470764 (88.4 MB) TX bytes:3455868 (3.4 MB)

eth1      Link encap:Ethernet HWaddr 00:0c:76:e0:1e:c5
          inet6 addr: fe80::20c:76ff:fee0:1ec5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:239 errors:0 dropped:0 overruns:0 frame:0
          TX packets:389 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:58024 (58.0 KB) TX bytes:75526 (75.5 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:30804 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30804 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10921200 (10.9 MB) TX bytes:10921200 (10.9 MB)

ns44923080-eb Link encap:Ethernet HWaddr 9a:db:d0:5a:ad:02
          inet6 addr: fe80::98db:d0ff:fe5a:ad02/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:621 errors:0 dropped:0 overruns:0 frame:0
          TX packets:289 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:119156 (119.1 KB) TX bytes:55649 (55.6 KB)

ns9cb8e46e-35 Link encap:Ethernet HWaddr 6e:f3:23:93:b4:11
          inet6 addr: fe80::6cf3:23ff:fe93:b411/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:637 errors:0 dropped:0 overruns:0 frame:0
          TX packets:271 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:121878 (121.8 KB) TX bytes:52144 (52.1 KB)

tap44923080-eb Link encap:Ethernet HWaddr ee:b3:16:a3:f9:ed
          inet6 addr: fe80::ecb3:16ff:fea3:f9ed/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:289 errors:0 dropped:0 overruns:0 frame:0
          TX packets:621 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:55649 (55.6 KB) TX bytes:119156 (119.1 KB)

tap8897281a-3f Link encap:Ethernet HWaddr 9a:2a:eb:a5:3d:60
          inet6 addr: fe80::982a:ebff:fea5:3d60/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2236 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3452 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1812589 (1.8 MB) TX bytes:2351741 (2.3 MB)

tap9cb8e46e-35 Link encap:Ethernet HWaddr 06:3c:cc:e5:30:4a
          inet6 addr: fe80::43c:ccff:fee5:304a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:271 errors:0 dropped:0 overruns:0 frame:0
          TX packets:637 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52144 (52.1 KB) TX bytes:121878 (121.8 KB)

virbr0    Link encap:Ethernet HWaddr e2:93:d0:a0:2c:f6
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ubuntu@ubuntu-System-Product-Name:~$ route -n

Kernel IP routing table
Destination     Gateway           Genmask             Flags Metric Ref    Use Iface
0.0.0.0            192.168.1.1       0.0.0.0          UG      0      0        0 eth0
10.254.1.0      192.168.10.15   255.255.255.0    UG        0      0        0 br-ex
172.17.0.0      0.0.0.0         255.255.0.0      U           0      0      0 docker0
192.168.1.0     0.0.0.0            255.255.255.0 U     1      0    0 eth0
192.168.10.0    0.0.0.0           255.255.255.0 U           0      0        0 br-ex
192.168.122.0   0.0.0.0            255.255.255.0 U       0      0        0 virbr0

****************************************
At this point run you may run
****************************************

cd devstack ; ./rejoin-stack.sh

and it will bring your devstack environment back

********************************************************************
Actually, on Ubuntu 14.04 box doing this kind of testing
********************************************************************
root@ubuntu-P5Q3 :~# cat /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
ip addr flush dev br-ex ;
ip addr add 192.168.10.15/24 dev br-ex ;
ip link set br-ex up ;
route add -net 10.254.1.0/24 gw 192.168.10.15 ;
exit 0

*****************************************************************
Establishing access to public devstack net from LAN

*****************************************************************

Run on Devstack Node
# Add route to LAN
$ sudo route add -net  192.168.1.0/24 gw 192.168.1.57


Run on LAN box
# Add route to devstack public network  via HOST_IP
$ sudo route add -net 192.168.10.0/24 gw 192.168.1.57

where 192.168.1.57 HOST_IP on Devstack Node
192.168.10.0/24   devstack's public  network
192.168.1.0/24    LAN address

Vncviewer started from Ubuntu VM with devstack environment installed
connecting to vncserver screen running on Ubuntu Rastasheep nova-docker instance

Running Glassfish 4.1 nova-docker container on real Ubuntu 14.04 box

SQLDeveloper connection to Oracle XE database running inside nova-docker
container

Launching nova-docker container via CLI on real Ubuntu 14.04 box

ubuntu@ubuntu-P5Q3 :~/devstack$ nova boot --image rastasheep/ubuntu-sshd:latest --flavor m1.small UbuntuRST
+--------------------------------------+----------------------------------------------------------------------+
| Property                             | Value                                                                |
+--------------------------------------+----------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                               |
| OS-EXT-AZ:availability_zone          | nova                                                                 |
| OS-EXT-STS:power_state               | 0                                                                    |
| OS-EXT-STS:task_state                | scheduling                                                           |
| OS-EXT-STS:vm_state                  | building                                                             |
| OS-SRV-USG:launched_at               | -                                                                    |
| OS-SRV-USG:terminated_at             | -                                                                    |
| accessIPv4                           |                                                                      |
| accessIPv6                           |                                                                      |
| adminPass                            | n56arrfUdTLY                                                         |
| config_drive                         |                                                                      |
| created                              | 2015-02-16T20:18:38Z                                                 |
| flavor                               | m1.small (2)                                                         |
| hostId                               |                                                                      |
| id                                   | 85acb8d4-2387-4a21-9b77-321480f03163                                 |
| image                                | rastasheep/ubuntu-sshd:latest (87956634-9708-4d63-8daf-cdd15d288d86) |
| key_name                             | -                                                                    |
| metadata                             | {}                                                                   |
| name                                 | UbuntuRST                                                            |
| os-extended-volumes:volumes_attached | []                                                                   |
| progress                             | 0                                                                    |
| security_groups                      | default                                                              |
| status                               | BUILD                                                                |
| tenant_id                            | 2f34beaaa0684e899f28c1b6fef521ac                                     |
| updated                              | 2015-02-16T20:18:38Z                                                 |
| user_id                              | a78cae8feb1f40b081db787629a407af                                     |
+--------------------------------------+----------------------------------------------------------------------+

ubuntu@ubuntu-P5Q3 :~/devstack$ nova list
+--------------------------------------+------------------+--------+------------+-------------+------------------------------------+
| ID                                   | Name             | Status | Task State | Power State | Networks                           |
+--------------------------------------+------------------+--------+------------+-------------+------------------------------------+
| 85acb8d4-2387-4a21-9b77-321480f03163 | UbuntuRST        | ACTIVE | -          | Running     | private=10.254.1.6                 |
| fc0a6180-d177-4f04-bdf6-382820c5f8da | derbyGlassfish41 | ACTIVE | -          | Running     |
| private=10.254.1.5, 192.168.10.152 |
+--------------------------------------+------------------+--------+------------+--------------

ubuntu@ubuntu-P5Q3 :~/devstack$ nova floating-ip-create
+----------------+-----------+----------+--------+
| Ip             | Server Id | Fixed Ip | Pool   |
+----------------+-----------+----------+--------+
| 192.168.10.153 | -         | -        | public |
+----------------+-----------+----------+--------+

ubuntu@ubuntu-P5Q3 :~/devstack$ nova floating-ip-associate UbuntuRST 192.168.10.153

ubuntu@ubuntu-P5Q3 :~/devstack$ ping -c 3 192.168.10.153
PING 192.168.10.153 (192.168.10.153) 56(84) bytes of data.
64 bytes from 192.168.10.153: icmp_seq=1 ttl=63 time=0.667 ms
64 bytes from 192.168.10.153: icmp_seq=2 ttl=63 time=0.274 ms
64 bytes from 192.168.10.153: icmp_seq=3 ttl=63 time=0.084 ms

--- 192.168.10.153 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.084/0.341/0.667/0.243 ms

ubuntu@ubuntu-P5Q3 :~/devstack$ ssh root@192.168.10.153
The authenticity of host '192.168.10.153 (192.168.10.153)' can't be established.
ECDSA key fingerprint is cf:f3:e5:fd:ce:d9:99:b6:79:2d:34:73:e8:a3:2e:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.10.153' (ECDSA) to the list of known hosts.
root@192.168.10.153's password:
root@instance-00000004:~# ps -ef
UID        PID PPID C STIME TTY          TIME CMD
root         1     0 0 20:18 ?        00:00:00 /usr/sbin/sshd -D
root         5     1 0 20:22 ?        00:00:00 sshd: root@pts/0
root         7     5 0 20:22 pts/0    00:00:00 -bash
root        18     7 0 20:22 pts/0    00:00:00 ps -ef

root@instance-00000004:~# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

nsa7183e2e-09 Link encap:Ethernet HWaddr fa:16:3e:3d:0f:68
          inet addr:10.254.1.6 Bcast:10.254.1.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe3d:f68/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2378 errors:0 dropped:12 overruns:0 frame:0
          TX packets:1425 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2586320 (2.5 MB) TX bytes:132646 (132.6 KB)

*************************************************************
Login via qdhcp-namespace into UbuntuRST
*************************************************************

ubuntu@ubuntu-P5Q3 :~/devstack$ sudo ip netns exec qdhcp-c9e35028-bb1b-4141-b02b-9f35c7524dd2 ssh root@10.254.1.6
The authenticity of host '10.254.1.6 (10.254.1.6)' can't be established.
ECDSA key fingerprint is cf:f3:e5:fd:ce:d9:99:b6:79:2d:34:73:e8:a3:2e:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.254.1.6' (ECDSA) to the list of known hosts.
root@10.254.1.6's password:

Last login: Mon Feb 16 20:22:28 2015 from 192.168.10.15
root@instance-00000004:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=19.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=18.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=19.2 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=55 time=18.4 ms

References
1. https://gist.github.com/charlesflynn/5576114

Friday, February 06, 2015

Set up Two Node RDO Juno ML2&OVS&VXLAN Cluster runnig Docker Hypervisor on Compute Node (CentOS 7, kernel 3.10.0-123.20.1.el7.x86_64)

It's quite obvious that Nova-Docker driver set up success for real application is important to get on Compute Nodes . It's nice when everything works on AIO
Juno host or Controller, but just as demonstration. Might be I did something wrong , might be due to some other reason but kernel version 3.10.0-123.20.1.el7.x86_64 seems to be the first brings success on RDO Juno Compute nodes.

Follow http://lxer.com/module/newswire/view/209851/index.html up to section
"Set up Nova-Docker on Controller&&Network Node"

***************************************************
Set up Nova-Docker Driver on Compute Node
***************************************************

# yum install python-pbr

# yum install docker-io -y
# git clone https://github.com/stackforge/nova-docker

# cd nova-docker
# git checkout stable/juno
# python setup.py install
# systemctl start docker
# systemctl enable docker
# chmod 660  /var/run/docker.sock 
#  mkdir /etc/nova/rootwrap.d

************************************************

Create the docker.filters file:

************************************************



vi /etc/nova/rootwrap.d/docker.filters



Insert Lines



# nova-rootwrap command filters for setting up network in the docker driver

# This file should be owned by (and only-writeable by) the root user

[Filters]

# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'

ln: CommandFilter, /bin/ln, root



*****************************************

Add line /etc/glance/glance-api.conf

*****************************************

container_formats=ami,ari,aki,bare,ovf,ova,docker

:wq

******************************
Update nova.conf
******************************
vi /etc/nova/nova.conf
set "compute_driver = novadocker.virt.docker.DockerDriver"

************************
Restart Services
************************

usermod -G docker nova

systemctl restart openstack-nova-compute (on Compute)
systemctl status openstack-nova-compute
systemctl restart openstack-glance-api (on Controller&&Network )

At this point `scp /root/keystonerc_admin compute:/root` from Controller to
Compute Node

*********************************************************************************
Test installation Nova-Docker Driver on Compute Node (RDO Juno , CentOS 7,
kernel 3.10.0-123.20.1.el7.x86_64 )
**********************************************************************************

*******************************************

Setup Ubuntu 14.04 with SSH access

*******************************************

First on Compute node

# docker pull rastasheep/ubuntu-sshd:14.04

# . keystonerc_admin

# docker save rastasheep/ubuntu-sshd:14.04 | glance image-create --is-public=True --container-format=docker --disk-format=raw --name rastasheep/ubuntu-sshd:14.04

Second on Controller node launch Nova-Docker container , running on Compute, via dashboard and assign floating IP address

*********************************************
Verify `docker ps ` on Compute Node
*********************************************
[root@juno1dev ~]# ssh 192.168.1.137
Last login: Fri Feb 6 15:38:49 2015 from juno1dev.localdomain

[root@juno2dev ~]# docker ps
CONTAINER ID        IMAGE                          COMMAND               CREATED             STATUS              PORTS               NAMES
ef23d030e35a        rastasheep/ubuntu-sshd:14.04   "/usr/sbin/sshd -D"   7 hours ago         Up 6 minutes                            nova-211bcb54-35ba-4f0a-a150-7e73546d8f46

[root@juno2dev ~]# ip netns
ef23d030e35af63c17698d1f4c6f7d8023c29455e9dff0288ce224657828993a
ca9aa6cb527f2302985817d3410a99c6f406f4820ed6d3f62485781d50f16590
fea73a69337334b36625e78f9a124e19bf956c73b34453f1994575b667e7401b
58834d3bbea1bffa368724527199d73d0d6fde74fa5d24de9cca41c29f978e31

********************************
On Controller run :-
********************************

[root@juno1dev ~]# ssh root@192.168.1.173
root@192.168.1.173's password:

Last login: Fri Feb 6 12:11:19 2015 from 192.168.1.127
root@instance-0000002b:~# apt-get update
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates Release [62.0 kB]
Get:4 http://archive.ubuntu.com trusty-security Release [62.0 kB]
Hit http://archive.ubuntu.com trusty/main Sources
Hit http://archive.ubuntu.com trusty/restricted Sources
Hit http://archive.ubuntu.com trusty/universe Sources
Hit http://archive.ubuntu.com trusty/main amd64 Packages
Hit http://archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://archive.ubuntu.com trusty/universe amd64 Packages
Get:5 http://archive.ubuntu.com trusty-updates/main Sources [208 kB]
Get:6 http://archive.ubuntu.com trusty-updates/restricted Sources [1874 B]
Get:7 http://archive.ubuntu.com trusty-updates/universe Sources [124 kB]
Get:8 http://archive.ubuntu.com trusty-updates/main amd64 Packages [524 kB]
Get:9 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [14.8 kB]
Get:10 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [318 kB]
Get:11 http://archive.ubuntu.com trusty-security/main Sources [79.8 kB]
Get:12 http://archive.ubuntu.com trusty-security/restricted Sources [1874 B]
Get:13 http://archive.ubuntu.com trusty-security/universe Sources [19.1 kB]
Get:14 http://archive.ubuntu.com trusty-security/main amd64 Packages [251 kB]
Get:15 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [14.8 kB]
Get:16 http://archive.ubuntu.com trusty-security/universe amd64 Packages [110 kB]
Fetched 1793 kB in 9s (199 kB/s)
Reading package lists... Done

If network operations like `apt-get install ... ` run afterwards with no problems
Nova-Docker driver is installed and works on Compute Node

**************************************************************************************
Finally I've set up openstack-nova-compute on Controller , to run several instances with Qemu/Libvirt driver :-
**************************************************************************************

Xen Virtualization on Linux and Solaris

Sunday, February 15, 2015

Testing the most recent Nova-Docker driver on Ubuntu 14.04 in devstack environment recoverable between reboots

Friday, February 06, 2015

Set up Two Node RDO Juno ML2&OVS&VXLAN Cluster runnig Docker Hypervisor on Compute Node (CentOS 7, kernel 3.10.0-123.20.1.el7.x86_64)

About Me

Blog Archive