Friday, April 01, 2016

Setup Docker Hypervisor on Multi Node DVR Cluster RDO Mitaka

UPDATE 04/03/2016
  In meantime  use  repositories for RC1, rather then Delorean trunks
DVR && Nova-Docker Driver (stable/mitaka) tested fine on RDO Mitaka (build 20160329) with no issues described in previous post for RDO Liberty
So, create DVR deployment with Contrpoller/Network + N(*)Compute Nodes. Switch to Docker Hypervisor on each Compute Node and make requiered updates to glance and filters file on Controller. You are all set. Nova-Dockers instances FIP(s) are available from outside via Neutron Distributed Router (DNAT) using "fg" inteface ( fip-namespace ) residing on same host as Docker Hypervisor. South-North traffic is not related with VXLAN tunneling on DVR systems.

Why DVR come into concern ?

  Refreshing in memory similar problem with Nova-Docker Driver (Kilo)
with which I had same kind of problems (VXLAN connection Controller <==> Compute) on F22 (OVS 2.4.0) when the same driver worked fine on CentOS 7.1 (OVS 2.3.1). I just guess that Nova-Docker driver has a problem with OVS 2.4.0 no matter of stable/kilo, stable/liberty, stable/mitaka branches been checked out for driver build.

I have to notice that issue is related specifically with ML2&OVS&VXLAN setup,
RDO Mitaka deployment ML2&OVS&VLAN  works with Nova-Docker (stable/mitaka) with no problems.

  I have not run ovs-ofctl dump-flows at br-tun bridges ant etc,
because even having proved malfunctinality I cannot file it to BZ.
Nova-Docker Driver is not packaged for RDO so it's upstream stuff,
Upstream won't consider issue which involves build driver from source
on RDO Mitaka (RC1).

Thus as quick and efficient workaround I suggest DVR deployment setup,
to kill two birds with one stone. It will result South-North traffic
to be forwarded right away from host running Docker Hypervisor to Internet
and vice/versa due to basic "fg" functionality ( outgoing interface of
fip-namespace,residing on Compute node having L3 agent running in "dvr"
agent_mode ).

Procedure in details
First install repositories for RDO Mitaka (the most recent build passed CI):-

# yum -y install yum-plugin-priorities
# cd /etc/yum.repos.d
# curl -O
# curl -O
# yum -y install openstack-packstack (Controller only)

Now proceed as follows :-

1. Here is   Answer file to deploy pre DVR Cluster
2. Convert cluster to DVR as advised in  "RDO Liberty DVR Neutron workflow on CentOS 7.2"

Just one notice on RDO Mitaka on each compute node run

# ovs-vsctl add-br br-ex
# ovs-vsctl add-port br-ex eth0

Then configure

On Controller (X=2) and Computes X=(3,4) update :-
# cat ifcfg-br-ex


# cat ifcfg-eth0

Then run script
#!/bin/bash -x
chkconfig network on
systemctl stop NetworkManager
systemctl disable NetworkManager
service network restart

Reboot node.

Nova-Docker Setup on each Compute
# curl -sSL | sh
# usermod -aG docker nova      ( seems not help to set 660 for docker.sock )
# systemctl start docker
# systemctl enable docker
# chmod 666  /var/run/docker.sock (add to /etc/rc.d/rc.local)
# easy_install pip
# git clone -b stable/mitaka

Driver build
# cd nova-docker
# pip install -r requirements.txt
# python install

Switch nova-compute to DockerDriver
vi /etc/nova/nova.conf

Next on Controller/Network Node and each Compute Node

mkdir /etc/nova/rootwrap.d
vi /etc/nova/rootwrap.d/docker.filters
# nova/virt/docker/ 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root

Nova Compute Service restart on Compute Nodes
# systemctl restart openstack-nova-compute

Glance API Service restart on Controller
vi /etc/glance/glance-api.conf

# systemctl restart openstack-glance-api

Network flow on Compute in a bit more details

When floating IP gets assigned to  VM ,  what actually happens ( [1] ) :-
The same explanation may be found in ([4]) , the only style would not be in step by step manner, in particular it contains detailed description of reverse network flow and ARP Proxy functionality.

1.The fip- namespace is created on the local compute node 
  (if it does not already exist)
2.A new port rfp- gets created on the qrouter- namespace 
  (if it does not already exist)
3.The rfp port on the qrouter namespace is assigned the associated floating IP address
4.The fpr port on the fip namespace gets created and linked via point-to-point  network to the rfp port of the qrouter namespace
5.The fip namespace gateway port fg- is assigned an additional address
  from the public network range to set up  ARP proxy point
6.The fg- is configured as a Proxy ARP

Flow itself  ( [1] ):

1.The VM, initiating transmission, sends a packet via default gateway
   and br-int forwards the traffic to the local DVR gateway port (qr-).
2.DVR routes the packet using the routing table to the rfp- port
3.The packet is applied NAT rule, replacing the source-IP of VM to
   the assigned floating IP, and then it gets sent through the rfp- port,
   which connects to the fip namespace via point-to-point network
4. The packet is received on the fpr- port in the fip namespace
and then routed outside through the fg- port

Two Node deployment snapshots (Controller/Network + Compute)


   Three Node deployment snapshots (Controller/Network + 2*Compute)


Build on Compute GlassFish 4.1 docker image per  and upload to glance :-

[root@ip-192-169-142-137 ~(keystone_admin)]# docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED              SIZE
derby/docker-glassfish41   latest              3a6b84ec9206        About a minute ago   1.155 GB
rastasheep/ubuntu-sshd     latest              70e0ac74c691        2 days ago           251.6 MB
phusion/baseimage          latest              772dd063a060        3 months ago         305.1 MB
tutum/tomcat               latest              2edd730bbedd        7 months ago         539.9 MB
larsks/thttpd              latest              a31ab5050b67        15 months ago        1.058 MB

[root@ip-192-169-142-137 ~(keystone_admin)]# docker save derby/docker-glassfish41 |  openstack image create  derby/docker-glassfish41  --public --container-format docker --disk-format raw
| Field            | Value                                                |
| checksum         | 9bea6dd0bcd8d0d7da2d82579c0e658a                     |
| container_format | docker                                               |
| created_at       | 2016-04-01T14:29:20Z                                 |
| disk_format      | raw                                                  |
| file             | /v2/images/acf03d15-b7c5-4364-b00f-603b6a5d9af2/file |
| id               | acf03d15-b7c5-4364-b00f-603b6a5d9af2                 |
| min_disk         | 0                                                    |
| min_ram          | 0                                                    |
| name             | derby/docker-glassfish41                             |
| owner            | 31b24d4b1574424abe53b9a5affc70c8                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 1175020032                                           |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2016-04-01T14:30:13Z                                 |
| virtual_size     | None                                                 |
| visibility       | public                                               |

[root@ip-192-169-142-137 ~(keystone_admin)]# docker ps
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS              PORTS               NAMES
8f551d35f2d7        derby/docker-glassfish41   "/sbin/my_init"       39 seconds ago      Up 31 seconds                           nova-faba725e-e031-4edb-bf2c-41c6dfc188c1
dee4425261e8        tutum/tomcat               "/"             About an hour ago   Up About an hour                        nova-13450558-12d7-414c-bcd2-d746495d7a57
41d2ebc54d75        rastasheep/ubuntu-sshd     "/usr/sbin/sshd -D"   2 hours ago         Up About an hour                        nova-04ddea42-10a3-4a08-9f00-df60b5890ee9

[root@ip-192-169-142-137 ~(keystone_admin)]# docker logs 8f551d35f2d7
*** Running /etc/my_init.d/
No SSH host key available. Generating one...
*** Running /etc/my_init.d/
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Creating SSH2 ED25519 key; this may take some time ...
invoke-rc.d: policy-rc.d denied execution of restart.
SSH KEYS regenerated by Boris just in case !
SSHD started !
*** Running /etc/my_init.d/
Derby database started !
*** Running /etc/my_init.d/
Bad Network Configuration.  DNS can not resolve the hostname: instance-00000006: instance-00000006: unknown error
Waiting for domain1 to start ......
Successfully started the domain : domain1
domain  Location: /opt/glassfish4/glassfish/domains/domain1
Log File: /opt/glassfish4/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
=> Modifying password of admin to random in Glassfish
spawn asadmin --user admin change-admin-password
Enter the admin password>
Enter the new admin password>
Enter the new admin password again>
Command change-admin-password executed successfully.

Fairly hard docker image been built by "docker expert" as myself ;)
gets launched and nova-docker instance seems to run properly
several daemons at a time ( sshd enabled )

[boris@fedora23wks Downloads]$ ssh root@
root@'s password:
Last login: Fri Apr  1 15:33:06 2016 from
root@instance-00000006:~# ps -ef
root         1     0  0 14:32 ?        00:00:00 /usr/bin/python3 -u /sbin/my_init
root       100     1  0 14:33 ?        00:00:00 /bin/bash /etc/my_init.d/
root       103     1  0 14:33 ?        00:00:00 /usr/sbin/sshd
root       170     1  0 14:33 ?        00:00:03 /opt/jdk1.8.0_25/bin/java -Djava.library.path=/op
root       427   100  0 14:33 ?        00:00:02 java -jar /opt/glassfish4/bin/../glassfish/lib/cl
root       444   427  2 14:33 ?        00:01:23 /opt/jdk1.8.0_25/bin/java -cp /opt/glassfish4/gla

root      1078     0  0 15:32 ?        00:00:00 bash
root      1110   103  0 15:33 ?        00:00:00 sshd: root@pts/0
root      1112  1110  0 15:33 pts/0    00:00:00 -bash
root      1123  1112  0 15:33 pts/0    00:00:00 ps -ef

Glassfish is running indeed