Tuesday, March 29, 2016

Nova-Docker driver Multi Node Setup on RDO Mitaka (build 20160329)

 UPDATE 04/01/2016

  DVR && Nova-Docker Driver (stable/mitaka) tested fine on RDO Mitaka  ( build 20160329 ) with no issues discribed in link for RDO Liberty.
So , create DVR deployment with Contrpoller/Network + N(*)Compute
Nodes. Switch to Docker Hypervisor on each Compute Node and
make requiered  updates to glance and filters file on Controller.
You are all set. Nova-Dockers instances FIP(s) are available from outside
via Neutron Distributed Router (DNAT) using "fg" inteface ( fip-namespace )
residing on same host as Docker Hypervisor. South-North traffic is not
related with VXLAN tunneling on DVR systems.

Why DVR come into concern ?

 Refreshing in memory similar problem with Nova-Docker Driver (Kilo)
with which I had same kind of problems (VXLAN connection Controller <==> Compute) on F22 (OVS 2.4.0) when the same driver worked fine on CentOS 7.1 (OVS 2.3.1). I just guess that Nova-Docker driver has a problem with OVS 2.4.0 
no matter of stable/kilo, stable/liberty, stable/mitaka branches been checked out for driver build.

  I have not run ovs-ofctl dump-flows at br-tun bridges ant etc,
because even having proved malfunctinality I cannot file it to BZ.
Nova-Docker Driver is not packaged for RDO so it's upstream stuff,
Upstream won't consider issue which involves build driver from source
on RDO Mitaka (RC1).

Thus as quick and efficient workaround I suggest DVR deployment setup,
to kill two birds with one stone. It will result South-North traffic
to be forwarded right away from host running Docker Hypervisor to Internet
and vice/versa due to basic "fg" functionality (outgoing interface of
fip-namespace,residing on Compute node having L3 agent running in "dvr"


Following bellow is procedure of  setting up the most recent Nova-Docker driver
based on  branch stable/mitaka  of https://github.com/openstack/nova-docker
on the current build of RDO Mitaka, which has passed continuous integration.
Testing has been done on AIO RDO instance created via straight forward
packstack run. In meantime procedure of uploading docker image to Glance
has been changed since Kilo times. SELINUX was set to permissive mode
after rdo set up, to get floating ip(s) working for Nova-Docker instances.

First install repositories for RDO Mitaka (the most recent build passed CI):-

# yum -y install yum-plugin-priorities
# cd /etc/yum.repos.d
# curl -O https://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
# curl -O https://trunk.rdoproject.org/centos7-mitaka/current-passed-ci/delorean.repo

# yum -y install openstack-packstack

[root@SeverMitaka01 ~]# rpm -qa \*openstack-packstack\*

Run AIO install , set SELINUX to PERMISSIVE mode and reboot

Nova-Docker Setup itself
# curl -sSL https://get.docker.com/ | sh

I clearly see after this command installed package ( not signed warning )
[root@ServerCentOS722 ~]# rpm -qa \*docker-engine-selinux\*

However, it causes problems even in AIO environment.

# usermod -aG docker nova      ( seems not help to set 660 for docker.sock )
# systemctl start docker
# systemctl enable docker
# chmod 666  /var/run/docker.sock (add to /etc/rc.d/rc.local)
# easy_install pip
# git clone -b stable/mitaka   https://github.com/openstack/nova-docker

Driver set up
# cd nova-docker
# pip install -r requirements.txt
# python setup.py install

Update and create following files
vi /etc/nova/nova.conf

Reminder - missing next step results failure openstack-nova-scheduler
and generates "No availables hosts found"

mkdir /etc/nova/rootwrap.d
vi /etc/nova/rootwrap.d/docker.filters

# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root

Nova Compute Service restart
# systemctl restart openstack-nova-compute

Glance API Service restart
vi /etc/glance/glance-api.conf

# systemctl restart openstack-glance-api

[root@ip-192-169-142-57 ~(keystone_admin)]# docker pull eugeneware/docker-wordpress-nginx
Using default tag: latest
latest: Pulling from eugeneware/docker-wordpress-nginx
e9c5e611068d: Pull complete
c29de585b225: Pull complete
0b3e3644d782: Pull complete
a3ed95caeb02: Pull complete
fea6537ba52d: Pull complete
8e03e1523751: Pull complete
5eeb66c69c02: Pull complete
08a6186db887: Pull complete
bfc0c4ec6477: Pull complete
862560bef6ea: Pull complete
d40676c8c305: Pull complete
92c8f8e16f5b: Pull complete
218eceb27482: Pull complete
676daea49f2f: Pull complete
c6c1b1189a3a: Pull complete
ca0f66e4f666: Pull complete
e861ed388e7f: Pull complete
4d632d738841: Pull complete
9e14ab1859ff: Pull complete
ae63cbaa7162: Pull complete
5266cf748ad1: Pull complete
6145eef78a90: Pull complete
fef9938ad991: Pull complete
e7b95771b42e: Pull complete
32446e158ec8: Pull complete
e3b3b36d5798: Pull complete
d1c4b64eb915: Pull complete
97cff020f5b5: Pull complete
01d73c2c35e7: Pull complete
Digest: sha256:d505ed12e146b22d2eabac6840ee707561dfa0dd9a51de3f5cd64d487f0f5330
Status: Downloaded newer image for eugeneware/docker-wordpress-nginx:latest

[root@ip-192-169-142-57 ~(keystone_admin)]# docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
rastasheep/ubuntu-sshd                latest              f1a87341f481        10 days ago         251.6 MB
eugeneware/docker-wordpress-nginx     latest              54e8e5b794ec        8 months ago        602.1 MB
larsks/thttpd                         latest              a31ab5050b67        14 months ago       1.058 MB

Uploading docker image to Glance

[root@ip-192-169-142-57 ~(keystone_admin)]# docker save eugeneware/docker-wordpress-nginx | openstack image create eugeneware/docker-wordpress-nginx  --public --container-format docker --disk-format raw

| Field            | Value                                                |
| checksum         | 620992bb2c498d6079c5b0e14f8009ac                     |
| container_format | docker                                               |
| created_at       | 2016-03-29T12:51:17Z                                 |
| disk_format      | raw                                                  |
| file             | /v2/images/6afca60c-e36e-4314-9406-085e67ed1f5f/file |
| id               | 6afca60c-e36e-4314-9406-085e67ed1f5f                 |
| min_disk         | 0                                                    |
| min_ram          | 0                                                    |
| name             | eugeneware/docker-wordpress-nginx                    |
| owner            | 545ee91ee03d4a13a9b60edc3611be79                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 621040128                                            |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2016-03-29T12:52:05Z                                 |
| virtual_size     | None                                                 |
| visibility       | public                                               |

Launching Nova-Docker containers via dashboard and assigning floating
ip(s) for access from Internet . External network


   Access via floating IP to Nova-Docker container running GlassFish 4.1

   Access via floating IP to Nova-Docker container running Wordpress

   Access via floating IP to Nova-Docker container running Apache/Tomcat8

 Via command line nova-dockers containers running

[root@ip-192-169-142-57 ~(keystone_demo)]# nova list
| ID                                   | Name           | Status | Task State | Power State | Networks                              |
| c1580857-04f1-4106-a627-4f19d8937728 | TomCat         | ACTIVE | -          | Running     | demo_network=, |
| 27449afb-e99c-4420-a0c2-c82dd0271280 | UbuntuSSH      | ACTIVE | -          | Running     | demo_network=, |
| 69034f99-0f64-4c8f-ae22-274ed474e764 | WordpresNGINX  | ACTIVE | -          | Running     | demo_network=, |
| e89b4166-36aa-4c50-8ae5-6d5c5d28ae08 | dbaGlassfish01 | ACTIVE | -          | Running     | demo_network=, |

[root@ip-192-169-142-57 ~(keystone_admin)]# docker ps -a
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                   PORTS               NAMES

c32413d1f38c        dba/docker-glassfish41              "/sbin/my_init"          26 minutes ago      Up 26 minutes                                nova-e89b4166-36aa-4c50-8ae5-6d5c5d28ae08
41f6c41e1335        tutum/tomcat                        "/run.sh"                3 hours ago         Up 3 hours                                   nova-c1580857-04f1-4106-a627-4f19d8937728
344bc6a48a61        991158fe2789                        "/bin/sh -c 'wget -q "   4 hours ago         Exited (5) 4 hours ago                       jolly_lovelace
83f6a7caad13        eugeneware/docker-wordpress-nginx   "/bin/bash /start.sh"    6 hours ago         Up 6 minutes                                 nova-69034f99-0f64-4c8f-ae22-274ed474e764
f69b0a5eff19        rastasheep/ubuntu-sshd              "/usr/sbin/sshd -D"      8 hours ago         Up 3 hours                                   nova-27449afb-e99c-4420-a0c2-c82dd0271280

Access to Nova-Docker containers (L2) running on VM(L1)@F23 KVM Hypervisor  from F23 Virtualization box
    Runing tutum/tomcat container access via floating IP

  Running Nova-Docker rastasheep/ubuntu container access via floating IP

1. http://egonzalez.org/nova-docker-driver/