UPDATE 04/01/2016
DVR && Nova-Docker Driver (stable/mitaka) tested fine on RDO Mitaka ( build 20160329 ) with no issues discribed in link for RDO Liberty.
So , create DVR deployment with Contrpoller/Network + N(*)Compute
Nodes. Switch to Docker Hypervisor on each Compute Node and
make requiered updates to glance and filters file on Controller.
You are all set. Nova-Dockers instances FIP(s) are available from outside
via Neutron Distributed Router (DNAT) using "fg" inteface ( fip-namespace )
residing on same host as Docker Hypervisor. South-North traffic is not
related with VXLAN tunneling on DVR systems.
Why DVR come into concern ?
Refreshing in memory similar problem with Nova-Docker Driver (Kilo)
with which I had same kind of problems (VXLAN connection Controller <==> Compute) on F22 (OVS 2.4.0) when the same driver worked fine on CentOS 7.1 (OVS 2.3.1). I just guess that Nova-Docker driver has a problem with OVS 2.4.0
no matter of stable/kilo, stable/liberty, stable/mitaka branches been checked out for driver build.
I have not run ovs-ofctl dump-flows at br-tun bridges ant etc,
because even having proved malfunctinality I cannot file it to BZ.
Nova-Docker Driver is not packaged for RDO so it's upstream stuff,
Upstream won't consider issue which involves build driver from source
on RDO Mitaka (RC1).
Thus as quick and efficient workaround I suggest DVR deployment setup,
to kill two birds with one stone. It will result South-North traffic
to be forwarded right away from host running Docker Hypervisor to Internet
and vice/versa due to basic "fg" functionality (outgoing interface of
fip-namespace,residing on Compute node having L3 agent running in "dvr"
agent_mode).
END UPDATE
Following bellow is procedure of setting up the most recent Nova-Docker driver
based on branch stable/mitaka of https://github.com/openstack/nova-docker
on the current build of RDO Mitaka, which has passed continuous integration.
Testing has been done on AIO RDO instance created via straight forward
packstack run. In meantime procedure of uploading docker image to Glance
has been changed since Kilo times. SELINUX was set to permissive mode
after rdo set up, to get floating ip(s) working for Nova-Docker instances.
First install repositories for RDO Mitaka (the most recent build passed CI):-
# yum -y install yum-plugin-priorities
# cd /etc/yum.repos.d
# curl -O https://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
# curl -O https://trunk.rdoproject.org/centos7-mitaka/current-passed-ci/delorean.repo
# yum -y install openstack-packstack
[root@SeverMitaka01 ~]# rpm -qa \*openstack-packstack\*
openstack-packstack-puppet-8.0.0-0.20160316101826.9fd26e8.el7.centos.noarch
openstack-packstack-8.0.0-0.20160316101826.9fd26e8.el7.centos.noarch
Run AIO install , set SELINUX to PERMISSIVE mode and reboot
****************************
Nova-Docker Setup itself
****************************
# curl -sSL https://get.docker.com/ | sh
I clearly see after this command installed package ( not signed warning )
[root@ServerCentOS722 ~]# rpm -qa \*docker-engine-selinux\*
docker-engine-selinux-1.10.3-1.el7.centos.noarch
However, it causes problems even in AIO environment.
# usermod -aG docker nova ( seems not help to set 660 for docker.sock )
# systemctl start docker
# systemctl enable docker
# chmod 666 /var/run/docker.sock (add to /etc/rc.d/rc.local)
# easy_install pip
# git clone -b stable/mitaka https://github.com/openstack/nova-docker
*******************
Driver set up
*******************
# cd nova-docker
# pip install -r requirements.txt
# python setup.py install
********************************************
Update and create following files
********************************************
vi /etc/nova/nova.conf
compute_driver=novadocker.virt.docker.DockerDriver
Reminder - missing next step results failure openstack-nova-scheduler
and generates "No availables hosts found"
mkdir /etc/nova/rootwrap.d
vi /etc/nova/rootwrap.d/docker.filters
[Filters]
# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root
********************************
Nova Compute Service restart
********************************
# systemctl restart openstack-nova-compute
********************************
Glance API Service restart
********************************
vi /etc/glance/glance-api.conf
container_formats=ami,ari,aki,bare,ovf,ova,docker
# systemctl restart openstack-glance-api
[root@ip-192-169-142-57 ~(keystone_admin)]# docker pull eugeneware/docker-wordpress-nginx
Using default tag: latest
latest: Pulling from eugeneware/docker-wordpress-nginx
e9c5e611068d: Pull complete
c29de585b225: Pull complete
0b3e3644d782: Pull complete
a3ed95caeb02: Pull complete
fea6537ba52d: Pull complete
8e03e1523751: Pull complete
5eeb66c69c02: Pull complete
08a6186db887: Pull complete
bfc0c4ec6477: Pull complete
862560bef6ea: Pull complete
d40676c8c305: Pull complete
92c8f8e16f5b: Pull complete
218eceb27482: Pull complete
676daea49f2f: Pull complete
c6c1b1189a3a: Pull complete
ca0f66e4f666: Pull complete
e861ed388e7f: Pull complete
4d632d738841: Pull complete
9e14ab1859ff: Pull complete
ae63cbaa7162: Pull complete
5266cf748ad1: Pull complete
6145eef78a90: Pull complete
fef9938ad991: Pull complete
e7b95771b42e: Pull complete
32446e158ec8: Pull complete
e3b3b36d5798: Pull complete
d1c4b64eb915: Pull complete
97cff020f5b5: Pull complete
01d73c2c35e7: Pull complete
Digest: sha256:d505ed12e146b22d2eabac6840ee707561dfa0dd9a51de3f5cd64d487f0f5330
Status: Downloaded newer image for eugeneware/docker-wordpress-nginx:latest
[root@ip-192-169-142-57 ~(keystone_admin)]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rastasheep/ubuntu-sshd latest f1a87341f481 10 days ago 251.6 MB
eugeneware/docker-wordpress-nginx latest 54e8e5b794ec 8 months ago 602.1 MB
larsks/thttpd latest a31ab5050b67 14 months ago 1.058 MB
*********************************************
Uploading docker image to Glance
*********************************************
[root@ip-192-169-142-57 ~(keystone_admin)]# docker save eugeneware/docker-wordpress-nginx | openstack image create eugeneware/docker-wordpress-nginx --public --container-format docker --disk-format raw
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | 620992bb2c498d6079c5b0e14f8009ac |
| container_format | docker |
| created_at | 2016-03-29T12:51:17Z |
| disk_format | raw |
| file | /v2/images/6afca60c-e36e-4314-9406-085e67ed1f5f/file |
| id | 6afca60c-e36e-4314-9406-085e67ed1f5f |
| min_disk | 0 |
| min_ram | 0 |
| name | eugeneware/docker-wordpress-nginx |
| owner | 545ee91ee03d4a13a9b60edc3611be79 |
| protected | False |
| schema | /v2/schemas/image |
| size | 621040128 |
| status | active |
| tags | |
| updated_at | 2016-03-29T12:52:05Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------+
Launching Nova-Docker containers via dashboard and assigning floating
ip(s) for access from Internet . External network 172.124.4.0/24
Access via floating IP to Nova-Docker container running GlassFish 4.1
Access via floating IP to Nova-Docker container running Wordpress
Access via floating IP to Nova-Docker container running Apache/Tomcat8
************************************************************************
Via command line nova-dockers containers running
*************************************************************************
[root@ip-192-169-142-57 ~(keystone_demo)]# nova list
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
| c1580857-04f1-4106-a627-4f19d8937728 | TomCat | ACTIVE | - | Running | demo_network=50.0.0.17, 172.124.4.157 |
| 27449afb-e99c-4420-a0c2-c82dd0271280 | UbuntuSSH | ACTIVE | - | Running | demo_network=50.0.0.11, 172.124.4.154 |
| 69034f99-0f64-4c8f-ae22-274ed474e764 | WordpresNGINX | ACTIVE | - | Running | demo_network=50.0.0.16, 172.124.4.156 |
| e89b4166-36aa-4c50-8ae5-6d5c5d28ae08 | dbaGlassfish01 | ACTIVE | - | Running | demo_network=50.0.0.18, 172.124.4.158 |
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
[root@ip-192-169-142-57 ~(keystone_admin)]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c32413d1f38c dba/docker-glassfish41 "/sbin/my_init" 26 minutes ago Up 26 minutes nova-e89b4166-36aa-4c50-8ae5-6d5c5d28ae08
41f6c41e1335 tutum/tomcat "/run.sh" 3 hours ago Up 3 hours nova-c1580857-04f1-4106-a627-4f19d8937728
344bc6a48a61 991158fe2789 "/bin/sh -c 'wget -q " 4 hours ago Exited (5) 4 hours ago jolly_lovelace
83f6a7caad13 eugeneware/docker-wordpress-nginx "/bin/bash /start.sh" 6 hours ago Up 6 minutes nova-69034f99-0f64-4c8f-ae22-274ed474e764
f69b0a5eff19 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 8 hours ago Up 3 hours nova-27449afb-e99c-4420-a0c2-c82dd0271280
****************************************************************************************
Access to Nova-Docker containers (L2) running on VM(L1)@F23 KVM Hypervisor from F23 Virtualization box
****************************************************************************************
Runing tutum/tomcat container access via floating IP
Running Nova-Docker rastasheep/ubuntu container access via floating IP
References
1. http://egonzalez.org/nova-docker-driver/
DVR && Nova-Docker Driver (stable/mitaka) tested fine on RDO Mitaka ( build 20160329 ) with no issues discribed in link for RDO Liberty.
So , create DVR deployment with Contrpoller/Network + N(*)Compute
Nodes. Switch to Docker Hypervisor on each Compute Node and
make requiered updates to glance and filters file on Controller.
You are all set. Nova-Dockers instances FIP(s) are available from outside
via Neutron Distributed Router (DNAT) using "fg" inteface ( fip-namespace )
residing on same host as Docker Hypervisor. South-North traffic is not
related with VXLAN tunneling on DVR systems.
Why DVR come into concern ?
Refreshing in memory similar problem with Nova-Docker Driver (Kilo)
with which I had same kind of problems (VXLAN connection Controller <==> Compute) on F22 (OVS 2.4.0) when the same driver worked fine on CentOS 7.1 (OVS 2.3.1). I just guess that Nova-Docker driver has a problem with OVS 2.4.0
no matter of stable/kilo, stable/liberty, stable/mitaka branches been checked out for driver build.
I have not run ovs-ofctl dump-flows at br-tun bridges ant etc,
because even having proved malfunctinality I cannot file it to BZ.
Nova-Docker Driver is not packaged for RDO so it's upstream stuff,
Upstream won't consider issue which involves build driver from source
on RDO Mitaka (RC1).
Thus as quick and efficient workaround I suggest DVR deployment setup,
to kill two birds with one stone. It will result South-North traffic
to be forwarded right away from host running Docker Hypervisor to Internet
and vice/versa due to basic "fg" functionality (outgoing interface of
fip-namespace,residing on Compute node having L3 agent running in "dvr"
agent_mode).
END UPDATE
Following bellow is procedure of setting up the most recent Nova-Docker driver
based on branch stable/mitaka of https://github.com/openstack/nova-docker
on the current build of RDO Mitaka, which has passed continuous integration.
Testing has been done on AIO RDO instance created via straight forward
packstack run. In meantime procedure of uploading docker image to Glance
has been changed since Kilo times. SELINUX was set to permissive mode
after rdo set up, to get floating ip(s) working for Nova-Docker instances.
First install repositories for RDO Mitaka (the most recent build passed CI):-
# yum -y install yum-plugin-priorities
# cd /etc/yum.repos.d
# curl -O https://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo
# curl -O https://trunk.rdoproject.org/centos7-mitaka/current-passed-ci/delorean.repo
# yum -y install openstack-packstack
[root@SeverMitaka01 ~]# rpm -qa \*openstack-packstack\*
openstack-packstack-puppet-8.0.0-0.20160316101826.9fd26e8.el7.centos.noarch
openstack-packstack-8.0.0-0.20160316101826.9fd26e8.el7.centos.noarch
Run AIO install , set SELINUX to PERMISSIVE mode and reboot
****************************
Nova-Docker Setup itself
****************************
# curl -sSL https://get.docker.com/ | sh
I clearly see after this command installed package ( not signed warning )
[root@ServerCentOS722 ~]# rpm -qa \*docker-engine-selinux\*
docker-engine-selinux-1.10.3-1.el7.centos.noarch
However, it causes problems even in AIO environment.
# usermod -aG docker nova ( seems not help to set 660 for docker.sock )
# systemctl start docker
# systemctl enable docker
# chmod 666 /var/run/docker.sock (add to /etc/rc.d/rc.local)
# easy_install pip
# git clone -b stable/mitaka https://github.com/openstack/nova-docker
*******************
Driver set up
*******************
# cd nova-docker
# pip install -r requirements.txt
# python setup.py install
********************************************
Update and create following files
********************************************
vi /etc/nova/nova.conf
compute_driver=novadocker.virt.docker.DockerDriver
Reminder - missing next step results failure openstack-nova-scheduler
and generates "No availables hosts found"
mkdir /etc/nova/rootwrap.d
vi /etc/nova/rootwrap.d/docker.filters
[Filters]
# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root
********************************
Nova Compute Service restart
********************************
# systemctl restart openstack-nova-compute
********************************
Glance API Service restart
********************************
vi /etc/glance/glance-api.conf
container_formats=ami,ari,aki,bare,ovf,ova,docker
# systemctl restart openstack-glance-api
[root@ip-192-169-142-57 ~(keystone_admin)]# docker pull eugeneware/docker-wordpress-nginx
Using default tag: latest
latest: Pulling from eugeneware/docker-wordpress-nginx
e9c5e611068d: Pull complete
c29de585b225: Pull complete
0b3e3644d782: Pull complete
a3ed95caeb02: Pull complete
fea6537ba52d: Pull complete
8e03e1523751: Pull complete
5eeb66c69c02: Pull complete
08a6186db887: Pull complete
bfc0c4ec6477: Pull complete
862560bef6ea: Pull complete
d40676c8c305: Pull complete
92c8f8e16f5b: Pull complete
218eceb27482: Pull complete
676daea49f2f: Pull complete
c6c1b1189a3a: Pull complete
ca0f66e4f666: Pull complete
e861ed388e7f: Pull complete
4d632d738841: Pull complete
9e14ab1859ff: Pull complete
ae63cbaa7162: Pull complete
5266cf748ad1: Pull complete
6145eef78a90: Pull complete
fef9938ad991: Pull complete
e7b95771b42e: Pull complete
32446e158ec8: Pull complete
e3b3b36d5798: Pull complete
d1c4b64eb915: Pull complete
97cff020f5b5: Pull complete
01d73c2c35e7: Pull complete
Digest: sha256:d505ed12e146b22d2eabac6840ee707561dfa0dd9a51de3f5cd64d487f0f5330
Status: Downloaded newer image for eugeneware/docker-wordpress-nginx:latest
[root@ip-192-169-142-57 ~(keystone_admin)]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rastasheep/ubuntu-sshd latest f1a87341f481 10 days ago 251.6 MB
eugeneware/docker-wordpress-nginx latest 54e8e5b794ec 8 months ago 602.1 MB
larsks/thttpd latest a31ab5050b67 14 months ago 1.058 MB
*********************************************
Uploading docker image to Glance
*********************************************
[root@ip-192-169-142-57 ~(keystone_admin)]# docker save eugeneware/docker-wordpress-nginx | openstack image create eugeneware/docker-wordpress-nginx --public --container-format docker --disk-format raw
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | 620992bb2c498d6079c5b0e14f8009ac |
| container_format | docker |
| created_at | 2016-03-29T12:51:17Z |
| disk_format | raw |
| file | /v2/images/6afca60c-e36e-4314-9406-085e67ed1f5f/file |
| id | 6afca60c-e36e-4314-9406-085e67ed1f5f |
| min_disk | 0 |
| min_ram | 0 |
| name | eugeneware/docker-wordpress-nginx |
| owner | 545ee91ee03d4a13a9b60edc3611be79 |
| protected | False |
| schema | /v2/schemas/image |
| size | 621040128 |
| status | active |
| tags | |
| updated_at | 2016-03-29T12:52:05Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------+
Launching Nova-Docker containers via dashboard and assigning floating
ip(s) for access from Internet . External network 172.124.4.0/24
Access via floating IP to Nova-Docker container running GlassFish 4.1
Access via floating IP to Nova-Docker container running Wordpress
Access via floating IP to Nova-Docker container running Apache/Tomcat8
************************************************************************
Via command line nova-dockers containers running
*************************************************************************
[root@ip-192-169-142-57 ~(keystone_demo)]# nova list
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
| c1580857-04f1-4106-a627-4f19d8937728 | TomCat | ACTIVE | - | Running | demo_network=50.0.0.17, 172.124.4.157 |
| 27449afb-e99c-4420-a0c2-c82dd0271280 | UbuntuSSH | ACTIVE | - | Running | demo_network=50.0.0.11, 172.124.4.154 |
| 69034f99-0f64-4c8f-ae22-274ed474e764 | WordpresNGINX | ACTIVE | - | Running | demo_network=50.0.0.16, 172.124.4.156 |
| e89b4166-36aa-4c50-8ae5-6d5c5d28ae08 | dbaGlassfish01 | ACTIVE | - | Running | demo_network=50.0.0.18, 172.124.4.158 |
+--------------------------------------+----------------+--------+------------+-------------+---------------------------------------+
[root@ip-192-169-142-57 ~(keystone_admin)]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c32413d1f38c dba/docker-glassfish41 "/sbin/my_init" 26 minutes ago Up 26 minutes nova-e89b4166-36aa-4c50-8ae5-6d5c5d28ae08
41f6c41e1335 tutum/tomcat "/run.sh" 3 hours ago Up 3 hours nova-c1580857-04f1-4106-a627-4f19d8937728
344bc6a48a61 991158fe2789 "/bin/sh -c 'wget -q " 4 hours ago Exited (5) 4 hours ago jolly_lovelace
83f6a7caad13 eugeneware/docker-wordpress-nginx "/bin/bash /start.sh" 6 hours ago Up 6 minutes nova-69034f99-0f64-4c8f-ae22-274ed474e764
f69b0a5eff19 rastasheep/ubuntu-sshd "/usr/sbin/sshd -D" 8 hours ago Up 3 hours nova-27449afb-e99c-4420-a0c2-c82dd0271280
****************************************************************************************
Access to Nova-Docker containers (L2) running on VM(L1)@F23 KVM Hypervisor from F23 Virtualization box
****************************************************************************************
Runing tutum/tomcat container access via floating IP
References
1. http://egonzalez.org/nova-docker-driver/
