Step by step instruction for setting up Nova-Docker driver && Openstack Kilo on Ubuntu 14.04 in devstack environment recoverable between reboots. Also routing across LAN is described for remote access to Devstack
(stack.sh) public network. I've tried to cover all known as of time of writing problems preventing ./rejoin-stack.sh from successful run. However , I cannot
issue any warranty that tomorrow another daemon wouldn't rejoin stack instance
after reboot. The only way for safe solution would be RDO Kilo Release expected in may of 2015 (for myself of course).
This post is written with major concern of successful loading by Kilo Nova-Docker Driver in development environment.
Proceed as follows :-
$ sudo apt-get update
$ sudo apt-get -y install git git-review python-pip python-dev
$ sudo apt-get -y upgrade
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main \
> /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker
*********************************************
Update /etc/default/docker and setting:
*********************************************
DOCKER_OPTS='-G ubuntu'
#service docker restart
Logout from root and verify
ubuntu@ubuntu-WKS:~$ docker -v
Docker version 1.6.0, build 4749651
*******************************
Installing nova-docker
*******************************
$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ sudo pip install .
Configuring devstack
*****************************
Now we're ready to get devstack up and running. Start by cloning the repository:
$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack
$ git checkout -b kilo origin/stable/kilo
******************************************
Create local.conf under devstack
******************************************
[[local|localrc]]
HOST_IP=192.168.1.142
ADMIN_PASSWORD=secret
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
SERVICE_PASSWORD=secret
FLOATING_RANGE=192.168.12.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.12.150,end=192.168.12.254
PUBLIC_NETWORK_GATEWAY=192.168.12.15
SERVICE_TOKEN=super-secret-admin-token
VIRT_DRIVER=novadocker.virt.docker.DockerDriver
DEST=/opt/stack
SERVICE_DIR=$DEST/status
DATA_DIR=$DEST/data
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs
# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1
# Services
disable_service n-net
enable_service n-cauth
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest
# Introduce glance to docker images
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker
# Configure nova to use the nova-docker driver
[[post-config|$NOVA_CONF]]
[DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver
**************************************
Corresponding iptables entry
**************************************
# iptables -t nat -A POSTROUTING -o eth0 -j
At this point you are ready to run :-
$ ./stack.sh
*****************************************************************************
Attention skipping this step causes message "No hosts available"
when launching, either causes failure to launch nova-docker instances
in case of stack.sh rerun after ./unstack.sh
******************************************************************************
$ sudo cp nova-docker/etc/nova/rootwrap.d/docker.filters \
/etc/nova/rootwrap.d/
$ . openrc admin
For docker pull && docker save
************************************************************************
To enable security rules and launch NovaDocker Container :-
************************************************************************
$ cd dev*
$ . openrc demo
$ neutron security-group-rule-create --protocol icmp \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
$ neutron security-group-rule-create --protocol tcp \
--port-range-min 22 --port-range-max 22 \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
$ neutron security-group-rule-create --protocol tcp \
--port-range-min 80 --port-range-max 80 \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
Uploading docker image to glance
$ . openrc admin
$ docker pull rastasheep/ubuntu-sshd:14.04
$ docker save rastasheep/ubuntu-sshd:14.04 | glance image-create --is-public=True --container-format=docker --disk-format=raw --name rastasheep/ubuntu-sshd:14.04
Launch new instance via uploaded image :-
$ . openrc demo
$ nova boot --image "rastasheep/ubuntu-sshd:14.04" --flavor m1.tiny
--nic net-id=private-net-id UbuntuDocker
Enable security rules via dashboard :-
*******************************************************************************
You have run `sudo ./unstack.sh` , rebooted box hosting devstack instance and OVS bridge "br-ex" came up with no IP no matter which one of local.conf has been used for ./stack.sh deployment.
Before running ./rejoin-stack.sh following actions have to be undertaken
(just add to /etc/rc.local)
*******************************************************************************
sudo ip addr flush dev br-ex
sudo ip addr add 192.168.12.15/24 dev br-ex
sudo ip link set br-ex up
sudo route add -net 10.254.1.0/24 gw 192.168.12.15
You also might experience problems with rejoining cinder-volume daemon, in this case copy to &HOME folder corresponding files and run before ./rejoin-stack.sh from your $HOME ./cinder.sh
ubuntu@ubuntu-WKS:~$ cat cinder.sh
cp stack-volumes-default-backing-file /opt/stack/data/stack-volumes-default-backing-file
cp stack-volumes-lvmdriver-1-backing-file /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
sudo losetup /dev/loop0 /opt/stack/data/stack-volumes-default-backing-file
sudo losetup /dev/loop1 /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
*******************************************************************************
To access NovaDocker instances running within stack (devstack) AIO instance on Ubuntu 14.04 host 192.168.1.142 from another boxes located on the same
office LAN having address 192.168.1.0/24 manage as follows :-
*******************************************************************************
***************************
Run on Devstack Node
***************************
# Add route to LAN
$ sudo route add -net 192.168.1.0/24 gw 192.168.1.142
**************************
Run on LAN's box
**************************
# Add route to devstack public network via HOST_IP
$ sudo route add -net 192.168.12.0/24 gw 192.168.1.142
where 192.168.1.142 HOST_IP of Devstack Node running stack instance
192.168.12.0/24 is Devstack's public network. 192.168.1.0/24 is LAN address
Routing table on Devstack node should look like :-
ubuntu@ubuntu-WKS:~$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
10.254.1.0 192.168.12.150 255.255.255.0 UG 0 0 0 br-ex
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.1.0 192.168.1.142 255.255.255.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
192.168.12.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
References
http://blog.oddbit.com/2015/02/11/installing-novadocker-with-devstack/
Proceed as follows :-
$ sudo apt-get update
$ sudo apt-get -y install git git-review python-pip python-dev
$ sudo apt-get -y upgrade
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main \
> /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker
*********************************************
Update /etc/default/docker and setting:
*********************************************
DOCKER_OPTS='-G ubuntu'
#service docker restart
Logout from root and verify
ubuntu@ubuntu-WKS:~$ docker -v
Docker version 1.6.0, build 4749651
*******************************
Installing nova-docker
*******************************
$ git clone http://github.com/stackforge/nova-docker.git
$ cd nova-docker
$ sudo pip install .
iptables -t nat -A POSTROUTING -o eth0 -j
*****************************Configuring devstack
*****************************
Now we're ready to get devstack up and running. Start by cloning the repository:
$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack
$ git checkout -b kilo origin/stable/kilo
******************************************
Create local.conf under devstack
******************************************
[[local|localrc]]
HOST_IP=192.168.1.142
ADMIN_PASSWORD=secret
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
SERVICE_PASSWORD=secret
FLOATING_RANGE=192.168.12.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.12.150,end=192.168.12.254
PUBLIC_NETWORK_GATEWAY=192.168.12.15
SERVICE_TOKEN=super-secret-admin-token
VIRT_DRIVER=novadocker.virt.docker.DockerDriver
DEST=/opt/stack
SERVICE_DIR=$DEST/status
DATA_DIR=$DEST/data
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs
# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1
# Services
disable_service n-net
enable_service n-cauth
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest
# Introduce glance to docker images
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker
# Configure nova to use the nova-docker driver
[[post-config|$NOVA_CONF]]
[DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver
**************************************
Corresponding iptables entry
**************************************
# iptables -t nat -A POSTROUTING -o eth0 -j
MASQUERADEAt this point you are ready to run :-
$ ./stack.sh
*****************************************************************************
Attention skipping this step causes message "No hosts available"
when launching, either causes failure to launch nova-docker instances
in case of stack.sh rerun after ./unstack.sh
******************************************************************************
$ sudo cp nova-docker/etc/nova/rootwrap.d/docker.filters \
/etc/nova/rootwrap.d/
$ . openrc admin
For docker pull && docker save
************************************************************************
To enable security rules and launch NovaDocker Container :-
************************************************************************
$ cd dev*
$ . openrc demo
$ neutron security-group-rule-create --protocol icmp \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
$ neutron security-group-rule-create --protocol tcp \
--port-range-min 22 --port-range-max 22 \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
$ neutron security-group-rule-create --protocol tcp \
--port-range-min 80 --port-range-max 80 \
--direction ingress --remote-ip-prefix 0.0.0.0/0 default
Uploading docker image to glance
$ . openrc admin
$ docker pull rastasheep/ubuntu-sshd:14.04
$ docker save rastasheep/ubuntu-sshd:14.04 | glance image-create --is-public=True --container-format=docker --disk-format=raw --name rastasheep/ubuntu-sshd:14.04
Launch new instance via uploaded image :-
$ . openrc demo
$ nova boot --image "rastasheep/ubuntu-sshd:14.04" --flavor m1.tiny
--nic net-id=private-net-id UbuntuDocker
Enable security rules via dashboard :-
*******************************************************************************
You have run `sudo ./unstack.sh` , rebooted box hosting devstack instance and OVS bridge "br-ex" came up with no IP no matter which one of local.conf has been used for ./stack.sh deployment.
Before running ./rejoin-stack.sh following actions have to be undertaken
(just add to /etc/rc.local)
*******************************************************************************
sudo ip addr flush dev br-ex
sudo ip addr add 192.168.12.15/24 dev br-ex
sudo ip link set br-ex up
sudo route add -net 10.254.1.0/24 gw 192.168.12.15
You also might experience problems with rejoining cinder-volume daemon, in this case copy to &HOME folder corresponding files and run before ./rejoin-stack.sh from your $HOME ./cinder.sh
ubuntu@ubuntu-WKS:~$ cat cinder.sh
cp stack-volumes-default-backing-file /opt/stack/data/stack-volumes-default-backing-file
cp stack-volumes-lvmdriver-1-backing-file /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
sudo losetup /dev/loop0 /opt/stack/data/stack-volumes-default-backing-file
sudo losetup /dev/loop1 /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
*******************************************************************************
To access NovaDocker instances running within stack (devstack) AIO instance on Ubuntu 14.04 host 192.168.1.142 from another boxes located on the same
office LAN having address 192.168.1.0/24 manage as follows :-
*******************************************************************************
***************************
Run on Devstack Node
***************************
# Add route to LAN
$ sudo route add -net 192.168.1.0/24 gw 192.168.1.142
**************************
Run on LAN's box
**************************
# Add route to devstack public network via HOST_IP
$ sudo route add -net 192.168.12.0/24 gw 192.168.1.142
where 192.168.1.142 HOST_IP of Devstack Node running stack instance
192.168.12.0/24 is Devstack's public network. 192.168.1.0/24 is LAN address
Routing table on Devstack node should look like :-
ubuntu@ubuntu-WKS:~$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
10.254.1.0 192.168.12.150 255.255.255.0 UG 0 0 0 br-ex
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.1.0 192.168.1.142 255.255.255.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
192.168.12.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
References
http://blog.oddbit.com/2015/02/11/installing-novadocker-with-devstack/
