Posting is addressing question been asked at ask.openstack.org ([1])
Question :-
Can meta-data co-exist in qrouter and qdhcp namespace at the same time
so that LANs without Routers involved can access meta-data ?
Answer is as follows :-
All private networks (having neutron router) created
before or after this change will continue provide metadata via
neutron-ns-metadata-proxy running in corresponding qrouter-namespace for
theirs VMs.
******************************************************************
For routable qdhcp-namespace created before update dhcp_agent.ini
******************************************************************
[root@vfedora22wks ~(keystone_admin)]# ip netns exec \
qdhcp-e86eebdb-71bd-4929-937c-2ab57db30e18 netstat -4 -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 50.0.0.10:53 0.0.0.0:*
LISTEN 6773/dnsmasq
tcp 0 0 169.254.169.254:53 0.0.0.0:*
LISTEN 6773/dnsmasq
tcp 0 0 50.0.0.10:42011 50.0.0.15:22
ESTABLISHED 2784/ssh
So it still gets access to metadata via qrouter's ns-metadata-proxy
******************************************************************************
For isolated qdhcp-namespaces /bin/neutron-ns-metadata-proxy
gets started in corresponding qdhcp-namespace
******************************************************************************
[root@vfedora22wks ~(keystone_admin)]# ip netns exec \
qdhcp-e0f08063-2002-4cc9-b7b1-611925ad01e5 netstat -4 -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
6333/python2
tcp 0 0 30.0.0.10:53 0.0.0.0:* LISTEN
6771/dnsmasq
tcp 0 0 169.254.169.254:53 0.0.0.0:* LISTEN
6771/dnsmasq
[root@vfedora22wks ~(keystone_admin)]# ps -f --pid 6333 | fold -s -w 82
UID PID PPID C STIME TTY TIME CMD
neutron 6333 1 0 20:38 ? 00:00:00 /usr/bin/python2
/bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/e0f08063-2002-4cc9-b7b1-611925ad01e5.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy <====
--network_id=e0f08063-2002-4cc9-b7b1-611925ad01e5 --state_path=/var/lib/neutron
--metadata_port=80 --metadata_proxy_user=983 --metadata_proxy_group=977 --verbose
--log-file=neutron-ns-metadata-proxy-e0f08063-2002-4cc9-b7b1-611925ad01e5.log
--log-dir=/var/log/neutron
For private_network having neutron router and created immediately after update
"enable_isolated_metadata=True" and service restart
[root@vfedora22wks ~(keystone_admin)]# ip netns exec \
qdhcp-6e4646d8-2c5f-4adc-a4dc-51884f090d09 netstat -4 -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:*
LISTEN 8654/python2
tcp 0 0 60.0.0.10:53 0.0.0.0:*
LISTEN 8626/dnsmasq
tcp 0 0 169.254.169.254:53 0.0.0.0:*
LISTEN 8626/dnsmasq
[root@vfedora22wks ~(keystone_admin)]# ps -f --pid 8654 | fold -s -w 82
UID PID PPID C STIME TTY TIME CMD
neutron 8654 1 0 20:43 ? 00:00:00 /usr/bin/python2
/bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/6e4646d8-2c5f-4adc-a4dc-51884f090d09.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy <=====
--network_id=6e4646d8-2c5f-4adc-a4dc-51884f090d09 --state_path=/var/lib/neutron
--metadata_port=80 --metadata_proxy_user=983 --metadata_proxy_group=977 --verbose
--log-file=neutron-ns-metadata-proxy-6e4646d8-2c5f-4adc-a4dc-51884f090d09.log
--log-dir=/var/log/neutron
However,I
have noticed that nodes restart disables neutron-ns-metadata-proxy for
route-able tenant's networks, e.g. neutron-router port exists on
corresponding network.
VM's metadata request is routed via qdhcp-namespace to qrouter-namespace
VMs get metadata from neutron-ns-metadata-proxy running in qrouter-namespace.
For isolated qdhcp-namespaces node reboot still keeps neutron-ns-metadata-proxy in corresponding qdhcp-namespace
*********************************************************************************************
All
Nodes rebooted neutron-ns-metadata-proxy is no longer kept in
route-able qdhcp-namespaces. VMs are serving via
neutron-ns-metadata-proxy running in qrouter-namespace. *********************************************************************************************
Launching CirrOS instance via isolated network
Verification neutron-ns-metadata-proxy status
Launching VF22Devs07 VM via route-able tenant's network demo_network Verification neutron-ns-metadata-proxy status
1.
Neutron DVR implements the fip-namespace on every Compute Node where
the VMs are running. Thus VMs with FloatingIPs can forward the traffic
to the External Network without routing it via Network Node.
(North-South Routing).
2. Neutron DVR implements the L3 Routers
across the Compute Nodes, so that tenants intra VM communication will
occur with Network Node not involved. (East-West Routing).
3.
Neutron Distributed Virtual Router provides the legacy SNAT behavior for
the default SNAT for all private VMs. SNAT service is not distributed,
it is centralized and the service node will host the service.
Three CentOS 7.1 VMs (4 GB RAM, 4 VCPU, 2 VNICs ) has been built for testing
at
Fedora 22 KVM Hypervisor.
Two libvirt sub-nets were used first
"openstackvms" for emulating External && Mgmt Networks
192.169.142.0/24 gateway virbr1 (192.169.142.1) and "vteps" 10.0.0.0/24
to support two VXLAN tunnels between Controller and Compute Nodes.
# virsh net-define openstackvms.xml
# virsh net-start openstackvms
# virsh net-autostart openstackvms
Second libvirt sub-net maybe defined and started same way.
***************************************************************************************
The last entry for [agent] is important for DVR configuration on Kilo ( vs Juno )
Command been run on Compute :- rsync -av root@controller:/etc/neutron/plugins/ml2 /etc/neutron/plugins
as suggested in http://schmaustech.blogspot.com/2014/12/configuring-dvr-in-openstack-juno.html
would work for you on Juno. On Kilo files /etc/neutron/plugins/ml2/ml2_conf.ini
are different on Controller/Network and on Compute nodes. Missing [agent]
section on Kilo will result VXLAN tunnels not come up after nodes reboot.
***************************************************************************************
Per http://funcptr.net/2014/09/29/openstack-resizing-of-instances/
During the resize process, the node where the instance is currently
running will use SSH to connect to another compute node where the re-sized
instance will live, and copy over the instance and associated files.
Actually, there is an option to change this behavior and perform resize instance on the same compute node .
View :- http://www.madorn.com/resize-on-single-compute.html#.Vgb1wrNRpFB
Just one notice it requires restart all nova services on Controller and
openstack-nova-compute on Compute Node.
Posting bellow presumes that there are at least 2 Compute nodes :-
compute01
compute02
There are a couple of assumptions which will be made:
Nova and qemu user both have the same UID on all compute nodes
The path for your instances is the same on all of your compute nodes
*************************
On Controller
*************************
Verify nova account has no any shell cat /etc/passwd | grep nova
In this case add /bin/bash to account "nova" usermod -s /bin/bash nova
****************************************
Generate SSH key and Configuration
****************************************
After doing this the next steps are all run as the nova user.
# su - nova
Now to generate an SSH key:
$ ssh-keygen -t rsa
Save the key without a passphrase.
Next we need to configure SSH to not do host key verification,
*******************************************************
Then user nova creates tar ball to replicate it
between all Compute Nodes
*******************************************************
# su - nova
$ id
$ pwd
/var/lib/nova
$ tar -cvf ssh.tar .ssh/*
^D
# cd ~nova
# scp ssh.tar compute01:/var/lib/nova
# scp ssh.tar compute02:/var/lib/nova
# ssh compute01
**********************
In other terminal
**********************
# ssh compute02
****************************************
On Compute01 and Compute02
****************************************
# usermod -s /bin/bash nova
# cd /var/lib/nova
# chown nova:nova ssh.tar
# su - nova
$ ls -la
$ tar -xvf ssh.tar
At this point : You should be able ssh as "nova" from contoller
to compute01 , compute02 and between compute nodes without
password prompt.
Compute nodes are trusting Controller and each other via account "nova"
*****************************
Now run on Controller
*****************************
[root@ip-192-169-142-127 ~(keystone_admin)]# . keystonerc_demo
[root@ip-192-169-142-127 ~(keystone_demo)]# nova list
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
| 64b84cb7-c249-4808-b16e-0071d4d288e8 | CirrOSDevs | ACTIVE | - | Running | demo_network=40.0.0.15, 172.24.4.234 |
| b6a9c438-3d7c-4f7e-aa4d-3ad47178eeac | VF22Devs15 | SHUTOFF | - | Shutdown | demo_network=40.0.0.13, 172.24.4.232 |
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
[root@ip-192-169-142-127 ~(keystone_demo)]# nova resize CirrOSDevs 2 --poll
Server resizing... 100% complete
Finished
[root@ip-192-169-142-127 ~(keystone_demo)]# nova list
+--------------------------------------+------------+---------------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+---------------+------------+-------------+--------------------------------------+
| 64b84cb7-c249-4808-b16e-0071d4d288e8 | CirrOSDevs | VERIFY_RESIZE | - | Running | demo_network=40.0.0.15, 172.24.4.234 |
| b6a9c438-3d7c-4f7e-aa4d-3ad47178eeac | VF22Devs15 | SHUTOFF | - | Shutdown | demo_network=40.0.0.13, 172.24.4.232 |
+--------------------------------------+------------+---------------+------------+-------------+--------------------------------------+
[root@ip-192-169-142-127 ~(keystone_demo)]# nova resize-confirm 64b84cb7-c249-4808-b16e-0071d4d288e8
[root@ip-192-169-142-127 ~(keystone_demo)]# nova list
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
| 64b84cb7-c249-4808-b16e-0071d4d288e8 | CirrOSDevs | ACTIVE | - | Running | demo_network=40.0.0.15, 172.24.4.234 |
| b6a9c438-3d7c-4f7e-aa4d-3ad47178eeac | VF22Devs15 | SHUTOFF | - | Shutdown | demo_network=40.0.0.13, 172.24.4.232 |
+--------------------------------------+------------+---------+------------+-------------+--------------------------------------+
*************************************************************************************
UPDATE 10/01/2015
Would you experience VXLAN tunnels disappiaring issue like it happens
on RDO Kilo add following lines to ml2_conf.ini on each Compute Node :- [agent] l2_population = True
followed by `openstack-service restart`
I also have to notice that Nested KVM does provide significant performance
improvement on Haswell i5,i7 CPUs
*************************************************************************************
Per http://specs.openstack.org/openstack/neutron-specs/specs/juno/neutron-ovs-dvr.html
1. Neutron DVR implements the fip-namespace
on every Compute Node where the VMs are running. Thus VMs with
FloatingIPs can forward the traffic to the External Network without routing it via Network Node. (North-South Routing).
2. Neutron DVR implements the L3 Routers across the
Compute Nodes, so that tenants intra VM communication will occur with Network Node not involved. (East-West Routing).
3. Neutron Distributed Virtual Router provides the legacy SNAT behavior for
the default SNAT for all private VMs. SNAT service is not distributed, it
is centralized and the service node will host the service.
- (2x) Compute node: Nova (nova-compute), Neutron (openvswitch-agent,l3-agent,metadata-agent )
Three CentOS 7.1 VMs (4 GB RAM, 4 VCPU, 2 VNICs ) has been built for testing
at
Fedora 22 KVM Hypervisor. Two libvirt sub-nets were used first
"openstackvms" for emulating External && Mgmt Networks
192.169.142.0/24 gateway virbr1 (192.169.142.1) and "vteps" 10.0.0.0/24
to support two VXLAN tunnels between Controller and Compute Nodes.
********************************************************************************
Please, be asvised that command like ([1]) :-
# rsync -av root@192.169.142.127:/etc/neutron/plugins/ml2 /etc/neutron/plugins
been run on Liberty Compute Node 192.169.142.147 will overwrite file
/etc/neutron/plugins/ml2/openvswitch_agent.ini
So, local_ip after this command should be turned backed to it's initial value.
********************************************************************************
[root@ip-192-169-142-147 ~]# ip netns
qrouter-6f638e97-7621-4d05-b9bc-50147b29d6b8
fip-fc3e1bf8-bb39-4468-b5cb-8cdc79837be1
qrouter-97a1c79a-178b-4507-8eb8-f0d6f8958858
[root@ip-192-169-142-147 ~]# ip netns exec fip-fc3e1bf8-bb39-4468-b5cb-8cdc79837be1 ip a | grep "inet"
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 169.254.31.29/31 scope global fpr-97a1c79a-1
inet6 fe80::40c0:3aff:fe04:50e5/64 scope link
inet 169.254.31.239/31 scope global fpr-6f638e97-7
inet6 fe80::84e6:42ff:fe56:33e2/64 scope link
inet 192.169.142.158/24 brd 192.169.142.255 scope global fg-b97c737e-17
inet6 fe80::f816:3eff:fe5f:ce5d/64 scope link
[root@ip-192-169-142-147 ~]# ip netns exec fip-fc3e1bf8-bb39-4468-b5cb-8cdc79837be1 ip route
default via 192.169.142.1 dev fg-b97c737e-17
169.254.31.28/31 dev fpr-97a1c79a-1 proto kernel scope link src 169.254.31.29
169.254.31.238/31 dev fpr-6f638e97-7 proto kernel scope link src 169.254.31.239
192.169.142.0/24 dev fg-b97c737e-17 proto kernel scope link src 192.169.142.158
192.169.142.157 via 169.254.31.28 dev fpr-97a1c79a-1
192.169.142.161 via 169.254.31.238 dev fpr-6f638e97-7
