It is actually an update of http://techbackground.blogspot.ie/2013/06/metadata-via-quantum-router.html for Neutron on Juno ( original blog considers Quantum implementation on Grizzly ). From my standpoint understanding of core architecture of Neutron openstack flow in regards of nova-api metadata service access (and getting proper response from nova -api ) by VMs launching via nova causes a lot of problems due to leak of understanding of core concepts.
Neutron proxies metadata requests to Nova adding HTTP headers which Nova uses to identify the source instance. Neutron actually uses two proxies to do this: a namespace proxy and a metadata agent.
This post shows how a metadata request gets from an instance to the Nova metadata service via a namespace proxy running in a Neutron router.
1. Instance makes request
[fedora@vf20rsx2211 ~]$ curl http://169.254.169.254/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups[fedora@vf20rsx2211 ~]$
[fedora@vf20rsx2211 ~]$ ip -4 address show dev eth0
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 50.0.0.44/24 brd 50.0.0.255 scope global dynamic eth0
valid_lft 70518sec preferred_lft 70518sec
[fedora@vf20rsx2211 ~]$ ip route
default via 50.0.0.1 dev eth0 proto static metric 1024
50.0.0.0/24 dev eth0 proto kernel scope link src 50.0.0.44
[fedora@vf20rsx2211 ~]$ ip route get 169.254.169.254
169.254.169.254 via 50.0.0.1 dev eth0 src 50.0.0.44
cache
2. Namespace proxy receives request
The default gateway 50.0.0.1 exists within a Neutron router namespace on the network node. The Neutron-l3-agent started a namespace proxy in this namespace and added some iptables rules to redirect metadata requests to it.
There are no special routes, so the request goes out the default gateway.
of course a Neutron router needs to have an interface on the subnet.
[root@juno1 ~(keystone_admin)]# ip netns exec qdhcp-45577666-657d-4f75-a3ab-9bc232f15203 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 50.0.0.1 0.0.0.0 UG 0 0 0 tap7a12f9b0-a4
50.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 tap7a12f9b0-a4
[root@juno1 ~(keystone_admin)]# neutron router-list
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| 1cf08ea2-959f-4206-b2f1-a9b4708399c1 | router4 | {"network_id": "65cbd354-daae-41bb-9d3c-e58b1062be19", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "147d5ecd-fe39-489e-8901-3b20a2c50148", "ip_address": "192.168.1.173"}]} | False | False |
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 ifconfig
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-7b037650-10: flags=4163 mtu 1500
inet 192.168.1.173 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::f816:3eff:fee5:de97 prefixlen 64 scopeid 0x20
ether fa:16:3e:e5:de:97 txqueuelen 0 (Ethernet)
RX packets 63929 bytes 87480425 (83.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36523 bytes 5286278 (5.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-17ddee14-9f: flags=4163 mtu 1500
inet 50.0.0.1 netmask 255.255.255.0 broadcast 50.0.0.255
inet6 fe80::f816:3eff:fe6f:a8e7 prefixlen 64 scopeid 0x20
ether fa:16:3e:6f:a8:e7 txqueuelen 0 (Ethernet)
RX packets 36643 bytes 5304620 (5.0 MiB)
RX errors 0 dropped 5 overruns 0 frame 0
TX packets 62940 bytes 87350558 (83.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 ip -4 address show dev qr-17ddee14-9f
16: qr-17ddee14-9f: mtu 1500 qdisc noqueue state UNKNOWN
inet 50.0.0.1/24 brd 50.0.0.255 scope global qr-17ddee14-9f
valid_lft forever preferred_lft forever
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 iptables-save| grep 9697
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 6755/python
[root@juno1 ~(keystone_admin)]# ps -f --pid 6755 | fold -s -w 82
UID PID PPID C STIME TTY TIME CMD
root 6755 1 0 08:01 ? 00:00:00 /usr/bin/python
/bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/1cf08ea2-959f-4206-b2f1-a9b4708399c1.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=1cf08ea2-959f-4206-b2f1-a9b4708399c1 --state_path=/var/lib/neutron
--metadata_port=9697 --verbose
--log-file=neutron-ns-metadata-proxy-1cf08ea2-959f-4206-b2f1-a9b4708399c1.log
--log-dir=/var/log/neutron
The nameserver proxy adds two HTTP headers to the request:
X-Forwarded-For: with the instance's IP address
X-Neutron-Router-ID: with the uuid of the Neutron router
and proxies it to a Unix domain socket with name
/var/lib/Neutron/metadata_proxy.
3. Metadata agent receives request and queries the Neutron service
The metadata agent listens on this Unix socket. It is a normal Linux service that runs in the main operating system IP namespace, and so it is able to reach the Neutron and Nova metadata services. Its configuration file has all the information required to do so.
[root@juno1 ~(keystone_admin)]# netstat -lxp | grep metadata
unix 2 [ ACC ] STREAM LISTENING 36027 1589/python /var/lib/neutron/metadata_proxy
[root@juno1 ~(keystone_admin)]# lsof /var/lib/neutron/metadata_proxy
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
neutron-m 1589 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
neutron-m 3412 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
neutron-m 3413 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
[root@juno1 ~(keystone_admin)]# ps -f --pid 1589 | fold -w 80 -s
UID PID PPID C STIME TTY TIME CMD
neutron 1589 1 0 07:59 ? 00:00:03 /usr/bin/python
/usr/bin/neutron-metadata-agent --config-file
/usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/metadata_agent.ini --log-file
/var/log/neutron/metadata-agent.log
[root@juno1 neutron(keystone_admin)]# grep -v '^#\|^\s*$' /etc/neutron/metadata_agent.ini
[DEFAULT]
debug = False
auth_url = http://192.168.1.127:35357/v2.0
auth_region = RegionOne
auth_insecure = False
admin_tenant_name = services
admin_user = neutron
admin_password = 808e36e154bd4cee
nova_metadata_ip = 192.168.1.127
nova_metadata_port = 8775
metadata_proxy_shared_secret =a965cd23ed2f4502
metadata_workers =2
metadata_backlog = 4096
It reads the X-Forwarded-For and X-Neutron-Router-ID headers in the request and queries the Neutron service to find the ID of the instance that created the request.
4. Metadata agent proxies request to Nova metadata service
It then adds these headers:
X-Instance-ID: the instance ID returned from Neutron
X-Instance-ID-Signature: instance ID signed with the shared-secret
X-Forwarded-For: the instance's IP address
and proxies the request to the Nova metadata service.
[root@juno1 nova(keystone_admin)]# grep metadata /etc/nova/nova.conf | grep -v ^# | grep -v ^$
enabled_apis=ec2,osapi_compute,metadata
metadata_listen=0.0.0.0
metadata_workers=2
metadata_host=192.168.1.127
neutron_metadata_proxy_shared_secret=a965cd23ed2f4502
service_neutron_metadata_proxy=True
[root@juno1 nova(keystone_admin)]# grep metadata /var/log/nova/nova-api.log | tail -15
2014-11-24 08:20:39.208 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-keys/ HTTP/1.1" status: 200 len: 125 time: 0.0013790
2014-11-24 08:20:39.217 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/ami-id HTTP/1.1" status: 200 len: 120 time: 0.0014508
2014-11-24 08:20:39.227 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/kernel-id HTTP/1.1" status: 200 len: 120 time: 0.0014200
2014-11-24 08:20:39.237 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-action HTTP/1.1" status: 200 len: 120 time: 0.0013640
2014-11-24 08:20:39.247 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 130 time: 0.0014000
2014-11-24 08:20:39.256 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ HTTP/1.1" status: 200 len: 130 time: 0.0013840
2014-11-24 08:20:39.265 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/ami-manifest-path HTTP/1.1" status: 200 len: 121 time: 0.0013070
2014-11-24 08:20:39.275 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/security-groups HTTP/1.1" status: 200 len: 116 time: 0.0013120
2014-11-24 08:20:39.285 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-type HTTP/1.1" status: 200 len: 124 time: 0.0013220
2014-11-24 08:20:39.294 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 200 len: 127 time: 0.0012989
2014-11-24 08:20:39.304 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/placement/availability-zone HTTP/1.1" status: 200 len: 120 time: 0.0013518
2014-11-24 08:20:39.313 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-keys/0/openssh-key HTTP/1.1" status: 200 len: 517 time: 0.0013201
2014-11-24 08:20:39.323 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ami HTTP/1.1" status: 200 len: 119 time: 0.0013349
2014-11-24 08:20:39.333 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ebs0 HTTP/1.1" status: 200 len: 124 time: 0.0013509
2014-11-24 08:20:39.342 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/root HTTP/1.1" status: 200 len: 124 time: 0.0013192
Neutron proxies metadata requests to Nova adding HTTP headers which Nova uses to identify the source instance. Neutron actually uses two proxies to do this: a namespace proxy and a metadata agent.
This post shows how a metadata request gets from an instance to the Nova metadata service via a namespace proxy running in a Neutron router.
1. Instance makes request
[fedora@vf20rsx2211 ~]$ curl http://169.254.169.254/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups[fedora@vf20rsx2211 ~]$
[fedora@vf20rsx2211 ~]$ ip -4 address show dev eth0
2: eth0:
inet 50.0.0.44/24 brd 50.0.0.255 scope global dynamic eth0
valid_lft 70518sec preferred_lft 70518sec
[fedora@vf20rsx2211 ~]$ ip route
default via 50.0.0.1 dev eth0 proto static metric 1024
50.0.0.0/24 dev eth0 proto kernel scope link src 50.0.0.44
[fedora@vf20rsx2211 ~]$ ip route get 169.254.169.254
169.254.169.254 via 50.0.0.1 dev eth0 src 50.0.0.44
cache
2. Namespace proxy receives request
The default gateway 50.0.0.1 exists within a Neutron router namespace on the network node. The Neutron-l3-agent started a namespace proxy in this namespace and added some iptables rules to redirect metadata requests to it.
There are no special routes, so the request goes out the default gateway.
of course a Neutron router needs to have an interface on the subnet.
[root@juno1 ~(keystone_admin)]# ip netns exec qdhcp-45577666-657d-4f75-a3ab-9bc232f15203 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 50.0.0.1 0.0.0.0 UG 0 0 0 tap7a12f9b0-a4
50.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 tap7a12f9b0-a4
[root@juno1 ~(keystone_admin)]# neutron router-list
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| 1cf08ea2-959f-4206-b2f1-a9b4708399c1 | router4 | {"network_id": "65cbd354-daae-41bb-9d3c-e58b1062be19", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "147d5ecd-fe39-489e-8901-3b20a2c50148", "ip_address": "192.168.1.173"}]} | False | False |
+--------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 ifconfig
lo: flags=73
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-7b037650-10: flags=4163
inet 192.168.1.173 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::f816:3eff:fee5:de97 prefixlen 64 scopeid 0x20
ether fa:16:3e:e5:de:97 txqueuelen 0 (Ethernet)
RX packets 63929 bytes 87480425 (83.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36523 bytes 5286278 (5.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-17ddee14-9f: flags=4163
inet 50.0.0.1 netmask 255.255.255.0 broadcast 50.0.0.255
inet6 fe80::f816:3eff:fe6f:a8e7 prefixlen 64 scopeid 0x20
ether fa:16:3e:6f:a8:e7 txqueuelen 0 (Ethernet)
RX packets 36643 bytes 5304620 (5.0 MiB)
RX errors 0 dropped 5 overruns 0 frame 0
TX packets 62940 bytes 87350558 (83.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 ip -4 address show dev qr-17ddee14-9f
16: qr-17ddee14-9f:
inet 50.0.0.1/24 brd 50.0.0.255 scope global qr-17ddee14-9f
valid_lft forever preferred_lft forever
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 iptables-save| grep 9697
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT
[root@juno1 ~(keystone_admin)]# ip netns exec qrouter-1cf08ea2-959f-4206-b2f1-a9b4708399c1 netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 6755/python
[root@juno1 ~(keystone_admin)]# ps -f --pid 6755 | fold -s -w 82
UID PID PPID C STIME TTY TIME CMD
root 6755 1 0 08:01 ? 00:00:00 /usr/bin/python
/bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/1cf08ea2-959f-4206-b2f1-a9b4708399c1.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=1cf08ea2-959f-4206-b2f1-a9b4708399c1 --state_path=/var/lib/neutron
--metadata_port=9697 --verbose
--log-file=neutron-ns-metadata-proxy-1cf08ea2-959f-4206-b2f1-a9b4708399c1.log
--log-dir=/var/log/neutron
The nameserver proxy adds two HTTP headers to the request:
X-Forwarded-For: with the instance's IP address
X-Neutron-Router-ID: with the uuid of the Neutron router
and proxies it to a Unix domain socket with name
/var/lib/Neutron/metadata_proxy.
3. Metadata agent receives request and queries the Neutron service
The metadata agent listens on this Unix socket. It is a normal Linux service that runs in the main operating system IP namespace, and so it is able to reach the Neutron and Nova metadata services. Its configuration file has all the information required to do so.
[root@juno1 ~(keystone_admin)]# netstat -lxp | grep metadata
unix 2 [ ACC ] STREAM LISTENING 36027 1589/python /var/lib/neutron/metadata_proxy
[root@juno1 ~(keystone_admin)]# lsof /var/lib/neutron/metadata_proxy
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
neutron-m 1589 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
neutron-m 3412 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
neutron-m 3413 neutron 5u unix 0xffff8800c269a580 0t0 36027 /var/lib/neutron/metadata_proxy
[root@juno1 ~(keystone_admin)]# ps -f --pid 1589 | fold -w 80 -s
UID PID PPID C STIME TTY TIME CMD
neutron 1589 1 0 07:59 ? 00:00:03 /usr/bin/python
/usr/bin/neutron-metadata-agent --config-file
/usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/metadata_agent.ini --log-file
/var/log/neutron/metadata-agent.log
[root@juno1 neutron(keystone_admin)]# grep -v '^#\|^\s*$' /etc/neutron/metadata_agent.ini
[DEFAULT]
debug = False
auth_url = http://192.168.1.127:35357/v2.0
auth_region = RegionOne
auth_insecure = False
admin_tenant_name = services
admin_user = neutron
admin_password = 808e36e154bd4cee
nova_metadata_ip = 192.168.1.127
nova_metadata_port = 8775
metadata_proxy_shared_secret =a965cd23ed2f4502
metadata_workers =2
metadata_backlog = 4096
It reads the X-Forwarded-For and X-Neutron-Router-ID headers in the request and queries the Neutron service to find the ID of the instance that created the request.
4. Metadata agent proxies request to Nova metadata service
It then adds these headers:
X-Instance-ID: the instance ID returned from Neutron
X-Instance-ID-Signature: instance ID signed with the shared-secret
X-Forwarded-For: the instance's IP address
and proxies the request to the Nova metadata service.
5. Nova metadata service receives request
The metadata service was started by nova-api. The handler checks
the X-Instance-ID-Signature with the shared key, looks up the data and
returns the response which travels back via the two proxies to the
instance.[root@juno1 nova(keystone_admin)]# grep metadata /etc/nova/nova.conf | grep -v ^# | grep -v ^$
enabled_apis=ec2,osapi_compute,metadata
metadata_listen=0.0.0.0
metadata_workers=2
metadata_host=192.168.1.127
neutron_metadata_proxy_shared_secret=a965cd23ed2f4502
service_neutron_metadata_proxy=True
[root@juno1 nova(keystone_admin)]# grep metadata /var/log/nova/nova-api.log | tail -15
2014-11-24 08:20:39.208 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-keys/ HTTP/1.1" status: 200 len: 125 time: 0.0013790
2014-11-24 08:20:39.217 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/ami-id HTTP/1.1" status: 200 len: 120 time: 0.0014508
2014-11-24 08:20:39.227 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/kernel-id HTTP/1.1" status: 200 len: 120 time: 0.0014200
2014-11-24 08:20:39.237 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-action HTTP/1.1" status: 200 len: 120 time: 0.0013640
2014-11-24 08:20:39.247 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 130 time: 0.0014000
2014-11-24 08:20:39.256 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ HTTP/1.1" status: 200 len: 130 time: 0.0013840
2014-11-24 08:20:39.265 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/ami-manifest-path HTTP/1.1" status: 200 len: 121 time: 0.0013070
2014-11-24 08:20:39.275 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/security-groups HTTP/1.1" status: 200 len: 116 time: 0.0013120
2014-11-24 08:20:39.285 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-type HTTP/1.1" status: 200 len: 124 time: 0.0013220
2014-11-24 08:20:39.294 4972 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 200 len: 127 time: 0.0012989
2014-11-24 08:20:39.304 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/placement/availability-zone HTTP/1.1" status: 200 len: 120 time: 0.0013518
2014-11-24 08:20:39.313 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/public-keys/0/openssh-key HTTP/1.1" status: 200 len: 517 time: 0.0013201
2014-11-24 08:20:39.323 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ami HTTP/1.1" status: 200 len: 119 time: 0.0013349
2014-11-24 08:20:39.333 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/ebs0 HTTP/1.1" status: 200 len: 124 time: 0.0013509
2014-11-24 08:20:39.342 4970 INFO nova.metadata.wsgi.server [-] 50.0.0.44,192.168.1.127 "GET /2009-04-04/meta-data/block-device-mapping/root HTTP/1.1" status: 200 len: 124 time: 0.0013192
