Multiple external networks with a single L3 agent testing on RDO Liberty per Lars Kellogg-Stedman

Following bellow is supposed to test in multi node environment
Multiple external networks with a single L3 agent by Lars Kellogg-Stedman

However, current post contains an attempt to analyze and understand how traffic to/from external network flows through br-int when provider external networks has been involved

I was also hit by  Bug  neutron-openvswitch-agent is crashing with "invalid literal for int() with base 10" error
and patch https://review.openstack.org/#/c/225001/   was also applied

Basic 3 VM node setup was done per https://www.linux.com/community/blogs/133-general-linux/854587-rdo-liberty-beta-set-up-for-three-vm-nodes-controllernetworkcompute-ml2aovsavxlan-on-centos71/

Nested KVM was enable for all VM hosting RDO Liberty nodes.

Create to two Libvirt sub-nets external3,external4 on KVM Virtualization Host (F22)

[root@fedora22wksr ~]# cat external3.xml
<network>
   <name>external3</name>
   <uuid>d0e9964b-f95d-40c2-b749-b609aed52cf2</uuid>
   <forward mode='nat'>
     <nat>
       <port start='1024' end='65535'/>
     </nat>
   </forward>
   <bridge name='virbr6' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6d'/>
   <ip address='10.3.0.1' netmask='255.255.255.0'>
     <dhcp>
       <range start='10.3.0.1' end='10.3.0.254' />
     </dhcp>
   </ip>
</network>
[root@fedora22wksr ~]# cat external4.xml
<network>
   <name>external4</name>
   <uuid>d0e9964b-f97d-40c2-b749-b609aed52cf2</uuid>
   <forward mode='nat'>
     <nat>
       <port start='1024' end='65535'/>
     </nat>
   </forward>
   <bridge name='virbr7' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6d'/>
   <ip address='10.4.0.1' netmask='255.255.255.0'>
     <dhcp>
       <range start='10.4.0.1' end='10.4.0.254' />
     </dhcp>
   </ip>
</network>

Shutdown VM hosting Network Node and add two VNICs eth3 belongs
external3 , eth4 belongs  external4
Startup VM and create corresponding files ifcfg-eth3,ifcfg-eth4 with static
IP addresses.

# service network restart

or reboot Nerwork Node.

*************************
On Network Node
*************************
# ovs-vsctl add-br br-ex
# ovs-vsctl add-port br-ex eth2
# ovs-vsctl add-br br-eth3
# ovs-vsctl add-port br-eth3 eth3
# ovs-vsctl add-br br-eth4
# ovs-vsctl add-port br-eth4 eth4

******************************
Update l3_agent.ini file
******************************
external_network_bridge =
external_network_id =

***********************************************************************
Update /etc/neutron/plugins/ml2/openvswitch_agent.ini
***********************************************************************
[ovs]
network_vlan_ranges =physnet1, physnet3,physnet4
bridge_mappings = physnet1:br-ex,physnet3:br-eth3,physnet4:br-eth4

Then copy  /etc/neutron/plugins/ml2/openvswitch_agent.ini
to /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini

************************************************************************
SSH to Controller 192.169.142.127 and update ml2_conf.ini
************************************************************************
[ml2]
type_drivers = local,flat,gre,vxlan

[ml2_type_flat]
flat_networks = *

# openstack-service restart on Controller

**********************************************************
Get back to VM hosting Network Node
**********************************************************
# openstack-service restart neutron
# systemctl | grep neutron

[root@ip-192-169-142-147 ~]# systemctl| grep neutron
neutron-dhcp-agent.service                                                          loaded active running   OpenStack Neutron DHCP Agent
neutron-l3-agent.service                                                            loaded active running   OpenStack Neutron Layer 3 Agent
neutron-metadata-agent.service                                                      loaded active running   OpenStack Neutron Metadata Agent
neutron-openvswitch-agent.service                                                   loaded active running   OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service                                                         loaded active exited    OpenStack Neutron Open vSwitch Cleanup Utility

****************************************
External networks creation
****************************************
# source keystonerc_admin
# neutron net-create external3 -- --router:external  \
  --provider:network_type=flat \
  --provider:physical_network=physnet3

# neutron net-create external4 -- --router:external  \
  --provider:network_type=flat \
  --provider:physical_network=physnet4

# neutron subnet-create --disable-dhcp external3 10.3.0.0/24
# neutron subnet-create --disable-dhcp external4 10.4.0.0/24

# neutron net-create public1 --provider:network_type flat \
 --provider:physical_network physnet1 --router:external

# neutron subnet-create public1\
 --gateway 172.24.4.225  172.24.4.224/28 \
 --allocation-pool start=172.24.4.226,end=172.24.4.238 \
 --enable_dhcp=False

*************************************************
Then login as demo and create
*************************************************
RouterExt3 with gateway to external3
RouterExt4 with gateway to external4
RouterDemo with gateway to public1

Then create private networks private1, demo-network4,demo_network5
Attach first to RouterDemo , second to RouterExt4, third to RouterExt3

Notice that qg-xxxxxxx interfaces from all qrouter-namespaces are attached to br-int

While using provider external networks,traffic to/from external network flows through br-int. 
br-int and br-ex will be connected using veth pair int-br-ex and phy-br-ex.
br-int and br-eth3 will be connected using veth pair int-br-eth3 and phy-br-eth3. 
br-int and br-eth4 will be connected using veth pair int-br-eth4 and phy-br-eth4. 
This will be automatically created by neutron-openvswitch-agent based on the bridge_mappings configured earlier.  

[root@ip-192-169-142-147 ~(keystone_admin)]# ovs-vsctl show
38e920e3-da61-4a1b-876a-052a49d777a2
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a000089"
            Interface "vxlan-0a000089"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.0.0.147", out_key=flow, remote_ip="10.0.0.137"}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge "br-eth4"
        Port "br-eth4"
            Interface "br-eth4"
                type: internal
        Port "phy-br-eth4"
            Interface "phy-br-eth4"
                type: patch
                options: {peer="int-br-eth4"}
        Port "eth4"
            Interface "eth4"
    Bridge br-int
        fail_mode: secure
        Port "tap7ce0a427-fd"
            tag: 5
            Interface "tap7ce0a427-fd"
                type: internal
        Port "qr-45110e77-5b"
            tag: 1
            Interface "qr-45110e77-5b"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "qr-a99aa111-1d"
            tag: 3
            Interface "qr-a99aa111-1d"
                type: internal
        Port "qg-615baaa8-a6"
            tag: 6
            Interface "qg-615baaa8-a6"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tap709fbf6f-ab"
            tag: 3
            Interface "tap709fbf6f-ab"
                type: internal
        Port "int-br-eth3"
            Interface "int-br-eth3"
                type: patch
                options: {peer="phy-br-eth3"}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-b7d78d6d-dd"
            tag: 5
            Interface "qr-b7d78d6d-dd"
                type: internal
        Port "int-br-eth4"
            Interface "int-br-eth4"
                type: patch
                options: {peer="phy-br-eth4"}
        Port "qg-c28dfe1c-44"
            tag: 2
            Interface "qg-c28dfe1c-44"
                type: internal
        Port "qg-54aa0373-dd"
            tag: 4
            Interface "qg-54aa0373-dd"
                type: internal
        Port "tap06adaf37-d4"
            tag: 1
            Interface "tap06adaf37-d4"
                type: internal
    Bridge br-ex
        Port "eth2"
            Interface "eth2"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge "br-eth3"
        Port "eth3"
            Interface "eth3"
        Port "phy-br-eth3"
            Interface "phy-br-eth3"
                type: patch
                options: {peer="int-br-eth3"}
        Port "br-eth3"
            Interface "br-eth3"
                type: internal
    ovs_version: "2.3.1"