Friday, February 09, 2018

Attempt of evaluation TripleO QuickStart (Master) overcloud containerized HA deployment on 32 GB VIRTHOST (Second Test)

Following bellow is a sample of tuning deploy-configHA.yaml which allows to obtain
fairly functional overcloud topology with PCS HA Controller's cluster, then create
Networks, HA Router and launch F27 VM. Install on VM links text browser and surf the Net
pretty fast and smoothly. Sizes of PCS Controller's were tuned rather then default size
of undercloud as well as number VCPUS running undercloud VM. 
Initial swap area after deployment is 800 MB
Setup Repos 
[boris@fedora27workstation ~]$git clone \
   https://github.com/openstack/tripleo-quickstart
[boris@fedora27workstation ~]$bash ./tripleo-quickstart/quickstart.sh \
  --install-deps 
Configure Environment
[boris@fedora27workstation ~]$ export CONFIG= ~boris/deploy-configHA.yaml 
[boris@fedora27workstation ~]$ cat  ~boris/deploy-configHA.yaml 
# 32 GB VIRTHOST 
controller_memory: 7500
compute memory: 6500 
 overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230
  - name: control_1
    flavor: control
    virtualbmc_port: 6231
  - name: control_2
    flavor: control
    virtualbmc_port: 6232

  - name: compute_0
    flavor: compute
    virtualbmc_port: 6233
node_count: 4
containerized_overcloud: true
delete_docker_cache: true
enable_pacemaker: true
run_tempest: false
extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org
  --control-scale 3
  --compute-scale 1
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml
[boris@fedora27workstation ~]$ export CONFIG=~boris/deploy-configHA.yaml
[boris@fedora27workstation ~]$ bash ./tripleo-quickstart/quickstart.sh  \
                        --clean \
                        --release master \
                        --teardown all \
                        --tags all \
                        -e @$CONFIG \
                         $VIRTHOST 
 . . . . . . . 
Friday 09 February 2018  12:30:38 +0300 (0:00:01.772)       3:23:18.694 ******* 
=============================================================================== 
overcloud-deploy : Deploy the overcloud ----------------------------------------------- 4339.72s
/home/boris/.quickstart/usr/local/share/ansible/roles/overcloud-deploy/tasks/deploy-overcloud.yml:1 
overcloud-prep-containers : Prepare for the containerized deployment ------------------ 3348.90s
/home/boris/.quickstart/usr/local/share/ansible/roles/overcloud-prep-containers/tasks/overcloud-prep-containers.yml:28 
undercloud-deploy : Install the undercloud -------------------------------------------- 1565.50s
/home/boris/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/install-undercloud.yml:20 
modify-image : Run virt-customize on the provided image -------------------------------- 490.30s
/home/boris/.quickstart/usr/local/share/ansible/roles/modify-image/tasks/libguestfs.yml:46 -----
tripleo-validations : Run validations tests through Mistral ---------------------------- 396.76s
/home/boris/.quickstart/usr/local/share/ansible/roles/tripleo-validations/tasks/main.yml:21 ----
overcloud-prep-images : Prepare the overcloud images for deploy ------------------------ 288.70s
/home/boris/.quickstart/usr/local/share/ansible/roles/overcloud-prep-images/tasks/overcloud-prep-images.yml:1 
tripleo-validations : Run validations tests through Mistral ---------------------------- 211.02s
/home/boris/.quickstart/usr/local/share/ansible/roles/tripleo-validations/tasks/main.yml:21 ----
tripleo-validations : Run validations tests through Mistral ---------------------------- 150.33s
/home/boris/.quickstart/usr/local/share/ansible/roles/tripleo-validations/tasks/main.yml:21 ----
convert-image : convert image ---------------------------------------------------------- 141.78s
/home/boris/.quickstart/tripleo-quickstart/roles/convert-image/tasks/main.yml:25 ---------------
setup/undercloud : Upload undercloud volume to storage pool ----------------------------- 98.55s
/home/boris/.quickstart/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:280 ---
setup/undercloud : Perform selinux relabel on undercloud image -------------------------- 92.20s
/home/boris/.quickstart/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:162 ---
setup/undercloud : Inject additional images --------------------------------------------- 49.99s
/home/boris/.quickstart/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:67 ----
overcloud-prep-flavors : Prepare the scripts for overcloud flavors ---------------------- 44.48s
/home/boris/.quickstart/usr/local/share/ansible/roles/overcloud-prep-flavors/tasks/overcloud-prep-flavors.yml:1 
tripleo-validations : Run negative tests for pre-introspection group -------------------- 36.70s
/home/boris/.quickstart/usr/local/share/ansible/roles/tripleo-validations/tasks/main.yml:55 ----
validate-tempest : Install openstack services tempest plugins --------------------------- 31.58s
/home/boris/.quickstart/usr/local/share/ansible/roles/validate-tempest/tasks/pre-tempest.yml:32 
undercloud-deploy : Start the Virtual BMCs ---------------------------------------------- 27.20s
/home/boris/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/configure-vbmc.yml:107 
setup/undercloud : iptables ------------------------------------------------------------- 20.20s
/home/boris/.quickstart/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:387 ---
convert-image : Resize the undercloud image using qemu-image resize --------------------- 19.27s
/home/boris/.quickstart/tripleo-quickstart/roles/convert-image/tasks/main.yml:19 ---------------
fetch-images : Get tar images from cache ------------------------------------------------ 19.22s
/home/boris/.quickstart/tripleo-quickstart/roles/fetch-images/tasks/fetch.yml:186 --------------
undercloud-deploy : Create the Virtual BMCs --------------------------------------------- 14.47s
/home/boris/.quickstart/usr/local/share/ansible/roles/undercloud-deploy/tasks/configure-vbmc.yml:75 
+ set +x
##################################
Virtual Environment Setup Complete
##################################

Access the undercloud by:

    ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

Follow the documentation in the link below to complete your deployment.

    http://ow.ly/c44w304begR

##################################
Virtual Environment Setup Complete
##################################
[boris@fedora27workstation ~]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Fri Feb  9 09:29:56 2018 from gateway
[stack@undercloud ~]$ . stackrc
(undercloud) [stack@undercloud ~]$ cat overcloudrc
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.0.0.5,192.168.24.6
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://10.0.0.5:13000//v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
export OS_PASSWORD=bM6Yy84qEAdHjXvWu7fcXHnMU
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi
(undercloud) [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 6bd06df3-c2e9-46b7-b212-2d7472a3db10 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
| 581cb5be-7ee0-4f8c-be59-603f65fa60bf | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.14 |
| 73151c8f-6678-48bc-8d04-e1ed97007411 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.13 |
| 7eb8f0e5-a743-42a5-83be-83615d5d3f03 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.9  |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
(undercloud) [stack@undercloud ~]$ ssh heat-admin@192.168.24.16
The authenticity of host '192.168.24.16 (192.168.24.16)' can't be established.
ECDSA key fingerprint is SHA256:w3fm1lLcwNKHDAo2/tad+mliZe8aJQPUZT5734JOhCo.
ECDSA key fingerprint is MD5:71:3e:23:32:d6:80:36:02:4a:81:ea:05:09:22:50:87.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.16' (ECDSA) to the list of known hosts.
Last login: Fri Feb  9 09:29:14 2018 from 192.168.24.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.16-12.el7_4.7-94ff4df) - partition with quorum
Last updated: Fri Feb  9 09:33:12 2018
Last change: Fri Feb  9 09:20:26 2018 by root via cibadmin on overcloud-controller-0

12 nodes configured
37 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.24.1:8787/master/centos-binary-rabbitmq:pcmklatest]
   rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0
   rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1
   rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2
 Docker container set: galera-bundle [192.168.24.1:8787/master/centos-binary-mariadb:pcmklatest]
   galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-controller-0
   galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-controller-1
   galera-bundle-2 (ocf::heartbeat:galera): Master overcloud-controller-2
 Docker container set: redis-bundle [192.168.24.1:8787/master/centos-binary-redis:pcmklatest]
   redis-bundle-0 (ocf::heartbeat:redis): Master overcloud-controller-0
   redis-bundle-1 (ocf::heartbeat:redis): Slave overcloud-controller-1
   redis-bundle-2 (ocf::heartbeat:redis): Slave overcloud-controller-2
 ip-192.168.24.6 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 ip-10.0.0.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 ip-172.16.2.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 ip-172.16.2.7 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 ip-172.16.1.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 ip-172.16.3.7 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 Docker container set: haproxy-bundle [192.168.24.1:8787/master/centos-binary-haproxy:pcmklatest]
   haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0
   haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1
   haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2
 Docker container: openstack-cinder-volume [192.168.24.1:8787/master/centos-binary-cinder-volume:pcmklatest]
   openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 ~]# vi overcloudrc
[root@overcloud-controller-0 ~]# . overcloudrc
(overcloud) [root@overcloud-controller-0 ~]# nova service-list
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id                                   | Binary           | Host                                | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 971cc1ce-1d70-4950-8f53-81d2c5c06138 | nova-scheduler   | overcloud-controller-1.localdomain  | internal | enabled | up    | 2018-02-09T09:34:41.000000 | -               | False       |
| c22939e2-caca-48d2-bdbf-b3dfb761a207 | nova-scheduler   | overcloud-controller-0.localdomain  | internal | enabled | up    | 2018-02-09T09:34:42.000000 | -               | False       |
| 0cbc89d4-8ae6-47bb-90bf-ccbf40008a9a | nova-scheduler   | overcloud-controller-2.localdomain  | internal | enabled | up    | 2018-02-09T09:34:38.000000 | -               | False       |
| 0439fbf2-dbbd-45f7-8aa3-3fe4a6bf3549 | nova-consoleauth | overcloud-controller-1.localdomain  | internal | enabled | up    | 2018-02-09T09:34:36.000000 | -               | False       |
| 3c6af83d-999d-47bb-8c75-d40c6206c7df | nova-consoleauth | overcloud-controller-0.localdomain  | internal | enabled | up    | 2018-02-09T09:34:35.000000 | -               | False       |
| b60e0945-9daa-4476-8c92-82e385f1376e | nova-consoleauth | overcloud-controller-2.localdomain  | internal | enabled | up    | 2018-02-09T09:34:41.000000 | -               | False       |
| 0d17978f-70f9-4180-ad75-e4b7fa3b46e9 | nova-conductor   | overcloud-controller-1.localdomain  | internal | enabled | up    | 2018-02-09T09:34:35.000000 | -               | False       |
| 76a526cd-d127-426a-958f-bc6e6bd85553 | nova-compute     | overcloud-novacompute-0.localdomain | nova     | enabled | up    | 2018-02-09T09:34:39.000000 | -               | False       |
| 4ef0ead0-77c8-4a3e-837b-ad136cbc2bb9 | nova-conductor   | overcloud-controller-0.localdomain  | internal | enabled | up    | 2018-02-09T09:34:35.000000 | -               | False       |
| aa8b0547-dccb-4460-a5da-a101c0d58ebb | nova-conductor   | overcloud-controller-2.localdomain  | internal | enabled | up    | 2018-02-09T09:34:40.000000 | -               | False       |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(overcloud) [root@overcloud-controller-0 ~]# neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                                | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| 143ea92b-3e32-436b-9d59-87934bbac856 | DHCP agent         | overcloud-controller-1.localdomain  | nova              | :-)   | True           | neutron-dhcp-agent        |
| 19b15be3-26bf-44fd-b3f7-fa1bc1ea81ff | Open vSwitch agent | overcloud-controller-0.localdomain  |                   | :-)   | True           | neutron-openvswitch-agent |
| 39a919d1-2906-4681-ad82-786a7dde6be0 | Metadata agent     | overcloud-controller-1.localdomain  |                   | :-)   | True           | neutron-metadata-agent    |
| 3ae45bcf-ba22-4a3f-a72c-e93b03130084 | Metadata agent     | overcloud-controller-0.localdomain  |                   | :-)   | True           | neutron-metadata-agent    |
| 46ad2f5f-d557-4f2a-bbdd-e2326b267371 | L3 agent           | overcloud-controller-0.localdomain  | nova              | :-)   | True           | neutron-l3-agent          |
| 5fd9b8ab-a015-4516-ad13-626765c0ea19 | Open vSwitch agent | overcloud-novacompute-0.localdomain |                   | :-)   | True           | neutron-openvswitch-agent |
| 64ee300e-af31-4c9a-ae94-a354be509145 | Metadata agent     | overcloud-controller-2.localdomain  |                   | :-)   | True           | neutron-metadata-agent    |
| 707a31b0-65f2-4a05-86c8-640bb5925329 | DHCP agent         | overcloud-controller-0.localdomain  | nova              | :-)   | True           | neutron-dhcp-agent        |
| 7d979f97-3419-4617-96a6-3713bbc00557 | Open vSwitch agent | overcloud-controller-2.localdomain  |                   | :-)   | True           | neutron-openvswitch-agent |
| 8493c8e2-ddc9-49e3-b4bd-c34e4c153e03 | Open vSwitch agent | overcloud-controller-1.localdomain  |                   | :-)   | True           | neutron-openvswitch-agent |
| aa48c36e-d6a2-4980-93a8-5a8e6229392b | L3 agent           | overcloud-controller-2.localdomain  | nova              | :-)   | True           | neutron-l3-agent          |
| b5ec07fc-b8b9-4618-bcb9-7cb829c34606 | L3 agent           | overcloud-controller-1.localdomain  | nova              | :-)   | True           | neutron-l3-agent          |
| de863a15-9f8c-4bee-97cf-69e454646066 | DHCP agent         | overcloud-controller-2.localdomain  | nova              | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
(overcloud) [root@overcloud-controller-0 ~]# docker ps
CONTAINER ID        IMAGE                                                                                  COMMAND                  CREATED             STATUS                      PORTS               NAMES
5a1c45f0304c        192.168.24.1:8787/master/centos-binary-gnocchi-statsd:current-tripleo-rdo              "kolla_start"            14 minutes ago      Up 14 minutes                                   gnocchi_statsd
360975b254a5        192.168.24.1:8787/master/centos-binary-gnocchi-api:current-tripleo-rdo                 "kolla_start"            14 minutes ago      Up 14 minutes                                   gnocchi_api
50caf9b1c5a1        192.168.24.1:8787/master/centos-binary-cinder-volume:pcmklatest                        "/bin/bash /usr/local"   14 minutes ago      Up 14 minutes                                   openstack-cinder-volume-docker-0
efaeadc581d8        192.168.24.1:8787/master/centos-binary-gnocchi-metricd:current-tripleo-rdo             "kolla_start"            16 minutes ago      Up 16 minutes                                   gnocchi_metricd
f21e683f899d        192.168.24.1:8787/master/centos-binary-neutron-openvswitch-agent:current-tripleo-rdo   "kolla_start"            19 minutes ago      Up 19 minutes (healthy)                         neutron_ovs_agent
7f028960e827        192.168.24.1:8787/master/centos-binary-neutron-l3-agent:current-tripleo-rdo            "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         neutron_l3_agent
14465000cebe        192.168.24.1:8787/master/centos-binary-neutron-metadata-agent:current-tripleo-rdo      "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         neutron_metadata_agent
aa1e3dde0344        192.168.24.1:8787/master/centos-binary-neutron-dhcp-agent:current-tripleo-rdo          "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         neutron_dhcp
1560cc6facbf        192.168.24.1:8787/master/centos-binary-glance-api:current-tripleo-rdo                  "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         glance_api
1e41468ae39d        192.168.24.1:8787/master/centos-binary-panko-api:current-tripleo-rdo                   "kolla_start"            20 minutes ago      Up 20 minutes                                   panko_api
33004ee80524        192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo                    "kolla_start"            20 minutes ago      Up 20 minutes                                   nova_metadata
2916e53225cf        192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo                    "kolla_start"            20 minutes ago      Up 20 minutes (unhealthy)                       nova_api
d01c3adcbcc0        192.168.24.1:8787/master/centos-binary-cron:current-tripleo-rdo                        "kolla_start"            20 minutes ago      Up 20 minutes                                   logrotate_crond
12500a9a6310        192.168.24.1:8787/master/centos-binary-heat-api-cfn:current-tripleo-rdo                "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         heat_api_cfn
d12983cbb630        192.168.24.1:8787/master/centos-binary-neutron-server:current-tripleo-rdo              "kolla_start"            20 minutes ago      Up 20 minutes                                   neutron_api
29a6eacae287        192.168.24.1:8787/master/centos-binary-aodh-listener:current-tripleo-rdo               "kolla_start"            20 minutes ago      Up 20 minutes (healthy)                         aodh_listener
6dd790d2baa1        192.168.24.1:8787/master/centos-binary-swift-container:current-tripleo-rdo             "kolla_start"            20 minutes ago      Up 20 minutes                                   swift_container_auditor
349ebafec130        192.168.24.1:8787/master/centos-binary-heat-api:current-tripleo-rdo                    "kolla_start"            20 minutes ago      Up 20 minutes                                   heat_api_cron
70135a559934        192.168.24.1:8787/master/centos-binary-swift-proxy-server:current-tripleo-rdo          "kolla_start"            20 minutes ago      Up 20 minutes                                   swift_object_expirer
23f575566615        192.168.24.1:8787/master/centos-binary-swift-object:current-tripleo-rdo                "kolla_start"            20 minutes ago      Up 20 minutes                                   swift_object_updater
3b464c18f79d        192.168.24.1:8787/master/centos-binary-swift-container:current-tripleo-rdo             "kolla_start"            20 minutes ago      Up 20 minutes                                   swift_container_replicator
e7fbf14023bf        192.168.24.1:8787/master/centos-binary-swift-account:current-tripleo-rdo               "kolla_start"            20 minutes ago      Up 20 minutes                                   swift_account_auditor
73e12b0ab2b8        192.168.24.1:8787/master/centos-binary-cinder-api:current-tripleo-rdo                  "kolla_start"            21 minutes ago      Up 20 minutes                                   cinder_api_cron
9e7c13b4fb35        192.168.24.1:8787/master/centos-binary-swift-account:current-tripleo-rdo               "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         swift_account_server
d79177eb61db        192.168.24.1:8787/master/centos-binary-nova-conductor:current-tripleo-rdo              "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         nova_conductor
1b4eb184e953        192.168.24.1:8787/master/centos-binary-cinder-scheduler:current-tripleo-rdo            "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         cinder_scheduler
52f38b6914e4        192.168.24.1:8787/master/centos-binary-swift-object:current-tripleo-rdo                "kolla_start"            21 minutes ago      Up 21 minutes                                   swift_object_replicator
0d6e29855cdf        192.168.24.1:8787/master/centos-binary-swift-container:current-tripleo-rdo             "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         swift_container_server
b8bfbc8b8fce        192.168.24.1:8787/master/centos-binary-heat-engine:current-tripleo-rdo                 "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         heat_engine
2846456f2e24        192.168.24.1:8787/master/centos-binary-aodh-api:current-tripleo-rdo                    "kolla_start"            21 minutes ago      Up 21 minutes                                   aodh_api
4900e4d63dfd        192.168.24.1:8787/master/centos-binary-swift-object:current-tripleo-rdo                "kolla_start"            21 minutes ago      Up 21 minutes                                   swift_rsync
79326d6c58e8        192.168.24.1:8787/master/centos-binary-nova-novncproxy:current-tripleo-rdo             "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         nova_vnc_proxy
e5fd37a26f98        192.168.24.1:8787/master/centos-binary-ceilometer-notification:current-tripleo-rdo     "kolla_start"            21 minutes ago      Up 21 minutes (healthy)                         ceilometer_agent_notification
81547ad2b6cd        192.168.24.1:8787/master/centos-binary-swift-account:current-tripleo-rdo               "kolla_start"            21 minutes ago      Up 21 minutes                                   swift_account_reaper
77f1f1cbfe85        192.168.24.1:8787/master/centos-binary-nova-consoleauth:current-tripleo-rdo            "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         nova_consoleauth
d8a62ab1d810        192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo                    "kolla_start"            22 minutes ago      Up 22 minutes                                   nova_api_cron
8c3b3a3f6897        192.168.24.1:8787/master/centos-binary-aodh-notifier:current-tripleo-rdo               "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         aodh_notifier
eeff860cc81f        192.168.24.1:8787/master/centos-binary-ceilometer-central:current-tripleo-rdo          "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         ceilometer_agent_central
5a4ba18c6b1a        192.168.24.1:8787/master/centos-binary-swift-account:current-tripleo-rdo               "kolla_start"            22 minutes ago      Up 22 minutes                                   swift_account_replicator
db51288026b9        192.168.24.1:8787/master/centos-binary-swift-object:current-tripleo-rdo                "kolla_start"            22 minutes ago      Up 22 minutes                                   swift_object_auditor
c8a587bab30a        192.168.24.1:8787/master/centos-binary-heat-api:current-tripleo-rdo                    "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         heat_api
91fa1ae1977c        192.168.24.1:8787/master/centos-binary-swift-proxy-server:current-tripleo-rdo          "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         swift_proxy
d40422d89e16        192.168.24.1:8787/master/centos-binary-cinder-api:current-tripleo-rdo                  "kolla_start"            22 minutes ago      Up 22 minutes                                   cinder_api
fdf830388534        192.168.24.1:8787/master/centos-binary-swift-object:current-tripleo-rdo                "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         swift_object_server
d3eaf313b9c7        192.168.24.1:8787/master/centos-binary-nova-scheduler:current-tripleo-rdo              "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         nova_scheduler
0e60007aa22c        192.168.24.1:8787/master/centos-binary-aodh-evaluator:current-tripleo-rdo              "kolla_start"            22 minutes ago      Up 22 minutes (healthy)                         aodh_evaluator
774c17fe3d5d        192.168.24.1:8787/master/centos-binary-swift-container:current-tripleo-rdo             "kolla_start"            22 minutes ago      Up 22 minutes                                   swift_container_updater
21ac20c76fce        192.168.24.1:8787/master/centos-binary-keystone:current-tripleo-rdo                    "/bin/bash -c '/usr/l"   27 minutes ago      Up 27 minutes                                   keystone_cron
a0c4b0ca68d3        192.168.24.1:8787/master/centos-binary-keystone:current-tripleo-rdo                    "kolla_start"            28 minutes ago      Up 28 minutes (healthy)                         keystone
dec5a64ac7cd        192.168.24.1:8787/master/centos-binary-iscsid:current-tripleo-rdo                      "kolla_start"            28 minutes ago      Up 28 minutes                                   iscsid
6398cb621da9        192.168.24.1:8787/master/centos-binary-nova-placement-api:current-tripleo-rdo          "kolla_start"            28 minutes ago      Up 28 minutes                                   nova_placement
03da3bfdfa10        192.168.24.1:8787/master/centos-binary-horizon:current-tripleo-rdo                     "kolla_start"            28 minutes ago      Up 28 minutes                                   horizon
9fdf6b1c9515        192.168.24.1:8787/master/centos-binary-haproxy:pcmklatest                              "/bin/bash /usr/local"   31 minutes ago      Up 31 minutes                                   haproxy-bundle-docker-0
d7cf18e15f66        192.168.24.1:8787/master/centos-binary-redis:pcmklatest                                "/bin/bash /usr/local"   33 minutes ago      Up 33 minutes                                   redis-bundle-docker-0
e39d1e54368e        192.168.24.1:8787/master/centos-binary-mariadb:current-tripleo-rdo                     "kolla_start"            34 minutes ago      Up 34 minutes                                   clustercheck
313125f4753a        192.168.24.1:8787/master/centos-binary-mariadb:pcmklatest                              "/bin/bash /usr/local"   35 minutes ago      Up 35 minutes                                   galera-bundle-docker-0
2af3ccc8e168        192.168.24.1:8787/master/centos-binary-rabbitmq:pcmklatest                             "/bin/bash /usr/local"   37 minutes ago      Up 37 minutes                                   rabbitmq-bundle-docker-0
94dd3d7fe881        192.168.24.1:8787/master/centos-binary-memcached:current-tripleo-rdo                   "/bin/bash -c 'source"   40 minutes ago      Up 40 minutes                                   memcached

(overcloud) [root@overcloud-controller-0 ~]# ovs-vsctl show
779bc1af-d1d6-4a74-91de-7be0503d744c
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-ex
            Interface br-ex
                type: internal
        Port "vlan40"
            tag: 40
            Interface "vlan40"
                type: internal
        Port "vlan20"
            tag: 20
            Interface "vlan20"
                type: internal
        Port "vlan30"
            tag: 30
            Interface "vlan30"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "eth0"
            Interface "eth0"
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                type: internal
        Port "vlan50"
            tag: 50
            Interface "vlan50"
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-ac100005"
            Interface "vxlan-ac100005"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.0.10", out_key=flow, remote_ip="172.16.0.5"}
        Port "vxlan-ac10000c"
            Interface "vxlan-ac10000c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.0.10", out_key=flow, remote_ip="172.16.0.12"}
        Port "vxlan-ac100008"
            Interface "vxlan-ac100008"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.0.10", out_key=flow, remote_ip="172.16.0.8"}
    ovs_version: "2.8.1"

  
  VIRTHOST INITIAL STATUS