Sunday, August 21, 2016

Emulation Triple0 QuickStart HA Controller's Cluster failover

Procedure bellow identify Controller which has RouterDSA in active state and
shutdown/startup  this Controller ( controller-1 in particular case).
Then  log into conntroller-1 and restart pcs cluster on particular  Controller,
afterwards  runs `pacemaker resource cleanup` for several resources what
results bringing back cluster nodes in proper status


 
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id                          | host                        | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f-         | overcloud-                  | True           | :-)   | active   |
| 85f2-9bb9835cf360           | controller-1.localdomain    |           |       |              |
| ae0f67ce-732b-                  | overcloud-                  | True           | :-)   | standby  |
| 4cb2-9b52-d15c22211972      | controller-0.localdomain    |          |       |          |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud-         | True           | :-)   | standby  |
| fd1c3489b                      | controller-2.localdomain    |            |       |                |
+-----------------------------+-----------------------------+----------------+-------+----------+
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova stop overcloud-controller-1
Request to stop server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                    | Status  | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE  | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE  | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE  | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE  | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova start overcloud-controller-1
Request to start server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
The authenticity of host '192.0.2.14 (192.0.2.14)' can't be established.
ECDSA key fingerprint is a3:e6:de:2e:2b:45:e4:33:3d:d0:75:e5:b7:7f:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.14' (ECDSA) to the list of known hosts.

[heat-admin@overcloud-controller-1 ~]$ sudo su -
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:12:39 2016        Last change: Sun Aug 21 13:24:42 2016 by root via cibadmin on overcloud-controller-1
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.12    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-2 ]
     Slaves: [ overcloud-controller-1 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Stopped
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 ]
     Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]

Failed Actions:
* rabbitmq_monitor_10000 on overcloud-controller-0 'not running' (7): call=81, status=complete, exitreason='none',
    last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
* rabbitmq_monitor_10000 on overcloud-controller-2 'not running' (7): call=79, status=complete, exitreason='none',
    last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-1 ~]# pcs cluster stop
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
[root@overcloud-controller-1 ~]# pcs cluster start
Starting Cluster...
[root@overcloud-controller-1 ~]#
Broadcast message from systemd-journald@overcloud-controller-1.localdomain (Sun 2016-08-21 15:16:07 UTC):

haproxy[16997]: proxy nova_ec2 has no server available!

======================================
Script start.sh  [ 1 ]
======================================
#!/bin/bash -x
pcs resource cleanup rabbitmq-clone ;
sleep 10
pcs resource cleanup neutron-server-clone ;
sleep 10
pcs resource cleanup openstack-nova-api-clone ;
sleep 10
pcs resource cleanup openstack-nova-consoleauth-clone ;
sleep 10
pcs resource cleanup openstack-heat-engine-clone ;
sleep 10
pcs resource cleanup openstack-cinder-api-clone ;
sleep 10
pcs resource cleanup openstack-glance-registry-clone ;
sleep 10
pcs resource cleanup httpd-clone
=======================================


[root@overcloud-controller-1 ~]# . ./start.sh
Waiting for 3 replies from the CRMd... OK
Cleaning up rabbitmq:0 on overcloud-controller-0, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-1, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-2, removing fail-count-rabbitmq

Waiting for 3 replies from the CRMd... OK
Cleaning up neutron-server:0 on overcloud-controller-0, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-1, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-2, removing fail-count-neutron-server

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-api:0 on overcloud-controller-0, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-1, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-2, removing fail-count-openstack-nova-api

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-0, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-1, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-2, removing fail-count-openstack-nova-consoleauth

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-heat-engine:0 on overcloud-controller-0, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-1, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-2, removing fail-count-openstack-heat-engine

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-cinder-api:0 on overcloud-controller-0, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-1, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-2, removing fail-count-openstack-cinder-api

Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-glance-registry:0 on overcloud-controller-0, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-1, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-2, removing fail-count-openstack-glance-registry

Waiting for 3 replies from the CRMd... OK
Cleaning up httpd:0 on overcloud-controller-0, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-1, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-2, removing fail-count-httpd

[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:18:04 2016        Last change: Sun Aug 21 15:17:57 2016 by hacluster via crmd on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.12    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-1 ~]# logout
[heat-admin@overcloud-controller-1 ~]$ logout
Connection to 192.0.2.14 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.16
Last login: Sun Aug 21 15:08:18 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sun Aug 21 15:08:24 UTC 2016 on pts/0
[root@overcloud-controller-0 ~]# .  keysstonerc_admin
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id                          | host                        | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f-         | overcloud-                  | True      | :-)   | standby  |
| 85f2-9bb9835cf360           | controller-1.localdomain    |      |       |                |
| ae0f67ce-732b-              | overcloud-                  | True           | :-)  | standby  |
| 4cb2-9b52-d15c22211972      | controller-0.localdomain    |          |       |       |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud-                  | True   :-)   | active   |
| fd1c3489b                   | controller-2.localdomain        |                |     |          |
+-----------------------------+-----------------------------+----------------+-------+----------+

====================================
Verification Galera DB sync on Controllers
====================================

[root@overcloud-controller-0 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.

[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.

[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
Last login: Sun Aug 21 15:12:27 2016 from 192.0.2.1
[heat-admin@overcloud-controller-1 ~]$ sudo su -
Last login: Sun Aug 21 15:12:34 UTC 2016 on pts/0

[root@overcloud-controller-1 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.


 ==================
 Setup details
==================

[boris@fedora24wks tripleo-quickstart]$ cat ./config/general_config/ha.yml
# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6144
compute_memory: 6144
default_vcpu: 2

undercloud_memory: 8192

# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 2

# Create three controller nodes and one compute node.
overcloud_nodes:
  - name: control_0
    flavor: control
  - name: control_1
    flavor: control
  - name: control_2
    flavor: control

  - name: compute_0
    flavor: compute
  - name: compute_1
    flavor: compute

# We don't need introspection in a virtual environment (because we are
# creating all the "hardware" we really know the necessary
# information).
step_introspect: true

# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
  --control-scale 3 --compute-scale 2 --neutron-network-type vxlan
  --neutron-tunnel-types vxlan
  --ntp-server pool.ntp.org
test_tempest: false
test_ping: true
enable_pacemaker: true

##################################
Virtual Environment Setup Complete
##################################

Access the undercloud by:

    ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

There are scripts in the home directory to continue the deploy:

    overcloud-deploy.sh will deploy the overcloud
    overcloud-deploy-post.sh will do any post-deploy configuration
    overcloud-validate.sh will run post-deploy validation

Alternatively, you can ignore these scripts and follow the upstream docs,
starting from the overcloud deploy section:

    http://ow.ly/1Vc1301iBlb

##################################
Virtual Environment Setup Complete
##################################

[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.1.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Wed Aug 24 12:13:16 2016 from gateway
[stack@undercloud ~]$ sudo su


[root@undercloud stack]# cd /etc/yum.repos.d
[root@undercloud yum.repos.d]# ls -l
total 40
-rw-r--r--. 1 root root 1664 Dec  9  2015 CentOS-Base.repo
-rw-r--r--. 1 root root 1057 Aug 24 02:58 CentOS-Ceph-Hammer.repo
-rw-r--r--. 1 root root 1309 Dec  9  2015 CentOS-CR.repo
-rw-r--r--. 1 root root  649 Dec  9  2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  290 Dec  9  2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root  630 Dec  9  2015 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Dec  9  2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec  9  2015 CentOS-Vault.repo
-rw-r--r--. 1 root root  162 Aug 24 02:58 delorean-deps.repo
-rw-r--r--. 1 root root  220 Aug 24 02:58 delorean.repo


====================================================
Delorean repos file been installed via quickstart on undercloud
====================================================
[root@undercloud yum.repos.d]# cat delorean-deps.repo
[delorean-mitaka-testing]
name=dlrn-mitaka-testing
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-mitaka/
enabled=1
gpgcheck=0
priority=2

[root@undercloud yum.repos.d]# cat delorean.repo
[delorean]
name=delorean-openstack-rally-3909299306233247d547bad265a1adb78adfb3d4
baseurl=http://trunk.rdoproject.org/centos7-mitaka/39/09/3909299306233247d547bad265a1adb78adfb3d4_4e6dfa3c
enabled=1
gpgcheck=0