Procedure bellow identify Controller which has RouterDSA in active state and
shutdown/startup this Controller ( controller-1 in particular case).
Then log into conntroller-1 and restart pcs cluster on particular Controller,
afterwards runs `pacemaker resource cleanup` for several resources what
results bringing back cluster nodes in proper status
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f- | overcloud- | True | :-) | active |
| 85f2-9bb9835cf360 | controller-1.localdomain | | | |
| ae0f67ce-732b- | overcloud- | True | :-) | standby |
| 4cb2-9b52-d15c22211972 | controller-0.localdomain | | | |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud- | True | :-) | standby |
| fd1c3489b | controller-2.localdomain | | | |
+-----------------------------+-----------------------------+----------------+-------+----------+
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova stop overcloud-controller-1
Request to stop server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | SHUTOFF | - | Shutdown | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova start overcloud-controller-1
Request to start server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
The authenticity of host '192.0.2.14 (192.0.2.14)' can't be established.
ECDSA key fingerprint is a3:e6:de:2e:2b:45:e4:33:3d:d0:75:e5:b7:7f:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.14' (ECDSA) to the list of known hosts.
[heat-admin@overcloud-controller-1 ~]$ sudo su -
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:12:39 2016 Last change: Sun Aug 21 13:24:42 2016 by root via cibadmin on overcloud-controller-1
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured
Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-192.0.2.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
ip-172.16.3.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-2 ]
Slaves: [ overcloud-controller-1 ]
Clone Set: memcached-clone [memcached]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-10.0.0.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.1.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-core-clone [openstack-core]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-0 ]
Slaves: [ overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: mongod-clone [mongod]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped
Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-clone [openstack-heat-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
Started: [ overcloud-controller-0 ]
Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: delay-clone [delay]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-server-clone [neutron-server]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Failed Actions:
* rabbitmq_monitor_10000 on overcloud-controller-0 'not running' (7): call=81, status=complete, exitreason='none',
last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
* rabbitmq_monitor_10000 on overcloud-controller-2 'not running' (7): call=79, status=complete, exitreason='none',
last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@overcloud-controller-1 ~]# pcs cluster stop
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
[root@overcloud-controller-1 ~]# pcs cluster start
Starting Cluster...
[root@overcloud-controller-1 ~]#
Broadcast message from systemd-journald@overcloud-controller-1.localdomain (Sun 2016-08-21 15:16:07 UTC):
haproxy[16997]: proxy nova_ec2 has no server available!
======================================
Script start.sh [ 1 ]
======================================
#!/bin/bash -x
pcs resource cleanup rabbitmq-clone ;
sleep 10
pcs resource cleanup neutron-server-clone ;
sleep 10
pcs resource cleanup openstack-nova-api-clone ;
sleep 10
pcs resource cleanup openstack-nova-consoleauth-clone ;
sleep 10
pcs resource cleanup openstack-heat-engine-clone ;
sleep 10
pcs resource cleanup openstack-cinder-api-clone ;
sleep 10
pcs resource cleanup openstack-glance-registry-clone ;
sleep 10
pcs resource cleanup httpd-clone
=======================================
[root@overcloud-controller-1 ~]# . ./start.sh
Waiting for 3 replies from the CRMd... OK
Cleaning up rabbitmq:0 on overcloud-controller-0, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-1, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-2, removing fail-count-rabbitmq
Waiting for 3 replies from the CRMd... OK
Cleaning up neutron-server:0 on overcloud-controller-0, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-1, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-2, removing fail-count-neutron-server
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-api:0 on overcloud-controller-0, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-1, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-2, removing fail-count-openstack-nova-api
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-0, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-1, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-2, removing fail-count-openstack-nova-consoleauth
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-heat-engine:0 on overcloud-controller-0, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-1, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-2, removing fail-count-openstack-heat-engine
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-cinder-api:0 on overcloud-controller-0, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-1, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-2, removing fail-count-openstack-cinder-api
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-glance-registry:0 on overcloud-controller-0, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-1, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-2, removing fail-count-openstack-glance-registry
Waiting for 3 replies from the CRMd... OK
Cleaning up httpd:0 on overcloud-controller-0, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-1, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-2, removing fail-count-httpd
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:18:04 2016 Last change: Sun Aug 21 15:17:57 2016 by hacluster via crmd on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured
Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-192.0.2.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
ip-172.16.3.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: memcached-clone [memcached]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-10.0.0.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.1.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-core-clone [openstack-core]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-0 ]
Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
Clone Set: mongod-clone [mongod]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0
Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-clone [openstack-heat-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: delay-clone [delay]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-server-clone [neutron-server]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@overcloud-controller-1 ~]# logout
[heat-admin@overcloud-controller-1 ~]$ logout
Connection to 192.0.2.14 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.16
Last login: Sun Aug 21 15:08:18 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sun Aug 21 15:08:24 UTC 2016 on pts/0
[root@overcloud-controller-0 ~]# . keysstonerc_admin
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f- | overcloud- | True | :-) | standby |
| 85f2-9bb9835cf360 | controller-1.localdomain | | | |
| ae0f67ce-732b- | overcloud- | True | :-) | standby |
| 4cb2-9b52-d15c22211972 | controller-0.localdomain | | | |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud- | True :-) | active |
| fd1c3489b | controller-2.localdomain | | | |
+-----------------------------+-----------------------------+----------------+-------+----------+
====================================
Verification Galera DB sync on Controllers
====================================
[root@overcloud-controller-0 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
Last login: Sun Aug 21 15:12:27 2016 from 192.0.2.1
[heat-admin@overcloud-controller-1 ~]$ sudo su -
Last login: Sun Aug 21 15:12:34 UTC 2016 on pts/0
[root@overcloud-controller-1 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
==================
Setup details
==================
[boris@fedora24wks tripleo-quickstart]$ cat ./config/general_config/ha.yml
# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6144
compute_memory: 6144
default_vcpu: 2
undercloud_memory: 8192
# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 2
# Create three controller nodes and one compute node.
overcloud_nodes:
- name: control_0
flavor: control
- name: control_1
flavor: control
- name: control_2
flavor: control
- name: compute_0
flavor: compute
- name: compute_1
flavor: compute
# We don't need introspection in a virtual environment (because we are
# creating all the "hardware" we really know the necessary
# information).
step_introspect: true
# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
--control-scale 3 --compute-scale 2 --neutron-network-type vxlan
--neutron-tunnel-types vxlan
--ntp-server pool.ntp.org
test_tempest: false
test_ping: true
enable_pacemaker: true
##################################
Virtual Environment Setup Complete
##################################
Access the undercloud by:
ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
There are scripts in the home directory to continue the deploy:
overcloud-deploy.sh will deploy the overcloud
overcloud-deploy-post.sh will do any post-deploy configuration
overcloud-validate.sh will run post-deploy validation
Alternatively, you can ignore these scripts and follow the upstream docs,
starting from the overcloud deploy section:
http://ow.ly/1Vc1301iBlb
##################################
Virtual Environment Setup Complete
##################################
[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.1.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Wed Aug 24 12:13:16 2016 from gateway
[stack@undercloud ~]$ sudo su
[root@undercloud stack]# cd /etc/yum.repos.d
[root@undercloud yum.repos.d]# ls -l
total 40
-rw-r--r--. 1 root root 1664 Dec 9 2015 CentOS-Base.repo
-rw-r--r--. 1 root root 1057 Aug 24 02:58 CentOS-Ceph-Hammer.repo
-rw-r--r--. 1 root root 1309 Dec 9 2015 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Dec 9 2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 290 Dec 9 2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Dec 9 2015 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Dec 9 2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec 9 2015 CentOS-Vault.repo
-rw-r--r--. 1 root root 162 Aug 24 02:58 delorean-deps.repo
-rw-r--r--. 1 root root 220 Aug 24 02:58 delorean.repo
====================================================
Delorean repos file been installed via quickstart on undercloud
====================================================
[root@undercloud yum.repos.d]# cat delorean-deps.repo
[delorean-mitaka-testing]
name=dlrn-mitaka-testing
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-mitaka/
enabled=1
gpgcheck=0
priority=2
[root@undercloud yum.repos.d]# cat delorean.repo
[delorean]
name=delorean-openstack-rally-3909299306233247d547bad265a1adb78adfb3d4
baseurl=http://trunk.rdoproject.org/centos7-mitaka/39/09/3909299306233247d547bad265a1adb78adfb3d4_4e6dfa3c
enabled=1
gpgcheck=0
shutdown/startup this Controller ( controller-1 in particular case).
Then log into conntroller-1 and restart pcs cluster on particular Controller,
afterwards runs `pacemaker resource cleanup` for several resources what
results bringing back cluster nodes in proper status
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f- | overcloud- | True | :-) | active |
| 85f2-9bb9835cf360 | controller-1.localdomain | | | |
| ae0f67ce-732b- | overcloud- | True | :-) | standby |
| 4cb2-9b52-d15c22211972 | controller-0.localdomain | | | |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud- | True | :-) | standby |
| fd1c3489b | controller-2.localdomain | | | |
+-----------------------------+-----------------------------+----------------+-------+----------+
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova stop overcloud-controller-1
Request to stop server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | SHUTOFF | - | Shutdown | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+---------+------------+-------------+---------------------+
[stack@undercloud ~]$ nova start overcloud-controller-1
Request to start server overcloud-controller-1 has been accepted.
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 5387385d-69a1-40ab-a77a-40d97949dc16 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 456031a7-21c4-497f-a7d8-baa3d403ee2f | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.14 |
| 80b6ce3a-23a0-42d3-a1b3-fec22ca8f615 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.17 |
| b5a8c17c-e170-4f66-a5dd-846546afcfce | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| c10e25b3-6732-4afb-b51c-5d9f859bd7d6 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
The authenticity of host '192.0.2.14 (192.0.2.14)' can't be established.
ECDSA key fingerprint is a3:e6:de:2e:2b:45:e4:33:3d:d0:75:e5:b7:7f:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.14' (ECDSA) to the list of known hosts.
[heat-admin@overcloud-controller-1 ~]$ sudo su -
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:12:39 2016 Last change: Sun Aug 21 13:24:42 2016 by root via cibadmin on overcloud-controller-1
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured
Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-192.0.2.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
ip-172.16.3.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-2 ]
Slaves: [ overcloud-controller-1 ]
Clone Set: memcached-clone [memcached]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-10.0.0.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.1.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-core-clone [openstack-core]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-0 ]
Slaves: [ overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: mongod-clone [mongod]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped
Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-clone [openstack-heat-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
Started: [ overcloud-controller-0 ]
Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: delay-clone [delay]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: neutron-server-clone [neutron-server]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ overcloud-controller-0 overcloud-controller-2 ]
Stopped: [ overcloud-controller-1 ]
Failed Actions:
* rabbitmq_monitor_10000 on overcloud-controller-0 'not running' (7): call=81, status=complete, exitreason='none',
last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
* rabbitmq_monitor_10000 on overcloud-controller-2 'not running' (7): call=79, status=complete, exitreason='none',
last-rc-change='Sun Aug 21 15:11:13 2016', queued=0ms, exec=0ms
PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@overcloud-controller-1 ~]# pcs cluster stop
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
[root@overcloud-controller-1 ~]# pcs cluster start
Starting Cluster...
[root@overcloud-controller-1 ~]#
Broadcast message from systemd-journald@overcloud-controller-1.localdomain (Sun 2016-08-21 15:16:07 UTC):
haproxy[16997]: proxy nova_ec2 has no server available!
======================================
Script start.sh [ 1 ]
======================================
#!/bin/bash -x
pcs resource cleanup rabbitmq-clone ;
sleep 10
pcs resource cleanup neutron-server-clone ;
sleep 10
pcs resource cleanup openstack-nova-api-clone ;
sleep 10
pcs resource cleanup openstack-nova-consoleauth-clone ;
sleep 10
pcs resource cleanup openstack-heat-engine-clone ;
sleep 10
pcs resource cleanup openstack-cinder-api-clone ;
sleep 10
pcs resource cleanup openstack-glance-registry-clone ;
sleep 10
pcs resource cleanup httpd-clone
=======================================
[root@overcloud-controller-1 ~]# . ./start.sh
Waiting for 3 replies from the CRMd... OK
Cleaning up rabbitmq:0 on overcloud-controller-0, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-1, removing fail-count-rabbitmq
Cleaning up rabbitmq:0 on overcloud-controller-2, removing fail-count-rabbitmq
Waiting for 3 replies from the CRMd... OK
Cleaning up neutron-server:0 on overcloud-controller-0, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-1, removing fail-count-neutron-server
Cleaning up neutron-server:0 on overcloud-controller-2, removing fail-count-neutron-server
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-api:0 on overcloud-controller-0, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-1, removing fail-count-openstack-nova-api
Cleaning up openstack-nova-api:0 on overcloud-controller-2, removing fail-count-openstack-nova-api
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-0, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-1, removing fail-count-openstack-nova-consoleauth
Cleaning up openstack-nova-consoleauth:0 on overcloud-controller-2, removing fail-count-openstack-nova-consoleauth
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-heat-engine:0 on overcloud-controller-0, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-1, removing fail-count-openstack-heat-engine
Cleaning up openstack-heat-engine:0 on overcloud-controller-2, removing fail-count-openstack-heat-engine
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-cinder-api:0 on overcloud-controller-0, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-1, removing fail-count-openstack-cinder-api
Cleaning up openstack-cinder-api:0 on overcloud-controller-2, removing fail-count-openstack-cinder-api
Waiting for 3 replies from the CRMd... OK
Cleaning up openstack-glance-registry:0 on overcloud-controller-0, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-1, removing fail-count-openstack-glance-registry
Cleaning up openstack-glance-registry:0 on overcloud-controller-2, removing fail-count-openstack-glance-registry
Waiting for 3 replies from the CRMd... OK
Cleaning up httpd:0 on overcloud-controller-0, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-1, removing fail-count-httpd
Cleaning up httpd:0 on overcloud-controller-2, removing fail-count-httpd
[root@overcloud-controller-1 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Aug 21 15:18:04 2016 Last change: Sun Aug 21 15:17:57 2016 by hacluster via crmd on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured
Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-192.0.2.12 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
ip-172.16.3.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: memcached-clone [memcached]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-10.0.0.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.2.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
ip-172.16.1.4 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-core-clone [openstack-core]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-0 ]
Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
Clone Set: mongod-clone [mongod]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0
Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-clone [openstack-heat-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: delay-clone [delay]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: neutron-server-clone [neutron-server]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
PCSD Status:
overcloud-controller-0: Online
overcloud-controller-1: Online
overcloud-controller-2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@overcloud-controller-1 ~]# logout
[heat-admin@overcloud-controller-1 ~]$ logout
Connection to 192.0.2.14 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.16
Last login: Sun Aug 21 15:08:18 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sun Aug 21 15:08:24 UTC 2016 on pts/0
[root@overcloud-controller-0 ~]# . keysstonerc_admin
[root@overcloud-controller-0 ~]# neutron l3-agent-list-hosting-router RouterDSA
+-----------------------------+-----------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+-----------------------------+-----------------------------+----------------+-------+----------+
| 558fe2d4-a709-482f- | overcloud- | True | :-) | standby |
| 85f2-9bb9835cf360 | controller-1.localdomain | | | |
| ae0f67ce-732b- | overcloud- | True | :-) | standby |
| 4cb2-9b52-d15c22211972 | controller-0.localdomain | | | |
| fd9bfd34-9e36-4dac-a350-d18 | overcloud- | True :-) | active |
| fd1c3489b | controller-2.localdomain | | | |
+-----------------------------+-----------------------------+----------------+-------+----------+
====================================
Verification Galera DB sync on Controllers
====================================
[root@overcloud-controller-0 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
[root@overcloud-controller-0 ~]# logout
[heat-admin@overcloud-controller-0 ~]$ logout
Connection to 192.0.2.16 closed.
[stack@undercloud ~]$ ssh heat-admin@192.0.2.14
Last login: Sun Aug 21 15:12:27 2016 from 192.0.2.1
[heat-admin@overcloud-controller-1 ~]$ sudo su -
Last login: Sun Aug 21 15:12:34 UTC 2016 on pts/0
[root@overcloud-controller-1 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
==================
Setup details
==================
[boris@fedora24wks tripleo-quickstart]$ cat ./config/general_config/ha.yml
# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6144
compute_memory: 6144
default_vcpu: 2
undercloud_memory: 8192
# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 2
# Create three controller nodes and one compute node.
overcloud_nodes:
- name: control_0
flavor: control
- name: control_1
flavor: control
- name: control_2
flavor: control
- name: compute_0
flavor: compute
- name: compute_1
flavor: compute
# We don't need introspection in a virtual environment (because we are
# creating all the "hardware" we really know the necessary
# information).
step_introspect: true
# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
--control-scale 3 --compute-scale 2 --neutron-network-type vxlan
--neutron-tunnel-types vxlan
--ntp-server pool.ntp.org
test_tempest: false
test_ping: true
enable_pacemaker: true
##################################
Virtual Environment Setup Complete
##################################
Access the undercloud by:
ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
There are scripts in the home directory to continue the deploy:
overcloud-deploy.sh will deploy the overcloud
overcloud-deploy-post.sh will do any post-deploy configuration
overcloud-validate.sh will run post-deploy validation
Alternatively, you can ignore these scripts and follow the upstream docs,
starting from the overcloud deploy section:
http://ow.ly/1Vc1301iBlb
##################################
Virtual Environment Setup Complete
##################################
[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.1.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Wed Aug 24 12:13:16 2016 from gateway
[stack@undercloud ~]$ sudo su
[root@undercloud stack]# cd /etc/yum.repos.d
[root@undercloud yum.repos.d]# ls -l
total 40
-rw-r--r--. 1 root root 1664 Dec 9 2015 CentOS-Base.repo
-rw-r--r--. 1 root root 1057 Aug 24 02:58 CentOS-Ceph-Hammer.repo
-rw-r--r--. 1 root root 1309 Dec 9 2015 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Dec 9 2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 290 Dec 9 2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Dec 9 2015 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Dec 9 2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec 9 2015 CentOS-Vault.repo
-rw-r--r--. 1 root root 162 Aug 24 02:58 delorean-deps.repo
-rw-r--r--. 1 root root 220 Aug 24 02:58 delorean.repo
====================================================
Delorean repos file been installed via quickstart on undercloud
====================================================
[root@undercloud yum.repos.d]# cat delorean-deps.repo
[delorean-mitaka-testing]
name=dlrn-mitaka-testing
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-mitaka/
enabled=1
gpgcheck=0
priority=2
[root@undercloud yum.repos.d]# cat delorean.repo
[delorean]
name=delorean-openstack-rally-3909299306233247d547bad265a1adb78adfb3d4
baseurl=http://trunk.rdoproject.org/centos7-mitaka/39/09/3909299306233247d547bad265a1adb78adfb3d4_4e6dfa3c
enabled=1
gpgcheck=0
