Wednesday, August 17, 2016

TripleO QuickStart HA Setup && Keeping undercloud persistent between cold reboots ( newly polished )

UPDATE 09/03/2016


Undercloud VM gets created with AutoStart at boot up
in meantime.So just change permissions and allow services
to start on undercloud (5 min - 7 min )

Up on deployment completed
[stack@ServerTQS72 ~]$ virsh dominfo undercloud | grep -i autostart
Autostart:      enable


END UPDATE


UPDATE 08/21/216


In case when virt tools (virsh,virt-manger ) stop to recognise running
qemu-kvm of undercloud as VM  issue `sudo shutdown -P now` via connection
 `ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud`.
It will result graceful shutdown of undercloud's qemu-kvm process on VIRTHOST.


END UPDATE 



This post follows up http://lxer.com/module/newswire/view/230814/index.html
and might work as timer saver unless status undecloud.qcow2 per
http://artifacts.ci.centos.org/artifacts/rdo/images/mitaka/delorean/stable/
requires fresh installation to be done from scratch.
Current update allows to automate procedure via /etc/rc.d/rc.local and exports
in stack's shell variables which allow to start virt-manager right away , presuming that xhost+ was issued in root's shell.

Thus, we intend to survive VIRTHOST cold reboot (downtime) and keep previous version of undercloud VM been able to bring it up avoiding build via quickstart.sh and restart procedure from logging into undercloud and immediately run overcloud deployment. Proceed as follows :-

1. System shutdown
    Cleanly commit :-
    [stack@undercloud~] $ openstack stack delete overcloud
2. Login into VIRTHOST as stack and gracefully shutdown undercloud
    [stack@ServerCentOS72 ~]$ virsh shutdown undercloud


=====================
 Make following updates
=====================

[root@ServerTQS72 ~]# cat /etc/rc.d/rc.local
#!/bin/bash
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
mkdir -p /run/user/1001
chown -R stack /run/user/1001

if [ $? -ne 0 ]
then
       exit 0   
fi
chgrp -R stack /run/user/1001

touch /var/lock/subsys/local

========================
In stack's .bashrc
========================

[stack@ServerTQS72 ~]$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
# BEGIN ANSIBLE MANAGED BLOCK
# Make sure XDG_RUNTIME_DIR is set (used by libvirt
# for creating config and sockets for qemu:///session
# connections)
: ${XDG_RUNTIME_DIR:=/run/user/$(id -u)}
export XDG_RUNTIME_DIR
export DISPLAY=:0.0
export NO_AT_BRIDGE=1

# END ANSIBLE MANAGED BLOCK

=================
REBOOT  VIRTHOST
=================

$ sudo su -
# xhost +
# su - stack

[stack@ServerTQS72 ~]$ virt-manager --connect qemu:///session

Start VM undercloud
 


  Virt-tools misbehavior (UPDATE 08/21/16) . Six qemu-kvm are up and running

   1. Undercloud
   2. 3 Node HA Controller (Pacemaker/Corosync) cluster
   3. 2 Compute Nodes (nested KVM enabled )

 

=====================================
 Log into undercloud from Ansible Server via  :-
=====================================
[boris@fedora24wks tripleo-quickstart]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud

Deploy overcloud using old overcloud-deploy.sh

# Deploy the overcloud!
openstack overcloud deploy \
 --templates /usr/share/openstack-tripleo-heat-templates \
 --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute \
 --ceph-storage-flavor oooq_ceph --timeout 90 -\
 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
 -e $HOME/network-environment.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
 --control-scale 3 --compute-scale 2 \
 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server pool.ntp.org \
    ${DEPLOY_ENV_YAML:+-e $DEPLOY_ENV_YAML} "$@" || true

# We don't always get a useful error code from the openstack deploy command,
# so check `heat stack-list` for a CREATE_FAILED status.
if heat stack-list | grep -q 'CREATE_FAILED'; then

    # get the failures list
    openstack stack failures list overcloud > failed_deployment_list.log || true

    # get any puppet related errors
    for failed in $(heat resource-list \
        --nested-depth 5 overcloud | grep FAILED |
        grep 'StructuredDeployment ' | cut -d '|' -f3)
    do
       echo "heat deployment-show out put for deployment: $failed" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       heat deployment-show $failed >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       echo "puppet standard error for deployment: $failed" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
       # the sed part removes color codes from the text
       heat deployment-show $failed |
           jq -r .output_values.deploy_stderr |
           sed -r "s:\x1B\[[0-9;]*[mK]::g" >> failed_deployments.log
       echo "######################################################" >> failed_deployments.log
    done
fi

[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| b6f105e8-3854-4939-99d9-73c16cf233fd | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| 30979d6e-773b-4d79-9446-1cd25bade373 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 256627a8-2202-4986-86b6-8cd6e46c21db | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| 9dc029bf-b096-4be6-b5a3-14b39ac098a4 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| 16d0e195-c6a2-4286-a368-6fe9851ccd82 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+