Saturday, January 07, 2017

TripleO QuickStart HA&&CEPH Deployment on Fedora 25 VIRTHOST 32 GB

##################
UPDATE 01/22/2017
##################

Regular upgrade F25's kernel to 4.9.X makes overcloud deployment on Server
Fedora 25 pretty stable,e.g. building heat stack "overcloud" no longer randomly
hangs , what caused deleting deleting stack and recreating it from scratch.
Details here :-
   TripleO QuickStart HA&&CEPH Deployment on Fedora 25 Server VIRTHOST 

Clean up Server F25 for Tripleo QuickStart redeployment
# rm -fr /home/stack
# userdel stack

##################
END UPDATE
##################

##################
UPDATE 01/12/2017
##################
It's much more  safe to issue
  # systemctl set-default multi-user.target
  # reboot
before deployment. I didn't  get a chance to test F25 Server, seems to be an optimal solution.  Overcloud  Deployment  in text mode




##############   
END UPDATE
##############
The most recent commits in  https://github.com/openstack/tripleo-quickstart
allow to use Fedora 25 Workstaion (32 GB) as target VIRTHOST for TripleO
Quickstart HA Deployments  and benefit from QEMU's  (2.7.0) && Libvirt's (2.2.0)  the most recent KVM virtualization features coming with last Fedora release.


Prior to deployment on VIRTHOST install KSM and enable ksm.service  :-

# dnf -y install python2-dnf ksm
# systemctl start sshd ksm
# systemctl enable sshd ksm

On workstation :-
[boris@fedora24wks]$ export VIRTHOST=192.168.0.74
[boris@fedora24wks]$ git clone \
   https://github.com/openstack/tripleo-quickstart
[boris@fedora24wks]$ cd tripleo-quickstart
[boris@fedora24wks]$ sudo bash quickstart.sh --install-deps
[boris@fedora24wks]$ sudo dnf install redhat-rpm-config 
[boris@fedora24wks]$ ssh-keygen
[boris@fedora24wks]$ ssh-copy-id root@$VIRTHOST
[boris@fedora24wks]$ ssh root@$VIRTHOST uname -a
 
[boris@fedora24wks general_config]$ cat ha.yml
# Deploy an HA openstack environment.
control_memory: 6500
compute_memory: 6500

undercloud_memory: 8192

# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 4

# Since HA has more machines, we set the cpu for controllers and
# compute nodes to 1
default_vcpu: 1

# This enables TLS for the undercloud which will also make haproxy bind to the
# configured public-vip and admin-vip.
undercloud_generate_service_certificate: True

# Create three controller nodes and one compute node.
overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230

  - name: control_1
    flavor: control
    virtualbmc_port: 6231

  - name: control_2
    flavor: control
    virtualbmc_port: 6232

  - name: compute_0
    flavor: compute
    virtualbmc_port: 6233

  - name: ceph_0
    flavor: ceph
    virtualbmc_port: 6234


# We do introspection in a virtual environment

step_introspect: true

# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
  --control-scale 3
  --compute-scale 1 
  --ceph-storage-scale 1
  --ntp-server pool.ntp.org
  -e {{overcloud_templates_path}}/environments/storage-environment.yaml
test_ping: true
enable_pacemaker: true

run_tempest: false
 
[boris@fedora24wks tripleo-quickstart]$ bash quickstart.sh --no-clone -e supported_distro_check=false -R newton --config config/general_config/ha.yml $VIRTHOST


******   undercloud deployment  *******


[boris@fedora24wks ~]$ ssh -F $HOME/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Sun Jan  8 06:51:39 2017 from gateway
[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ ls -l
total 1624856
-rwxr-xr-x. 1 stack stack        770 Jan  8 05:34 containers-default-parameters.yaml
-rw-rw-r--. 1 stack stack      18382 Jan  8 05:14 instackenv.json
-rw-r--r--. 1 root  root   355802133 Jan  3 08:59 ironic-python-agent.initramfs
-rwxr-xr-x. 1 root  root     5393328 Jan  3 08:59 ironic-python-agent.kernel
-rw-r--r--. 1 stack stack        474 Jan  8 05:34 network-environment.yaml
-rw-rw-r--. 1 stack stack        208 Jan  8 05:40 neutronl3ha.yaml
-rw-------. 1 stack stack       1675 Jan  8 07:21 oskey010817.pem
-rw-rw-r--. 1 stack stack          0 Jan  8 05:34 overcloud_custom_tht_script.log
-rwxr-xr-x. 1 stack stack        293 Jan  8 05:34 overcloud-custom-tht-script.sh
-rwxr-xr-x. 1 stack stack       1012 Jan  8 05:40 overcloud-deploy-post.sh
-rwxr-xr-x. 1 stack stack       2876 Jan  8 05:40 overcloud-deploy.sh
-rw-rw-r--. 1 stack stack       4211 Jan  8 05:59 overcloud-env.json
-rw-r--r--. 1 root  root    46800999 Jan  3 08:59 overcloud-full.initrd
-rw-r--r--. 1 root  root  1250130432 Jan  3 08:59 overcloud-full.qcow2
-rwxr-xr-x. 1 root  root     5393328 Jan  3 09:00 overcloud-full.vmlinuz
-rwxr-xr-x. 1 stack stack       3905 Jan  8 05:34 overcloud-prep-containers.sh
-rw-rw-r--. 1 stack stack       7336 Jan  8 05:40 overcloud_prep_flavors.log
-rwxr-xr-x. 1 stack stack       3672 Jan  8 05:39 overcloud-prep-flavors.sh
-rw-rw-r--. 1 stack stack       4885 Jan  8 05:39 overcloud_prep_images.log
-rwxr-xr-x. 1 stack stack        746 Jan  8 05:34 overcloud-prep-images.sh
-rw-rw-r--. 1 stack stack       1315 Jan  8 05:40 overcloud_prep_network.log
-rwxr-xr-x. 1 stack stack        861 Jan  8 05:40 overcloud-prep-network.sh
-rw-rw-r--. 1 stack stack        391 Jan  8 06:48 overcloudrc
-rw-------. 1 stack stack        351 Jan  8 05:19 quickstart-hieradata-overrides.yaml
-rw-------. 1 stack stack        587 Jan  8 05:33 stackrc
-rw-rw-r--. 1 stack stack        444 Jan  8 06:48 tempest-deployer-input.conf
-rw-------. 1 stack stack       7868 Jan  8 05:19 undercloud.conf
-rw-rw-r--. 1 stack stack     191200 Jan  8 05:34 undercloud_install.log
-rwxr-xr-x. 1 stack stack        151 Jan  8 05:19 undercloud-install.sh
-rw-rw-r--. 1 stack stack       1650 Jan  8 05:19 undercloud-passwords.conf
-rwxr-xr-x. 1 stack stack        463 Jan  8 05:34 upload_images_to_local_registry.py



**************************************************************************************
Update  neutronl3ha.yaml to create in overcloud router with ha=True by default
**************************************************************************************

[stack@undercloud ~]$ cat neutronl3ha.yaml
# Note: we need to enable the L3 HA for Neutron if we want to use pacemaker
# corosync 3 node controller.


parameter_defaults:
  NeutronL3HA: true

***********************************************************
[stack@undercloud ~]$ cat overcloud-deploy.sh
***********************************************************

#!/bin/bash

set -eux

### --start_docs
## Deploying the overcloud
## =======================

## Prepare Your Environment
## ------------------------

## * Source in the undercloud credentials.
## ::

source /home/stack/stackrc

### --stop_docs
# Wait until there are hypervisors available.
while true; do
    count=$(openstack hypervisor stats show -c count -f value)
    if [ $count -gt 0 ]; then
        break
    fi
done

### --start_docs


## * Deploy the overcloud!
## ::
openstack overcloud deploy \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage --swift-storage-flavor oooq_objectstorage --timeout 90  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /home/stack/neutronl3ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml   --validation-warnings-fatal --control-scale 3 --compute-scale 1  --ceph-storage-scale 1 --ntp-server pool.ntp.org -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
    ${DEPLOY_ENV_YAML:+-e $DEPLOY_ENV_YAML} "$@" && status_code=0 || status_code=$?


### --stop_docs
# We don't always get a useful error code from the openstack deploy command,
# so check `heat stack-list` for a CREATE_FAILED status.
if heat stack-list | grep -q 'CREATE_FAILED'; then
        # get the failures list
    openstack stack failures list overcloud > failed_deployment_list.log || true
   
    # get any puppet related errors
    for failed in $(heat resource-list \
        --nested-depth 5 overcloud | grep FAILED |
        grep 'StructuredDeployment ' | cut -d '|' -f3)
    do
    echo "heat deployment-show out put for deployment: $failed" >> failed_deployments.log
    echo "######################################################" >> failed_deployments.log
    heat deployment-show $failed >> failed_deployments.log
    echo "######################################################" >> failed_deployments.log
    echo "puppet standard error for deployment: $failed" >> failed_deployments.log
    echo "######################################################" >> failed_deployments.log
    # the sed part removes color codes from the text
    heat deployment-show $failed |
        jq -r .output_values.deploy_stderr |
        sed -r "s:\x1B\[[0-9;]*[mK]::g" >> failed_deployments.log
    echo "######################################################" >> failed_deployments.log
    # We need to exit with 1 because of the above || true
    done
fi
[stack@undercloud ~]$./overcloud-deploy.sh

   . . . . . . . . . . 
  


[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks               |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 64a90c37-2a71-4be3-afd5-23d92229ecd9 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.12 |
| 252895f9-b825-4499-910c-6b6385e2a5c1 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.18 |
| 96caa2c5-fec3-46f0-90a4-c8b2975a6bb9 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.9  |
| 647322d8-64c9-4534-916f-fe0208df5e97 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
| ee522eeb-673b-4782-b136-3706b7eaef99 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+

[root@overcloud-controller-0 ~]# ceph status
    cluster 06f2817c-d564-11e6-98fe-00cf70d8c1e2
     health HEALTH_OK
     monmap e1: 3 mons at {overcloud-controller-0=172.16.1.8:6789/0,overcloud-controller-1=172.16.1.17:6789/0,overcloud-controller-2=172.16.1.7:6789/0}
election epoch 8, quorum 0,1,2 overcloud-controller-2,overcloud-controller-0,overcloud-controller-1
     osdmap e19: 1 osds: 1 up, 1 in
            flags sortbitwise
      pgmap v1075: 224 pgs, 6 pools, 4762 MB data, 1508 objects
            13248 MB used, 37939 MB / 51187 MB avail
                 224 active+clean
[root@overcloud-controller-0 ~]# ceph osd df tree
ID WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE  VAR  PGS TYPE NAME                        
-1 0.04880        - 51187M 13248M 37939M 25.88 1.00   0 root default                     
-2 0.04880        - 51187M 13248M 37939M 25.88 1.00   0     host overcloud-cephstorage-0 
 0 0.04880  1.00000 51187M 13248M 37939M 25.88 1.00 224         osd.0                    
              TOTAL 51187M 13248M 37939M 25.88                                           
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
[root@overcloud-controller-0 ~]# ceph quorum_status
{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["overcloud-controller-2","overcloud-
controller-0","overcloud-controller-1"],"quorum_leader_name":"overcloud-controller-2",
"monmap":{"epoch":1,"fsid":"06f2817c-d564-11e6-98fe-00cf70d8c1e2",
"modified":"2017-01-08 06:22:24.893808","created":"2017-01-08 06:22:24.893808",
"mons":[{"rank":0,"name":"overcloud-controller-2","addr":"172.16.1.7:6789\/0"},
{"rank":1,"name":"overcloud-controller-0","addr":"172.16.1.8:6789\/0"},
{"rank":2,"name":"overcloud-controller-1","addr":"172.16.1.17:6789\/0"}]}}


  
  

   Status of memory allocation on VIRTHOST
 
 
    Virsh reports in stack's session on Fedora 25 VIRTHOST