Xen Virtualization on Linux and Solaris: Attempt to reproduce Deploying Kubernetes on Openstack using Heat by Ales Nosek (CentOS 7.2)

UPDATE 09/07/2016
Issue with RDO Mitaka ( CentOS repos based ) escalated to RH
"Bug 1374183 - Import Error for python-senlinclient python-zaqarclient python-magnumclient python-mistralclient"
END UPDATE

UPDATE 09/05/2016
Attempt on RDO Newton M3 results kubernetes stack CREATE_IN_PROGRESS to hang, reporting waiting for Master in heat logs.
Conditions from http://kubernetes.io/docs/getting-started-guides/openstack-heat/
for python clients are sartisfied in Newton (Master is running )
However, RDO Newton M3 itself fails with simple `nova boot ... ` issued on Compute Node.
END UPDATE

UPDATE 08/27/2016
I tested updated CentOS-7-x86_64-GenericCloud-1607.qcow2 with python2-boto 2.41 preinstalled.It eliminates "ERROR" during Master boot and provides the option to login into Master via ssh-keypair, exported in build environment. There is no any httpd daemon in SSL mode running into VM.Obviously https://Master-IP fails.
END UPDATE

I got negative results attempting to reproduce blog http://alesnosek.com/blog/2016/06/26/deploying-kubernetes-on-openstack-using-heat/ . Following bellow is my step by step procedure which finally
builds kubernetes heat stack which is not functional in meantime and troubleshooting kubernetes VM's boot logs having ERRORS . The last ones been fixed still don't make kubernetes stack functional.

Two Node Cluster Controller/Network/Compute && Storage deployed on RDO
Mitaka.

====================================
Environment set up for kubernetes stack build via heat
====================================
[boris@CentOS72Server ~(keystone_build)]$ cat openrc.sh
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=dda05d8fb4554e93
export OS_AUTH_URL=http://192.168.1.52:5000/v3
export PS1='[\u@\h \W(keystone_build)]\$ '

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_TENANT_ID=6e72c704971d4da3845f0ae9982bca6b

[boris@CentOS72Server ~(keystone_build)]$ cat openstack-heat.sh
export KUBERNETES_PROVIDER=openstack-heat
export STACK_NAME=kubernetes
export KUBERNETES_KEYPAIR_NAME=oskey082316
export NUMBER_OF_MINIONS=1
export MAX_NUMBER_OF_MINIONS=1
export EXTERNAL_NETWORK=public
export CREATE_IMAGE=false
export DOWNLOAD_IMAGE=false
export IMAGE_ID=7133dcf8-21a7-4beb-be1d-4a1f9d972cd8
export DNS_SERVER=83.221.202.254
export SWIFT_SERVER_URL=http://192.168.1.54:8080/v1/AUTH_6e72c704971d4da3845f0ae9982bca6b

1. Storage node separated during packstack deployment ( localhost:8080 causes issue on AIO box due to proxy-swift default endpoint )
2. SSL connection via horizon enabled in packstack deployment.
3. Security rules provide access to ports 443,80,22
========
Results
========
[root@CentOS72Server ~(keystone_admin)]# nova list

+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| ID                                   | Name                     | Status | Task State | Power State | Networks                                                      |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| f72bcec6-2def-4103-bb84-fcdc4a8af65e | CentOS72Devs01           | ACTIVE | -          | Running     | private=10.0.0.3, 192.168.1.150                               |
| 462e5122-fe5b-486e-8b1d-4379345271d6 | kubernetes-master        | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.3, 192.168.1.155 |
| 9c0f4e2c-1e9c-4370-8906-6b104b9bedbd | kubernetes-node-FhUQ6AJz | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.4, 192.168.1.156 |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# openstack stack list

+------------------------+------------+-----------------+---------------------+--------------+
| ID                     | Stack Name | Stack Status    | Creation Time       | Updated Time |
+------------------------+------------+-----------------+---------------------+--------------+
| 57b4511f-d264-4a29     | kubernetes | CREATE_COMPLETE | 2016-08-23T14:29:43 | None         |
| -ab8c-9ce273a4d9bb     |            |                 |                     |              |
+------------------------+------------+-----------------+---------------------+--------------+

[root@CentOS72Server ~(keystone_admin)]# nova secgroup-list

+--------------------------------------+-----------------------------------------+------------------------+
| Id                                   | Name                                    | Description            |

+--------------------------------------+-----------------------------------------+------------------------+

| 9763cead-5816-40c5-a6e0-50a821347e52 | default                                 | Default security group |

| fc918814-db18-4be9-a319-4d8988b9060f | kubernetes-secgroup_base-7raauykt5owy   |                        |

| 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 | kubernetes-secgroup_master-ztdnfr6paudu |                        |

| 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 | kubernetes-secgroup_node-dt77fol3a7og   |                        |

+--------------------------------------+-----------------------------------------+------------------------+

[boris@CentOS72Server kubernetes(keystone_build)]$ ./cluster/kube-up.sh
... Starting cluster using provider: openstack-heat
... calling verify-prereqs
swift client installed
glance client installed
nova client installed
heat client installed
openstack client installed
... calling kube-up
kube-up for provider openstack-heat
[INFO] Execute commands to create Kubernetes cluster
[INFO] Uploading kubernetes-server-linux-amd64.tar.gz
kubernetes-server.tar.gz
[INFO] Uploading kubernetes-salt.tar.gz
kubernetes-salt.tar.gz
[INFO] Key pair already exists
Stack not found: kubernetes
[INFO] Create stack kubernetes

+---------------------+-------------------------------------------------------------------------+
| Field               | Value                                                                   |
+---------------------+-------------------------------------------------------------------------+
| id                  | 57b4511f-d264-4a29-ab8c-9ce273a4d9bb                                    |
| stack_name          | kubernetes                                                              |
| description         | Kubernetes cluster with one master and one or more worker nodes (as     |
|                     | specified by the number_of_minions parameter, which defaults to 3).     |
|                     |                                                                         |
| creation_time       | 2016-08-23T14:29:43                                                     |
| updated_time        | None                                                                    |
| stack_status        | CREATE_IN_PROGRESS                                                      |
| stack_status_reason |                                                                         |
+---------------------+-------------------------------------------------------------------------+

... calling validate-cluster
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_COMPLETE
cluster "openstack-kubernetes" set.
user "openstack-kubernetes" set.
context "openstack-kubernetes" set.
switched to context "openstack-kubernetes".
Wrote config for openstack-kubernetes to /home/boris/.kube/config
Done, listing cluster services:

The connection to the server 192.168.1.155 was refused - did you specify the right host or port?
=========================================
Status of heat-engine.log up on successful completition
As far as I understand
python-senlinclient
python-zaqarclient
are not packaged with RDO Mitaka on CentOS 7.2
See also :-
https://bugs.launchpad.net/heat/+bug/1544220
https://bugs.launchpad.net/heat/+bug/1597593
https://bugzilla.redhat.com/show_bug.cgi?id=1294489
=========================================
[boris@CentOS72Server kubernetes(keystone_build)]$ cat /home/boris/.kube/config
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://192.168.1.155
name: openstack-kubernetes
contexts:
- context:
    cluster: openstack-kubernetes
    user: openstack-kubernetes
name: openstack-kubernetes
current-context: openstack-kubernetes
kind: Config
preferences: {}
users:
- name: openstack-kubernetes
user:

==========================
Kubernetes Master VM boot log contains
===========================

[[32m  OK  [0m] Started Update UTMP about System Runlevel Changes.
[  380.104758] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  455.439213] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  469.546079] cloud-init[4161]: [WARNING ] /usr/lib/python2.7/site-packages/salt/states/cmd.py:1041: DeprecationWarning: The legacy user/group arguments are deprecated. Replace them with runas. These arguments will be removed in Salt Oxygen.
[  521.559170] cloud-init[4161]: [WARNING ] State for file: /var/log/kube-apiserver.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
[  521.723063] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.

Even if I checkout branch :-

$ git clone https://github.com/kubernetes/kubernetes.git

$ cd kubernetes

$ git checkout origin/release-1.3.0

$ make quick-release

Same error in Master VM boot log.

I believe CentOS 7.2 image has to be updated up to python2-boto 2.41 via EPEL 7 during cloud-init run ( first boot )

References
http://alesnosek.com/blog/2016/06/26/deploying-kubernetes-on-openstack-using-heat/

Xen Virtualization on Linux and Solaris

Tuesday, August 23, 2016

Attempt to reproduce Deploying Kubernetes on Openstack using Heat by Ales Nosek (CentOS 7.2)

About Me

Blog Archive