Tuesday, August 23, 2016

Attempt to reproduce Deploying Kubernetes on Openstack using Heat by Ales Nosek (CentOS 7.2)

UPDATE 09/07/2016
Issue with RDO Mitaka ( CentOS repos based ) escalated to RH
"Bug 1374183 - Import Error for python-senlinclient python-zaqarclient python-magnumclient python-mistralclient"
END UPDATE

UPDATE 09/05/2016
Attempt on RDO Newton M3 results kubernetes stack CREATE_IN_PROGRESS  to hang, reporting waiting for Master in heat logs.
Conditions from http://kubernetes.io/docs/getting-started-guides/openstack-heat/
for python clients are sartisfied in Newton (Master is running )
However, RDO Newton M3 itself fails with simple `nova boot ... ` issued on Compute Node.
END UPDATE
 
UPDATE 08/27/2016
I tested updated CentOS-7-x86_64-GenericCloud-1607.qcow2 with python2-boto 2.41 preinstalled.It eliminates "ERROR" during Master boot and provides the option to login into Master via ssh-keypair, exported in build environment. There is no any httpd daemon in SSL mode running into VM.Obviously https://Master-IP fails.
END UPDATE

I got negative results attempting to reproduce blog http://alesnosek.com/blog/2016/06/26/deploying-kubernetes-on-openstack-using-heat/  .  Following bellow is my step by step procedure which finally
builds kubernetes heat stack which is not functional  in meantime and troubleshooting kubernetes VM's boot logs  having ERRORS . The last ones been fixed still don't make kubernetes stack functional.

Two Node Cluster Controller/Network/Compute  && Storage deployed on RDO
Mitaka.

====================================
Environment set up for kubernetes stack build via heat
====================================
[boris@CentOS72Server ~(keystone_build)]$ cat openrc.sh
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=dda05d8fb4554e93
export OS_AUTH_URL=http://192.168.1.52:5000/v3
export PS1='[\u@\h \W(keystone_build)]\$ '

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_TENANT_ID=6e72c704971d4da3845f0ae9982bca6b

[boris@CentOS72Server ~(keystone_build)]$ cat openstack-heat.sh
export KUBERNETES_PROVIDER=openstack-heat
export STACK_NAME=kubernetes
export KUBERNETES_KEYPAIR_NAME=oskey082316
export NUMBER_OF_MINIONS=1
export MAX_NUMBER_OF_MINIONS=1
export EXTERNAL_NETWORK=public
export CREATE_IMAGE=false
export DOWNLOAD_IMAGE=false
export IMAGE_ID=7133dcf8-21a7-4beb-be1d-4a1f9d972cd8
export DNS_SERVER=83.221.202.254
export SWIFT_SERVER_URL=http://192.168.1.54:8080/v1/AUTH_6e72c704971d4da3845f0ae9982bca6b

1. Storage node separated during packstack deployment ( localhost:8080 causes issue on AIO box due to proxy-swift default endpoint )
2. SSL connection via horizon enabled in packstack deployment.
3. Security rules provide access to ports  443,80,22
========
Results
========
[root@CentOS72Server ~(keystone_admin)]# nova list
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| ID                                   | Name                     | Status | Task State | Power State | Networks                                                      |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+
| f72bcec6-2def-4103-bb84-fcdc4a8af65e | CentOS72Devs01           | ACTIVE | -          | Running     | private=10.0.0.3, 192.168.1.150                               |
| 462e5122-fe5b-486e-8b1d-4379345271d6 | kubernetes-master        | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.3, 192.168.1.155 |
| 9c0f4e2c-1e9c-4370-8906-6b104b9bedbd | kubernetes-node-FhUQ6AJz | ACTIVE | -          | Running     | kubernetes-fixed_network-htt6bujn7umv=10.0.0.4, 192.168.1.156 |
+--------------------------------------+--------------------------+--------+------------+-------------+---------------------------------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# openstack stack list
+------------------------+------------+-----------------+---------------------+--------------+
| ID                     | Stack Name | Stack Status    | Creation Time       | Updated Time |
+------------------------+------------+-----------------+---------------------+--------------+
| 57b4511f-d264-4a29     | kubernetes | CREATE_COMPLETE | 2016-08-23T14:29:43 | None         |
| -ab8c-9ce273a4d9bb     |            |                 |                     |              |
+------------------------+------------+-----------------+---------------------+--------------+
[root@CentOS72Server ~(keystone_admin)]# nova secgroup-list
+--------------------------------------+-----------------------------------------+------------------------+
| Id                                   | Name                                    | Description            |

+--------------------------------------+-----------------------------------------+------------------------+

| 9763cead-5816-40c5-a6e0-50a821347e52 | default                                 | Default security group |

| fc918814-db18-4be9-a319-4d8988b9060f | kubernetes-secgroup_base-7raauykt5owy   |                        |

| 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 | kubernetes-secgroup_master-ztdnfr6paudu |                        |

| 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 | kubernetes-secgroup_node-dt77fol3a7og   |                        |

+--------------------------------------+-----------------------------------------+------------------------+


[boris@CentOS72Server kubernetes(keystone_build)]$ ./cluster/kube-up.sh
... Starting cluster using provider: openstack-heat
... calling verify-prereqs
swift client installed
glance client installed
nova client installed
heat client installed
openstack client installed
... calling kube-up
kube-up for provider openstack-heat
[INFO] Execute commands to create Kubernetes cluster
[INFO] Uploading kubernetes-server-linux-amd64.tar.gz
kubernetes-server.tar.gz
[INFO] Uploading kubernetes-salt.tar.gz
kubernetes-salt.tar.gz
[INFO] Key pair already exists
Stack not found: kubernetes
[INFO] Create stack kubernetes
+---------------------+-------------------------------------------------------------------------+
| Field               | Value                                                                   |
+---------------------+-------------------------------------------------------------------------+
| id                  | 57b4511f-d264-4a29-ab8c-9ce273a4d9bb                                    |
| stack_name          | kubernetes                                                              |
| description         | Kubernetes cluster with one master and one or more worker nodes (as     |
|                     | specified by the number_of_minions parameter, which defaults to 3).     |
|                     |                                                                         |
| creation_time       | 2016-08-23T14:29:43                                                     |
| updated_time        | None                                                                    |
| stack_status        | CREATE_IN_PROGRESS                                                      |
| stack_status_reason |                                                                         |
+---------------------+-------------------------------------------------------------------------+

... calling validate-cluster
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_COMPLETE
cluster "openstack-kubernetes" set.
user "openstack-kubernetes" set.
context "openstack-kubernetes" set.
switched to context "openstack-kubernetes".
Wrote config for openstack-kubernetes to /home/boris/.kube/config
Done, listing cluster services:

The connection to the server 192.168.1.155 was refused - did you specify the right host or port?
=========================================
Status of heat-engine.log up on successful completition
As far as I understand
python-senlinclient
python-zaqarclient
are not packaged with RDO Mitaka on CentOS 7.2
See also :-
https://bugs.launchpad.net/heat/+bug/1544220
https://bugs.launchpad.net/heat/+bug/1597593
https://bugzilla.redhat.com/show_bug.cgi?id=1294489
=========================================
[boris@CentOS72Server kubernetes(keystone_build)]$ cat  /home/boris/.kube/config
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://192.168.1.155
  name: openstack-kubernetes
contexts:
- context:
    cluster: openstack-kubernetes
    user: openstack-kubernetes
  name: openstack-kubernetes
current-context: openstack-kubernetes
kind: Config
preferences: {}
users:
- name: openstack-kubernetes
  user:

  
 

=======
Finally
=======
[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 fc918814-db18-4be9-a319-4d8988b9060f
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 83e43587-1f6f-4f1b-b8b9-85e353b4d030 |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | fc918814-db18-4be9-a319-4d8988b9060f |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 29a1ff1d-be63-4bec-bac7-fdfa00a9c551
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 275f5b0b-4521-4b40-abb8-97bc1ab9566f |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

[root@CentOS72Server ~(keystone_admin)]# nova secgroup-list
+--------------------------------------+-----------------------------------------+------------------------+
| Id                                   | Name                                    | Description            |
+--------------------------------------+-----------------------------------------+------------------------+
| 9763cead-5816-40c5-a6e0-50a821347e52 | default                                 | Default security group |
| fc918814-db18-4be9-a319-4d8988b9060f | kubernetes-secgroup_base-7raauykt5owy   |                        |
| 29a1ff1d-be63-4bec-bac7-fdfa00a9c551 | kubernetes-secgroup_master-ztdnfr6paudu |                        |
| 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 | kubernetes-secgroup_node-dt77fol3a7og   |                        |
+--------------------------------------+-----------------------------------------+------------------------+

[root@CentOS72Server ~(keystone_admin)]# neutron security-group-rule-create --protocol icmp  --direction ingress --remote-ip-prefix 0.0.0.0/0 08d5e1d7-0223-4acb-bf74-ed7230e98bf1
Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| description       |                                      |
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 8ef7ae78-42ff-4f82-baab-ce41e5e90cc8 |
| port_range_max    |                                      |
| port_range_min    |                                      |
| protocol          | icmp                                 |
| remote_group_id   |                                      |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | 08d5e1d7-0223-4acb-bf74-ed7230e98bf1 |
| tenant_id         | 6e72c704971d4da3845f0ae9982bca6b     |
+-------------------+--------------------------------------+

Can ping 192.168.1.155,192.168.1.156

Security rules for each kubernetes secgroup have ports 1 - 6535  open , however


==========================
 Kubernetes Master VM boot log contains
===========================

  
[[32m  OK  [0m] Started Update UTMP about System Runlevel Changes.
[  380.104758] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  455.439213] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.
[  469.546079] cloud-init[4161]: [WARNING ] /usr/lib/python2.7/site-packages/salt/states/cmd.py:1041: DeprecationWarning: The legacy user/group arguments are deprecated. Replace them with runas. These arguments will be removed in Salt Oxygen.
[  521.559170] cloud-init[4161]: [WARNING ] State for file: /var/log/kube-apiserver.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
[  521.723063] cloud-init[4161]: [ERROR   ] boto_route53 requires at least boto 2.35.0.

Even if I checkout branch :-

$ git clone https://github.com/kubernetes/kubernetes.git 
$ cd kubernetes 
$ git checkout origin/release-1.3.0
$ make quick-release
 
Same error in Master VM boot log.

I believe CentOS 7.2 image has to be updated up to python2-boto 2.41 via EPEL 7 during cloud-init run ( first boot )


  References
  http://alesnosek.com/blog/2016/06/26/deploying-kubernetes-on-openstack-using-heat/