This post is focused on tuning replica 2 gluster volume when building RDO Juno Gluster Cluster on CentOS 7. Steps undertaken come from Gluster 3.5.2 Release Notes (http://blog.nixpanic.net/2014_07_01_archive.html) and make integration Qemu (1.5.3) && libgfapi really working
- Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open vSwitch plugin )
- Compute node: Nova (nova-compute), Neutron (openvswitch-agent)
juno1.localdomain - Controller (192.168.1.127)
juno2.localdomain - Compute (192.168.1.137)
Download from http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/EPEL.repo/epel-7/SRPMS/
glusterfs-3.5.2-1.el7.src.rpm
$ rpm -iv glusterfs-3.5.2-1.el7.src.rpm
$ sudo yum install bison flex gcc automake libtool ncurses-devel readline-devel libxml2-devel openssl-devel libaio-devel lvm2-devel glib2-devel libattr-devel libibverbs-devel librdmacm-devel fuse-devel
$ rpmbuild -bb glusterfs.spec
. . . . . . . . . . . . . . . . . . . . . . .
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/noarch/glusterfs-resource-agents-3.5.2-1.el7.centos.noarch.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-regression-tests-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-debuginfo-3.5.2-1.el7.centos.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Sigc7l
+ umask 022
+ cd /home/boris/rpmbuild/BUILD
+ cd glusterfs-3.5.2
+ rm -rf /home/boris/rpmbuild/BUILDROOT/glusterfs-3.5.2-1.el7.centos.x86_64
+ exit 0
[boris@juno1 x86_64]$ cat install
sudo yum install glusterfs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm
$ sudo service glusterd start
1. First step is tuning /etc/sysconfig/iptables for IPv4 iptables firewall (service firewalld should be disabled) :-
Update /etc/sysconfig/iptables on both nodes:-
-A INPUT -p tcp -m multiport --dport 24007:24047 -j ACCEPT
-A INPUT -p tcp --dport 111 -j ACCEPT
-A INPUT -p udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m multiport --dport 38465:38485 -j ACCEPT
Comment out lines bellow , ignoring instruction
# -A FORWARD -j REJECT --reject-with icmp-host-prohibited
# -A INPUT -j REJECT --reject-with icmp-host-prohibited
Restart service iptables on both nodes
2. Second step:-
On juno1, run the following commands :
# ssh-keygen (Hit Enter to accept all of the defaults)
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@juno2
On both nodes run :-
# ./install
# service glusterd start
On juno1
#gluster peer probe juno2.localdomain
Should return "success"
[root@juno1 ~(keystone_admin)]# gluster peer status
Number of Peers: 1
Hostname: juno2.localdomain
Uuid: 3ca6490b-c44a-4601-ac13-51fec99e9caf
State: Peer in Cluster (Connected)
[root@juno1 ~(keystone_admin)]# ssh 192.168.1.137
Last login: Thu Aug 14 17:53:41 2014
[root@juno2 ~]# gluster peer status
Number of Peers: 1
Hostname: 192.168.1.127
Uuid: 051e7528-8c2b-46e1-abb6-6d84b2f2e45b
State: Peer in Cluster (Connected)
*************************************************************************
On Controller (192.168.1.127) and on Compute (192.168.1.137)
*************************************************************************
Verify ports availability:-
[root@juno1 ~(keystone_admin)]# netstat -lntp | grep gluster
tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 5453/glusterfsd
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38468 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38469 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 2667/glusterd
tcp 0 0 0.0.0.0:978 0.0.0.0:* LISTEN 5458/glusterfs
************************************
Switching Cinder to Gluster volume
************************************
# gluster volume create cinder-volumes57 \
replica 2 juno1.localdomain:/data5/data-volumes juno2.localdomain:/data5/data-volumes
# gluster volume start cinder-volumes57
# gluster volume set cinder-volumes57 auth.allow 192.168.1.*
The following configuration changes are necessary for 'qemu' and '
samba vfs plugin' integration with libgfapi to work seamlessly:
1. First step
gluster volume set cinder-volumes57 server.allow-insecure on
2. Restarting is required
gluster volume stop cinder-volumes57
gluster volume start cinder-volumes57
3. Edit /etc/glusterfs/glusterd.vol to have a line :
option rpc-auth-allow-insecure on
4. Restart glusterd is required :
service glusterd restart
Nova.conf (on Compute Node) should have entry :-
qemu_allowed_storage_drivers = gluster
[root@juno1 ~]# gluster volume info
Volume Name: cinder-volumes57
Type: Replicate
Volume ID: c1f2e1d2-0b11-426e-af3d-7af0d1d24d5e
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: juno1.localdomain:/data5/data-volumes
Brick2: juno2.localdomain:/data5/data-volumes
Options Reconfigured:
auth.allow: 192.168.1.*
server.allow-insecure: on
[root@juno1 ~]# gluster volume status
Status of volume: cinder-volumes57
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick juno1.localdomain:/data5/data-volumes 49152 Y 3346
Brick juno2.localdomain:/data5/data-volumes 49152 Y 3113
NFS Server on localhost 2049 Y 3380
Self-heal Daemon on localhost N/A Y 3387
NFS Server on juno2.localdomain 2049 Y 3911
Self-heal Daemon on juno2.localdomain N/A Y 3916
Task Status of Volume cinder-volumes57
------------------------------------------------------------------------------
There are no active volume tasks
##############################
Create entries in /etc/cinder/cinder.conf
##############################
enabled_backends=gluster
[gluster]
volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config = /etc/cinder/shares.conf
glusterfs_mount_point_base = /var/lib/cinder/volumes
volume_backend_name=GLUSTER
# vi /etc/cinder/shares.conf
192.168.1.127:/cinder-volumes57
:wq
[root@juno1 ~(keystone_admin)]# cinder type-create gluster
+--------------------------------------+---------+
| ID | Name |
+--------------------------------------+---------+
| 29917269-d73f-4c28-b295-59bfbda5d044 | gluster |
+--------------------------------------+---------+
[root@juno1 ~(keystone_admin)]# cinder type-key gluster set volume_backend_name=GLUSTER
Next step is cinder services restart :-
[root@juno1 ~(keystone_demo)]# for i in api scheduler volume ; do service openstack-cinder-${i} restart ; done
[root@juno1 ~(keystone_admin)]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos01-root00 147G 43G 105G 29% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 152K 3.9G 1% /dev/shm
tmpfs 3.9G 26M 3.8G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 1.9G 6.0M 1.7G 1% /srv/node/swift_loopback
/dev/sda3 477M 146M 302M 33% /boot
/dev/mapper/centos01-data5 98G 15G 83G 16% /data5
192.168.1.127:/cinder-volumes57 98G 15G 83G 16% /var/lib/cinder/volumes/8478b56ad61cf67ab9839fb0a5296965
tmpfs 3.9G 26M 3.8G 1% /run/netns
###################################################
How to verify implementation success. Boot nova instance
( with instance-id say 00000049) based on cinder volume.
###################################################
On Compute Node grep /var/log/libvirt/qemu/instance-00000049.log looking for
"gluster" entry . You are supposed to find a string highlighted down here
# cd /var/log/libvirt/qemu
# [root@juno2 qemu]# cat instance-00000049.log | grep gluster
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00000049 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 92151b16-c7b4-48d1-b49f-1e310e005c80 -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2014.2-2.el7.centos,serial=5dff0de4-c27d-453d-85b4-b2d9af514fcd,uuid=92151b16-c7b4-48d1-b49f-1e310e005c80 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000049.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://192.168.1.127:24007/cinder-volumes57/volume-179b9782-d2b7-4891-ba89-5198b71c6188,if=none,id=drive-virtio-disk0,format=raw,serial=179b9782-d2b7-4891-ba89-5198b71c6188,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:8b:9f:6c,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/92151b16-c7b4-48d1-b49f-1e310e005c80/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
At the same time issue on Controller following commands :-
[root@juno1 ~(keystone_boris)]# cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| 179b9782-d2b7-4891-ba89-5198b71c6188 | in-use | Win2012GLSVOL01 | 20 | gluster | true | 92151b16-c7b4-48d1-b49f-1e310e005c80 |
| ca0694ae-7e8d-4c84-aad8-3f178416dec6 | in-use | VF20LVG520711 | 7 | lvms | true | 51a20959-0a0c-4ef6-81ec-2edeab6e3588 |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
[root@juno1 ~(keystone_boris)]# nova list
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711 | SUSPENDED | - | Shutdown | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05 | SUSPENDED | - | Shutdown | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
[root@juno1 ~(keystone_boris)]# nova show 92151b16-c7b4-48d1-b49f-1e310e005c80 | grep 179b9782-d2b7-4891-ba89-5198b71c6188
| os-extended-volumes:volumes_attached | [{"id": "179b9782-d2b7-4891-ba89-5198b71c6188"}] |
[root@juno2 ~]# . keystonerc_boris
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
| 57640068-3ab7-466a-8eae-cf132359b233 | UbuntuUTRX1211 | ACTIVE | - | Running | private_boris=50.0.0.26, 192.168.1.174 |
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711 | SUSPENDED | - | Shutdown | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05 | SUSPENDED | - | Shutdown | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
<source protocol='gluster' name='cinder-volumes57/volume-bf448475-50c8-4491-92aa-77d36666f296'>
- Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open vSwitch plugin )
- Compute node: Nova (nova-compute), Neutron (openvswitch-agent)
juno1.localdomain - Controller (192.168.1.127)
juno2.localdomain - Compute (192.168.1.137)
Download from http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/EPEL.repo/epel-7/SRPMS/
glusterfs-3.5.2-1.el7.src.rpm
$ rpm -iv glusterfs-3.5.2-1.el7.src.rpm
$ sudo yum install bison flex gcc automake libtool ncurses-devel readline-devel libxml2-devel openssl-devel libaio-devel lvm2-devel glib2-devel libattr-devel libibverbs-devel librdmacm-devel fuse-devel
$ rpmbuild -bb glusterfs.spec
. . . . . . . . . . . . . . . . . . . . . . .
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/noarch/glusterfs-resource-agents-3.5.2-1.el7.centos.noarch.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-regression-tests-3.5.2-1.el7.centos.x86_64.rpm
Wrote: /home/boris/rpmbuild/RPMS/x86_64/glusterfs-debuginfo-3.5.2-1.el7.centos.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Sigc7l
+ umask 022
+ cd /home/boris/rpmbuild/BUILD
+ cd glusterfs-3.5.2
+ rm -rf /home/boris/rpmbuild/BUILDROOT/glusterfs-3.5.2-1.el7.centos.x86_64
+ exit 0
[boris@juno1 x86_64]$ cat install
sudo yum install glusterfs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-api-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-cli-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-devel-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-extra-xlators-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-fuse-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-geo-replication-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-libs-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-rdma-3.5.2-1.el7.centos.x86_64.rpm \
glusterfs-server-3.5.2-1.el7.centos.x86_64.rpm
$ sudo service glusterd start
1. First step is tuning /etc/sysconfig/iptables for IPv4 iptables firewall (service firewalld should be disabled) :-
Update /etc/sysconfig/iptables on both nodes:-
-A INPUT -p tcp -m multiport --dport 24007:24047 -j ACCEPT
-A INPUT -p tcp --dport 111 -j ACCEPT
-A INPUT -p udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m multiport --dport 38465:38485 -j ACCEPT
Comment out lines bellow , ignoring instruction
# -A FORWARD -j REJECT --reject-with icmp-host-prohibited
# -A INPUT -j REJECT --reject-with icmp-host-prohibited
Restart service iptables on both nodes
2. Second step:-
On juno1, run the following commands :
# ssh-keygen (Hit Enter to accept all of the defaults)
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@juno2
On both nodes run :-
# ./install
# service glusterd start
On juno1
#gluster peer probe juno2.localdomain
Should return "success"
[root@juno1 ~(keystone_admin)]# gluster peer status
Number of Peers: 1
Hostname: juno2.localdomain
Uuid: 3ca6490b-c44a-4601-ac13-51fec99e9caf
State: Peer in Cluster (Connected)
[root@juno1 ~(keystone_admin)]# ssh 192.168.1.137
Last login: Thu Aug 14 17:53:41 2014
[root@juno2 ~]# gluster peer status
Number of Peers: 1
Hostname: 192.168.1.127
Uuid: 051e7528-8c2b-46e1-abb6-6d84b2f2e45b
State: Peer in Cluster (Connected)
*************************************************************************
On Controller (192.168.1.127) and on Compute (192.168.1.137)
*************************************************************************
Verify ports availability:-
[root@juno1 ~(keystone_admin)]# netstat -lntp | grep gluster
tcp 0 0 0.0.0.0:49152 0.0.0.0:* LISTEN 5453/glusterfsd
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38465 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38466 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38468 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:38469 0.0.0.0:* LISTEN 5458/glusterfs
tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 2667/glusterd
tcp 0 0 0.0.0.0:978 0.0.0.0:* LISTEN 5458/glusterfs
************************************
Switching Cinder to Gluster volume
************************************
# gluster volume create cinder-volumes57 \
replica 2 juno1.localdomain:/data5/data-volumes juno2.localdomain:/data5/data-volumes
# gluster volume start cinder-volumes57
# gluster volume set cinder-volumes57 auth.allow 192.168.1.*
The following configuration changes are necessary for 'qemu' and '
samba vfs plugin' integration with libgfapi to work seamlessly:
1. First step
gluster volume set cinder-volumes57 server.allow-insecure on
2. Restarting is required
gluster volume stop cinder-volumes57
gluster volume start cinder-volumes57
3. Edit /etc/glusterfs/glusterd.vol to have a line :
option rpc-auth-allow-insecure on
4. Restart glusterd is required :
service glusterd restart
Nova.conf (on Compute Node) should have entry :-
qemu_allowed_storage_drivers = gluster
[root@juno1 ~]# gluster volume info
Volume Name: cinder-volumes57
Type: Replicate
Volume ID: c1f2e1d2-0b11-426e-af3d-7af0d1d24d5e
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: juno1.localdomain:/data5/data-volumes
Brick2: juno2.localdomain:/data5/data-volumes
Options Reconfigured:
auth.allow: 192.168.1.*
server.allow-insecure: on
[root@juno1 ~]# gluster volume status
Status of volume: cinder-volumes57
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick juno1.localdomain:/data5/data-volumes 49152 Y 3346
Brick juno2.localdomain:/data5/data-volumes 49152 Y 3113
NFS Server on localhost 2049 Y 3380
Self-heal Daemon on localhost N/A Y 3387
NFS Server on juno2.localdomain 2049 Y 3911
Self-heal Daemon on juno2.localdomain N/A Y 3916
Task Status of Volume cinder-volumes57
------------------------------------------------------------------------------
There are no active volume tasks
##############################
Create entries in /etc/cinder/cinder.conf
##############################
enabled_backends=gluster
[gluster]
volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config = /etc/cinder/shares.conf
glusterfs_mount_point_base = /var/lib/cinder/volumes
volume_backend_name=GLUSTER
# vi /etc/cinder/shares.conf
192.168.1.127:/cinder-volumes57
:wq
[root@juno1 ~(keystone_admin)]# cinder type-create gluster
+--------------------------------------+---------+
| ID | Name |
+--------------------------------------+---------+
| 29917269-d73f-4c28-b295-59bfbda5d044 | gluster |
+--------------------------------------+---------+
[root@juno1 ~(keystone_admin)]# cinder type-key gluster set volume_backend_name=GLUSTER
Next step is cinder services restart :-
[root@juno1 ~(keystone_demo)]# for i in api scheduler volume ; do service openstack-cinder-${i} restart ; done
[root@juno1 ~(keystone_admin)]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos01-root00 147G 43G 105G 29% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 152K 3.9G 1% /dev/shm
tmpfs 3.9G 26M 3.8G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 1.9G 6.0M 1.7G 1% /srv/node/swift_loopback
/dev/sda3 477M 146M 302M 33% /boot
/dev/mapper/centos01-data5 98G 15G 83G 16% /data5
192.168.1.127:/cinder-volumes57 98G 15G 83G 16% /var/lib/cinder/volumes/8478b56ad61cf67ab9839fb0a5296965
tmpfs 3.9G 26M 3.8G 1% /run/netns
###################################################
How to verify implementation success. Boot nova instance
( with instance-id say 00000049) based on cinder volume.
###################################################
On Compute Node grep /var/log/libvirt/qemu/instance-00000049.log looking for
"gluster" entry . You are supposed to find a string highlighted down here
# cd /var/log/libvirt/qemu
# [root@juno2 qemu]# cat instance-00000049.log | grep gluster
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00000049 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn,+osxsave,+xsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 92151b16-c7b4-48d1-b49f-1e310e005c80 -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2014.2-2.el7.centos,serial=5dff0de4-c27d-453d-85b4-b2d9af514fcd,uuid=92151b16-c7b4-48d1-b49f-1e310e005c80 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000049.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://192.168.1.127:24007/cinder-volumes57/volume-179b9782-d2b7-4891-ba89-5198b71c6188,if=none,id=drive-virtio-disk0,format=raw,serial=179b9782-d2b7-4891-ba89-5198b71c6188,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:8b:9f:6c,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/92151b16-c7b4-48d1-b49f-1e310e005c80/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
At the same time issue on Controller following commands :-
[root@juno1 ~(keystone_boris)]# cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
| 179b9782-d2b7-4891-ba89-5198b71c6188 | in-use | Win2012GLSVOL01 | 20 | gluster | true | 92151b16-c7b4-48d1-b49f-1e310e005c80 |
| ca0694ae-7e8d-4c84-aad8-3f178416dec6 | in-use | VF20LVG520711 | 7 | lvms | true | 51a20959-0a0c-4ef6-81ec-2edeab6e3588 |
+--------------------------------------+--------+-----------------+------+-------------+----------+--------------------------------------+
[root@juno1 ~(keystone_boris)]# nova list
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711 | SUSPENDED | - | Shutdown | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05 | SUSPENDED | - | Shutdown | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+--------------+-----------+------------+-------------+----------------------------------------+
[root@juno1 ~(keystone_boris)]# nova show 92151b16-c7b4-48d1-b49f-1e310e005c80 | grep 179b9782-d2b7-4891-ba89-5198b71c6188
| os-extended-volumes:volumes_attached | [{"id": "179b9782-d2b7-4891-ba89-5198b71c6188"}] |
##############################################
Another way of verification - run on Compute Node:-
##############################################
[root@juno1 ~(keystone_boris)]# ssh 192.168.1.137
Last login: Tue Nov 11 17:12:04 2014 from juno1.localdomain[root@juno2 ~]# . keystonerc_boris
[root@juno2 ~(keystone_boris)]# nova list
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
| 57640068-3ab7-466a-8eae-cf132359b233 | UbuntuUTRX1211 | ACTIVE | - | Running | private_boris=50.0.0.26, 192.168.1.174 |
| 51a20959-0a0c-4ef6-81ec-2edeab6e3588 | VF20RX520711 | SUSPENDED | - | Shutdown | private_boris=50.0.0.12, 192.168.1.175 |
| 92151b16-c7b4-48d1-b49f-1e310e005c80 | Win2012SRV05 | SUSPENDED | - | Shutdown | private_boris=50.0.0.25, 192.168.1.179 |
+--------------------------------------+----------------+-----------+------------+-------------+----------------------------------------+
[root@juno2 ~(keystone_boris)]# virsh dumpxml 57640068-3ab7-466a-8eae-cf132359b233 | grep -E 'source (file|protocol)'
[root@juno2 ~(keystone_boris)]# nova show 57640068-3ab7-466a-
8eae-cf132359b233 | grep bf448475-50c8-4491-92aa-77d36666f296
| os-extended-volumes:volumes_attached | [{"id": "bf448475-50c8-4491-92aa-77d36666f296"}] |
