Description of problem: Red Hat Storage nodes are added to a 'Gluster Enabled Cluster' on a POSIX compliant FS Data Center, and a volume is created, optimised for 'virt store', and started. On attempting to add this volume as Storage Domain to the Data Center, the operation fails. The entry in the vdsm logs, from the Hypervisor host, for the event is given below. ---------------------------------------------- Thread-1210::INFO::2013-08-01 21:35:51,968::logUtils::40::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'lizzie.lab.eng.blr.redhat.com:/ice', 'iqn': '', 'portal': '', 'user': '', 'vfs_type': 'glusterfs', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000'}], options=None) Thread-1210::DEBUG::2013-08-01 21:35:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs lizzie.lab.eng.blr.redhat.com:/ice /rhev/data-center/mnt/lizzie.lab.eng.blr.redhat.com:_ice' (cwd None) Thread-1210::ERROR::2013-08-01 21:35:52,103::hsm::2298::Storage.HSM::(connectStorageServer) Could not connect to storageServer Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2295, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 208, in connect fileSD.validateDirAccess(self.getMountObj().getRecord().fs_file) File "/usr/share/vdsm/storage/mount.py", line 244, in getRecord (self.fs_spec, self.fs_file)) OSError: [Errno 2] Mount of `lizzie.lab.eng.blr.redhat.com:/ice` at `/rhev/data-center/mnt/lizzie.lab.eng.blr.redhat.com:_ice` does not exist ---------------------------------------------- Some digging into the issue seems to suggest, that the firewall at the RHS server is blocking the mount access. This should have been opened during the boot-strapping of the RHS server, while being added to the 'Gluster Enabled Cluster' The volume information at the RHS server: ---------------------------------------------- [root@lizzie ~]# gluster volume info Volume Name: ice Type: Replicate Volume ID: 18120203-4f1a-4103-88a2-1c9adf52e408 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: lizzie.lab.eng.blr.redhat.com:/rhs/brick1/ice Brick2: mack.lab.eng.blr.redhat.com:/rhs/brick2/ice Options Reconfigured: storage.owner-gid: 36 storage.owner-uid: 36 network.remote-dio: enable cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: on nfs.disable: off [root@lizzie ~]# [root@lizzie ~]# [root@lizzie ~]# [root@lizzie ~]# gluster volume status Status of volume: ice Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick lizzie.lab.eng.blr.redhat.com:/rhs/brick1/ice 49152 Y 11278 Brick mack.lab.eng.blr.redhat.com:/rhs/brick2/ice 49152 Y 8397 NFS Server on localhost 2049 Y 11290 Self-heal Daemon on localhost N/A Y 11297 NFS Server on mack.lab.eng.blr.redhat.com 2049 Y 8409 Self-heal Daemon on mack.lab.eng.blr.redhat.com N/A Y 8416 There are no active volume tasks [root@lizzie ~]# ---------------------------------------------- The network and firewall settings at the RHS server: ---------------------------------------------- [root@lizzie ~]# ifconfig eth0 Link encap:Ethernet HWaddr 52:54:00:33:54:84 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15874 errors:0 dropped:0 overruns:0 frame:0 TX packets:3676 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1300589 (1.2 MiB) TX bytes:1129648 (1.0 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:4480 errors:0 dropped:0 overruns:0 frame:0 TX packets:4480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:420130 (410.2 KiB) TX bytes:420130 (410.2 KiB) rhevm Link encap:Ethernet HWaddr 52:54:00:33:54:84 inet addr:10.70.34.106 Bcast:10.70.35.255 Mask:255.255.254.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15734 errors:0 dropped:0 overruns:0 frame:0 TX packets:3675 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1071863 (1.0 MiB) TX bytes:1129630 (1.0 MiB) [root@lizzie ~]# [root@lizzie ~]# iptables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 8223 841K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 916 104K ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 487 29220 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:161 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:24007 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:111 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38465 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38466 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38467 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:39543 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:55863 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38468 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:963 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:965 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4379 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:139 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:445 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:24009:24108 143 19811 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 8263 packets, 1524K bytes) pkts bytes target prot opt in out source destination [root@lizzie ~]# ---------------------------------------------- For testing, a bypass rule was added to the firewall, at top of the INPUT chain, to allow all incoming packets. ---------------------------------------------- [root@lizzie ~]# iptables -I INPUT 1 -j ACCEPT ---------------------------------------------- Then the volume from the RHS server cluster was added as Storage Domain without any issue. Version-Release number of selected component (if applicable): Red Hat Enterprise Virtualization Manager Version: 3.2.2-0.41.el6ev How reproducible: Steps to Reproduce: 1.Add Hypervisor node(s) to 'Virt Enabled Cluster' on a POSIX compliant FS Data Center 2.Add RHS nodes to a separate 'Gluster Enabled Cluster' on the Data Center 2.Create a volume, optimize it for 'virt store', and start the volume. 3.Try to add the volume as Storage Domain to the Data Center Actual results: Volume created for VM Image Store, on Red Hat Storage nodes added to 'Gluster Enabled Cluster', cannot be added as Storage Domain, to POSIX compliant FS Data Center, possibly due to firewall block. Expected results: Volume created for VM Image Store, on Red Hat Storage nodes added to 'Gluster Enabled Cluster', should be added as Storage Domain, to POSIX compliant FS Data Center successfully. The boot-strapping process of the RHS server, when it is added to the 'Gluster Enabled Cluster', should ensure that all the required ports are opened. Additional info:
For RHS 2.1 nodes, we need to open these ports too - -A INPUT -p tcp -m tcp --dport 49152:49251 -j ACCEPT and port 2049 for NFS
The ports required for RHS 2.1 are opened by RHEV-M 3.3 during the bootstrap process
Verified on RHEVM 3.3 IS11 build Firewall rules after boot-strapping of RHS node: ----------------------------------------------- # iptables -nvL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 12035 1247K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 1395 164K ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 729 43740 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321 1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:161 2 120 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:24007 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:111 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38465 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38466 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:111 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38467 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2049 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:39543 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:55863 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:38468 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:963 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:965 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4379 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:139 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:445 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:24009:24108 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:49152:49251 211 27404 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT 12128 packets, 2312K bytes) pkts bytes target prot opt in out source destination ----------------------------------------------- Ports 2049 (tcp), 111 (tcp), and 49152:49251 (tcp) are the new ones to be opened. Of these, 111(tcp) was not mentioned previously in this BZ.
Closing - RHEV 3.3 Released