Bug 991145 - [RHEV+RHS] Volume created for VM Image Store, on Red Hat Storage nodes added to 'Gluster Enabled Cluster', cannot be added as Storage Domain, to POSIX compliant FS Data Center, possibly due to firewall block
[RHEV+RHS] Volume created for VM Image Store, on Red Hat Storage nodes added ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-webadmin-portal (Show other bugs)
3.2.0
All Linux
high Severity high
: ---
: 3.3.0
Assigned To: Shubhendu Tripathi
Rejy M Cyriac
gluster
: Regression, ZStream
Depends On:
Blocks: 993014
  Show dependency treegraph
 
Reported: 2013-08-01 13:36 EDT by Rejy M Cyriac
Modified: 2016-02-10 13:59 EST (History)
10 users (show)

See Also:
Fixed In Version: is7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 993014 (view as bug list)
Environment:
virt rhev rhs integration
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Gluster
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
scohen: Triaged+


Attachments (Terms of Use)

  None (edit)
Description Rejy M Cyriac 2013-08-01 13:36:41 EDT
Description of problem:
Red Hat Storage nodes are added to a 'Gluster Enabled Cluster' on a POSIX compliant FS Data Center, and a volume is created, optimised for 'virt store', and started. On attempting to add this volume as Storage Domain to the Data Center, the operation fails.

The entry in the vdsm logs, from the Hypervisor host, for the event is given below.

----------------------------------------------

Thread-1210::INFO::2013-08-01 21:35:51,968::logUtils::40::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': 'lizzie.lab.eng.blr.redhat.com:/ice', 'iqn': '', 'portal': '', 'user': '', 'vfs_type': 'glusterfs', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000'}], options=None)
Thread-1210::DEBUG::2013-08-01 21:35:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs lizzie.lab.eng.blr.redhat.com:/ice /rhev/data-center/mnt/lizzie.lab.eng.blr.redhat.com:_ice' (cwd None)
Thread-1210::ERROR::2013-08-01 21:35:52,103::hsm::2298::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2295, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 208, in connect
    fileSD.validateDirAccess(self.getMountObj().getRecord().fs_file)
  File "/usr/share/vdsm/storage/mount.py", line 244, in getRecord
    (self.fs_spec, self.fs_file))
OSError: [Errno 2] Mount of `lizzie.lab.eng.blr.redhat.com:/ice` at `/rhev/data-center/mnt/lizzie.lab.eng.blr.redhat.com:_ice` does not exist

----------------------------------------------

Some digging into the issue seems to suggest, that the firewall at the RHS server is blocking the mount access. This should have been opened during the boot-strapping of the RHS server, while being added to the 'Gluster Enabled Cluster'

The volume information at the RHS server:

----------------------------------------------

[root@lizzie ~]# gluster volume info
 
Volume Name: ice
Type: Replicate
Volume ID: 18120203-4f1a-4103-88a2-1c9adf52e408
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: lizzie.lab.eng.blr.redhat.com:/rhs/brick1/ice
Brick2: mack.lab.eng.blr.redhat.com:/rhs/brick2/ice
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
auth.allow: *
user.cifs: on
nfs.disable: off
[root@lizzie ~]# 
[root@lizzie ~]# 
[root@lizzie ~]# 
[root@lizzie ~]# gluster volume status
Status of volume: ice
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick lizzie.lab.eng.blr.redhat.com:/rhs/brick1/ice	49152	Y	11278
Brick mack.lab.eng.blr.redhat.com:/rhs/brick2/ice	49152	Y	8397
NFS Server on localhost					2049	Y	11290
Self-heal Daemon on localhost				N/A	Y	11297
NFS Server on mack.lab.eng.blr.redhat.com		2049	Y	8409
Self-heal Daemon on mack.lab.eng.blr.redhat.com		N/A	Y	8416
 
There are no active volume tasks
[root@lizzie ~]# 

----------------------------------------------

The network and firewall settings at the RHS server:

----------------------------------------------

[root@lizzie ~]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 52:54:00:33:54:84  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15874 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3676 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1300589 (1.2 MiB)  TX bytes:1129648 (1.0 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4480 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4480 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:420130 (410.2 KiB)  TX bytes:420130 (410.2 KiB)

rhevm     Link encap:Ethernet  HWaddr 52:54:00:33:54:84  
          inet addr:10.70.34.106  Bcast:10.70.35.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15734 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3675 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1071863 (1.0 MiB)  TX bytes:1129630 (1.0 MiB)

[root@lizzie ~]# 
[root@lizzie ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 8223  841K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
  916  104K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
  487 29220 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:54321 
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:161 
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:24007 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:111 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38465 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38466 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38467 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:39543 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:55863 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38468 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:963 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:965 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:4379 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:139 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:445 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpts:24009:24108 
  143 19811 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT 8263 packets, 1524K bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@lizzie ~]# 

----------------------------------------------

For testing, a bypass rule was added to the firewall, at top of the INPUT chain, to allow all incoming packets.

----------------------------------------------

[root@lizzie ~]# iptables -I INPUT 1 -j ACCEPT

----------------------------------------------

Then the volume from the RHS server cluster was added as Storage Domain without any issue.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Manager Version: 3.2.2-0.41.el6ev 

How reproducible:


Steps to Reproduce:
1.Add Hypervisor node(s) to 'Virt Enabled Cluster' on a POSIX compliant FS Data Center
2.Add RHS nodes to a separate 'Gluster Enabled Cluster' on the Data Center
2.Create a volume, optimize it for 'virt store', and start the volume.
3.Try to add the volume as Storage Domain to the Data Center

Actual results:
Volume created for VM Image Store, on Red Hat Storage nodes added to 'Gluster Enabled Cluster',  cannot be added as Storage Domain, to POSIX compliant FS Data Center, possibly due to firewall block.

Expected results:
Volume created for VM Image Store, on Red Hat Storage nodes added to 'Gluster Enabled Cluster',  should be added as Storage Domain, to POSIX compliant FS Data Center successfully. The boot-strapping process of the RHS server, when it is added to the 'Gluster Enabled Cluster', should ensure that all the required ports are opened.

Additional info:
Comment 1 Sahina Bose 2013-08-02 06:26:45 EDT
For RHS 2.1 nodes, we need to open these ports too -
-A INPUT -p tcp -m tcp --dport 49152:49251 -j ACCEPT
and port 2049 for NFS
Comment 6 Sahina Bose 2013-08-05 07:27:50 EDT
The ports required for RHS 2.1 are opened by RHEV-M 3.3 during the bootstrap process
Comment 8 Rejy M Cyriac 2013-08-28 07:08:15 EDT
Verified on RHEVM 3.3 IS11 build

Firewall rules after boot-strapping of RHS node:

-----------------------------------------------

# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
12035 1247K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
 1395  164K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
  729 43740 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:54321 
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:161 
    2   120 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:24007 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:111 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38465 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38466 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:111 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38467 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:2049 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:39543 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:55863 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:38468 
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:963 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:965 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:4379 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:139 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:445 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpts:24009:24108 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpts:49152:49251 
  211 27404 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT 12128 packets, 2312K bytes)
 pkts bytes target     prot opt in     out     source               destination         

-----------------------------------------------

Ports 2049 (tcp), 111 (tcp), and 49152:49251 (tcp) are the new ones to be opened.

Of these, 111(tcp) was not mentioned previously in this BZ.
Comment 9 Itamar Heim 2014-01-21 17:29:55 EST
Closing - RHEV 3.3 Released
Comment 10 Itamar Heim 2014-01-21 17:29:59 EST
Closing - RHEV 3.3 Released
Comment 11 Itamar Heim 2014-01-21 17:32:56 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.