Description of problem: ******************************** After updating the system to latest glusterfs and few other packages the following tests failed from the swift test suite for GSWAUTH: testAddAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testDeleteAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testListAcounts (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testSwauthPrep (test.functional_auth.gswauth.test_gswauth_cli.TestSwauthPrep) ... FAIL testDeleteUser (test.functional_auth.gswauth.test_gswauth_cli.TestUser) ... FAIL testaddUser (test.functional_auth.gswauth.test_gswauth_cli.TestUser) ... FAIL ***************************************************************************** ====================================================================== FAIL: testaddUser (test.functional_auth.gswauth.test_gswauth_cli.TestUser) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/swiftonfile/test/functional_auth/gswauth/test_gswauth_cli.py", line 380, in testaddUser 'Invalid add user request accepted(wrong admin-url provided): %s' % output) AssertionError: Invalid add user request accepted(wrong admin-url provided): Traceback (most recent call last): File "/usr/bin/gswauth-add-user", line 93, in <module> except socket.error, (errno, msg): ValueError: need more than 1 value to unpack ------------------------------------------------- The same tests passed for tempauth. The packages that got update are as follows: *************************************************** Updated: bind-libs.x86_64 32:9.9.4-29.el7_2.2 bind-libs-lite.x86_64 32:9.9.4-29.el7_2.2 bind-license.noarch 32:9.9.4-29.el7_2.2 bind-utils.x86_64 32:9.9.4-29.el7_2.2 glusterfs.x86_64 0:3.7.5-17.el7rhgs glusterfs-api.x86_64 0:3.7.5-17.el7rhgs glusterfs-cli.x86_64 0:3.7.5-17.el7rhgs glusterfs-client-xlators.x86_64 0:3.7.5-17.el7rhgs glusterfs-fuse.x86_64 0:3.7.5-17.el7rhgs glusterfs-geo-replication.x86_64 0:3.7.5-17.el7rhgs glusterfs-libs.x86_64 0:3.7.5-17.el7rhgs glusterfs-rdma.x86_64 0:3.7.5-17.el7rhgs glusterfs-server.x86_64 0:3.7.5-17.el7rhgs gnutls.x86_64 0:3.3.8-14.el7_2 gnutls-dane.x86_64 0:3.3.8-14.el7_2 gnutls-utils.x86_64 0:3.3.8-14.el7_2 grub2.x86_64 1:2.02-0.34.el7_2 grub2-efi.x86_64 1:2.02-0.34.el7_2 grub2-tools.x86_64 1:2.02-0.34.el7_2 ioprocess.x86_64 0:0.15.0-4.el7 libmcrypt.x86_64 0:2.5.8-13.el7 nagios-common.x86_64 0:4.0.8-2.el7 nagios-plugins.x86_64 0:2.0.3-3.el7 nagios-plugins-ide_smart.x86_64 0:2.0.3-3.el7 nagios-plugins-procs.x86_64 0:2.0.3-3.el7 nrpe.x86_64 0:2.15-7.el7 nss.x86_64 0:3.19.1-19.el7_2 nss-sysinit.x86_64 0:3.19.1-19.el7_2 nss-tools.x86_64 0:3.19.1-19.el7_2 ntp.x86_64 0:4.2.6p5-22.el7_2.1 ntpdate.x86_64 0:4.2.6p5-22.el7_2.1 openssh.x86_64 0:6.6.1p1-23.el7_2 openssh-clients.x86_64 0:6.6.1p1-23.el7_2 openssh-server.x86_64 0:6.6.1p1-23.el7_2 openssl.x86_64 1:1.0.1e-51.el7_2.2 openssl-libs.x86_64 1:1.0.1e-51.el7_2.2 python-cpopen.x86_64 0:1.4-0.el7 python-eventlet.noarch 0:0.15.2-1.el7 python-gluster.noarch 0:3.7.5-17.el7rhgs python-ioprocess.noarch 0:0.15.0-4.el7 python-iso8601.noarch 0:0.1.10-1.el7 python-keyring.noarch 0:5.0-1.el7 python-perf.x86_64 0:3.10.0-327.4.5.el7 rpcbind.x86_64 0:0.2.0-33.el7_2 userspace-rcu.x86_64 0:0.7.16-1.el7 Version-Release number of selected component (if applicable): **************************************** rpm -qa | grep glusterfs glusterfs-libs-3.7.5-17.el7rhgs.x86_64 glusterfs-rdma-3.7.5-17.el7rhgs.x86_64 glusterfs-cli-3.7.5-17.el7rhgs.x86_64 samba-vfs-glusterfs-4.2.4-12.el7rhgs.x86_64 glusterfs-client-xlators-3.7.5-17.el7rhgs.x86_64 glusterfs-server-3.7.5-17.el7rhgs.x86_64 glusterfs-api-3.7.5-17.el7rhgs.x86_64 glusterfs-geo-replication-3.7.5-17.el7rhgs.x86_64 glusterfs-fuse-3.7.5-17.el7rhgs.x86_64 glusterfs-3.7.5-17.el7rhgs.x86_64 How reproducible: ************************ Always Steps to Reproduce: 1. Do the setup required to run swift test suite 2. start tests with tempauth and gswauth 3. Update the system 4. again run the test suite. Actual results: ******************** Tests with user account Add,delete,list user delete user add with gswauth failed. testAddAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testDeleteAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testListAcounts (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... FAIL testSwauthPrep (test.functional_auth.gswauth.test_gswauth_cli.TestSwauthPrep) ... FAIL testDeleteUser (test.functional_auth.gswauth.test_gswauth_cli.TestUser) ... FAIL testaddUser (test.functional_auth.gswauth.test_gswauth_cli.TestUser) ... FAIL Expected results: All the tests should passed. Additional info: ******************** The same tests passed with tempauth. Also before updating the above mentioned packages all tempauth and gswauth cases passed.
This minor regression was probably introduced with update of "python-eventlet" package. The code that catches "socket.error" exception raised by "eventlet.green.httplib.HTTPConnection.getresponse" is trying to unpack the exception assuming that it's a tuple. Since it isn't a tuple and is a single value, you get an error. The change that introduced this code is this: http://review.gluster.org/#/c/6431/
The package shipped in RHGS 3.1.2 ISO is "python-eventlet.noarch 0:0.14.0-1.el7ost" which is compatible with gluster-swift code. When EPEL repo (we pull packages required for running tests) is installed and then a "yum update" is run, python-eventlet will be updated to "python-eventlet.noarch 0:0.15.2-1.el7" which is incompatible. In production environment, I'm not sure if customers will enable and use EPEL repo. This incompatible behavior, although not critical, can be easily fixed as a precautionary measure.
Prashanth, I tested with RHGS 3.1.2 RC and I could able to see non ost package for python-eventlet - python-eventlet-0.14.0-1.el6.noarch and I see errors after upgrading from RHGS 3.1.1 (based on RHEL 6 ) to RHGS 3.1.2 (based on RHEL6) [root@]# gswauth-list -A http://127.0.0.1:8080/auth -K redhat123 saltsugar List failed: 500 Server Error [root@]# gswauth-add-account -A http://127.0.0.1:8080/auth -K redhat123 saltsugar Account creation failed: 500 Server Error
@Satheesaran: Could you please post the content of logs at /var/log/messages when you do these operations ?
(In reply to Prashanth Pai from comment #5) > @Satheesaran: Could you please post the content of logs at /var/log/messages > when you do these operations ? <snip> Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting to memcached: localhost:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached. py", line 239, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in get#012 fp, sock = self._parent_class_gette r()#012 File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get#012 created = self.create()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in connect#012 while not socket_connect(fd, address):#012 File "/usr/lib/python2.6/site-package s/eventlet/greenio.py", line 39, in socket_connect#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 13] EACCES (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting to memcached: localhost:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached. py", line 239, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in connect#012 while not socket_connect(fd, address):#012 File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 39, in socket_connect#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 13] E ACCES (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 account-server: STDOUT: ERROR:root:Getting volume info failed, make sure to have passwordless ssh on localhost (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 account-server: STDOUT: ERROR:root:No export found in [] matching drive, gsmetadata (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 proxy-server: ERROR Insufficient Storage 127.0.0.1:6012/gsmetadata (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 proxy-server: Account HEAD returning 503 for [507] (txn: txa8c55edc565545a9b80d3-0056d16bbb) Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting to memcached: localhost:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached. py", line 239, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in connect#012 while not socket_connect(fd, address):#012 File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 39, in socket_connect#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 13] EACCES (txn: txa8c55edc565545a9b80d3-0056d16bbb) </snip>
Created attachment 1131069 [details] var-log-messages from the RHGS node1
Prashanth, I tested gswauth in RHGS 3.1.2 based on RHEL 6 and it doesn't seem to work I got the same errors as below : [root@]# gswauth-list -A http://127.0.0.1:8080/auth -K redhat123 vol1 List failed: 500 Server Error [root@]# gswauth-add-account -A http://127.0.0.1:8080/auth -K redhat123 vol1 Account creation failed: 500 Server Error For your information, I have removed the existing python-eventlet ( python-eventlet-0.14.0-1.el6.noarch ) and installed the ost flavour of the same (python-eventlet-0.14.0-3.el6ost.noarch.rpm ) as available in brew[1] [1] - https://brewweb.devel.redhat.com/buildinfo?buildID=448441 gswauth account creation, user addition, account-listing, user-listing, user authentication works now <snip> [root@]# gswauth-add-account vol1 -A http://10.70.37.64:8080/auth -K redhat123 [root@]# gswauth-list -A http://10.70.37.64:8080/auth -K redhat123 +----------+ | Accounts | +----------+ | vol1 | +----------+ [root@ ~]# gswauth-add-user -A http://10.70.37.64:8080/auth -K redhat123 -a vol1 satheesaran redhat789 [root@ ~]# gswauth-list -A http://10.70.37.64:8080/auth -K redhat123 vol1 +-------------+ | Users | +-------------+ | satheesaran | +-------------+ [root@ ~]# curl -v -H 'X-Auth-User: vol1:satheesaran' -H 'X-Auth-key:redhat789' -k http://127.0.0.1:8080/auth/v1.0 * About to connect() to 127.0.0.1 port 8080 (#0) * Trying 127.0.0.1... connected * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET /auth/v1.0 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 127.0.0.1:8080 > Accept: */* > X-Auth-User: vol1:satheesaran > X-Auth-key:redhat789 > < HTTP/1.1 200 OK < Content-Length: 80 < X-Auth-Token-Expires: 86399 < X-Auth-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111 < X-Storage-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111 < X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_vol1 < Content-Type: text/html; charset=UTF-8 < X-Trans-Id: tx969d736b6c31478a8b233-0056d1dbdf < Date: Sat, 27 Feb 2016 17:24:47 GMT < * Connection #0 to host 127.0.0.1 left intact * Closing connection #0 {"storage": {"default": "local", "local": "http://127.0.0.1:8080/v1/AUTH_vol1"}} [root@ ~]# curl -v -X PUT -H 'X-Auth-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111' http://127.0.0.1:8080/v1/AUTH_vol1/cont1 * About to connect() to 127.0.0.1 port 8080 (#0) * Trying 127.0.0.1... connected * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > PUT /v1/AUTH_vol1/cont1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 127.0.0.1:8080 > Accept: */* > X-Auth-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111 > < HTTP/1.1 201 Created < Content-Length: 0 < Content-Type: text/html; charset=UTF-8 < X-Trans-Id: tx81da22af36ec45fa9efad-0056d1dc23 < Date: Sat, 27 Feb 2016 17:25:55 GMT < * Connection #0 to host 127.0.0.1 left intact * Closing connection #0 [root@dhcp37-64 ~]# curl -v -X GET -H 'X-Auth-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111' http://127.0.0.1:8080/v1/AUTH_vol1 * About to connect() to 127.0.0.1 port 8080 (#0) * Trying 127.0.0.1... connected * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET /v1/AUTH_vol1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 127.0.0.1:8080 > Accept: */* > X-Auth-Token: AUTH_tk9c1a1ec1aed2459e98a3681f6b8f8111 > < HTTP/1.1 200 OK < Content-Length: 6 < X-Account-Container-Count: 0 < Accept-Ranges: bytes < X-Account-Object-Count: 0 < X-Bytes-Used: 0 < X-Timestamp: 1456591302.31800 < X-Object-Count: 0 < X-Account-Bytes-Used: 0 < X-Type: Account < Content-Type: text/plain; charset=utf-8 < X-Container-Count: 0 < X-Trans-Id: tx9d22a475d9e54d6abe6b2-0056d1dc28 < Date: Sat, 27 Feb 2016 17:26:00 GMT < cont1 * Connection #0 to host 127.0.0.1 left intact * Closing connection #0 </snip>
(In reply to SATHEESARAN from comment #6) > (In reply to Prashanth Pai from comment #5) > > @Satheesaran: Could you please post the content of logs at /var/log/messages > > when you do these operations ? > > <snip> > Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting > to memcached: localhost:11211#012Traceback (most recent call last):#012 > File "/usr/lib/python2.6/site-packages/swift/common/memcached. > py", line 239, in _get_conns#012 fp, sock = > self._client_cache[server].get()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 132, in > get#012 fp, sock = self._parent_class_gette > r()#012 File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line > 119, in get#012 created = self.create()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in > create#012 > sock.connect((host, int(port)))#012 File > "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in > connect#012 while not socket_connect(fd, address):#012 File > "/usr/lib/python2.6/site-package > s/eventlet/greenio.py", line 39, in socket_connect#012 raise > socket.error(err, errno.errorcode[err])#012error: [Errno 13] EACCES (txn: > txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting > to memcached: localhost:11211#012Traceback (most recent call last):#012 > File "/usr/lib/python2.6/site-packages/swift/common/memcached. > py", line 239, in _get_conns#012 fp, sock = > self._client_cache[server].get()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in > get#012 fp, sock = self.create()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in > create#012 sock.connect((host, int(port)))#012 File > "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in > connect#012 > while not socket_connect(fd, address):#012 File > "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 39, in > socket_connect#012 raise socket.error(err, > errno.errorcode[err])#012error: [Errno 13] E > ACCES (txn: txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 account-server: STDOUT: ERROR:root:Getting volume > info failed, make sure to have passwordless ssh on localhost (txn: > txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 account-server: STDOUT: ERROR:root:No export found > in [] matching drive, gsmetadata (txn: txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 proxy-server: ERROR Insufficient Storage > 127.0.0.1:6012/gsmetadata (txn: txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 proxy-server: Account HEAD returning 503 for [507] > (txn: txa8c55edc565545a9b80d3-0056d16bbb) > Feb 27 04:26:19 dhcp37-61 proxy-server: STDOUT: ERROR:root:Error connecting > to memcached: localhost:11211#012Traceback (most recent call last):#012 > File "/usr/lib/python2.6/site-packages/swift/common/memcached. > py", line 239, in _get_conns#012 fp, sock = > self._client_cache[server].get()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 135, in > get#012 fp, sock = self.create()#012 File > "/usr/lib/python2.6/site-packages/swift/common/memcached.py", line 128, in > create#012 sock.connect((host, int(port)))#012 File > "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 190, in > connect#012 while not socket_connect(fd, address):#012 File > "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 39, in > socket_connect#012 raise socket.error(err, > errno.errorcode[err])#012error: [Errno 13] EACCES (txn: > txa8c55edc565545a9b80d3-0056d16bbb) > > </snip> Apologies for escalating this issue as it turns out to be manual error. I have removed fs.conf file from /etc/swift accidentally and that has caused all this problems. All works well with RHGS 3.1.2 based on RHEL 6
After upgrading the rpm's to swiftonfile-2.3.0-2.el7rhgs.noarch and running the regression test suite, all the tests passed. testAddAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testAddAccountNonSuperAdminUsers (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testDeleteAccount (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testDeleteAccountNonSuperAdminUsers (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testListAcounts (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testListAcountsNonSuperAdminUsers (test.functional_auth.gswauth.test_gswauth_cli.TestAccount) ... ok testCleanUPToken (test.functional_auth.gswauth.test_gswauth_cli.TestCleanUPToken) ... ok Marking the BZ to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2016:1289