Bug 924792 - Gluster-swift does not allow operations on multiple volumes concurrently.
Summary: Gluster-swift does not allow operations on multiple volumes concurrently.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: object-storage
Version: 3.4.0-alpha
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: crisbud@redhat.com
QA Contact: pushpesh sharma
URL:
Whiteboard:
Depends On:
Blocks: 909053
TreeView+ depends on / blocked
 
Reported: 2013-03-22 14:10 UTC by Alex Wheeler
Modified: 2014-08-04 22:31 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.5.0
Doc Type: Bug Fix
Doc Text:
Clone Of: 909053
Environment:
Last Closed: 2014-04-17 11:41:08 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions: Not reproducible on latest upstream G4S.(Havana based)


Attachments (Terms of Use)
This patch adds the missing functions to ring.py (2.39 KB, patch)
2013-03-27 23:20 UTC, Alex Wheeler
no flags Details | Diff
Updates the test_ring.py to test for this situation (3.60 KB, patch)
2013-03-28 01:00 UTC, Alex Wheeler
no flags Details | Diff
Added more test coverage (3.71 KB, patch)
2013-03-28 01:41 UTC, Alex Wheeler
no flags Details | Diff
Re-factored, and corrects the return type for get_more_nodes (4.08 KB, patch)
2013-03-29 14:26 UTC, Alex Wheeler
no flags Details | Diff
Broke the test cases out to multiple, meaningfully named tests (4.92 KB, patch)
2013-03-29 14:49 UTC, Alex Wheeler
no flags Details | Diff

Description Alex Wheeler 2013-03-22 14:10:55 UTC
+++ This bug was initially created as a clone of Bug #909053 +++

Description of problem:
At any given point of time, users can only perform operations on only one gluster volume. To use the other volume, the ring files have to be recreated.

Version-Release number of selected component (if applicable):
Master

How reproducible:
Always

Steps to Reproduce:
1. Try to do operations on two different accounts at the same time.
2. Operations on one the account's will fail.
  
Actual results:
Operations on one the account's will fail.

Expected results:
Operations on both the account should complete successfully.

Additional info:

--- Additional comment from Scott Haines on 2013-02-13 17:19:01 EST ---

Per Feb-13 bug triage meeting, targeting for 2.1.0.

--- Additional comment from Vijay Bellur on 2013-03-07 03:55:49 EST ---

CHANGE: http://review.gluster.org/4485 (object-storage: Restoring multi volume support in UFO.) merged in master by Vijay Bellur (vbellur@redhat.com)

--- Additional comment from Junaid on 2013-03-14 04:52:47 EDT ---

Fixed in upstream.

--- Additional comment from Alex Wheeler on 2013-03-19 08:48:03 EDT ---

This only partially corrects the problem.  When this patch is applied against 3.4 Alpha 1, and against 3.3.1-11, everything works fine with just one volume.
With a second volume, all works well when just referencing the first volume, but when referencing the second volume, the container-server requests are against the wrong volume.  Here's a snippet of output from my storage1.log file:
Mar 18 22:01:03 r3master account-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "HEAD /test/0/AUTH_test" 204 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0114 ""
Mar 18 22:01:03 r3master account-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "PUT /test/0/AUTH_test/test.txt" 201 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0036 ""
Mar 18 22:01:03 r3master container-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "PUT /admin/0/AUTH_test/test.txt" 202 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0147

As you can see, the container-server PUT is trying to access the admin volume, instead of the test volume.

Comment 1 Junaid 2013-03-25 06:44:52 UTC
Hi Alex,

If you read the commit message there, to enable the volumes to be exposed by UFO the users have to rebuild the ring files by running gluster-swift-gen-builders with the list of volume names.

   $ gluster-swift-gen-builders <vol-name1> [<vol-name2>]...

Did you try this out?

Comment 2 Alex Wheeler 2013-03-25 12:54:43 UTC
Yes I did, and in fact I had to do that to get this far, otherwise I get the 'No export found in %r matching drive, %s', el, drive' error.

Looking at the snippet above, there are three requests logged.
The first, a HEAD, correctly sets the account as /test/.
The second, a PUT, also correctly set the account as /test/.
For the third though, something has gone wrong, and rather than continue to use the correct volume it is converted to the first volume that was passed in the gluster-swift-gen-builders.

So, for this sample, the command, executed in /etc/swift, was:
gluster-swift-gen-builders admin test

When I invert the order, as gluster-swift-gen-builders test admin, and try a PUT on the admin volume, I get the same behavior as seen above, where the first two requests are for admin, and the last for test.

Comment 3 Alex Wheeler 2013-03-27 23:20:38 UTC
Created attachment 717356 [details]
This patch adds the missing functions to ring.py

For the container-server, the function get_part_nodes was being called instead of get_nodes.  As the gluster version of ring.py didn't implement that, the swift version was called, and was returning the wrong value.

Comment 4 Alex Wheeler 2013-03-28 01:00:11 UTC
Created attachment 717381 [details]
Updates the test_ring.py to test for this situation

Comment 5 Alex Wheeler 2013-03-28 01:41:39 UTC
Created attachment 717392 [details]
Added more test coverage

Comment 6 Alex Wheeler 2013-03-29 14:26:52 UTC
Created attachment 718091 [details]
Re-factored, and corrects the return type for get_more_nodes

I'll be submitting this patch to Gerrit shortly.

Comment 7 Alex Wheeler 2013-03-29 14:49:32 UTC
Created attachment 718100 [details]
Broke the test cases out to multiple, meaningfully named tests

Comment 8 Vijay Bellur 2013-03-29 15:04:33 UTC
REVIEW: http://review.gluster.org/4748 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on master by Alex Wheeler (wheelear@gmail.com)

Comment 9 Anand Avati 2013-04-19 17:54:38 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 10 Anand Avati 2013-04-19 18:52:04 UTC
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#2) for review on release-3.4 by Kaleb KEITHLEY (kkeithle@redhat.com)

Comment 11 Anand Avati 2013-04-19 19:22:36 UTC
REVIEW: http://review.gluster.org/4860 (object-storage: Restoring multi volume support in UFO.) posted (#1) for review on release-3.3 by Alex Wheeler (wheelear@gmail.com)

Comment 12 Anand Avati 2013-04-19 19:22:54 UTC
REVIEW: http://review.gluster.org/4861 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on release-3.3 by Alex Wheeler (wheelear@gmail.com)

Comment 13 Anand Avati 2013-04-19 20:08:29 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#2) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 14 Anand Avati 2013-04-19 20:37:55 UTC
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#4) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 15 Anand Avati 2013-04-19 20:47:20 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#3) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 16 Anand Avati 2013-04-22 14:52:37 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#4) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 17 Anand Avati 2013-04-29 16:08:15 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#5) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 18 Anand Avati 2013-04-29 16:08:35 UTC
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#5) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 19 Anand Avati 2013-05-03 17:16:40 UTC
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#6) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 20 Anand Avati 2013-05-03 17:17:00 UTC
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#6) for review on release-3.4 by Alex Wheeler (wheelear@gmail.com)

Comment 22 crisbud@redhat.com 2014-01-31 07:00:54 UTC
Just doing the book keeping part here  : 

patch is merged in master in ufo repository. 
http://review.gluster.org/#/c/4748/ 

This needs to be reverified with HAVANA based code and updated. 

Thanks,
Chetan Risbud.

Comment 23 pushpesh sharma 2014-01-31 10:08:29 UTC
Not reproducible on latest upstream G4S.(Havana based)

Comment 24 Niels de Vos 2014-04-17 11:41:08 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report.

glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.