+++ This bug was initially created as a clone of Bug #909053 +++ Description of problem: At any given point of time, users can only perform operations on only one gluster volume. To use the other volume, the ring files have to be recreated. Version-Release number of selected component (if applicable): Master How reproducible: Always Steps to Reproduce: 1. Try to do operations on two different accounts at the same time. 2. Operations on one the account's will fail. Actual results: Operations on one the account's will fail. Expected results: Operations on both the account should complete successfully. Additional info: --- Additional comment from Scott Haines on 2013-02-13 17:19:01 EST --- Per Feb-13 bug triage meeting, targeting for 2.1.0. --- Additional comment from Vijay Bellur on 2013-03-07 03:55:49 EST --- CHANGE: http://review.gluster.org/4485 (object-storage: Restoring multi volume support in UFO.) merged in master by Vijay Bellur (vbellur) --- Additional comment from Junaid on 2013-03-14 04:52:47 EDT --- Fixed in upstream. --- Additional comment from Alex Wheeler on 2013-03-19 08:48:03 EDT --- This only partially corrects the problem. When this patch is applied against 3.4 Alpha 1, and against 3.3.1-11, everything works fine with just one volume. With a second volume, all works well when just referencing the first volume, but when referencing the second volume, the container-server requests are against the wrong volume. Here's a snippet of output from my storage1.log file: Mar 18 22:01:03 r3master account-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "HEAD /test/0/AUTH_test" 204 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0114 "" Mar 18 22:01:03 r3master account-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "PUT /test/0/AUTH_test/test.txt" 201 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0036 "" Mar 18 22:01:03 r3master container-server 127.0.0.1 - - [18/Mar/2013:22:01:03 +0000] "PUT /admin/0/AUTH_test/test.txt" 202 - "txe0a73e5b77fa4d6497c20b5377427388" "-" "-" 0.0147 As you can see, the container-server PUT is trying to access the admin volume, instead of the test volume.
Hi Alex, If you read the commit message there, to enable the volumes to be exposed by UFO the users have to rebuild the ring files by running gluster-swift-gen-builders with the list of volume names. $ gluster-swift-gen-builders <vol-name1> [<vol-name2>]... Did you try this out?
Yes I did, and in fact I had to do that to get this far, otherwise I get the 'No export found in %r matching drive, %s', el, drive' error. Looking at the snippet above, there are three requests logged. The first, a HEAD, correctly sets the account as /test/. The second, a PUT, also correctly set the account as /test/. For the third though, something has gone wrong, and rather than continue to use the correct volume it is converted to the first volume that was passed in the gluster-swift-gen-builders. So, for this sample, the command, executed in /etc/swift, was: gluster-swift-gen-builders admin test When I invert the order, as gluster-swift-gen-builders test admin, and try a PUT on the admin volume, I get the same behavior as seen above, where the first two requests are for admin, and the last for test.
Created attachment 717356 [details] This patch adds the missing functions to ring.py For the container-server, the function get_part_nodes was being called instead of get_nodes. As the gluster version of ring.py didn't implement that, the swift version was called, and was returning the wrong value.
Created attachment 717381 [details] Updates the test_ring.py to test for this situation
Created attachment 717392 [details] Added more test coverage
Created attachment 718091 [details] Re-factored, and corrects the return type for get_more_nodes I'll be submitting this patch to Gerrit shortly.
Created attachment 718100 [details] Broke the test cases out to multiple, meaningfully named tests
REVIEW: http://review.gluster.org/4748 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on master by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#2) for review on release-3.4 by Kaleb KEITHLEY (kkeithle)
REVIEW: http://review.gluster.org/4860 (object-storage: Restoring multi volume support in UFO.) posted (#1) for review on release-3.3 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4861 (Adds missing functions to ring.py, and more thorough tests.) posted (#1) for review on release-3.3 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#2) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#4) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#3) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#4) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#5) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#5) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4858 (Adds missing functions to ring.py, and more thorough tests.) posted (#6) for review on release-3.4 by Alex Wheeler (wheelear)
REVIEW: http://review.gluster.org/4857 (object-storage: Restoring multi volume support in UFO.) posted (#6) for review on release-3.4 by Alex Wheeler (wheelear)
Just doing the book keeping part here : patch is merged in master in ufo repository. http://review.gluster.org/#/c/4748/ This needs to be reverified with HAVANA based code and updated. Thanks, Chetan Risbud.
Not reproducible on latest upstream G4S.(Havana based)
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user