Bug 1129486 - adding bricks to mounted volume causes client failures
Summary: adding bricks to mounted volume causes client failures
Keywords:
Status: CLOSED EOL
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.5.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-12 21:17 UTC by Scott Merrill
Modified: 2016-06-17 15:56 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-17 15:56:26 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
mnt.log from the client (55.65 KB, text/plain)
2014-08-12 21:17 UTC, Scott Merrill
no flags Details
glustershd.log from server gluster1 (14.97 KB, text/plain)
2014-08-12 21:17 UTC, Scott Merrill
no flags Details
glustershd.log from server gluster2 (13.89 KB, text/plain)
2014-08-12 21:18 UTC, Scott Merrill
no flags Details

Description Scott Merrill 2014-08-12 21:17:02 UTC
Created attachment 926217 [details]
mnt.log from the client

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
1. Create a replica volume
2. Mount the volume on a client
3. Write data to the volume
4. Add bricks
5. file bugzilla

Actual results:
smerrill@gluster-test:DEVELOPMENT:~> sudo mount -t glusterfs gluster1:/test /mnt
smerrill@gluster-test:DEVELOPMENT:~> ls -lha /mnt
total 8.0K
drwxr-xr-x   3 root root   23 Aug 12 16:59 .
dr-xr-xr-x. 25 root root 4.0K Aug 12 14:17 ..
smerrill@gluster-test:DEVELOPMENT:~> sudo chgrp web-dev /mnt
smerrill@gluster-test:DEVELOPMENT:~> sudo chmod g+w /mnt
smerrill@gluster-test:DEVELOPMENT:~> ls -lha /mnt
total 8.0K
drwxrwxr-x   3 root web-dev   23 Aug 12 16:59 .
dr-xr-xr-x. 25 root root    4.0K Aug 12 14:17 ..
smerrill@gluster-test:DEVELOPMENT:~> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
                      5.0G  1.6G  3.2G  33% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             485M   39M  421M   9% /boot
/dev/mapper/rootvg-tmplv
                      2.0G   68M  1.9G   4% /tmp
/dev/mapper/rootvg-varlv
                      4.0G  1.2G  2.6G  32% /var
gluster1:/test        100G   33M  100G   1% /mnt
smerrill@gluster-test:DEVELOPMENT:~> for i in $(seq 10 20); do for j in $(seq 30 40); do mkdir -p /mnt/$i/$j; dd if=/dev/urandom of=/mnt/$i/$j/test$i bs=1M count=25; done; done;
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46319 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45505 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45736 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.47164 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.5578 s, 10.2 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.486 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43611 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45828 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43417 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43528 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44691 s, 10.7 MB/s
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/30/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/31/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/32/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/33/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/34/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/35/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/36/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/37/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/38/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/39/test11': No such file or directory
mkdir: cannot create directory `/mnt/11': Permission denied
dd: opening `/mnt/11/40/test11': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/30/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/31/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/32/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/33/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/34/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/35/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/36/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/37/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/38/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/39/test12': No such file or directory
mkdir: cannot create directory `/mnt/12': Permission denied
dd: opening `/mnt/12/40/test12': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/30/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/31/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/32/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/33/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/34/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/35/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/36/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/37/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/38/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/39/test13': No such file or directory
mkdir: cannot create directory `/mnt/13': Permission denied
dd: opening `/mnt/13/40/test13': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/30/test14': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/31/test14': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/32/test14': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/33/test14': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/34/test14': No such file or directory
mkdir: cannot create directory `/mnt/14': Permission denied
dd: opening `/mnt/14/35/test14': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.4298 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43561 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43241 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43229 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44489 s, 10.7 MB/s
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/30/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/31/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/32/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/33/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/34/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/35/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/36/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/37/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/38/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/39/test15': No such file or directory
mkdir: cannot create directory `/mnt/15': Permission denied
dd: opening `/mnt/15/40/test15': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/30/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/31/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/32/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/33/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/34/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/35/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/36/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/37/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/38/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/39/test16': No such file or directory
mkdir: cannot create directory `/mnt/16': Permission denied
dd: opening `/mnt/16/40/test16': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/30/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/31/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/32/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/33/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/34/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/35/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/36/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/37/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/38/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/39/test17': No such file or directory
mkdir: cannot create directory `/mnt/17': Permission denied
dd: opening `/mnt/17/40/test17': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/30/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/31/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/32/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/33/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/34/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/35/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/36/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/37/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/38/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/39/test18': No such file or directory
mkdir: cannot create directory `/mnt/18': Permission denied
dd: opening `/mnt/18/40/test18': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/30/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/31/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/32/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/33/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/34/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/35/test19': No such file or directory
mkdir: cannot create directory `/mnt/19': Permission denied
dd: opening `/mnt/19/36/test19': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44639 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.51232 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.45052 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48194 s, 10.6 MB/s
mkdir: cannot create directory `/mnt/20': Permission denied
dd: opening `/mnt/20/30/test20': No such file or directory
mkdir: cannot create directory `/mnt/20': Permission denied
dd: opening `/mnt/20/31/test20': No such file or directory
mkdir: cannot create directory `/mnt/20': Permission denied
dd: opening `/mnt/20/32/test20': No such file or directory
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.49898 s, 10.5 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.44481 s, 10.7 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.46215 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.437 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.43658 s, 10.8 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.53151 s, 10.4 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.48387 s, 10.6 MB/s
25+0 records in
25+0 records out
26214400 bytes (26 MB) copied, 2.52852 s, 10.4 MB/s
smerrill@gluster-test:DEVELOPMENT:~>

Expected results:
I did not expect any errors.

Additional info:
Attached is the mnt.log from the client.

Comment 1 Scott Merrill 2014-08-12 21:17:40 UTC
Created attachment 926218 [details]
glustershd.log from server gluster1

Comment 2 Scott Merrill 2014-08-12 21:18:00 UTC
Created attachment 926219 [details]
glustershd.log from server gluster2

Comment 3 Niels de Vos 2016-06-17 15:56:26 UTC
This bug is getting closed because the 3.5 is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.


Note You need to log in before you can comment on or make changes to this bug.