The failure occurs due to the vol file being created incorrectly. The vol file parsing errors due to having multiple "volume data-client-0" entries.

I suspect it has something to do with all the bricks for this volume residing on one single server.

(In reply to Leildin from comment #0)

Sorry, missed some stuff here:
 
> Description of problem:
> 
> Upgrading a distributed volume from 6 to 8 bricks.
> xfs disks mounted to /gluster/bricks/brick*
> added brick 7 and 8 without any problems

Volume is called data

the volume sees bricks fine and I want to push layout and files onto new bricks.
I try a gluster volume rebalance datat start.
It's successful and I get a job-id.

when I try to see status it says my job failed at 0%.

we tried to regenerate all .vol files in /var/lib/glusterd/vol/data by running pkill glusterd ; glusterd --xlator-option *.upgrade=on -N ; glusterd

The .vol files have the same problem as can be found in attachement and summarised in "Actual results"

So, a quick work-around is to change manually the .vol files that have an error, for me it was:

data-replication.vol
trusted-data.tcp-fuse.vol
data.tcp-fuse.vol

after these files are changed, the rebalance can be done.

After rebalance is done the files are rewritten and have the same bug put back in !

Clients can't mount the volume in this state. They have the same parsing error.

Is this bug is still valid ? It seems to be your problem is related to creation of vols , Can u change the bug accordingly

This bug is still valid. it happened after an add brick.
I had been using this volume no problem for a year+.

when I added the bricks all went fine and then tried to rebalance to get the bricks up and running with the rest.

I have changed one option since finding this bug, I removed:
network.frame-timeout: 45

# volume reset data network.frame-timeout

If you need any more info let me know, I have gluster-clients that can't mount volumes here and there, I have to unmount and remount them after correcting the config files a second time.

I *think* Jiffin was suggesting that you change the component to glusterd, but I'm not entirely sure.

Since this is an issue at add-brick API, I would request DHT team to look into it. Hence moving the component back to distribute

Just upgraded from 3.6.2 to 3.7.4 without much of a hitch.
rebooted and found my volume could not be mounted.
looked at .vol files, they were regenerated with false information just as above.

I would like to add I have a single server with all 8 bricks on it.
this server has never been part of a trusted pool

My apologies for the extremely delayed response.

I went through the code and the glusterd process generates the volfiles based on the info stored in /var/lib/glusterd/.  It looks like something might be wrong there.

glusterd uses the information in the /var/lib/glusterd/<volname>/bricks directory to generate the client info portion for the client vol files (this includes any fuse client, rebalance etc).

For example, I have a volume called loop with 3 bricks.

Volume Name: loop
Type: Distribute
Volume ID: 68b941df-b656-4950-bcfa-bdd940b774a7
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 192.168.122.9:/bricks/brick2/b2
Brick2: 192.168.122.9:/bricks/brick1/b2
Brick3: 192.168.122.8:/bricks/brick2/b2
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
diagnostics.client-log-level: INFO


If I check the brick info stored in /var/lib/glusterd/vols/loop/bricks, I see 

-rw------- 1 root root 179 Aug 17 13:39 192.168.122.8:-bricks-brick2-b2
-rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick1-b2
-rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick2-b2


These files contain the information which is used to generate the volfiles.


[root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick2-b2
hostname=192.168.122.9
path=/bricks/brick2/b2
real_path=/bricks/brick2/b2
listen-port=0
rdma.listen-port=0
decommissioned=0
brick-id=loop-client-0   <--- client 0
mount_dir=/b2
snap-status=0


[root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick1-b2
hostname=192.168.122.9
path=/bricks/brick1/b2
real_path=/bricks/brick1/b2
listen-port=0
rdma.listen-port=0
decommissioned=0
brick-id=loop-client-1    <--- client 1
mount_dir=/b2
snap-status=0

[root@nb-rhs3-srv1 bricks]# cat 192.168.122.8:-bricks-brick2-b2
hostname=192.168.122.8
path=/bricks/brick2/b2
real_path=/bricks/brick2/b2
listen-port=49152
rdma.listen-port=0
decommissioned=0
brick-id=loop-client-2   <--- client 2
mount_dir=/b2
snap-status=0


It sounds like the files in /var/lib/glusterd/data/bricks for the original 6 bricks have for some reason got the same brick-id. 

We do not know why this could have happened. If you have any steps to reproduce the issue, please let us know.

Can you please send across the contents of the /var/lib/glusterd/data on the server so we can confirm this theory?

If this is the case, this problem will show up everytime the volfiles are generated (if you were to change an option or add/remove bricks for example). You will need to edit the files and correct the brick ids in the same order as listed in the gluster volume info. 


Brick1: gls-safran1:/gluster/bricks/brick1/data   <-- data-client-0
Brick2: gls-safran1:/gluster/bricks/brick2/data   <-- data-client-1
Brick3: gls-safran1:/gluster/bricks/brick3/data   <-- data-client-2
Brick4: gls-safran1:/gluster/bricks/brick4/data   <-- data-client-3
Brick5: gls-safran1:/gluster/bricks/brick5/data   <-- data-client-4
Brick6: gls-safran1:/gluster/bricks/brick6/data   <-- data-client-5
Brick7: gls-safran1:/gluster/bricks/brick7/data   <-- data-client-6
Brick8: gls-safran1:/gluster/bricks/brick8/data   <-- data-client-7

Please let me know if you have any questions.

(In reply to Nithya Balachandran from comment #10)
> My apologies for the extremely delayed response.
> 
> I went through the code and the glusterd process generates the volfiles
> based on the info stored in /var/lib/glusterd/.  It looks like something
> might be wrong there.
> 
> glusterd uses the information in the /var/lib/glusterd/<volname>/bricks
> directory to generate the client info portion for the client vol files (this
> includes any fuse client, rebalance etc).
> 
> For example, I have a volume called loop with 3 bricks.
> 
> Volume Name: loop
> Type: Distribute
> Volume ID: 68b941df-b656-4950-bcfa-bdd940b774a7
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.122.9:/bricks/brick2/b2
> Brick2: 192.168.122.9:/bricks/brick1/b2
> Brick3: 192.168.122.8:/bricks/brick2/b2
> Options Reconfigured:
> transport.address-family: inet
> performance.readdir-ahead: on
> nfs.disable: on
> diagnostics.client-log-level: INFO
> 
> 
> If I check the brick info stored in /var/lib/glusterd/vols/loop/bricks, I
> see 
> 
> -rw------- 1 root root 179 Aug 17 13:39 192.168.122.8:-bricks-brick2-b2
> -rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick1-b2
> -rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick2-b2
> 
> 
> These files contain the information which is used to generate the volfiles.
> 
> 
> [root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick2-b2
> hostname=192.168.122.9
> path=/bricks/brick2/b2
> real_path=/bricks/brick2/b2
> listen-port=0
> rdma.listen-port=0
> decommissioned=0
> brick-id=loop-client-0   <--- client 0
> mount_dir=/b2
> snap-status=0
> 
> 
> [root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick1-b2
> hostname=192.168.122.9
> path=/bricks/brick1/b2
> real_path=/bricks/brick1/b2
> listen-port=0
> rdma.listen-port=0
> decommissioned=0
> brick-id=loop-client-1    <--- client 1
> mount_dir=/b2
> snap-status=0
> 
> [root@nb-rhs3-srv1 bricks]# cat 192.168.122.8:-bricks-brick2-b2
> hostname=192.168.122.8
> path=/bricks/brick2/b2
> real_path=/bricks/brick2/b2
> listen-port=49152
> rdma.listen-port=0
> decommissioned=0
> brick-id=loop-client-2   <--- client 2
> mount_dir=/b2
> snap-status=0
> 
> 
> It sounds like the files in /var/lib/glusterd/data/bricks for the original 6
> bricks have for some reason got the same brick-id. 
> 
> We do not know why this could have happened. If you have any steps to
> reproduce the issue, please let us know.
> 
> Can you please send across the contents of the /var/lib/glusterd/data on the
> server so we can confirm this theory?
> 
> If this is the case, this problem will show up everytime the volfiles are
> generated (if you were to change an option or add/remove bricks for
> example). You will need to edit the files and correct the brick ids in the
> same order as listed in the gluster volume info. 
> 
> 
> Brick1: gls-safran1:/gluster/bricks/brick1/data   <-- data-client-0
> Brick2: gls-safran1:/gluster/bricks/brick2/data   <-- data-client-1
> Brick3: gls-safran1:/gluster/bricks/brick3/data   <-- data-client-2
> Brick4: gls-safran1:/gluster/bricks/brick4/data   <-- data-client-3
> Brick5: gls-safran1:/gluster/bricks/brick5/data   <-- data-client-4
> Brick6: gls-safran1:/gluster/bricks/brick6/data   <-- data-client-5
> Brick7: gls-safran1:/gluster/bricks/brick7/data   <-- data-client-6
> Brick8: gls-safran1:/gluster/bricks/brick8/data   <-- data-client-7
> 
> Please let me know if you have any questions.

Hi,

I have since moved on to gluster 3.7.14 on all of my servers.
I can confirm that when I had the bug any rebalance, option change would corrupt the vol files.
I had to go back into them and make them right then upgrade to not have the bug anymore.
Do you still want the /var/lib/glusterd/vols/data files ?
They are correct and don't get corrupted anymore.

(In reply to Leildin from comment #11)
> (In reply to Nithya Balachandran from comment #10)
> > My apologies for the extremely delayed response.
> > 
> > I went through the code and the glusterd process generates the volfiles
> > based on the info stored in /var/lib/glusterd/.  It looks like something
> > might be wrong there.
> > 
> > glusterd uses the information in the /var/lib/glusterd/<volname>/bricks
> > directory to generate the client info portion for the client vol files (this
> > includes any fuse client, rebalance etc).
> > 
> > For example, I have a volume called loop with 3 bricks.
> > 
> > Volume Name: loop
> > Type: Distribute
> > Volume ID: 68b941df-b656-4950-bcfa-bdd940b774a7
> > Status: Started
> > Number of Bricks: 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: 192.168.122.9:/bricks/brick2/b2
> > Brick2: 192.168.122.9:/bricks/brick1/b2
> > Brick3: 192.168.122.8:/bricks/brick2/b2
> > Options Reconfigured:
> > transport.address-family: inet
> > performance.readdir-ahead: on
> > nfs.disable: on
> > diagnostics.client-log-level: INFO
> > 
> > 
> > If I check the brick info stored in /var/lib/glusterd/vols/loop/bricks, I
> > see 
> > 
> > -rw------- 1 root root 179 Aug 17 13:39 192.168.122.8:-bricks-brick2-b2
> > -rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick1-b2
> > -rw------- 1 root root 175 Aug 17 13:39 192.168.122.9:-bricks-brick2-b2
> > 
> > 
> > These files contain the information which is used to generate the volfiles.
> > 
> > 
> > [root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick2-b2
> > hostname=192.168.122.9
> > path=/bricks/brick2/b2
> > real_path=/bricks/brick2/b2
> > listen-port=0
> > rdma.listen-port=0
> > decommissioned=0
> > brick-id=loop-client-0   <--- client 0
> > mount_dir=/b2
> > snap-status=0
> > 
> > 
> > [root@nb-rhs3-srv1 bricks]# cat 192.168.122.9:-bricks-brick1-b2
> > hostname=192.168.122.9
> > path=/bricks/brick1/b2
> > real_path=/bricks/brick1/b2
> > listen-port=0
> > rdma.listen-port=0
> > decommissioned=0
> > brick-id=loop-client-1    <--- client 1
> > mount_dir=/b2
> > snap-status=0
> > 
> > [root@nb-rhs3-srv1 bricks]# cat 192.168.122.8:-bricks-brick2-b2
> > hostname=192.168.122.8
> > path=/bricks/brick2/b2
> > real_path=/bricks/brick2/b2
> > listen-port=49152
> > rdma.listen-port=0
> > decommissioned=0
> > brick-id=loop-client-2   <--- client 2
> > mount_dir=/b2
> > snap-status=0
> > 
> > 
> > It sounds like the files in /var/lib/glusterd/data/bricks for the original 6
> > bricks have for some reason got the same brick-id. 
> > 
> > We do not know why this could have happened. If you have any steps to
> > reproduce the issue, please let us know.
> > 
> > Can you please send across the contents of the /var/lib/glusterd/data on the
> > server so we can confirm this theory?
> > 
> > If this is the case, this problem will show up everytime the volfiles are
> > generated (if you were to change an option or add/remove bricks for
> > example). You will need to edit the files and correct the brick ids in the
> > same order as listed in the gluster volume info. 
> > 
> > 
> > Brick1: gls-safran1:/gluster/bricks/brick1/data   <-- data-client-0
> > Brick2: gls-safran1:/gluster/bricks/brick2/data   <-- data-client-1
> > Brick3: gls-safran1:/gluster/bricks/brick3/data   <-- data-client-2
> > Brick4: gls-safran1:/gluster/bricks/brick4/data   <-- data-client-3
> > Brick5: gls-safran1:/gluster/bricks/brick5/data   <-- data-client-4
> > Brick6: gls-safran1:/gluster/bricks/brick6/data   <-- data-client-5
> > Brick7: gls-safran1:/gluster/bricks/brick7/data   <-- data-client-6
> > Brick8: gls-safran1:/gluster/bricks/brick8/data   <-- data-client-7
> > 
> > Please let me know if you have any questions.
> 
> Hi,
> 
> I have since moved on to gluster 3.7.14 on all of my servers.
> I can confirm that when I had the bug any rebalance, option change would
> corrupt the vol files.
> I had to go back into them and make them right then upgrade to not have the
> bug anymore.
> Do you still want the /var/lib/glusterd/vols/data files ?
> They are correct and don't get corrupted anymore.



Hi,

If you corrected the files in /var/lib/glusterd/vols/data/bricks, then yes, this should not happen anymore and we don't need the files. 


In that case can we close the BZ?

If you have not corrected the files in /var/lib/glusterd/vols/data/bricks, please send them across.

(In reply to Nithya Balachandran from comment #13)
> If you have not corrected the files in /var/lib/glusterd/vols/data/bricks,
> please send them across.

I've corrected them since the bug, changed version and done all kinds of rebalance and option changing without issue since.
Let's close this BZ.

Thanks. Closing this BZ as per comment#14.