Hide Forgot
After installing the 3.1 beta RPMs on 4 RHEL 4 update 5 hosts with infiniband I started the gluster cli on the first host, added the others as peers, then was unable to create a volume that included the host I was on. Below is the output that I got: [root@iosrv-7-1 ~]# gluster gluster> peer status No peers present gluster> peer probe iosrv-7-2 Probe successful gluster> peer probe iosrv-7-3 Probe successful gluster> peer probe iosrv-7-4 Probe successful gluster> peer status Number of Peers: 3 hostname:iosrv-7-2, uuid:5268b25f-8533-43fb-ba79-0c26a15ee069, state:3 (connected) hostname:iosrv-7-3, uuid:19be1f71-ab84-461e-a3ae-988f5e704143, state:3 (connected) hostname:iosrv-7-4, uuid:3c7fddcf-3edc-4fa0-b6b7-ab15cc3360ce, state:3 (connected) gluster> volume create gluster-test transport rdma iosrv-7-1:/mnt/brick1 iosrv-7-2:/mnt/brick2 iosrv-7-3:/mnt/brick3 iosrv-7-4:/mnt/brick4 Creation of volume gluster-test has been unsuccessful Host iosrv-7-1 not a friend gluster> I tried to probe the local machines name, but that didn't work: gluster> peer probe iosrv-7-1 iosrv-7-1 is already part of another cluster Probe unsuccessful Probe failed gluster> Finally I started the CLI on the second host and found this: gluster> peer status Number of Peers: 3 hostname:172.16.2.4, uuid:4c649194-b909-4b27-bddc-e11000455fdb, state:3 (connected) hostname:iosrv-7-3, uuid:19be1f71-ab84-461e-a3ae-988f5e704143, state:3 (connected) hostname:iosrv-7-4, uuid:3c7fddcf-3edc-4fa0-b6b7-ab15cc3360ce, state:3 (connected) gluster> peer probe iosrv-7-1 Probe on host iosrv-7-1 port 6969 already a friend gluster> 172.16.2.4 is the IP of the first host. Based on the documentation I expected the 'volume create' to just work without any additional probing.
hi Michael, Could you please attach the 'uname -a' output, the log files located in /usr/local/var/log/glusterfs on all these machines. This should help us find the root cause of the issue. Thanks Pranith.
Created attachment 341 Logs are prepended with servername that they came from.
(In reply to comment #1) > hi Michael, > Could you please attach the 'uname -a' output, the log files located in > /usr/local/var/log/glusterfs on all these machines. This should help us find > the root cause of the issue. > > Thanks > Pranith. uname -a from all 4 servers iosrv-7-1: Linux iosrv-7-1.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux iosrv-7-3: Linux iosrv-7-3.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux iosrv-7-4: Linux iosrv-7-4.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux iosrv-7-2: Linux iosrv-7-2.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux
Any updates on this? I am unable to create any volumes. Today I updated to the latest code from the git.gluster.com repo and the status is the same.
(In reply to comment #4) > Any updates on this? I am unable to create any volumes. Today I updated to the > latest code from the git.gluster.com repo and the status is the same. Mike, As a workaround, add an entry for the initial server to /etc/hosts on the initial server.
(In reply to comment #5) > (In reply to comment #4) > > Any updates on this? I am unable to create any volumes. Today I updated to the > > latest code from the git.gluster.com repo and the status is the same. > > Mike, > > As a workaround, add an entry for the initial server to /etc/hosts on the > initial server. This is the contents of our /etc/hosts without any changes: 127.0.0.1 localhost.localdomain localhost 138.67.1.104 ra.mines.edu 172.16.2.4 iosrv-7-1.local iosrv-7-1 Do you see any problems with these entries? Mike
Is there anything I can do to help debug this and get a fix committed to git? I'm sitting dead in the water right now and I'm not seeing any evidence of progress. I'm willing to work on this, but need some assistance from a developer to know how to do it.
(In reply to comment #7) > Is there anything I can do to help debug this and get a fix committed to git? > I'm sitting dead in the water right now and I'm not seeing any evidence of > progress. I'm willing to work on this, but need some assistance from a > developer to know how to do it. Please check with latest 3.1 - if issue still existing ,as a work around ,please use ipaddress instead of hostnames.
The issue is still there with the latest 3.1, but the work around does help. I was able to probe peers and create a volume using IP addresses rather than hostname. Before doing that I was able to determine that the problem was due to the fact that the hostname command which I believe uses the same underlying system call as the glusterd code returns iosrv-7-1.local and the code is expecting to find no domain name attached. It would be nice to find a fix for this, but for now I can move on and will open a new bug report on the next problem that I've hit. hint: Client hangs on ls of glusterfs native mount Mike (In reply to comment #8) > (In reply to comment #7) > > Is there anything I can do to help debug this and get a fix committed to git? > > I'm sitting dead in the water right now and I'm not seeing any evidence of > > progress. I'm willing to work on this, but need some assistance from a > > developer to know how to do it. > > Please check with latest 3.1 - if issue still existing ,as a work around > ,please use ipaddress instead of hostnames.
*** Bug 2009 has been marked as a duplicate of this bug. ***
PATCH: http://patches.gluster.com/patch/5582 in master (mgmt/glusterd: glusterd_is_local_addr implementation)
fixed in 3.1.1qa1 release. http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa1.tar.gz
The reason for the bug is that in the previous implementation, the aliases are not handled properly for localhost. Gluster handles them now.
This patch works for me. I'm able to probe peers by hostname and create volumes and everything works. There is still one odd issue though. I start with a clean install on iosrv-7-1 and first probe iosrv-7-2. It shows up on iosrv-7-1 as iosrv-7-2, but when I look at the peer status on iosrv-7-2 it shows it only peer as being 172.16.2.4. This is reproducible with any of my hosts. Let me know if you need any debugging info to fix this minor annoyance. Thanks, Mike (In reply to comment #13) > The reason for the bug is that in the previous implementation, the aliases are > not handled properly for localhost. Gluster handles them now.
(In reply to comment #14) > This patch works for me. I'm able to probe peers by hostname and create volumes > and everything works. There is still one odd issue though. I start with a clean > install on iosrv-7-1 and first probe iosrv-7-2. It shows up on iosrv-7-1 as > iosrv-7-2, but when I look at the peer status on iosrv-7-2 it shows it only > peer as being 172.16.2.4. This is reproducible with any of my hosts. Let me > know if you need any debugging info to fix this minor annoyance. > > Thanks, > Mike > > (In reply to comment #13) > > The reason for the bug is that in the previous implementation, the aliases are > > not handled properly for localhost. Gluster handles them now. hi Michael, this is a known issue 1995. So I am marking this as resolved. Pranith
Need to check with Pranith/Vijay about what to update.
The following information is added in Creating Trusted Storage Pools section: After peer probe, in the remote machine, the peer machine information is stored with IP address instead of hostname.