Bug 763587 (GLUSTER-1855)

hi Michael,
     Could you please attach the 'uname -a' output, the log files located in /usr/local/var/log/glusterfs on all these machines. This should help us find the root cause of the issue.

Thanks
Pranith.

Created attachment 341


Logs are prepended with servername that they came from.

(In reply to comment #1)
> hi Michael,
>      Could you please attach the 'uname -a' output, the log files located in
> /usr/local/var/log/glusterfs on all these machines. This should help us find
> the root cause of the issue.
> 
> Thanks
> Pranith.

uname -a from all 4 servers

iosrv-7-1: Linux iosrv-7-1.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux
iosrv-7-3: Linux iosrv-7-3.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux
iosrv-7-4: Linux iosrv-7-4.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux
iosrv-7-2: Linux iosrv-7-2.local 2.6.18-128.7.1.el5.ddn3.l1.6.7.2.ddn5smp #1 SMP Sat Jun 19 01:54:00 CEST 2010 x86_64 x86_64 x86_64 GNU/Linux

Any updates on this? I am unable to create any volumes. Today I updated to the latest code from the git.gluster.com repo and the status is the same.

(In reply to comment #4)
> Any updates on this? I am unable to create any volumes. Today I updated to the
> latest code from the git.gluster.com repo and the status is the same.

Mike,

As a workaround, add an entry for the initial server to /etc/hosts on the initial server.

(In reply to comment #5)
> (In reply to comment #4)
> > Any updates on this? I am unable to create any volumes. Today I updated to the
> > latest code from the git.gluster.com repo and the status is the same.
> 
> Mike,
> 
> As a workaround, add an entry for the initial server to /etc/hosts on the
> initial server.

This is the contents of our /etc/hosts without any changes:

127.0.0.1 localhost.localdomain localhost
138.67.1.104 ra.mines.edu
172.16.2.4  iosrv-7-1.local  iosrv-7-1

Do you see any problems with these entries?

Mike

Is there anything I can do to help debug this and get a fix committed to git? I'm sitting dead in the water right now and I'm not seeing any evidence of progress. I'm willing to work on this, but need some assistance from a developer to know how to do it.

(In reply to comment #7)
> Is there anything I can do to help debug this and get a fix committed to git?
> I'm sitting dead in the water right now and I'm not seeing any evidence of
> progress. I'm willing to work on this, but need some assistance from a
> developer to know how to do it.

Please check with latest 3.1 - if issue still existing ,as a work around ,please use ipaddress instead of hostnames.

The issue is still there with the latest 3.1, but the work around does help. I was able to probe peers and create a volume using IP addresses rather than hostname. Before doing that I was able to determine that the problem was due to the fact that the hostname command which I believe uses the same underlying system call as the glusterd code returns iosrv-7-1.local and the code is expecting to find no domain name attached. It would be nice to find a fix for this, but for now I can move on and will open a new bug report on the next problem that I've hit. hint: Client hangs on ls of glusterfs native mount

Mike

(In reply to comment #8)
> (In reply to comment #7)
> > Is there anything I can do to help debug this and get a fix committed to git?
> > I'm sitting dead in the water right now and I'm not seeing any evidence of
> > progress. I'm willing to work on this, but need some assistance from a
> > developer to know how to do it.
> 
> Please check with latest 3.1 - if issue still existing ,as a work around
> ,please use ipaddress instead of hostnames.

*** Bug 2009 has been marked as a duplicate of this bug. ***

PATCH: http://patches.gluster.com/patch/5582 in master (mgmt/glusterd: glusterd_is_local_addr implementation)

fixed in 3.1.1qa1 release.
http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa1.tar.gz

The reason for the bug is that in the previous implementation, the aliases are not handled properly for localhost. Gluster handles them now.

This patch works for me. I'm able to probe peers by hostname and create volumes and everything works. There is still one odd issue though. I start with a clean install on iosrv-7-1 and first probe iosrv-7-2. It shows up on iosrv-7-1 as iosrv-7-2, but when I look at the peer status on iosrv-7-2 it shows it only peer as being 172.16.2.4. This is reproducible with any of my hosts. Let me know if you need any debugging info to fix this minor annoyance.

Thanks,
Mike

(In reply to comment #13)
> The reason for the bug is that in the previous implementation, the aliases are
> not handled properly for localhost. Gluster handles them now.

(In reply to comment #14)
> This patch works for me. I'm able to probe peers by hostname and create volumes
> and everything works. There is still one odd issue though. I start with a clean
> install on iosrv-7-1 and first probe iosrv-7-2. It shows up on iosrv-7-1 as
> iosrv-7-2, but when I look at the peer status on iosrv-7-2 it shows it only
> peer as being 172.16.2.4. This is reproducible with any of my hosts. Let me
> know if you need any debugging info to fix this minor annoyance.
> 
> Thanks,
> Mike
> 
> (In reply to comment #13)
> > The reason for the bug is that in the previous implementation, the aliases are
> > not handled properly for localhost. Gluster handles them now.

hi Michael,
    this is a known issue 1995. So I am marking this as resolved.

Pranith

Need to check with Pranith/Vijay about what to update.

The following information is added in Creating Trusted Storage Pools section: After peer probe, in the remote machine, the peer machine information is stored with IP address instead of hostname.