Bug 1730953 - mount generates errors after umount
Summary: mount generates errors after umount
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: mainline
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
Assignee: Amar Tumballi
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-18 02:54 UTC by Amgad
Modified: 2019-07-23 03:22 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-23 03:22:21 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Gluster.org Gerrit 23087 None Merged backup-volfile-server: ignore duplicate entries 2019-07-23 03:22:20 UTC

Description Amgad 2019-07-18 02:54:47 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Amgad 2019-07-18 03:04:37 UTC
Description of problem:

mount a volume after being umount introduces errors, however the mount was successful

Version-Release number of selected component (if applicable):
5.5-1

How reproducible:
Had 3-cluster nodes. Created a volume and /etc/fstab had the following entry to make it persistent

10.76.155.233:/bcmt-glusterfs /opt/bcmt/storage glusterfs backup-volfile-servers=10.76.155.237:10.76.155.236:10.76.155.233,_netdev 0 0
[root@gfs3 tasks]# umount /opt/bcmt/storage



Steps to Reproduce:
1. umount /opt/bcmt/storage
2. mount /opt/bcmt/storage




Actual results:

[root@gfs3 bricks]# 
[root@gfs3 bricks]# umount /opt/bcmt/storage

glusterfs-client log entry:
-------------------------
[2019-07-18 02:29:30.968978] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-bcmt-glusterfs-server: disconnecting connection from CTX_ID:4a34475c-9838-447f-93b2-b5ba33096c9b-GRAPH_ID:0-PID:24514-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0
[2019-07-18 02:29:30.969162] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-bcmt-glusterfs-server: Shutting down connection CTX_ID:4a34475c-9838-447f-93b2-b5ba33096c9b-GRAPH_ID:0-PID:24514-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0

[root@gfs3 bricks]# mount /opt/bcmt/storage
[2019-07-18 02:30:09.182263] E [glusterfsd.c:783:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists

glusterfs-client log entry:
--------------------------
[2019-07-18 02:30:09.182263] E [glusterfsd.c:783:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
[2019-07-18 02:30:09.218383] I [addr.c:54:compare_addr_and_update] 0-/data0/glusterfs: allowed = "*", received addr = "10.76.155.233"
[2019-07-18 02:30:09.218430] I [login.c:110:gf_auth] 0-auth/login: allowed user names: f9f0cc51-20cf-42ba-8288-257d57f28bcb
[2019-07-18 02:30:09.218443] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-bcmt-glusterfs-server: accepted client from CTX_ID:1b3a75a1-5868-43a2-b2c2-7e79cbb88f01-GRAPH_ID:0-PID:24762-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0 (version: 5.5)
[root@gfs3 bricks]#

Expected results:

Should not have the errors

Additional info:

Expected results:


Additional info:

Comment 2 Amgad 2019-07-18 03:05:32 UTC
Description of problem:

mount a volume after being umount introduces errors, however the mount was successful

Version-Release number of selected component (if applicable):
5.5-1

How reproducible:
Had 3-cluster nodes. Created a volume and /etc/fstab had the following entry to make it persistent

10.76.155.233:/bcmt-glusterfs /opt/bcmt/storage glusterfs backup-volfile-servers=10.76.155.237:10.76.155.236:10.76.155.233,_netdev 0 0
[root@gfs3 tasks]# umount /opt/bcmt/storage



Steps to Reproduce:
1. umount /opt/bcmt/storage
2. mount /opt/bcmt/storage




Actual results:

[root@gfs3 bricks]# 
[root@gfs3 bricks]# umount /opt/bcmt/storage

glusterfs-client log entry:
-------------------------
[2019-07-18 02:29:30.968978] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-bcmt-glusterfs-server: disconnecting connection from CTX_ID:4a34475c-9838-447f-93b2-b5ba33096c9b-GRAPH_ID:0-PID:24514-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0
[2019-07-18 02:29:30.969162] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-bcmt-glusterfs-server: Shutting down connection CTX_ID:4a34475c-9838-447f-93b2-b5ba33096c9b-GRAPH_ID:0-PID:24514-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0

[root@gfs3 bricks]# mount /opt/bcmt/storage
[2019-07-18 02:30:09.182263] E [glusterfsd.c:783:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists

glusterfs-client log entry:
--------------------------
[2019-07-18 02:30:09.182263] E [glusterfsd.c:783:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
[2019-07-18 02:30:09.218383] I [addr.c:54:compare_addr_and_update] 0-/data0/glusterfs: allowed = "*", received addr = "10.76.155.233"
[2019-07-18 02:30:09.218430] I [login.c:110:gf_auth] 0-auth/login: allowed user names: f9f0cc51-20cf-42ba-8288-257d57f28bcb
[2019-07-18 02:30:09.218443] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-bcmt-glusterfs-server: accepted client from CTX_ID:1b3a75a1-5868-43a2-b2c2-7e79cbb88f01-GRAPH_ID:0-PID:24762-HOST:gfs3-PC_NAME:bcmt-glusterfs-client-2-RECON_NO:-0 (version: 5.5)
[root@gfs3 bricks]#

Expected results:

Should not have the errors

Additional info:

Expected results:


Additional info:

Comment 3 Amar Tumballi 2019-07-18 04:42:41 UTC
Can you try below line?

'10.76.155.233:/bcmt-glusterfs /opt/bcmt/storage glusterfs backup-volfile-servers=10.76.155.237:10.76.155.236,_netdev 0 0'

(change is, no addition of 10.76.155.233 in backup-volfile-servers). Let us know if that fixes the problem.

Comment 4 Amgad 2019-07-21 17:26:06 UTC
Thanks Amar -- removing the "10.76.155.233" fixed the problem! Does that mean the "backup-volfile-servers" should not include all the servers?

The problem is we use a DNS name (glusterfs-server) (instead of IPs) that resolves to any of the three servers in case one is down. That's why needed all servers to be backup -- any suggestion?
--> 'glusterfs-server:/bcmt-glusterfs /opt/bcmt/storage glusterfs backup-volfile-servers=10.76.155.237:10.76.155.236,10.76.155.233,_netdev 0 0'

Comment 5 Amar Tumballi 2019-07-22 04:03:23 UTC
Thanks for confirming. The request seems genuine, instead of erroring out a duplicate entry in backup-volfile-server (it happens because 'gluster-server' resolved to one of the IP, and it was not expecting it in backup-volfile-servers. (as it was given as primary server itself).

For now, continue with the fix. I will see if we can fix the issue in code, and have a backport done.

Comment 6 Worker Ant 2019-07-22 05:05:01 UTC
REVIEW: https://review.gluster.org/23087 (backup-volfile-server: ignore duplicate entries) posted (#1) for review on master by Amar Tumballi

Comment 7 Worker Ant 2019-07-23 03:22:21 UTC
REVIEW: https://review.gluster.org/23087 (backup-volfile-server: ignore duplicate entries) merged (#2) on master by Amar Tumballi


Note You need to log in before you can comment on or make changes to this bug.