Bug 1237038

Description of problem:
Sometimes when the brick daemon is restarted e.g. after force kill from command line it doesn't work properly and cannot be started correctly for such brick. Only restart of the machine will help for me. The brick could be used for any volume but it will cause failures e.g. rebalance always fail. The probable reason is that on such brick nothing could be written from glusterfs.
Moreover if volume is compound only from such bricks it cannot be mounted. For such cases in logs there can be found 'connection refused'.

Not sure how to reproduce such state though, maybe killing the brick process (kill -9) do some operations (rebalance, create, remove data, remove-bricks etc) with volume and then stop start the volume could hit this.

Version-Release number of selected component (if applicable):
glusterfs-3.7.1-5

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
This I found in bad brick daemon log:
[2015-06-30 08:21:14.958022] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down
[2015-06-30 08:28:41.757378] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.1 (args: /usr/sbin/glusterfsd -s <IP> --volfile-id test.<IP>.rhgs-b3-b3 -p /var/lib/glusterd/vols/test/run/<IP>-rhgs-b3-b3.pid -S /var/run/gluster/ef06f8578fffc09daec4a35d0b3f56c2.socket --brick-name /rhgs/b3/b3 -l /var/log/glusterfs/bricks/rhgs-b3-b3.log --xlator-option *-posix.glusterd-uuid=a4e17c12-7898-4e83-b0e0-9895c0d18573 --brick-port 49178 --xlator-option test-server.listen-port=49178)
[2015-06-30 08:28:41.773389] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-06-30 08:28:41.785685] I [graph.c:269:gf_add_cmdline_options] 0-test-server: adding option 'listen-port' for volume 'test-server' with value '49178'
[2015-06-30 08:28:41.785735] I [graph.c:269:gf_add_cmdline_options] 0-test-posix: adding option 'glusterd-uuid' for volume 'test-posix' with value 'a4e17c12-7898-4e83-b0e0-9895c0d18573'
[2015-06-30 08:28:41.786242] I [MSGID: 115034] [server.c:397:_check_for_auth_option] 0-/rhgs/b3/b3: skip format check for non-addr auth option auth.login./rhgs/b3/b3.allow
[2015-06-30 08:28:41.786406] I [MSGID: 115034] [server.c:397:_check_for_auth_option] 0-/rhgs/b3/b3: skip format check for non-addr auth option auth.login.554f1a38-17f9-47c5-b066-6de6a5d46e04.password
[2015-06-30 08:28:41.786419] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2015-06-30 08:28:41.789410] I [rpcsvc.c:2213:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2015-06-30 08:28:41.789516] W [options.c:936:xl_opt_validate] 0-test-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2015-06-30 08:28:41.792466] E [ctr-helper.c:250:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.
[2015-06-30 08:28:41.792499] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096
[2015-06-30 08:28:41.792512] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-cachesize from params.Assigning default value: 1000
[2015-06-30 08:28:41.792524] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal
[2015-06-30 08:28:41.792588] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-wal-autocheckpoint from params.Assigning default value: 1000
[2015-06-30 08:28:41.792604] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: normal
[2015-06-30 08:28:41.792615] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-test-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none
[2015-06-30 08:28:41.793214] I [trash.c:2363:init] 0-test-trash: no option specified for 'eliminate', using NULL
[2015-06-30 08:28:41.794763] W [graph.c:357:_log_if_unknown_option] 0-test-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2015-06-30 08:28:41.794799] W [graph.c:357:_log_if_unknown_option] 0-test-server: option 'rpc-auth.auth-unix' is not recognized
[2015-06-30 08:28:41.794835] W [graph.c:357:_log_if_unknown_option] 0-test-server: option 'rpc-auth.auth-null' is not recognized
[2015-06-30 08:28:41.794877] W [graph.c:357:_log_if_unknown_option] 0-test-quota: option 'timeout' is not recognized
[2015-06-30 08:28:41.794914] W [graph.c:357:_log_if_unknown_option] 0-test-trash: option 'brick-path' is not recognized
[2015-06-30 08:28:41.807265] W [MSGID: 113026] [posix.c:1326:posix_mkdir] 0-test-posix: mkdir (/.trashcan/): gfid (00000000-0000-0000-0000-000000000005) isalready associated with directory (/rhgs/b3/b3/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/.trashcan). Hence,both directories will share same gfid and thiscan lead to inconsistencies.
[2015-06-30 08:28:41.807306] E [MSGID: 113027] [posix.c:1348:posix_mkdir] 0-test-posix: mkdir of /rhgs/b3/b3/.trashcan/ failed [File exists]
[2015-06-30 08:28:41.807464] W [MSGID: 113026] [posix.c:1326:posix_mkdir] 0-test-posix: mkdir (/.trashcan/internal_op): gfid (00000000-0000-0000-0000-000000000006) isalready associated with directory (/rhgs/b3/b3/.glusterfs/00/00/00000000-0000-0000-0000-000000000005/internal_op). Hence,both directories will share same gfid and thiscan lead to inconsistencies.
[2015-06-30 08:28:41.807507] E [MSGID: 113027] [posix.c:1348:posix_mkdir] 0-test-posix: mkdir of /rhgs/b3/b3/.trashcan/internal_op failed [File exists]
[2015-06-30 08:28:41.807525] E [trash.c:387:trash_internal_op_mkdir_cbk] 0-test-trash: mkdir failed for internal op directory : File exists