Description of problem: Kernel mount/fuse mount to Ceph FS hangs intermittently. This started happening recently in the latest few hours. Tried using all monitor server in mount command, still same problem. For the segment of strace output is below: read(8, "# /etc/security/limits.conf\n#\n#T"..., 4096) = 2422 read(8, "", 4096) = 0 close(8) = 0 munmap(0x7f5c84084000, 4096) = 0 openat(AT_FDCWD, "/etc/security/limits.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8 brk(NULL) = 0x7f5c861d1000 brk(0x7f5c861f5000) = 0x7f5c861f5000 getdents(8, /* 3 entries */, 32768) = 88 open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 9 fstat(9, {st_mode=S_IFREG|0644, st_size=26254, ...}) = 0 mmap(NULL, 26254, PROT_READ, MAP_SHARED, 9, 0) = 0x7f5c8407e000 close(9) = 0 futex(0x7f5c8360ba80, FUTEX_WAKE_PRIVATE, 2147483647) = 0 getdents(8, /* 0 entries */, 32768) = 0 brk(NULL) = 0x7f5c861f5000 brk(NULL) = 0x7f5c861f5000 brk(0x7f5c861ed000) = 0x7f5c861ed000 brk(NULL) = 0x7f5c861ed000 close(8) = 0 open("/etc/security/limits.d/20-nproc.conf", O_RDONLY) = 8 fstat(8, {st_mode=S_IFREG|0644, st_size=191, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5c8407d000 read(8, "# Default limit for number of us"..., 4096) = 191 read(8, "", 4096) = 0 close(8) = 0 munmap(0x7f5c8407d000, 4096) = 0 setrlimit(RLIMIT_NPROC, {rlim_cur=29182, rlim_max=29182}) = 0 setpriority(PRIO_PROCESS, 0, 0) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_AUDIT) = 8 fcntl(8, F_SETFD, FD_CLOEXEC) = 0 sendto(8, "\230\0\0\0Q\4\5\0\3\0\0\0\0\0\0\0op=PAM:session_o"..., 152, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 152 poll([{fd=8, events=POLLIN}], 1, 500) = 1 ([{fd=8, revents=POLLIN}]) recvfrom(8, "$\0\0\0\2\0\0\0\3\0\0\0D\357\377\377\0\0\0\0\230\0\0\0Q\4\5\0\3\0\0\0"..., 8988, MSG_PEEK|MSG_DONTWAIT, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 36 recvfrom(8, "$\0\0\0\2\0\0\0\3\0\0\0D\357\377\377\0\0\0\0\230\0\0\0Q\4\5\0\3\0\0\0"..., 8988, MSG_DONTWAIT, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 36 close(8) = 0 getrlimit(RLIMIT_NPROC, {rlim_cur=29182, rlim_max=29182}) = 0 setrlimit(RLIMIT_NPROC, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5c84078b10) = 18728 close(4) = 0 select(7, [3 6], [], NULL, NULL Version-Release number of selected component (if applicable): cat BaseOS-3.1.3-2017-11-22/installed-debs | grep ceph ii ceph-common 12.2.1-42redhat1xenial amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-fuse 12.2.1-42redhat1xenial amd64 FUSE-based client for the Ceph distributed file system ii libcephfs2 12.2.1-42redhat1xenial amd64 Ceph distributed file system client library ii python-cephfs 12.2.1-42redhat1xenial amd64 Python 2 libraries for the Ceph libcephfs library cat BaseOS-3.1.3-2017-11-22/uname Linux BaseOS-3.1.3-2017-11-22 4.13.0-25-generic #29~16.04.2-Ubuntu SMP Tue Jan 9 12:16:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux How reproducible: 100%reproduced Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Search the logs for when the first pg went to state "unknown" compare that to entries in the other logs (cluster, mon, etc) around that time. Something major happened for all pgs to go to state "unknown" and we should have logged something about it. pgs: 100.000% pgs unknown 2176 unknown
the first issue (not able to mount cephfs) can be related to the second isssue. To mount a client, mds need to records session information in object store. The second issue prevents osd from handling any request. So mount hangs
(In reply to Yan, Zheng from comment #19) > the first issue (not able to mount cephfs) can be related to the second > isssue. To mount a client, mds need to records session information in object > store. The second issue prevents osd from handling any request. So mount > hangs Hi Yan, The timeline we have says the first issue (not able to mount cephfs) was seen well before the second issue (MGR can not communicate with OSDs). "1. The original issue, where the cephFS was unmount able / intermittent mounts even when the ceph status showed OK ie all GOOD."
yes, I looks like balancer issue. It exports too much at a time.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1563