Bug 1559749 - [CephFS]: IO is hanging while doing rsync
Summary: [CephFS]: IO is hanging while doing rsync
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: CephFS
Version: 3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z3
: 3.0
Assignee: Patrick Donnelly
QA Contact: Persona non grata
Aron Gunn
URL:
Whiteboard:
: 1594760 (view as bug list)
Depends On:
Blocks: 1557269
TreeView+ depends on / blocked
 
Reported: 2018-03-23 08:20 UTC by Persona non grata
Modified: 2018-06-29 23:58 UTC (History)
14 users (show)

Fixed In Version: RHEL: ceph-12.2.4-10.el7cp Ubuntu: ceph_12.2.4-14redhat1xenial
Doc Type: Bug Fix
Doc Text:
.Reducing the number of active MDS daemons on CephFS no longer causes kernel client's I/O to hang Previously, reducing the number of active Metadata Server (MDS) daemons on a Ceph File System (CephFS) would cause kernel client's I/O to hang. When this happens, kernel clients were unable to connect to MDS ranks greater than or equal to `max_mds`. This issue has been fixed in this release.
Clone Of:
Environment:
Last Closed: 2018-05-15 18:20:31 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 22969 0 None None None 2018-04-25 17:13:01 UTC
Ceph Project Bug Tracker 23154 0 None None None 2018-04-25 17:03:44 UTC
Red Hat Product Errata RHBA-2018:1563 0 None None None 2018-05-15 18:21:25 UTC

Description Persona non grata 2018-03-23 08:20:03 UTC
Description of problem:
Running automated scripts for testing rsync module,but IOs were hung for long hours. Different IO are used(dd,fio,crefi,touch)

Version-Release number of selected component (if applicable):
ceph version 12.2.4-4.el7cp (bfc2b497ab362f2b3afa7bd1f9d0053f74b60d66) luminous (stable)

How reproducible:
Always

Steps to Reproduce:
1.Setup ceph cluster and mount on ceph-fuse and kernel clients on same mount point
2.Try to do IOs on machine and mount point,use rsync module to sync data from local to mount dir and data on mount dir to local.

Actual results:
IOs were hung

Expected results:

IOs should be successful and sync should happen

Additional info:
Logs of clients and 2 active mdss are attached

Comment 16 Yan, Zheng 2018-03-29 01:21:29 UTC
looks like ceph_mdsc_open_export_target_session(mdsc, target) return error. the function only return -ENOMEM (unlikely in this case) and -EINVAL. It return -EINVAL when "target >= mdsmap->m_max_mds". did you change max_mds from 2 to 1 during the test?

Comment 17 Persona non grata 2018-03-29 03:58:49 UTC
(In reply to Yan, Zheng from comment #16)
> looks like ceph_mdsc_open_export_target_session(mdsc, target) return error.
> the function only return -ENOMEM (unlikely in this case) and -EINVAL. It
> return -EINVAL when "target >= mdsmap->m_max_mds". did you change max_mds
> from 2 to 1 during the test?

Yes,for previous test cleanup

Comment 39 errata-xmlrpc 2018-05-15 18:20:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1563

Comment 40 Patrick Donnelly 2018-06-29 23:58:06 UTC
*** Bug 1594760 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.