Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1247752

Summary:	Log::reopen_log_file() must take the flusher lock to avoid closing an fd ::_flush() is still using
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Ken Dreyer (Red Hat) <kdreyer>
Component:	RADOS	Assignee:	Ken Dreyer (Red Hat) <kdreyer>
Status:	CLOSED ERRATA	QA Contact:	ceph-qe-bugs <ceph-qe-bugs>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	1.2.3	CC:	ceph-eng-bugs, ceph-qe-bugs, dzafman, flucifre, kchai, kdreyer, sjust, tmuthami, vumrao
Target Milestone:	rc
Target Release:	1.2.4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-0.80.8-16.el6cp ceph-0.80.8-16.el7cp	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1246694	Environment:
Last Closed:	2015-07-31 12:54:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 1 Ken Dreyer (Red Hat) 2015-07-29 14:04:19 UTC

upstream's firefly patch is https://github.com/ceph/ceph/pull/5406 so we'll use that.

Comment 4 Tamil 2015-07-30 01:24:20 UTC

steps to reproduce:
set logging in ceph.conf
 
debug ms = 20
debug osd = 20
debug filestore = 20
 
 
sudo rbd create image --size 1000000000
 
sudo rbd bench-write image --io-threads 256 --io-size 4096 --io-total 1000000000000 2>&1 >/dev/n
 
while the workload is in progress [in the background],
 
sudo vi /root/.gdbinit
set pagination off
set target-async on
set non-stop on
 
ps -ef | grep ceph-osd - look for pid of osd.0
 
sudo gdb
attach <pid of osd.0>
 
b Log.cc:117
 
c -a
 
check for sighup and where it breaks
 
sudo ceph osd dump
 
sudo ceph pg dump
 
sudo ceph pg scrub <pg.id>
 
watch for the objects to corrupt
 
 
[ubuntu@magna016 ~]$ sudo ceph -s
    cluster 8c89dca4-2ad2-46f9-b38f-d8450a2c6e0a
     health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean; recovery 3920/34408 objects degraded (11
     monmap e1: 1 mons at {magna016=10.8.128.16:6789/0}, election epoch 2, quorum 0 magna016
     osdmap e37: 3 osds: 2 up, 2 in
      pgmap v1723: 193 pgs, 4 pools, 60233 MB data, 15733 objects
            106 GB used, 1745 GB / 1852 GB avail
            3920/34408 objects degraded (11.393%)
                 192 active+degraded
                   1 active+clean
 
  client io 2336 kB/s wr, 1168 op/s

Comment 5 Samuel Just 2015-07-30 02:16:35 UTC

Degraded objects aren't what you are looking for.  What happened here is the osd died.  That might actually have been due to the bug causing corruption in something the osd then read back, or it might just be that the thread stopped by the gdb session eventually caused a timeout to fail and kill the osd.  You'll have to try it again and keep the osd log.

Comment 6 Ken Dreyer (Red Hat) 2015-07-30 04:00:28 UTC

For non-RHEL, the fix will be in the Ceph v0.80.8.4 packages.

Comment 7 Tamil 2015-07-31 00:45:23 UTC

works fine.

Comment 9 errata-xmlrpc 2015-07-31 12:54:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1527