Bug 1305831

Summary: Slow reboot/shutdown and strange message about removing ioctl and busy device
Product: [Fedora] Fedora Reporter: Fernando <fmuro>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 23CC: agk, anprice, bmarzins, bmr, bugzilla, dwysocha, extras-orphan, fat.lobyte9, gesserat, gmazyland, harald, heinzm, johannbg, jonathan, lnykryn, lvm-team, msekleta, msnitzer, muadda, palazzotti, prajnoha, prockai, rickrich, s, systemd-maint, zbyszek, zkabelac
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-20 18:35:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Shutdown log
none
Following Harald's instructions
none
Halting
none
first shot
none
second shot none

Description Fernando 2016-02-09 11:05:52 UTC
Created attachment 1122381 [details]
Shutdown log

Description of problem:

Shutting down takes sometimes a very long time. Even when it doesn't, iterations of the following message fill the screen before finally shutting down:

device-mapper: remove ioctl on fedora-root failed: Device or resource busy

I'm filing this as a systemd problem because this is my suspicion, but I'm far from a Linux expert, so I beg your pardon if I'm mistaken.

Version-Release number of selected component (if applicable):

The system is up to date. The kernel version is 4.3.5-300.fc23.x86_64 in case this helps.

How reproducible:

99% times I reboot/shutdown

Steps to Reproduce:

1. Reboot, halt or shutdown and watch the screen

Actual results:

It ends up shutting down, but with that suspicious message on the screen, and quite often after having to wait for a couple of minutes.

Expected results:

Smooth shutdown without screen messages.

Additional info:

I have proceeded as suggested in the following link:

https://freedesktop.org/wiki/Software/systemd/Debugging/#shutdowncompleteseventually

I attach the log. Towards the end, there are lots of systemd messages saying something like:

Failed to send unit remove signal for WHATEVER: Transport endpoint is not connected

Comment 1 Zbigniew Jędrzejewski-Szmek 2016-02-09 12:48:00 UTC
What type of filesystems do you have (ext4, btrfs)? RAID?

Comment 2 Fernando 2016-02-09 18:38:09 UTC
ext4 with root, swap and home in separate logical volumes. I'd say I'm not using RAID since cat /proc/mdstat gives

Personalities : 
unused devices: <none>

Comment 3 Fernando 2016-02-11 10:33:22 UTC
BTW, I don't get that iterated message with kernel 4.2.3, just with 4.3.5 and 4.3.5. In case this helps, with 4.2.3, after [ OK ] Reached target Shutdown I get:

dracut Warning: Killing all remaining processes
dracut Warning: Unmounted /oldroot.

Comment 4 Michal Sekletar 2016-02-11 12:54:17 UTC
I looked at the shutdown logs you've posted and they look pretty normal. Nothing out of the ordinary. I think that messages you see on the screen are from shutdown initrd environment. At that time systemd is not running anymore. Also you stated that with older kernel you don't experience the problems, so I really think this is probably device-mapper bug. Not sure whether userspace tools or kernel part is the culprit though.

Comment 5 Fernando 2016-02-11 17:07:23 UTC
Michal, thanks for your input. This is the first time I report a bug here, what should we do now? Shall I open a new report of we can transfer this one to the right people?

Comment 6 Michal Sekletar 2016-02-11 17:48:25 UTC
(In reply to Fernando from comment #5)
> Michal, thanks for your input. This is the first time I report a bug here,
> what should we do now? Shall I open a new report of we can transfer this one
> to the right people?

You don't need to submit separate bug report. Just change component of the existing one. 

Due to lack of the better idea I am reassigning to device-mapper. I hope they can reroute it to the proper place if necessary.

Comment 7 Harald Hoyer 2016-02-12 09:50:39 UTC
Does the file /run/initramfs/.need_shutdown exist?
# ls -l /run/initramfs/.need_shutdown

If you want to debug the final shutdown do:

# mkdir -p /run/initramfs/etc/cmdline.d
# echo "rd.debug rd.break=pre-shutdown rd.break=shutdown" \
  > /run/initramfs/etc/cmdline.d/debug.conf
# touch /run/initramfs/.need_shutdown

This will drop you to a shell in the end, and you will be able to take photos of the last messages.

Exit the shell:
# exit

You will be dropped to another shell... take a photo, then
# exit

Comment 8 Fernando 2016-02-12 10:07:24 UTC
Hi Harald, /run/initramfs/.need_shutdown does exist, but it's empty, I've done what you say (as super user) and nothing happens.

Comment 9 Harald Hoyer 2016-02-12 11:46:57 UTC
(In reply to Fernando from comment #8)
> Hi Harald, /run/initramfs/.need_shutdown does exist, but it's empty, I've
> done what you say (as super user) and nothing happens.

oh.. of course you have to shutdown/reboot

Comment 10 Fernando 2016-02-12 17:08:29 UTC
Created attachment 1123593 [details]
Following Harald's instructions

Comment 11 Fernando 2016-02-12 17:09:26 UTC
Created attachment 1123595 [details]
Halting

Comment 12 Fernando 2016-02-12 17:13:09 UTC
Oh, crap, I should have understood that those commands were not shutting down themselves... Anyway, I've followed your instructions but the first shell seems to have cleaned all previous output (see the picture). Nevertheless, I've also uploaded the result of a halt, in case this is what you're asking. As you can see, it's just the line in my first message repeated all over the screen. If I've failed again in following your instructions, please tell me, I'll repeat it. Thanks!

Comment 13 Harald Hoyer 2016-02-15 12:46:03 UTC
(In reply to Fernando from comment #12)
> Oh, crap, I should have understood that those commands were not shutting
> down themselves... Anyway, I've followed your instructions but the first
> shell seems to have cleaned all previous output (see the picture).
> Nevertheless, I've also uploaded the result of a halt, in case this is what
> you're asking. As you can see, it's just the line in my first message
> repeated all over the screen. If I've failed again in following your
> instructions, please tell me, I'll repeat it. Thanks!

you can always scroll up with <shift>+<pageup>

Comment 14 Fernando 2016-02-16 10:10:43 UTC
I'm afraid this isn't working. I've also tried other key combinations I've found in internet to no avail.

Comment 15 Harald Hoyer 2016-02-16 14:18:49 UTC
(In reply to Fernando from comment #14)
> I'm afraid this isn't working. I've also tried other key combinations I've
> found in internet to no avail.

did you remove "quiet" from the kernel command line?

Comment 16 Fernando 2016-02-16 16:43:23 UTC
Created attachment 1127632 [details]
first shot

Comment 17 Fernando 2016-02-16 16:43:46 UTC
Created attachment 1127633 [details]
second shot

Comment 18 Fernando 2016-02-16 16:44:23 UTC
No, I didn't, that was my mistake. Please, have a look at the two shots I've uploaded. Thanks!

Comment 19 Harald Hoyer 2016-02-17 14:39:55 UTC
(In reply to Fernando from comment #18)
> No, I didn't, that was my mistake. Please, have a look at the two shots I've
> uploaded. Thanks!

and exiting the last shell does not reboot?

Comment 20 Fernando 2016-02-17 14:57:39 UTC
(In reply to Harald Hoyer from comment #19)
> (In reply to Fernando from comment #18)
> > No, I didn't, that was my mistake. Please, have a look at the two shots I've
> > uploaded. Thanks!
> 
> and exiting the last shell does not reboot?

Yes, after the second shot 'exit' does reboot.

Comment 21 Harald Hoyer 2016-02-17 16:10:14 UTC
(In reply to Fernando from comment #20)
> (In reply to Harald Hoyer from comment #19)
> > (In reply to Fernando from comment #18)
> > > No, I didn't, that was my mistake. Please, have a look at the two shots I've
> > > uploaded. Thanks!
> > 
> > and exiting the last shell does not reboot?
> 
> Yes, after the second shot 'exit' does reboot.

and you have a problem to reboot without the debug changes of comment #7 ?

Comment 22 Fernando 2016-02-17 17:34:44 UTC
(In reply to Harald Hoyer from comment #21)
> (In reply to Fernando from comment #20)
> > (In reply to Harald Hoyer from comment #19)
> > > (In reply to Fernando from comment #18)
> > > > No, I didn't, that was my mistake. Please, have a look at the two shots I've
> > > > uploaded. Thanks!
> > > 
> > > and exiting the last shell does not reboot?
> > 
> > Yes, after the second shot 'exit' does reboot.
> 
> and you have a problem to reboot without the debug changes of comment #7 ?

Well, it always ends up rebooting/switching off, as I explained in the first message, quite often after a while, sometimes without the Fedora splash screen (it stays in a blocked Gnome session), and 99% of times with the following strange message:

device-mapper: remove ioctl on fedora-root failed: Device or resource busy

I also explain above that this started with a kernel upgrade.

Comment 23 Fedora End Of Life 2016-11-24 15:28:30 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 24 Alexander Korsunsky 2016-12-05 15:40:46 UTC
I still experience this bug in Fedora 25.

I get three messages:

device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
Command failed


Multiple times over the screen, right before shutdown.

My Root partition is an LVM partition with one single ext4 partition inside. 

I read somewhere that device-mapper has been deprecated, but I couldn't remove it because it's protected by systemd-udevd.

Do you need help debugging this issue? I don't get hangs on shutdown, but it looks kind of ugly.

Comment 25 Peter Rajnoha 2016-12-06 07:54:59 UTC
(In reply to Alexander Korsunsky from comment #24)
> I still experience this bug in Fedora 25.
> 
> I get three messages:
> 
> device-mapper: remove ioctl on fedora-root failed: Device or resource busy

The systemd/dracut shutdown hook is trying to deactivate a device which is still in use (not an LVM/device-mapper issue).

> Kernel not configured for semaphores (System V IPC). Not using udev
> synchronisation code.
> Command failed

This was already reported as bug #1359352 (which was closed as dup of bug #1385432 - an selinux issue in dracut environment at shutdown).

> 
> 
> Multiple times over the screen, right before shutdown.
> 
> My Root partition is an LVM partition with one single ext4 partition inside. 
> 
> I read somewhere that device-mapper has been deprecated, but I couldn't
> remove it because it's protected by systemd-udevd.


Device-mapper is definitely not deprecated.

Comment 26 Alexander Korsunsky 2016-12-06 11:01:51 UTC
(In reply to Peter Rajnoha from comment #25)
> (In reply to Alexander Korsunsky from comment #24)
> > device-mapper: remove ioctl on fedora-root failed: Device or resource busy
> 
> The systemd/dracut shutdown hook is trying to deactivate a device which is
> still in use (not an LVM/device-mapper issue).
So who's issue is that? Should I create a new bug for that? Which component?

> This was already reported as bug #1359352 (which was closed as dup of bug
> #1385432 - an selinux issue in dracut environment at shutdown).
Thanks for the links.


> Device-mapper is definitely not deprecated.
I must have misunderstood this comment from bug 1359352, comment 14


> By the way, the bug was missed because device-mapper (the fedora package, not the kernel functionality itself) is obsolete - orphaned.  It is now a sub-package of lvm2, and although I removed the bugzilla component when we did that, some fedora system has put it back!

Comment 27 Peter Rajnoha 2016-12-06 12:24:10 UTC
(In reply to Alexander Korsunsky from comment #26)
> (In reply to Peter Rajnoha from comment #25)
> > (In reply to Alexander Korsunsky from comment #24)
> > > device-mapper: remove ioctl on fedora-root failed: Device or resource busy
> > 
> > The systemd/dracut shutdown hook is trying to deactivate a device which is
> > still in use (not an LVM/device-mapper issue).
> So who's issue is that? Should I create a new bug for that? Which component?
> 

That would be "dracut" component I suppose (if not, they will change to proper component surely).

> > Device-mapper is definitely not deprecated.
> I must have misunderstood this comment from bug 1359352, comment 14

Well, that was meant only for "device-mapper bugzilla component", not device-mapper driver itself.

Comment 28 Alexander Korsunsky 2016-12-08 10:38:57 UTC
> (In reply to Alexander Korsunsky from comment #26)
> > (In reply to Peter Rajnoha from comment #25)
> > > (In reply to Alexander Korsunsky from comment #24)
> > So who's issue is that? Should I create a new bug for that? Which component?
> > 
> 
> That would be "dracut" component I suppose (if not, they will change to
> proper component surely).

For reference, I created the bug #1402073

Comment 29 Fedora End Of Life 2016-12-20 18:35:10 UTC
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.