Hi, could you please paste here output of lsblk command? It would help us to see what does your device stack look like

Here you go:

lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0 223,6G  0 disk  
├─sda1                                          8:1    0   500M  0 part  /boot
└─sda2                                          8:2    0 223,1G  0 part  
  └─luks-74c76887-424d-41f8-8a92-97ceb87a4c8f 253:0    0 223,1G  0 crypt 
    ├─fedora_latitude-root                    253:1    0    50G  0 lvm   /
    ├─fedora_latitude-swap                    253:2    0   7,8G  0 lvm   [SWAP]
    └─fedora_latitude-home                    253:3    0 165,3G  0 lvm   /home
sr0                                            11:0    1  1024M  0 rom

Popping in to say that I am seeing similar issues also with LUKS and LVM2 using Fedora 20. This was the only google result for "stoppping LVM" (three Ps, a typo in the shutdown log message.)

Jul 17 13:17:11 dhcp-17-12 systemd: Job dev-disk-by\x2did-lvm\x2dpv\x2duuid\x2dV0QAFp\x2dOTOK\x2do2xl\x2dfuOF\x2djgjh\x2dYHrB\x2dziVREP.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:11 dhcp-17-12 systemd: Job dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2d762c54869c7f4398875055e068cc12d5\x2dluks\x2d762c5486\x2d9c7f\x2d4398\x2d8750\x2d55e068cc12d5.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:11 dhcp-17-12 systemd: Job dev-disk-by\x2did-dm\x2dname\x2dluks\x2d762c5486\x2d9c7f\x2d4398\x2d8750\x2d55e068cc12d5.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:11 dhcp-17-12 systemd: Job dev-dm\x2d0.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:11 dhcp-17-12 systemd: Job dev-block-253:0.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:11 dhcp-17-12 systemd: Job sys-devices-virtual-block-dm\x2d0.device/stop timed out.
Jul 17 13:17:11 dhcp-17-12 systemd: Timed out stoppping LVM PV V0QAFp-OTOK-o2xl-fuOF-jgjh-YHrB-ziVREP on /dev/dm-0.
Jul 17 13:17:11 dhcp-17-12 systemd:
Jul 17 13:17:12 dhcp-17-12 systemd: user stopping timed out. Killing.



NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0 238.5G  0 disk  
├─sda1                                          8:1    0   1.6G  0 part  
├─sda2                                          8:2    0   1.4G  0 part  /boot
└─sda3                                          8:3    0 235.5G  0 part  
  └─luks-762c5486-9c7f-4398-8750-55e068cc12d5 253:0    0 235.5G  0 crypt 
    ├─HelpDeskRHEL6-root                      253:1    0 134.2G  0 lvm   /
    ├─HelpDeskRHEL6-Swap                      253:2    0     4G  0 lvm   
    ├─HelpDeskRHEL6-Home                      253:3    0     4G  0 lvm   
    ├─HelpDeskRHEL6-NotBackedUp               253:4    0     8G  0 lvm   
    ├─HelpDeskRHEL6-Root                      253:5    0  14.7G  0 lvm   
    └─HelpDeskRHEL6-VirtualMachines           253:6    0  29.3G  0 lvm   


Yeah, I know the volumes are named after RHEL6 -- counterintuitively, "root" is my F20 root, and "Root" is my RHEL6 root. Not intentional, but I missed my chance to name the partition in Anaconda while co-opting an existing LVM installation of RHEL6, so that's what happened.

Like Javiermon, this manifests for me also mainly in the form of very slow shutdown times.

I've got a similar log entry trying to shut down Fedora 21 here.


Shutdown takes minutes (I push the power button eventually)

lsblk:

NAME                                            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                               8:0    0 238.5G  0 disk  
├─sda1                                            8:1    0   200M  0 part  /boot/efi
├─sda2                                            8:2    0   500M  0 part  /boot
├─sda3                                            8:3    0 237.8G  0 part  
│ ├─vg-lv_root                                  253:0    0    50G  0 lvm   /
│ ├─vg-lv_swap                                  253:1    0   5.5G  0 lvm   [SWAP]
│ └─vg-lv_home                                  253:2    0 182.3G  0 lvm   
│   └─luks-768008b0-9ef1-440f-bfb0-756db9708c64 253:3    0 182.3G  0 crypt /home
└─sda4                                            8:4    0  1007K  0 part  

(swap isn't encrypted yet)

I have to assume some process keeps /home open.


Other lines in the logs.  Lots to do with WiFi:

Dec 22 09:49:18 klatch.company.net systemd[1]: wpa_supplicant.service: main process exited, code=exited, status=1/FAILURE
Dec 22 09:49:18 klatch.company.net systemd[1]: Unit wpa_supplicant.service entered failed state.
Dec 22 09:49:18 klatch.company.net systemd[1]: wpa_supplicant.service failed.
Dec 22 09:49:18 klatch.company.net gdm-Xorg-:0[1066]: (EE) Server terminated successfully (0). Closing log file.
Dec 22 09:49:18 klatch.company.net gdm[934]: Freeing conversation 'gdm-password' with active job
Dec 22 09:49:19 klatch.company.net bluetoothd[768]: Stopping SDP server
Dec 22 09:49:19 klatch.company.net bluetoothd[768]: Exit
Dec 22 09:49:19 klatch.company.net acpid[774]: exiting
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22628]: Failed to get rfkill device 'rfkill11': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22627]: Failed to get rfkill device 'rfkill12': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22630]: Failed to get rfkill device 'rfkill9': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22629]: Failed to get rfkill device 'rfkill10': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22626]: Failed to get rfkill device 'rfkill13': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22631]: Failed to get rfkill device 'rfkill8': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22633]: Failed to get rfkill device 'rfkill5': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22632]: Failed to get rfkill device 'rfkill7': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22635]: Failed to get rfkill device 'rfkill4': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22640]: Failed to get rfkill device 'rfkill6': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd-rfkill[22636]: Failed to get rfkill device 'rfkill2': No such file or directory
Dec 22 09:49:19 klatch.company.net systemd[1]: systemd-rfkill: control process exited, code=exited status=1
Dec 22 09:49:19 klatch.company.net systemd[1]: Unit systemd-rfkill entered failed state.
Dec 22 09:49:19 klatch.company.net systemd[1]: systemd-rfkill failed.
Dec 22 09:49:19 klatch.company.net systemd[1]: systemd-rfkill: control process exited, code=exited status=1
Dec 22 09:49:19 klatch.company.net systemd[1]: Unit systemd-rfkill entered failed state.
Dec 22 09:49:19 klatch.company.net systemd[1]: systemd-rfkill failed.

.....

and more lvm errors:

Dec 22 09:49:19 klatch.company.net lvm[22672]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Dec 22 09:49:19 klatch.company.net lvm[22672]: 3 logical volume(s) in volume group "vg" unmonitored


the LVM errors seem related to # 1152185

Created attachment 1001284 [details]
todays journalctl

todays journalctl about umounting luks /home

It also fails to umount /boot (which isn't luks)

shutdown never happens (not for several minutes)

This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

I've confirmed this in a laptop I've upgraded to Fedora 22.

I cannot change the version number in bugzilla.

Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

I seeing this issue in Fedora 23. Can someone please reopen this bug, or should I file a new one?

I think there is a good question in the post below blkdeactivate vs blk-availability, see https://lists.fedoraproject.org/archives/list/devel%40lists.fedoraproject.org/message/IOGQB2FNFCRK5FI46LACZ7PA5WG5HA7F/

Reopening this for now...

It's my post actually, and I did some more investigation afterwards which made things clearer for me.

I found out the cause of LVM not being deactivated: firewalld's service stop removes some modules from the kernel (specifically nf_conntrack), and that hangs, leaving an immortal userland process (rmmod) that keeps the root mount alive. The service stop never finishes until the shutdown timeout is reached, when systemd attempts to shutdown uncleanly (TERM-ing and KILL-ing everything). The LVM deactivation rightfully fails (as there is still a mounted filesystem on the LV), and by extension, so does the LUKS deactivation.

blkdeactivate actually does already exclude a bunch of hardcoded mountpoints (/ /boot /lib /lib64 /bin /sbin /var /usr /usr/lib /usr/lib64 /usr/sbin /usr/bin). It still seems to me like an insufficient solution: it's perfectly possible, if not likely, that an userland process part of a service that is not yet finished holds an open file in a non-excluded mountpoint. That will cause blkdeactivate to fail at it's job at unmounting (as it is currently called in blk-availability.service) and device deactivation.

Unmounting should be systemd's job, as it holds the dependency tree and can know with reasonable certainty that all userland processes have been properly terminated if at all possible. blkdeactivate could then only be run from the initramfs without doing any unmounting (it will have already been done if possible).

I also found a strange interaction that I suspect could cause some failures in unusual situations. Dracut's current shutdown procedure calls to /usr/lib/dracut/dracut-initramfs-restore through the stop command of dracut-shutdown.service. It prepares a new system root for late shutdown, that systemd will pivot to after being done. The script attempts to mount /boot read-only unconditionally, which is normally fine, as the service depends on boot.mount, and will only be stopped after it is unmounted. But if the unmounting fails and /boot is still mounted, the script will fail completely and there will be no pivot at all, possibly leaving activated block devices at poweroff time.

(In reply to Daniel Miranda from comment #11)
> It still seems to me like an insufficient solution: it's
> perfectly possible, if not likely, that an userland process part of a
> service that is not yet finished holds an open file in a non-excluded
> mountpoint. That will cause blkdeactivate to fail at it's job at unmounting
> (as it is currently called in blk-availability.service) and device
> deactivation.
> 

Actually, blkdeactivate was primarily intended for RHEL6 where systemd is not used. Before blkdeactivate, there was no script or hook within shutdown sequence that would take care of proper device stack deactivation (mainly device-mapper based devices - crypt mappings, LVM, mpath devices and so on). This sometimes lead to several problems by accessing devices which were no longer available and errored out (e.g. a stack of devices on top of iSCSI after iSCSI sessions were logged out).

> Unmounting should be systemd's job, as it holds the dependency tree and can
> know with reasonable certainty that all userland processes have been
> properly terminated if at all possible. blkdeactivate could then only be run
> from the initramfs without doing any unmounting (it will have already been
> done if possible).
> 

Current state is that systemd is using a simple loop (and there's also a limit for the iteration count as well IIRC) over devices at shutdown and it tries to deactivate them one by one as it sees it in the listing of devices - this is not quite perfect as this way systemd can end up trying to deactivate a device which is still in use (and this really ends up with warning messages about a device being in use while trying to remove it at shutdown sequence). So one has to look at the device stack from each subsystem's point of view which knows how to deal with the deactivation in proper order - and that's what blkdeactivate does - it traverses the lsblk (== sysfs) structure from top to bottom and when it hits known device subsystem (e.g. LVM), it calls subsystem-specific deactivation hook which can also deactivate grouped devices in one go (e.g. whole volume group), thus saving useless iterations.

At one time, I was looking at the "device" systemd units with an intention to enhance them and reuse systemd's knowledge somehow to do a proper shutdown or to make blkdeactivate and systemd better communicate with each other (or make this completely in the hands of systemd), but unfortunately, I've never got to this properly as I got the feeling that systemd team still thinks their simple deactivation loop is just enough (...it actually is useful for very simple setups, but once the stack gets a bit more complicated, consisting of different types, it's definitely not). Also, when talking to systemd guys about this more, current loop at shutdown is meant to be simple as it is by design because it's just the "very last chance to do SOME cleanup if all the other shutdown scripts haven't done so yet" - so it's a fallback action actually! IOW, each subsystem is supposed to be responsible for cleaning up its own stuff.

This is simply just a manifestation of the fact we're still missing complete, central and proper management of device stacks comprising of all possible device types (...or at least the most common used for starters).

The blkdeactivate script is just meant to provide at least the best-effort service for deactivation of these stacks for *now* till we come up with proper device stack management (where proper shutdown is just a part of that because there are other areas besides shutdown where the lack of device management from higher perspective makes problems). And blkdeactivate definitely is not complete (it handles only dm-based and MD devices with mount points on top). The complete solution is still something we don't have... but it's something that starts to be debated a lot recently among various teams which is, I think, a very good start.

There are already several projects that try to do some device management from this higher perspective and to encompass several subsystems, but each one is focused in some distinct area (e.g. storaged) so all these efforts need to be put together to make one complete whole.

Of course, we need to count with the fact that systemd is not the only service-managing mechanism out and we should not completely focus only on solution in this environment if this is going to be an upstream solution which should avoid useless and ever-appearing duplication of work and generating new specific wrappers once for all.

So that's just for the context around blkdeactivate and systemd and stuff. Simply, blkdeactivate is not perfect. It's a helper, providing best effort, but it certainly is not the final solution for deactivation (...that's why you can find things like hard-coded paths which should not be unmounted in the blkdeactivate script).

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.