This bug is related to open bug https://bugzilla.redhat.com/show_bug.cgi?id=1379044 .  I do not know who had the idea of leaving SELinux on during initrd shutdown, but I think at this point the right thing to do is to stop SELinux altogether right before initrd shutdown, until the right work is in place to make Dracut play nice with SELinux.

This change has been identified as the problem causer: https://github.com/fedora-selinux/selinux-policy/commit/015047e1d962173e3789af3fad86198a3b5e3ac2

More like this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1371991

These are the most relevant log problems for the work we are trying to push into ZoL:

mount: /oldsys/sys: filesystem mounted, but mount(8) failed: Permission denied
mount: /oldsys/proc: filesystem mounted, but mount(8) failed: Permission denied
mount: /oldsys/run: filesystem mounted, but mount(8) failed: Permission denied
mount: /oldsys/dev: filesystem mounted, but mount(8) failed: Permission denied
[  164.998027] dracut: Taking over mdmon processes.
[  165.000491] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[  165.829445] dracut Warning: Unmounted /oldroot.
[  166.198007] dracut: ZFS: Exporting ZFS storage pools...
[  166.242905] dracut Warning: zpool: error while loading shared libraries: libzpool.so.2: cannot enable executable stack as shared object requires: Permission denied
dracut Warning: zpool: error while loading shared libraries: libzpool.so.2: cannot enable executable stack as shared object requires: Permission denied

This change unmasked these errors:

https://github.com/zfsonlinux/zfs/pull/5287

Please help.  Thanks.

*** Bug 1359352 has been marked as a duplicate of this bug. ***

Having this issue on a Dell XPS 13 with Fedora 25 

Linux tardis-xps.localdomain 4.8.11-300.fc25.x86_64 #1 SMP Mon Nov 28 18:24:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I'm having the same problem on a Dell XPS 13 with kernel-4.8.14-300.fc25.x86_64

I am having the same bug on Fedora 25 (Dell Inspiron 15), but only when I am selecting disk encryption during the installation process. There is no problem when I don't use disk encryption.

(In reply to Shakhar Dasgupta from comment #8)
> I am having the same bug on Fedora 25 (Dell Inspiron 15), but only when I am
> selecting disk encryption during the installation process. There is no
> problem when I don't use disk encryption.

Now, it also started for the unencrypted installation (maybe after an update).

Having the same issue on a Lenovo T530 and a Lenovo X1 Carbon. This is in addition to the Dell XPS 13 I previously reported. None of my installations are encrypted. Is there any information or command I can run to assist in getting this issue moved forward?

Same issue with 4th gen Lenovo X1 carbon. I'm not using Fedora's disk encryption. F25 current as of today, Tue Jan 17 03:10:13 UTC 2017.
4.9.3-200.fc25.x86_64 #1 SMP Fri Jan 13 01:01:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I disabled selinux for a few boots and now get the following:

device-mapper: remove loctl on fedora-root failed: Device or resource busy
Command failed

This repeats several times and then shuts down.

Here is the device-mapper version

Installed Packages
Name        : device-mapper
Arch        : x86_64
Epoch       : 0
Version     : 1.02.136
Release     : 3.fc25
Size        : 291 k
Repo        : @System
From repo   : updates
Summary     : Device mapper utility
URL         : http://sources.redhat.com/dm
License     : GPLv2
Description : This package contains the supporting userspace utility, dmsetup,
            : for the kernel device-mapper.

I am running fedora 25 under Wayland.  This bug has been with me since sometime in fedora 24. However, I never had the total disaster in fedora 24 that I have in fedora 25.

Info: 
Linux fed25.localhost.localdomain 4.9.3-200.fc25.x86_64 #1 SMP Fri Jan 13 01:01:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Core 2 Duo Asrok motherboard 4 megs

This is my production computer and it is spinning 4 hard disks all formatted with lvm
 ACTIVE            '/dev/fedora-21/swap' [2,00 GiB] inherit
  ACTIVE            '/dev/fedora-21/root' [75,00 GiB] inherit
  ACTIVE            '/dev/fedora-21/home' [300,00 GiB] inherit
  ACTIVE            '/dev/fed25/root' [35,00 GiB] inherit
  ACTIVE            '/dev/fed25/swap' [5,00 GiB] inherit
  ACTIVE            '/dev/fed25/home' [104,04 GiB] inherit
  ACTIVE            '/dev/fed22/root' [50,00 GiB] inherit
  ACTIVE            '/dev/fed22/home' [149,01 GiB] inherit
  ACTIVE            '/dev/fed22/swap' [3,00 GiB] inherit
  ACTIVE            '/dev/fed24/root' [69,85 GiB] inherit
  ACTIVE            '/dev/fed24/home' [93,13 GiB] inherit
  ACTIVE            '/dev/fed24/swap' [3,73 GiB] inherit
  ACTIVE            '/dev/fed24/extra03' [116,42 GiB] inherit
  ACTIVE            '/dev/fed23/root' [60,54 GiB] inherit
  ACTIVE            '/dev/fed23/swap' [3,73 GiB] inherit
  ACTIVE            '/dev/fed23/home' [155,53 GiB] inherit

I boot off the fedora 25 boot sector /dev/sda1 without any encryption
devtmpfs                     1,8G     0  1,8G   0% /dev
tmpfs                        1,9G  268K  1,9G   1% /dev/shm
tmpfs                        1,9G  1,7M  1,9G   1% /run
tmpfs                        1,9G     0  1,9G   0% /sys/fs/cgroup
/dev/mapper/fed25-root        35G  8,5G   25G  26% /
tmpfs                        1,9G  144K  1,9G   1% /tmp
/dev/sdb2                    345G  9,5G  318G   3% /extra02
/dev/mapper/fed25-home       103G  2,5G   95G   3% /home
/dev/sdc2                    228G   96G  121G  45% /extra
/dev/mapper/fed24-extra03    115G   19G   91G  17% /extra03
/dev/mapper/fed22-home       147G   21G  119G  15% /home/tompolk/fed22
/dev/mapper/fedora--21-home  296G   23G  258G   9% /home/tompolk/fed21
/dev/mapper/fed24-home        92G   32G   55G  37% /home/tompolk/fed24
/dev/mapper/fed23-home       153G   28G  118G  20% /home/tompolk/fed23
/dev/sda1                    4,8G  189M  4,4G   5% /boot
tmpfs                        370M   20K  370M   1% /run/user/42
tmpfs                        370M  7,1M  363M   2% /run/user/1000

The system is backed up through a raspberry pi 1 

Files are shared on the system via smb

Every two or three weeks while working in Libreoffice calc using a drop down list, I will made a mistake and try to correct it.  The system hangs totally, and I have to press the power button to shut it down.

At that point the computer is totally borked.  The fedora 25 disk must be removed for the computer to boot.  I can repair the fedora 25 using system rescue, put the disk back in and all comes right!

I notice, the error "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code," on shutdown, but normally, the disk will get fsck on reboot and all is fine.

Any help on this?

Is it safe to ignore these messages during shutdown or are we risking filesystem corruption here?

With only the basic file systems mounts: /boot, /, /home - and nothing in /mnt or a custom mount point:

sudo restorecon -rv /
sudo dracut -f

Does this fix the problem or change it? I'm not seeing the problem on multiple F25 systems, although none are using dmcrypt.

Tried those commands, still getting the same error messages on shutdown. I am using encryption.

I tried both commands from #15 and that did not resolve the issue on any of the three laptops having the issue.

None of the laptops are encrypted.

Chris Murphy; is there anything else I should try? Would reinstalling F25 help?

I loaded another machine with Fedora 25 - using Fedora-WS-Live-25-1-3. No issues on shutdown. Ran sudo dnf update. Immediately following the update the problem starts.

This is now an issue on four difference systems.

I have found the only way to not have the machine borked  is simply to not reboot the machine.  Once the machine is borked I can only reboot by resetting CMOS, and manually cleaning up the file corruption on all previously mounted volumes.

However, the other day, after a new kernel was installed, I shut the machine down with the message "Kernel not configured for semaphores ..." still present, but successfully rebooted without disaster.  However, before the starting thereboot, I manually unmounted all mounted volumes including smb volumes.  I still get the error message, but on reboot all the volumes mount okay.

So far so good for a couple of weeks.  We will see.  Seems like this bug is a real show stopper on Fedora 25.

Same problem here. On my self-built desktop. Running Fedora 25 and KDE Plasma 5.2

Same issue on HP envy x360... I noticed I have some TPM options in bios... I'm tempted to change said options, but I'm not sure if they have anything to do with this error...

at the moment, I have selinux disabled... the errors aren't as prevelant, but they still show up during shutdown.

UPDATE: Disabling TPM and security options in bios did nothing... errors still annoying and a little disconcerting...

Confirmed, I am also getting spammed with "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper" on system shutdown/restart. This happens at fresh install up to latest update on kernel 4.9.9-200.fc25.x86_64. Running F25 on default configuration (no disk encryption)

However I have *no* delayed shutdown, no unclean mount/fsck request.

SELinux folks, are you waiting on something here? This seems like quite a significant and commonly encountered issue.

I can confirm this problem on Fedora 25. I have had the issue since Fedora 24 on two different laptops. One is a Dell Latiude E7240 and the other is a Dell Latitude E5540. Im using disc encyption on both of them.
I noticed this, when I ran journalctl -b -1:

Mar 02 08:31:10 e7240 systemd-cryptsetup[1925]: Failed to deactivate: Device or resource busy

Don't know if it has anything to do with the bug.

uname -r -> 4.9.12-200.fc25.x86_64

Any update on this issue?

I'm trying to get some priority on it.

There seems to be some confusion, though, now I look into this more closely. I'm not sure if the "Kernel not configured for semaphores" message alone actually indicates any significant problem; I think it may be one of those things which actually shows up *all the time*, but you don't usually notice it when shutdown proceeds cleanly. You just happen to see the message when shutdown is delayed.

I'm not totally sure, though. It may be useful to have input from kernel maintainers here, so CCing labbott and jforbes.

As shown in bug  1359352 which was duped to this bug, this is not a kernel issue. The kernel is configured for semaphores:

wintertmute:[~]>ipcs -l

------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398442373116
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767

Could you guys test it with following fix:

$ cat kernel_ipc_info.cil 
(allow init_t kernel_t (system (ipc_info)))

# semodule -i kernel_ipc_info.cil

and reproduce the issue? 

Thanks.

(In reply to Lukas Vrabec from comment #29)
> # echo "(allow init_t kernel_t (system (ipc_info)))" > kernel_ipc_info.cil
> # semodule -i kernel_ipc_info.cil

Did this.

> and reproduce the issue? 

1. Do the above
2. Reboot
3. Same errors many times, no difference, no more information on the screen

I also tried to check journalctl after the reboot, but haven't found anything that might have looked relevant (but as I don't know what to look for, I might have missed it). No message of interest about ipc, lock or selinux.

The only thing I saw, searching for fedora-root, is:

Mar 25 07:57:12 tuxedo audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-poweroff comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success
Mar 25 07:57:12 tuxedo systemd[1]: Stopped LVM2 PV scan on device 8:16.
Mar 25 07:57:12 tuxedo audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-pvscan@8:16 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 25 07:57:12 tuxedo audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-pvscan@8:16 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 25 07:57:12 tuxedo systemd[1]: Removed slice system-lvm2\x2dpvscan.slice.
Mar 25 07:57:12 tuxedo blkdeactivate[3884]:   [UMOUNT]: unmounting data-srv (dm-3) mounted on /srv... done
Mar 25 07:57:12 tuxedo blkdeactivate[3884]:   [UMOUNT]: unmounting fedora-home (dm-2) mounted on /home... skipping
Mar 25 07:57:12 tuxedo blkdeactivate[3884]:   [SKIP]: unmount of fedora-root (dm-0) mounted on /
Mar 25 07:57:12 tuxedo blkdeactivate[3884]:   [LVM]: deactivating Volume Group data... done
Mar 25 07:57:12 tuxedo systemd[1]: Stopped Availability of block devices.

Compared with the messages I get:

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed

Latest Fedora 25 after a fresh `dnf upgrade`, the issue appeared around time upgrading to Fedora 25, nothing encrypted here. As following the track to this bug, I saw a remark about VirtualBox: I have libvirt installed.

Let me know what I can do better or more to help debug this issue.

Same symptoms on Lenovo T440s with Fedora 25.

I see it on Fedora 25 and Fedora 26; regardless of whether rootfs is on an dm device (LVM or LUKS or none); however in all cases I've seen this intermittent problem, an LVM device is present.

Also, capturing this after journald shutdown is hard, these instructions require enforcing=0 and that seems to make the problem go away.
https://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1

I've had the same install on a thinkpad x230 since Fedora 23. I started getting the "Kernel not configured for semaphores" issue on shutdown sometime during Fedora 24. I upgraded to fedora 25 yesterday and still get this message, but I have no other symptoms other than the message.

-----------------------------------
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: Fedora
Description:    Fedora release 25 (Twenty Five)
Release:        25
Codename:       TwentyFive
-----------------------------------
kf5-plasma.i686 - 5.33.0-1.fc25
-----------------------------------

Yesterday I installed Fedora 25 from scratch on my Dell Inspiron 15R and I'm getting the same message when I shutdown OR restart the system. No other symptomns.

THe issue is the same after an upgrade to 4.10.8-200.fc25.x86_64

This still happens on a fresh install
- F25 KDE
- LVM w/o encryption
- 4.10.9-200.fc25.x86_64

Some more potentially relevant versions:
libblockdev-lvm.x86_64                 1.9-10.fc25                      @updates
llvm-libs.x86_64                       3.9.1-2.fc25                     @updates
lvm2.x86_64                            2.02.167-3.fc25                  @updates
lvm2-libs.x86_64                       2.02.167-3.fc25                  @updates
libselinux.x86_64                      2.5-13.fc25                      @updates
libselinux-python3.x86_64              2.5-13.fc25                      @updates
libselinux-utils.x86_64                2.5-13.fc25                      @updates
rpm-plugin-selinux.x86_64              4.13.0.1-1.fc25                  @updates
selinux-policy.noarch                  3.13.1-225.11.fc25               @updates
selinux-policy-targeted.noarch         3.13.1-225.11.fc25               @updates

This did not happen on my previous install that was using BTRFS.

Eric, 

Could you do following:

1. # semodule -DB
2. reproduce the issue
3. attach output of #ausearch -m AVC,USER_AVC 

THanks.

Created attachment 1272175 [details]
output from commands requested

This is the output from the requested commands on one of the three affected systems I have.

This bug has been approved for the list of Prioritized Bugs and Issues: https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues

Just thought i should mention since this bug seems to also have SELinux denials. I don't have any of those errors. The boot doesn't hang at all and additionally at the end i get this:

failed to read reboot parameter file: no such file or directory

Which i guess might be related to something like this: https://github.com/systemd/systemd/issues/5646

*** Bug 1450247 has been marked as a duplicate of this bug. ***

I would like to ask for a status update. Is there any progress on this bug ?
This bug is on the list of Prioritized bugs https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues and a fix is considered as important for Fedora users.

Thanks,
Jan

I'm getting it as well, encrypted LVM.
It's really annoying and aesthetically poor.

Same for me. And not only on Fedora. I get the same issue on Ubuntu and Kali Linux as well

The same thing here, on Fedora 26. It’s not just aesthetically poor, but it prevents the computer from shutting down at random, which is a problem an order of magnitude more serious. I’m using Btrfs on LVM on LUKS on LVM, which also qualifies as “encrypted LVM”.

I fixed all AVC from this BZ. Moving to POST. Fixes will be part of the latest build.

Hello to all,

Here is proof that this bug has everything to do with encrypted LVM (Fedora 25):
[SOLVED] kernel not configured for semaphore
http://www.forums.fedoraforum.org/showthread.php?t=314465

Lukas (Vrabec/Sparrow),

Could you, please, explain to us the fix you did to correct this bug? Or point to it somewhere to kernel/selinux policy git? :-)

Thank you in advance,
_nobody_

selinux-policy-3.13.1-225.18.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-45ca9bfcb6

selinux-policy-3.13.1-225.18.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-45ca9bfcb6

I just provided feedback on the bodhi website. Copying here just in case.

On a lenovo thinkpad x201, no encryption on the LVM groups, the "Kernel not configured for semaphores (System V IPC)" can still be seen when shutting down.

Currently running Fedora 25 xfce spin.

Last metadata expiration check: 9:14:19 ago on Sat Jun 10 21:53:09 2017. 
Installed Packages selinux-policy.noarch 3.13.1-225.18.fc25 @updates-testing

Have you tried 'touch /.autorelabel' and rebooting?

(In reply to Joseph D. Wagner from comment #51)
> Have you tried 'touch /.autorelabel' and rebooting?

Just tried your suggestion, but the result was the same.

Sometimes it just print one line of that message, other times it is printed several times before shutting down.

selinux-policy-3.13.1-225.18.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

After installing selinux-policy-3.13.1-225.18.fc25 and running  `touch /.autorelabel` and `dracut -f` and rebooting in between and after, the problem still persists.


On shutdown, a lot of these messages: 

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.

With selinux-policy-3.13.1-257.fc26.noarch, and restorecon -rv /, and then dracut -f. I still see the problem on every reboot and shutdown.

Same here - running with selinux-policy-3.13.1-255.18.fc25 on LVM encrypted setup the semaphore messages on shutdown still exist.

Still exists on my setup as well. LVM encrypted partition.

Same here - installed selinux-policy-3.13.1-225.18.fc25 as an update on LVM not encrypted and the problem is still there.  If I do not umount the extra LVM volumes mounted under home, the whole system can be borked (not always).  When it is borked I cannot get a terminal by booting the fed25 rescue entry in grub.  If I boot into fed24 which is on the system, it comes up okay.  I can mount the fed25-root partition and all seems fine.  Yet, when I boot to fed25, it hangs in a loop and I cannot get a terminal. Doing the touch /.autorelabel -f dracut bit relabels everything, but it still goes into a loop on booting.  I have tried all the selinux suggestions and you still get nothing but that nasty loop.
Only thing I can do once it is borked is reinstall the fed24-root.
However, as long as I umount all the mounted LVM volumes in my home directory and all mounted smb volumes before shut down, and then unplug the the motherboard for 15 seconds, all is well on the next boot!

Sorry reinstall the fed25-root, not the fed24-root.

same after update to selinux-policy-3.13.1-225.18.fc25.noarch the problem still continues.

still have this problem fedora 25 and full drive encryption

*** Bug 1466098 has been marked as a duplicate of this bug. ***

I don't understand how this bug is closed. a.) it's not fixed b.) where is the errata? Do we need new bugs filed for this for various specific conditions or what?

I have no encryption in use.

kernel-4.11.7-300.fc26.x86_64
selinux-policy-3.13.1-259.fc26.noarch
dracut-044-183.fc26.x86_64

Following those updates, I've done a restorecon, and then dracut -f, then reboot, then reboot again, the problem still happens on every reboot.

https://bugzilla.redhat.com/show_bug.cgi?id=1385432#c53 .

(In reply to Adam Williamson from comment #64)
> https://bugzilla.redhat.com/show_bug.cgi?id=1385432#c53 .

Yes I saw that so I don't understand the meaning of posting that without comment.

Two people reporting on Fedora 25 that they still have the problem after updating. And I've got a much newer version on Fedora 26 and it still happens there too.

You said you didn't understand why the bug was closed. That's why it was closed: an update marked as fixing it was pushed stable.

I also met this issue on my F25.
I just upgraded to F26, still can reproduce this issue.

After upgrading bare metal FC25 to FC26 with command: dnf system-upgrade download --refresh --releasever=26

The problem still persists.

_nobody_

FC26 fresh install, full drive enc: I still can reproduce this issue (Im crying)
I'm not an expert what can I do to contribute?

OS: Fedora 26 x86_64
Model: 80JE Lenovo G40-80
Kernel: 4.11.8-300.fc26.x86_64
CPU: Intel i5-5200U (4) @ 2.700GHz

Hi. This bug still occurs on both Fedora 25 and F26 on my laptop - Lenovo e320, Intel® Core™ i5-2410M

Hi, I have this same problem on my laptop Acer E14 ES1-411, the problem appears on Fedora 26, Mint (Rafaela, Rosa and Serena) and Ubuntu.

Hi, I have this same problem on my laptop Acer E14 ES1-411, the problem appears on Fedora 26, Mint (Rafaela, Rosa and Serena) and Ubuntu.

Issue continues under Fedora 26.

Guys, 

I have fresh F26 system installed with encrypted LVM partitions and I cannot reproduce it. Booting, rebooting and shutting down system work fast and also don't see any relevant info in journal.

Could somebody attach reproducer for this issue? 

For people who facing this issue, if you switch SELinux to permissive do you still have this issue? 

Thanks,
Lukas.

(In reply to Lukas Vrabec from comment #74)
> Guys, 
> 
> I have fresh F26 system installed with encrypted LVM partitions and I cannot
> reproduce it. Booting, rebooting and shutting down system work fast and also
> don't see any relevant info in journal.
> 
> Could somebody attach reproducer for this issue? 
> 
> For people who facing this issue, if you switch SELinux to permissive do you
> still have this issue? 
> 
> Thanks,
> Lukas.

Hi Lucas, I don't have F26 on my laptop right now to test, because it breaks one app that I need for work, but on F25 I can see those "device-mapper..." messages when shutting down the system.

Changing SELinux to permissive via /etc/selinux/config does not resolve the issue for me. I'm using X session with GNOME desktop, F25 up to date, disk encryption, LVM and user auto-login.

My hardware:
Lenovo Thinkpad e320, Intel® Core™ i5-2410M

Messages:
"device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed
device-mapper: remove ioctl on luks-9oDa... failed: Device or resource busy
Command failed
.
.
.
"

Because I don't think that this is SELinux issue.

The have this issue on Fedora, Ubuntu and Kali so it is not Fedora specific.

Regarding to comment#75 and comment#77 moving this issue to dracut, I don't see any issues with SELinux here.

I would agree that I do not feel this is an SELinux issue.

Does it help if you add "plymouth.enable=0" on the kernel command line? Or remove rhgb?

Created attachment 1304038 [details]
rd.debug console=ttyS0 vm capture

Clean Fedora 26 installation in a VM. Booting with rd.debug console=ttyS0, and then connecting with virsh console.

Midway is a login prompt which marks end of startup, and after that point is output from shutdown initation within GNOME.

Created attachment 1304054 [details]
rhgb removed, rd.debug console=ttyS0

Same as before but without rhgb. I have no idea why the output is so much more verbose, it looks like there's a ton of repeating, like some kind of race is happening.

Created attachment 1304056 [details]
plymouth.enabled=0 rhgb removed rd.debug console=ttyS0

Looks the same as 81. Problem still occurs.

Created attachment 1304067 [details]
no rhgb, debug rd.debug

Kinda repetitive, but figured I'd attached the thing that I'm going to comment on some snippets about. The boot param line is:  root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap debug rd.debug console=ttyS0


So, there's a whole bunch of suspicious stuff going on, and I'm not sure which of those are contributing to this bug, or are non-factors.

[   43.138772] systemd-shutdown[1]: Unmounting file systems.
[   43.139786] systemd-shutdown[1]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota'.
[   43.151865] systemd-shutdown[1]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota'.

I'm not sure why this remount ro happens twice.


[   43.155170] systemd-shutdown[1]: All filesystems unmounted.
[   43.155933] systemd-shutdown[1]: Deactivating swaps.
[   43.156641] systemd-shutdown[1]: All swaps deactivated.
[   43.157340] systemd-shutdown[1]: Detaching loop devices.
[   43.158422] systemd-shutdown[1]: device-enumerator: scan all dirs

Seems sane.

[   43.209733] device-mapper: ioctl: unable to remove open device fedora-pool00-tpool
[   43.211879] device-mapper: ioctl: unable to remove open device fedora-pool00_tdata
[   43.214151] device-mapper: ioctl: unable to remove open device fedora-pool00_tmeta
[   43.217972] device-mapper: ioctl: unable to remove open device fedora-pool00-tpool
[   43.220269] device-mapper: ioctl: unable to remove open device fedora-pool00_tdata
[   43.223258] device-mapper: ioctl: unable to remove open device fedora-pool00_tmeta
[   43.267056] shutdown: 34 output lines suppressed due to ratelimiting

All of those are thin provisioning related. That's a bit obscure still so I've got to ask if anyone else having this problem is *NOT* using any thin provisioning at all. I have a separate baremetal installation that's Btrfs based, but I have a thin pool I used just for VMs, and I do run into this bug when rebooting or shutting down that baremetal machine tool (I just can't capture anything from it, because the problem happens after remount ro and so nothing gets logged and I don't have a serial console there.)

I'm not sure what 34 lines suppressed really means, but it sounds to me like there are other errors we have no idea what they are (dropped message)?


//shutdown@15(main): stat -c %T -f /
/shutdown@15(main): '[' tmpfs = tmpfs ']'
/shutdown@16(main): mount -o remount,rw /
/shutdown@19(main): mkdir /oldsys
/shutdown@20(main): for i in sys proc run dev
/shutdown@21(main): mkdir /oldsys/sys
/shutdown@22(main): mount --move /oldroot/sys /oldsys/sys
mount: /oldsys/sys: filesystem mounted, but mount(8) failed: Permission denied


The last four lines repeats 3 more times. Seems screwy that it can't be moved.

/shutdown@67(main): warn 'Cannot umount /oldroot'
/lib/dracut-lib.sh@57(warn): check_quiet
/lib/dracut-lib.sh@474(check_quiet): '[' -z yes ']'
/lib/dracut-lib.sh@58(warn): echo '<28>dracut Warning: Cannot umount /oldroot'
/lib/dracut-lib.sh@59(warn): echo 'dracut Warning: Cannot umount /oldroot'
dracut Warning: Cannot umount /oldroot


Doesn't seem good. I know plymouth has a shutdown exemption from systemd that might cause this but if plymouth.enabled=0 really prevents it from being used, then that's probably not it.


/shutdown@70(main): case $_pi[   44.203595] dracut: Disassembling device-mapper devices
d in
-snip-
/shutdown@70(mai[   44.212582] device-mapper: ioctl: unable to remove open device fedora-root
n): case $_pid in


OK if /oldroot is fedora-root and fedora-root can't be umounted then it makes sense dm is going to be mad and can't remove fedora-root. And since fedora-root is a thin LV, it'd explain with tpool, tmeta, and tdata can't be removed either.


///lib[   44.237514] dracut: Disassembling device-mapper devices
/dracut/hooks/shutdown/30-dm-shutdown.sh@7(_do_dm_shutdown): dmsetup info -c --noheadings -o name
//lib/dracut/hooks/shutdown/30-dm-shutdown.sh@7(_do_dm_shutdown): for dev in $(dmsetup info -c --noheadings -o name)
//lib/dracut/hooks/shutdown/30-dm-shutdown.sh@8(_do_dm_shutdown): dmsetup -v --noudevsync remove fedora-root
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed


But I have no idea if the "Kernel not configured for semaphores" is related to the inability to disassemble and shutdown dm. Or if that's just spurious, and only causes noisy messages, and the real problem with the shutdown delay.

sudo dnf remove plymouth
sudo dracut -f
sudo reboot
And then same boot params as 84, virsh console, startup and shutdown, and I get the same results as 84.

Anyway, I can't tell if something is kill exempt and holding up the umount, and that's why dm is mad. Or if dm is just mad on its own and this is a new normal (something's changed in dm).

Created attachment 1304095 [details]
selinux=0 plymouth removed debug rd.debug console=ttyS0

root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap debug rd.debug selinux=0 console=ttyS0


And now the problem does not happen. Specifically:

- Two remount ro's still happen

[   36.750033] device-mapper: ioctl: unable to remove open device fedora-pool00-tpool
[   36.754135] device-mapper: ioctl: unable to remove open device fedora-pool00_tdata
[   36.760857] device-mapper: ioctl: unable to remove open device fedora-pool00_tmeta
[   36.768213] device-mapper: ioctl: unable to remove open device fedora-pool00-tpool
[   36.772104] device-mapper: ioctl: unable to remove open device fedora-pool00_tdata
[   36.775881] device-mapper: ioctl: unable to remove open device fedora-pool00_tmeta
[   36.827061] shutdown: 34 output lines suppressed due to ratelimiting

Those are probably normal because root is still ro mounted so dm can't remove them.




//shutdown@15(main): stat -c %T -f /
/shutdown@15(main): '[' tmpfs = tmpfs ']'
/shutdown@16(main): mount -o remount,rw /
/shutdown@19(main): mkdir /oldsys
/shutdown@20(main): for i in sys proc run dev
/shutdown@21(main): mkdir /oldsys/sys
/shutdown@22(main): mount --move /oldroot/sys /oldsys/sys
/shutdown@20(main): for i in sys proc run dev
/shutdown@21(main): mkdir /oldsys/proc
/shutdown@22(main): mount --move /oldroot/proc /oldsys/proc
/shutdown@20(main): for i in sys proc run dev
/shutdown@21(main): mkdir /oldsys/run
/shutdown@22(main): mount --move /oldroot/run /oldsys/run
/shutdown@20(main): for i in sys proc run dev
/shutdown@21(main): mkdir /oldsys/dev
/shutdown@22(main): mount --move /oldroot/dev /oldsys/dev
/shutdown@27(main): '[' poweroff = kexec ']'
/shutdown@34(main): trap 'emergency_shell --shutdown shutdown Signal caught!' 0

No fails! So the moving was failing, maybe due to selinux disallowing it. Hence the "mount: /oldsys/sys: filesystem mounted, but mount(8) failed: Permission denied" messages we were seeing when selinux was enforcing.



/shutdown@60(main): umount_a
[   37.688590] XFS (dm-3): Unmounting Filesystem
[   37.693948] dracut Warning: Unmounted /oldroot.

This worked! It failed before.



///lib/dracut/hooks/shutdown/30-dm-shutdown.sh@7(_do_dm_shutdown)
...

And all of those 30-dm-shutdown.sh complete without the "not configured for semaphores" messages.

Also FWIW, there is no encryption at all, so that's not it.

(In reply to Chris Murphy from comment #86)
> Created attachment 1304095 [details]
> selinux=0 plymouth removed debug rd.debug console=ttyS0
> 
> root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap
> debug rd.debug selinux=0 console=ttyS0
> 
> 
> And now the problem does not happen. Specifically:

[…]

> 
> No fails! So the moving was failing, maybe due to selinux disallowing it.
> Hence the "mount: /oldsys/sys: filesystem mounted, but mount(8) failed:
> Permission denied" messages we were seeing when selinux was enforcing.
> 
> 
> 
> /shutdown@60(main): umount_a
> [   37.688590] XFS (dm-3): Unmounting Filesystem
> [   37.693948] dracut Warning: Unmounted /oldroot.
> 
> This worked! It failed before.
> 

[…]

Thanks for debugging this down to selinux!

Reassigning

It has almost been a year...is there any fix or workaround for this?

I think they need to take "Triaged" off of the keywords list, because the triage didn't work (at least not for me).

Confirmed on Fedora 26 (Dell DM-061)
LVM filesystem
4.12.5-300.fc26.x86_64
selinux-policy-3.13.1-260.4.fc26

The PC doesn't shutdown properly and it is extremely slow to boot up.

It happens after the last system update. 
I have no issues when booting an old kernel (4.11.8-300.fc26) on the same system.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Please, anyone who is still observing this problem, please, try disabling selinux temporarily by adding "selinux=0" to kernel command line and see if the problem still appears.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(In reply to Peter Rajnoha from comment #93)
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> 
> Please, anyone who is still observing this problem, please, try disabling
> selinux temporarily by adding "selinux=0" to kernel command line and see if
> the problem still appears.
> 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

...for those who still observe:

"Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code."

Tried in a freshly installed VM:
Error still appears in Fedora 26 after installing the latest updates.
After adding "selinux=0" to the kernel command line in /etc/default/grub & a grub2-mkconfig the error message is gone.

(In reply to Dennis Knorr from comment #95)
> Tried in a freshly installed VM:
> Error still appears in Fedora 26 after installing the latest updates.
> After adding "selinux=0" to the kernel command line in /etc/default/grub & a
> grub2-mkconfig the error message is gone.

With selinux enabled and the problematic case, did you also see the "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code." message during shutdown? I'm particularly interested in this one at this moment...

(In reply to Peter Rajnoha from comment #96)

> With selinux enabled and the problematic case, did you also see the "Kernel
> not configured for semaphores (System V IPC). Not using udev synchronisation
> code." message during shutdown? I'm particularly interested in this one at
> this moment...
On one machine, they vanished, on a second one, I am still seeing them.

(In reply to Peter Rajnoha from comment #96)
> (In reply to Dennis Knorr from comment #95)
> > Tried in a freshly installed VM:
> > Error still appears in Fedora 26 after installing the latest updates.
> > After adding "selinux=0" to the kernel command line in /etc/default/grub & a
> > grub2-mkconfig the error message is gone.
> 
> With selinux enabled and the problematic case, did you also see the "Kernel
> not configured for semaphores (System V IPC). Not using udev synchronisation
> code." message during shutdown? I'm particularly interested in this one at
> this moment...

Yes, that's exactly the message I get when SELinux is enabled & set to enforcing (Policy version 3.13.1-260.4.fc26).
With SELinux disabled or in permissive mode, the message is gone.

Switching Kernels from "4.12.5-300.fc26" to version "4.11.8-300.fc26" makes no difference in my case.

Created attachment 1318261 [details]
Screenshot of errors with selinux enabled

Screenshot of the error messages I get when selinux is enabled.

Created attachment 1318262 [details]
Screenshot of errors with selinux disabled

Screenshot of the (different) error messages I get when selinux is disabled.

I see no difference booting with selinux=0.
I think it could be something related to how dracut builds initramfs. The only kernel that gives me no issues is the one provided by F26 installation media (4.11.8-300.fc26).
Both 4.12.5 and 4.12.8 can't shutdown and sleep.

Peter, (comment #93)
I did as you requested and added "selinux=0" to kernel command line.
After booting into the KDE Gui desktop I did following:

I did a reboot and did NOT (NOT) get the error below that I have been getting every time I reboot or shutdown. 

error message with selinux ENABLED when rebooting or shutting down:

"command failed
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on fedora_fedora--25-root failed: Device or resource busy
command failed"

[Note:  the 4 lines above repeat for more than 1 full screen.  Last line to display before reboot is below]

"Failed to read reboot parameter file: no such file or directory
Rebooting."

After this reboot, which did not generate the semaphores messages, I did a boot-up with selinux enabled.  I then rebooted again and the semaphores messages had returned.

So, on my machine this error is repeatable. Booting up with selinux disabled will cause the next reboot NOT to generate the semaphores messages. Booting up with selinux enabled will cause the next reboot to generate the semaphore messages.

I am on Fedora 25 using the following kernels which all display the reboot semaphore messages:
4.12.8-200.fc25.x86_64 (server edition)
4.11.12-200.fc25.x86_64  (server edition)
4.11.11-200.fc25.x86_64  (server edition)
My hardware is an intel NUC7i5.
I am using raid1, LVM, but no encryption.

Tried setting selinux=0 on Feodra 26. No difference. Using a Thinkpad Yoga

(In reply to Peter Rajnoha from comment #93)
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> 
> Please, anyone who is still observing this problem, please, try disabling
> selinux temporarily by adding "selinux=0" to kernel command line and see if
> the problem still appears.
> 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I am using a Dell XPS 13 9360 on a fully updated Fedora 26 and get the following error on shutdown or reboot:

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on luke-guid failed: Device or resource busy
Command failed


After disabling selinux I get:
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed
device-mapper: remove ioctl on luke-guid failed: Device or resource busy
Command failed

(In reply to Peter Rajnoha from comment #93)
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> 
> Please, anyone who is still observing this problem, please, try disabling
> selinux temporarily by adding "selinux=0" to kernel command line and see if
> the problem still appears.
> 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I am using a Dell XPS 13 9360 on a fully updated Fedora 26 and get the following error on shutdown or reboot:

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
device-mapper: remove ioctl on luks-guid failed: Device or resource busy
Command failed


After disabling selinux I get:
device-mapper: remove ioctl on fedora-root failed: Device or resource busy
Command failed
device-mapper: remove ioctl on luks-guid failed: Device or resource busy
Command failed

Paul, 

Do you have any idea how can SELinux blocking this? 

Thanks,
Lukas.

What is generating the "Kernel not configured for semaphores ..." message?  That seems to be the best place as any to start debugging this, but I can't seem to find it in the kernel's device-mapper code (and honestly, it doesn't look like it is coming from the kernel anyway).

Is this coming from some lvm command line?  Dracut/initrd?  Maybe even systemd?

It seems to come from libdm:
https://www.redhat.com/archives/lvm-devel/2010-August/msg00011.html

I have the same problem as described above.

note that from runlevel 5
# telinit 3
does not work.
from runlevel 5, press control-alt F2 ... control-alt F6
this opens a text based console.  from there, you can execute
# telinit 3
successfully.
because the gui was not involved,
# shutdown -h
**** DID **** work successfully from runlevel 3,
but I tried repeating the experiment and it failed the second time.

also tried shutting down from plymouth directly, result was no change.

So is there any way the video driver could have anything to do with
this shutdown issue?

when you boot, just before plymoth starts, this error message flashes on the screen at the top in text mode, high resolution:

[drm:si_dpm_set_power_state [radeon]] *ERROR* si_restrict_performancee_levels_before_switch failed

supposedly you can avoid this by adding the kernel parameter:
radeon.dpm=0
I tried this and nothing much changed.

replaced radeon with an nvidia video card and it still didn't shutdown
but it gave a different error on shutdown in journalctl, but still got the error about system V semaphores 

using BIOS not UEFI.  using MS-DOS not GPT partition table.
using /dev/sda1 with ext4 boot partition.
using /dev/sda2 with LVM physical volume, volume group, and several lvm logical partitions formatted as ext4.
motherboard intel DP35DP bios
-- ACPI suspend state:  S3 (not S1; that didn't work either).
-- on power off:  stay off (not last state; not power on).

# ipcs -l 
shows presence of system V semaphores

there was also some junk in journalctl about spice (kvm virtualization video) and "vdagent", so I turned virtualization kvm module off with
/etc/modprobe.d/blacklist-kvm.conf:
----------
blacklist kvm
blacklist kvm-amd
blacklist kvm-intel
----------
# systemctl disable libvirtd
reboot;


# systemctl status lvm2-lvmetad.service 
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2017-09-12 22:38:33 PDT; 9min ago
     Docs: man:lvmetad(8)
  Process: 558 ExecStart=/usr/sbin/lvmetad -f -t 3600 (code=exited, status=0/SUCCESS)
 Main PID: 558 (code=exited, status=0/SUCCESS)

Sep 12 21:38:28 **** systemd[1]: Started LVM2 metadata daemon.

# systemctl status lvm2-lvmetad.socket 
● lvm2-lvmetad.socket - LVM2 metadata daemon socket
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Tue 2017-09-12 21:38:28 PDT; 1h 9min ago
     Docs: man:lvmetad(8)
   Listen: /run/lvm/lvmetad.socket (Stream)

# systemctl status lvm2-monitor.service 
● lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polli
   Loaded: loaded (/usr/lib/systemd/system/lvm2-monitor.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2017-09-12 21:38:28 PDT; 1h 10min ago
     Docs: man:dmeventd(8)
           man:lvcreate(8)
           man:lvchange(8)
           man:vgchange(8)
  Process: 550 ExecStart=/usr/sbin/lvm vgchange --monitor y --ignoreskippedcluster (code=exited, sta
 Main PID: 550 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/lvm2-monitor.service

Sep 12 21:38:28 **** lvm[550]:   2 logical volume(s) in volume group "main" monitored
Sep 12 21:38:28 **** systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeven

# 

I see this in journalctl on reboot:

Sep 12 21:37:51 **** systemd[1]: Starting Reboot...
Sep 12 21:37:51 **** systemd[1]: Shutting down.
Sep 12 21:37:51 **** lvm[2639]:   3 logical volume(s) in volume group "main" unmonitored
Sep 12 21:37:51 **** systemd[1]: Hardware watchdog 'iTCO_wdt', version 0
Sep 12 21:37:51 **** kernel: watchdog: watchdog0: watchdog did not stop!
Sep 12 21:37:51 **** systemd[1]: Set hardware watchdog to 10min.
Sep 12 21:37:52 **** systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Sep 12 21:37:52 **** systemd-journald[540]: Journal stopped

the one thing I have not tried is switching off selinux.

previous post:
I am talking about:

Error message on shutdown.  "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code."

https://bugzilla.redhat.com/show_bug.cgi?id=1402421

https://bugzilla.redhat.com/show_bug.cgi?id=1359352

which refer to this post.

The message:

  "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code" 

comes from libdm that is used by dmsetup and which in turn is used by dracut's device-mapper module (/usr/lib/dracut/modules.d/90dm/dm-shutdown.sh) where dracut calls "dmsetup remove" to remove all remaining device-mapper-based devices.

I've managed to reproduce and also, I've added more debug messages (the errno) that the libdm code receives when it tries to check whether semaphores are supported, it uses this call:

  semctl(0, 0, SEM_INFO, arg)

The semctl fails, the errno message says:

  Permission denied


WHEN I DISABLE SELINUX, I DON'T HIT THIS PROBLEM. So this is actually a problem with default selinux configuration.


The part of the shutdown log exactly:
...
[   28.311311] dracut Warning: Unmounted /oldroot.
[   28.343303] dracut: Disassembling device-mapper devices
_check_semaphore_is_supported: SEM_INFO failed: Permission denied
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
Powering off.
...


The patch I used to get the errno:

      1 diff --git a/libdm/libdm-common.c b/libdm/libdm-common.c
      2 index e983b0392..e11d8f864 100644
      3 --- a/libdm/libdm-common.c
      4 +++ b/libdm/libdm-common.c
      5 @@ -2150,6 +2150,7 @@ static int _check_semaphore_is_supported(void)
      6         maxid = semctl(0, 0, SEM_INFO, arg);
      7  
      8         if (maxid < 0) {
      9 +               log_sys_error("SEM_INFO", "_check_semaphore_is_supported");
     10                 log_warn("Kernel not configured for semaphores (System V IPC). "
     11                          "Not using udev synchronisation code.");
     12                 return 0;

Peter Rajnoha wrote, "WHEN I DISABLE SELINUX, I DON'T HIT THIS PROBLEM."

Confirmed. The problem being "Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code"

Peter could you boot in permissive mode and see if you get any AVC's, also disable dontaudit rules, before you reboot.

semodule -DB
reboot in permissive mode

When machine comes back up

ausearch -m avc -ts recent
semodule -B

Created attachment 1332339 [details]
Output from ausearch -m avc -ts recent

OK, here's the output I collected from the ausearch (with permissive mode).

Peter I see nothing out of the ordinary there.

Peter this looks like a potential Kernel issue, so I think we should open a different bugzilla.  Not sure this has anything to do with Dracut exhibits AVCs during cleanup.

I agree here, I understand that you see this error with SELinux in enforcing, but I also blame kernel here, there is no AVCs related to this issue.

Can we explain the fact that this happens only in (shutdown) initramfs but not when running from root fs? It's the same kernel running... That's why I thought it might be just some configuration issue related to selinux. Also, does the ausearch log contain ALL the log, including very late shutdown ramfs environment?

The audit.log is supposed to be shutdown as late as possible.  You could look into the journal to see if there are any extra messages there that never made it to the audit log.

On my Desktop running a fully updated Fedora 26, I still get as the last message before shutdown, 

"Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code"

It boots fast and shuts down fast but I get that message on shutdown. Next boot is fine even though I got that message before shut down.

However, if during up time you get a libreoffice hang that requires me to ssh into the box, VPN hangup, nasty network hang or power outage (the system is on a backup), the system will not shut down properly and you get the string of "no semapores ..." message.  If and only if ... before shut down after a one of the above events, I unmount all mounted LVM volumes, smb shares, and any other mounts that have been accessed during the uptime, shut down, and unplug the main power cord to the box for 15 seconds, then I can boot okay and all disks will be fsck checked and corrected. 

If, I do not unmount the LVM volumes and smb shares after one of the above issues, the system will be borked.  Only way to boot again is reset the BIOS and manually fsck all disks in the box.  


Base Board Information
	Manufacturer: ASRock
	Product Name: G41M-VS3 

vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz

Linux fed26 4.12.14-300.fc26.x86_64 #1 SMP Wed Sep 20 16:28:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

linux16 /vmlinuz-4.12.14-300.fc26.x86_64 root=/dev/mapper/fed26-root ro rd.lvm.lv=fed26/root rd.lvm.lv=fed26/swap rhgb quiet LANG=en_US.UTF-8

Tom

(In reply to Daniel Walsh from comment #120)
> The audit.log is supposed to be shutdown as late as possible.  You could
> look into the journal to see if there are any extra messages there that
> never made it to the audit log.

It's too late for the journal even - the journal is stopped before we're hitting this problem in shutdown ramfs and the last audit log is from before the "shutdown.target". I'm getting the logs from serial console attached to the machine:

[  OK  ] Reached target Shutdown.

...

[   72.145587] raw.virt systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[   72.105238] raw.virt systemd-journald[582]: Journal stopped

...

   72.167756] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[   72.173932] systemd-shutdown[1]: Unmounting file systems.
[   72.177028] systemd-shutdown[1]: Remounting '/' read-only with options 'seclabel,data=ordered'.
[   72.187804] EXT4-fs (dm-0): re-mounted. Opts: data=ordered
[   72.195675] systemd-shutdown[1]: Remounting '/' read-only with options 'seclabel,data=ordered'.
[   72.198921] EXT4-fs (dm-0): re-mounted. Opts: data=ordered
[   72.201687] systemd-shutdown[1]: All filesystems unmounted.
[   72.204309] systemd-shutdown[1]: Deactivating swaps.
[   72.206722] systemd-shutdown[1]: All swaps deactivated.
[   72.209523] systemd-shutdown[1]: Detaching loop devices.
[   72.212516] systemd-shutdown[1]: device-enumerator: scan all dirs
[   72.247709] shutdown: 21 output lines suppressed due to ratelimiting
mount: /oldsys/sys: filesystem was mounted, but failed to update userspace mount table.
mount: /oldsys/proc: filesystem was mounted, but failed to update userspace mount table.
mount: /oldsys/run: filesystem was mounted, but failed to update userspace mount table.
mount: /oldsys/dev: filesystem was mounted, but failed to update userspace mount table.
[   72.298744] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[   72.367989] dracut Warning: Unmounted /oldroot.
[   72.387414] dracut: Disassembling device-mapper devices
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
Powering off.

This is still happening in Fedora 27 beta as well.

Encrypted LVM partition.

Could you please remove the keyword "Triaged"? I do not believe this bug is triaged in any way, other than disabling SELinux, which isn't an acceptable triage to me.

I agree with Joseph

I noticed that the error shows only on pc poweroff and not on reboot

@Joseph D. Wagner: The "Triaged" keyword is part of the "Prioritized bugs and issues" process: https://fedoraproject.org/wiki/Fedora_Program_Management/Prioritized_bugs_and_issues_-_the_process

I am getting this on all three of my computer still.

T530 -- fresh install of F27
Dell XPS 13 9343 -- fresh install of F27
Lenovo X1 Carbon -- upgraded from F25 to F27

---
All are using LVM but w/o encryption.

I found an interesting thread:
https://forums.fedoraforum.org/showthread.php?314465-kernel-not-configured-for-semaphore

One person in that thread stated that installing w/o LVM solved the issue. Could this be an LVM issue (encrypted or not)?

Hi, 

I added some fixes to raid SELinux policy could you please try it with following packages: 

https://koji.fedoraproject.org/koji/buildinfo?buildID=1002934 

or for rawhide: 

https://koji.fedoraproject.org/koji/buildinfo?buildID=1002931

Thanks,
Lukas.

That package did not produce any change for me.

The update did not fix it for me either.

Wow - difficult issue to debug.  I get this on all 6 of my f25 and f26 systems (desktop and laptop), all using LVM.

Bumping to Fedora 27. This occurs with a fresh Fedora Workstation install to a blank disk using installer defaults on a Lenovo ThinkPad T560. Looking at the comments, I'm not the only F27 user affected.

Here's my block device layout (with some truncation on the LUKS UUID for easier reading):

[straussd@t560 ~]$ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                     8:0    0   477G  0 disk  
├─sda1                  8:1    0   200M  0 part  /boot/efi
├─sda2                  8:2    0     1G  0 part  /boot
└─sda3                  8:3    0 475.8G  0 part  
  ├─fedora-root       253:0    0    50G  0 lvm   /
  ├─fedora-swap       253:1    0  15.7G  0 lvm   [SWAP]
  └─fedora-home       253:2    0   410G  0 lvm   /home
mmcblk0               179:0    0 119.1G  0 disk  
└─mmcblk0p1           179:1    0 119.1G  0 part  
  └─luks-799[...]929a 253:3    0 119.1G  0 crypt /run/media/straussd/DTS

I'm using Opal to encrypt most of my data (which should be transparent to the installer and OS), but I use LUKS for removable media like the SD card shown above.

From the talk I had with Lukas Vrabec, the issue is now to find a reproducer.

As such, I would like to ask people facing this bug whether they can work on a reproducer, if possible.

I am also adding a QA representative on need-info to check whether the team can help us with a reproducer.

@Jan

On two of my systems, I reproduced this by doing a fresh install of Fedora 26:

1.) Select LVM as partition option and let anaconda create the partition layout for you.

2.) Enable disk encryption

3.) (Optional) Delete the home volume and expand root volume to fill remaining space. I like to have my home on root. I am not sure if this is required to reproduce the issue.

4.) Install

5.) Boot into install and do a dnf update.

6.) Problem starts to occur on later reboots.


I've since updated to Fedora 27 and the problem still occurs. I haven't tried doing a fresh install of F27 yet.

(In reply to Benjamin Xiao from comment #135)

> 3.) (Optional) Delete the home volume and expand root volume to fill
> remaining space. I like to have my home on root. I am not sure if this is
> required to reproduce the issue.
> 

Probably not needed as I would never do that, but I've seen the issue described in this bug.

I can reliably reproduce this issue on minimal Fedora Server 26 and 27 without any updates running in VirtualBox (VirtualBox-5.1.30-2.fc27.x86_64) with EFI and LVM without disk encryption. What can I do to help reproduce this? Would sharing an appliance in OVF help?

Instal Fedora-Workstation-Live-x86_64-27-1.6 to Asus F7E laptop PC.
Install to entire SSD with automatic partitioning and no encryption. Same fault.
Install to first partition of SSD with manual partitioning and no encryption. No problem.
Install to second partition of SSD with automatic partitioning and no encryption.  Same fault.
I can reproduce the fault consistently with automatic partitioning.

John:

When you did the manual partioning did you use LVM?

No, that's my point. LVM causes the bug.

I'm curious if this is related to suspend in someway.  I noticed that I shutdown my laptop after being on for only a few minutes and I didn't see the error with 4.14.7-300.fc27.x86_64.  But when I booted it up again and had to walk away and closed the lid when I came back and shutdown I saw the errors again.  Probably just a random occurrence.  I'm guessing many people here have booted up and shutdown shortly after and still see this error?  Also I see mention of desktops involved which I'm making an assumption that people here don't suspend them as often as laptops which could go against my argument of suspend being related.

No, there have been plenty times when I never used suspend between cold boot and shutdown, but I still got this error. It's LVM + SELinux, whatever it is.

I can confirm that it seems to be a LVM issue or at least is caused by using LVM. Installed F27 with automatic partitioning, updated and the problem occurred during restart.

I then reinstalled without LVM using custom partioning, updated and now the problem is gone

I'm not seeing this on systems with LVM, but without rootfs being on LVM. So something about rootfs being on LVM, like I mention in comment 84:
device-mapper: ioctl: unable to remove open device

I think it's related, but I don't know if that's the cause of the problem, or just another symptom of the problem. I'd guess the fact I have the problem with rootfs on LVM, which would translate into an inability to remove an open device, could then cause these error messages, ultimately systemd gives up and reboots anyway.

Another factor might be plymouth is known to exempt itself from being quit by systemd at reboot/shutdown; and maybe if it's not quitting, that's what's preventing systemd from doing either remount ro or umount of rootfs, and hence dm being unable to remove open device.

Going to try install Fedora 27 with LVM, automated partitioning and withtout the rootfs, I hope I'll catch it.

I was experiencing this problem on a fresh install of Fedora 27 on an HP 15-bs015-dx.

The original installation was done with automatic partitioning and LVM. After reading through the thread I decided to back everything up and re-install using manual partitioning with no logical volume management. I kept the basic partition scheme the same, just didn't use LVM.

Since removing logical volume management the machine has run perfectly with no errors on shutdown for a bit more than 2 weeks now.

I just updated my kernel to the latest one available on F27 and now get the same error on shutdown. My LVM is not encrypted. Did not see this before with a previous kernel version.

> Linux localhost.localdomain 4.14.16-300.fc27.x86_64 #1 SMP Wed Jan 31 19:24:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Attaching a screenshot: https://i.imgur.com/XcxZ5kl.jpg

Same problem here, I've had it since I installed Fedora 27, just started digging into it. I always get the "Kernel not configured for semaphores (System V IPC)" message when shutting down.

For those new, it's SELinux + rootfs on LVM. Get rid of one or the other and it goes away. Encryption has nothing to do with it.

I got to workaround the problem adding to /usr/lib/systemd/system-shutdown/ the following "clean.shutdown" script:

#!/bin/bash
/usr/sbin/setenforce 0 # Be permissive
/usr/bin/plymouth quit # So rootfs can be unmounted
sleep 2

*** Bug 1543694 has been marked as a duplicate of this bug. ***

Created attachment 1393790 [details]
Screenshot

Created attachment 1393791 [details]
Screencast

Thanks Salvador Ortiz, quite helpful until enough SELinux rules are eventually added to remove all of the warnings/errors (or until someone decides to revert the problem causer found by Rudd-O in comment 2).

The device-mapper error itself is not related to SELinux: see bug 1402073.

(Adding myself to the CC list, since I've now encountered this too.)

Just adding a clarification here:

The message "Kernel not configured for semaphores..." emitted by libdm when dmsetup is trying to remove devices is not direct cause for system hang during shutdown.

Dmsetup will not abort device removal operation after getting EPERM on semctl.

On the other hand dmsetup is unable to remove the device because it's still open by some process, i.e. report in bug #1402073.

Both cryptsetup LUKS and LVM2 uses device-mapper block devices hence the bug could be observed in both setups.

I'm not sure how (or even if) the selinux policy fits in the issue. Just one minor oddity: the dmsetup executable is labeled tmpfs_t in shutdown image...

I'm still seeing this on Fedora 28.

Would it be feasible to set up o workaround that would set SELinux into permissive mode for the very last phase of the shutdown process (with all services already down)? If this is SELinux-related, an ugly workaround of that kind may at least prevent the error messages and (more importantly) occasional freezes... This is also related: https://bugzilla.redhat.com/show_bug.cgi?id=1402073#c36

Ondrej, 

Do we have any progress on this issue from LVM side? 

THanks,
Lukas.

I've been having this problem with Gentoo (LVM on LUKS). SELinux wasn't installed and even not compiled in kernel.

Ondrej, 

Based on comment#159, to which component should I move this bugzilla? 

Thanks,
Lukas.

There are (at least) two independent problems.

First, the error message due to miss-configured shutdown environment, (-EPERM on semctl.)

Another one, the device blocked by mounted filesystem or other stacked dm device on top.

The second one may be rerouted to bug #1402073.

The first one is beyond my domain.

Ondrej, any idea who might be able to help with the beyond-your-domain part? Or at least what kind of experts I might need to go looking for in order to get you help? :)

Basically, what are the next steps here?

Hi Matthew,

about the -EPERM on semctl? I may only suggest humbly or speculate. I've been told (hope I recall it correctly) that selinux is supposed to be turned off during shutdown. Is it race then? I mean race between: a) "turn off selinux" and b) "deactivate dm devices"?

Could similar race be a cause for failed umount commands? (Provided tools are wrongly labelled like dmsetup binary is in shutdown initramfs image, see comment #156).

I would ask people familiar with shutdown process in initramfs (systemd/dracut nowadays?) and kernel/lsm if the -EPERM on semctl is really unexpected in this case. But again, for device-mapper the -EPERM on semctl is not a blocker for device deactivation.

Joseph D. Wagner wrote "For those new, it's SELinux + rootfs on LVM. Get rid of one or the other and it goes away. Encryption has nothing to do with it." (Comment 149)

Confirmed.  I had several Fedora releases and an encrypted LVM with root filesystem on an SSD that has long had this bug.  Using Manjaro, I first installed an LVM with root filesystem and no encryption.  No problem.  I then installed SELinux which immediately triggered the bug.

Salvador Ortiz wrote "I got to workaround the problem adding to /usr/lib/systemd/system-shutdown/ the following "clean.shutdown" script:
#!/bin/bash
/usr/sbin/setenforce 0 # Be permissive
/usr/bin/plymouth quit # So rootfs can be unmounted
sleep 2" (Comment 150)

Confirmed.  Manjaro Linux does not include plymouth so that is irrelevant, nor rhgb so Chris Murphy's comments 83,84 re rhgb are also confirmed.

Piotr Szyszkowski wrote "I've been having this problem with Gentoo (LVM on LUKS). SELinux wasn't installed and even not compiled in kernel." Comment 159)

That was not my experience with Manjaro.  LVM on a Manjaro encrypted filesystem without SELinux worked fine for me.

I have 2 systems running F27 + LVM + SElinux (enforcing, targeted) and I have seen this error occur on only one system.  One system is a laptop the other is an old Dell desktop system.  I have only seen the problem on the desktop system and only after upgrading to F27, I don't remember ever seeing this error before then.

This bug is identified as a PrioritizedBug. Since it has had that designation since 2017-05-25, the triage team will revisit it at the 2018-08-29 meeting. If you have any updates, you can provide them here or email triage.org.

I'm still not able to reproduce it.

For guys who are able to reproduce it. Could you attach output of: 
# semodule -lfull | grep unconfined

lvm_t is part of unconfined_domain_type, so it must be some process which is not running under lvm_t SELinux domain.

(In reply to Lukas Vrabec from comment #167)
> I'm still not able to reproduce it.
> 
> For guys who are able to reproduce it. Could you attach output of: 
> # semodule -lfull | grep unconfined
> 
> lvm_t is part of unconfined_domain_type, so it must be some process which is
> not running under lvm_t SELinux domain.

* Thinkpad x201
* Fedora 27 Plasma spin
* SELinux: Enforcing/Targeted
 
This is the output from # semodule -lfull | grep unconfined

100 unconfined        pp         
100 unconfineduser    pp

Ditto to #168.

# semodule -lfull | grep unconfined
100 unconfined        pp         
100 unconfineduser    pp

To be honest, the issue seems to have disappeared for me since Fedora 28. For what it's worth:
# semodule -lfull | grep unconfined
100 unconfined        pp         
100 unconfineduser    pp

Eric, 

Was it fresh installation of Fedora 28 or you upgraded from previous version of Fedora release? 

Thanks,
Lukas.

In my current Fedora 28 (upgraded from F27) I removed my workaround from Comment#150 and can report the issue solved.

Thanks,
Sog

(In reply to Petronald Green from comment #103)
> Tried setting selinux=0 on Feodra 26. No difference. Using a Thinkpad Yoga

Ive had this issue since 25. Can confirm fixed on 28. Been using as primary OS for over a month now.

(In reply to Lukas Vrabec from comment #171)
> Was it fresh installation of Fedora 28 or you upgraded from previous version
> of Fedora release? 

Both: 2 upgrades, one with encrypted partition, one without, and 1 fresh installation (without encrypted partition). All look fine and all have SE Linux enforced.

In reviewing the PrioritizedBugs today, we agreed that this appears to be resolved: https://meetbot.fedoraproject.org/fedora-meeting/2018-10-10/fedora_prioritized_bugs_and_issues.2018-10-10-15.02.log.html#l-132

Magic!

Based on comment#175 and comment#175, Closing this bug as CURRENTRELEASE.