Description of problem: Kernel.orgs compiled and installed with Fedora 18 installations give bug 979692 but also sit forever on 'A start job is running for dev-mapper-swap.device'. Older kernels, from F17 days, boot fine. Version-Release number of selected component (if applicable): Fedora 18 current. How reproducible: Boot. Fix bug 979692 by doing `lvm vgchange -ay ; exit` in dracut shell. See box boot further. See boot sit at 'A start job is running for dev-mapper-swap.device' with the twirly thing going back and forth. Actual results: See above. Expected results: No wait forever. Quick, predictable, problemfree encrypted swap activation. Additional info:
Let me correct that: 'Older kernels, from F17 days, boot fine.' is false. Nothing boots with something in crypttab for swap and fstab. This explains the priority. A working system is severely disabled.
What info do you need to fix this bug?
Is it reproducible with a Fedora packaged kernel?
I do not know as I generally do not run redhat kernels. Is this all dependant on certain kernel magic nowadays? I did not change my side of the magic.
Only thing that changes was the kernel version besides the distro version.
# lsinitrd /boot/initramfs-$(uname -r).img etc/crypttab # Is that factor?
(In reply to udo from comment #6) > # lsinitrd /boot/initramfs-$(uname -r).img etc/crypttab > # > > Is that factor? Yes, that is used in the initramfs, if your root device is encrypted.
(In reply to Harald Hoyer from comment #7) > (In reply to udo from comment #6) > > # lsinitrd /boot/initramfs-$(uname -r).img etc/crypttab > > # > > > > Is that factor? > > Yes, that is used in the initramfs, if your root device is encrypted. Oh, you probably meant to say, that /etc/crypttab is empty in the initramfs.
What is your kernel command line? Does adding "rd.lvm.vg=<volumegroup_of_the_swap>" to the kernel command line fix your issue?
Command line: BOOT_IMAGE=/vmlinuz-3.9.9 root=/dev/mapper/my_vg-rootlv ro LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us acpi_enforce_resources=lax fbcon=font:SUN12x22 vga=0xF07 threadirqs radeon.pcie_gen2=1 radeon.modeset=1 video=HDMI-A-1:1280x720@50i plymouth.enable=0 rd.lvm.vg=my_vg swap is on /dev/sda2. # cat /etc/crypttab swap /dev/sda2 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 When I leave out the /dev/mapper/swap entry in /etc/fstab (but have this crypttab) the system does boot and does not hang. But we see: # swapon -a swapon: /dev/sda2: read swap header failed: Invalid argument # mkswap /dev/sda2 /dev/sda2: Device or resource busy # systemctl |grep swap dev-mapper-swap.device loaded inactive dead start dev-mapper-swap.device systemd-...p loaded active exited Cryptography Setup for swap dev-sda2.swap loaded failed failed /dev/sda2 swap.target loaded active active Swap So what is causing this?
I am not going to look into any more of your bugreports until you are willing to try to reproduce them on a Fedora kernel.
Where did I imply unwillingness? I explained my general way of doing thingswhich worked quite well. Please help explain: Where is the dependency of what in the kernel versus the on-disk configuration files? And what if it 'works' with a Fedora kernel?...
Installed: kernel.x86_64 0:3.9.6-200.fc18 And sure it works. Thank you. Now what? What is the special sause? Did the recipe change when I upgraded to Fedora 18? Because that is the question.
(In reply to udo from comment #13) > Now what? What is the special sause? I don't know, but you can find out: Adjust your custom kernel .config incrementally to approach Fedora's config until it starts working.
I'll make one guess anyway: CONFIG_DM_UEVENT ?
Well, when we boot with fstab not referencing /dev/mapper/swap we can boot OK and then edit fstab to make it point to /dev/mapper/swap. (remove a hash sign) If we then do `swapon -a` then we see a succesfull swap activation. This means that the job taht we see stuck is not mentioning the right step of the process. Without /dev/mapper/swap in fstab the cryptoswap is created OK and later manually activated. With /dev/mapper/swap in fstab the task does not complete. So what is the correlation between a line in /etc/fstab versus just doing what is in crypttab? The kernel does have support for the crypto target as it had for years. When the fedora kernel was installed teh ramdisk was built the same way as I do as I saw the same stuff occurring. Yet the results were different than a simpel `make modules_install install` ...
With no swap in /etc/fstab (all hashed out) teh box boots with a kernel.org kernel. Then I can remove a hash sign from the /dev/mapper/swap line in /etc/fstab and do a swapon -a without problems. So the issue is with parsing the fstab. # cat /etc/fstab # # /etc/fstab # Created by anaconda on Mon Mar 1 16:46:47 2010 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/my_vg-rootlv / ext4 defaults,noatime,nodiratime 1 1 /dev/mapper/my_vg-mythlv /Myth jfs defaults,noatime,nodiratime,nodev,nosuid 1 2 /dev/sda1 /boot ext2 defaults,noatime,nodiratime,nodev,nosuid,noacl 1 2 #UUID=2cded850-3ea8-4b35-a9fc-c9e8ee8c138e /boot ext3 defaults,noatime,nodev,nosuid 1 2 /dev/mapper/my_vg-homelv /home ext4 defaults,noatime,nodiratime,nodev 1 2 /dev/mapper/my_vg-usrlv /usr ext4 defaults,noatime,nodiratime,nodev 1 2 /dev/mapper/my_vg-srclv /usr/src ext4 defaults,noatime,nodiratime,nodev 1 2 /dev/mapper/my_vg-varlv /var ext4 defaults,noatime,nodiratime,nodev,nosuid 1 2 #UUID=a906a32d-8396-4002-ada4-cc57bcb1d3e3 swap swap defaults 0 0 /dev/mapper/swap swap swap defaults 0 0 #/dev/sda2 swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 tmpfs /tmp tmpfs defaults 0 0
(In reply to Michal Schmidt from comment #15) > I'll make one guess anyway: CONFIG_DM_UEVENT ? Thanks. # grep CONFIG_DM_UEVENT .config # CONFIG_DM_UEVENT is not set But it makes no difference. With swap in fstab it hangs. Without swap in fstab it boots and /dev/mapper/swao exists. And after changing fstab to have a proper reference to /dev/mapper/swap I can do swapon -a as it should work.
# mv /etc/crypttab /etc/crypttab.udo # dracut -f # reboot Same behaviour, it hangs on the /dev/mapper/swap job as that target does not exist. So that must be the cause in the normal situation. But /dev/mapper/swap DOES exist when we boot without swap mention in fstab....
It's not a dracut problem. (In reply to udo from comment #17) > With no swap in /etc/fstab (all hashed out) teh box boots with a kernel.org > kernel. > /dev/mapper/swap swap swap defaults 0 0 > #/dev/sda2 swap swap defaults 0 0 So, you declare /dev/sda2 _and_ /dev/mapper/swap as swap? That cannot work, because your /etc/crypttab says: swap /dev/sda2 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256 Which means: take "/dev/sda2", encrypt it and create /dev/mapper/swap. So, you must not specify /dev/sda2 anymore anywhere!
No. /dev/mapper/swap as swap is the normal situation. But because of the issue I had to work around. And thus using sda2 (without encryption) was considered. Please note the hash sign which effectively disables the line. Furthermore: The (systemd?) job that hangs the boot has no timeout while in Fedora 17 it had because of I noticed with the 'does not activate encrypted swap' bug that is still open. Because of the lack of a timeout I cannot see what it is trying to run, what any results of such tries are and thus I cannot conclude how I can fix the situation. Of course I may be overlooking features that allow me to experience a usable timeout, etc but please consider helping me out here.
A running start job for a *.device unit means that systemd is waiting for a notification from udev about the device becoming available. My suggestion is to follow my advice in comment #14.
So a lack of timeout where there was one once will be fixed soon? Thanks, great news! I do think that wildly changing kernel parameters is a not so optimal path towards a solution. I'd first want to have more insight in the dependencies of the process. We know from my previous tries that the crypto swap target /dev/mapper/swap is created fine. We just do not know how soon or how well. Because of the supposed wonders of udev and despite udev being configured -see comment 18-, we have an issue that was not there in the old SysV days. Even worse: we have less ways to debug the issue. As I noted I cannot even see the code that the dev-mapper-swap job is trying to execute. Let alone possible results. It is a closed, dark box. So please explain me where to find what is being executed in the /dev/mapper/swpa task.
(In reply to udo from comment #23) > I do think that wildly changing kernel parameters is a not so optimal path > towards a solution. Why "wildly"? We know it works with the Fedora kernel. We know it does not work with your kernel config. Narrowing the difference down to a specific config option would provide a great clue about the issue. > As I noted I cannot even see the code that the dev-mapper-swap job is trying > to execute. Let alone possible results. It is a closed, dark box. > So please explain me where to find what is being executed in the > /dev/mapper/swpa task. There is no active component in starting *.device units. It's just sitting there waiting for the udev notification to arrive.
# udevadm info -p /sys/class/block/dm-6 -q all P: /devices/virtual/block/dm-6 N: dm-6 S: disk/by-id/dm-name-swap S: disk/by-id/dm-uuid-CRYPT-PLAIN-swap S: mapper/swap E: DEVLINKS=/dev/disk/by-id/dm-name-swap /dev/disk/by-id/dm-uuid-CRYPT-PLAIN-swap /dev/mapper/swap E: DEVNAME=/dev/dm-6 E: DEVPATH=/devices/virtual/block/dm-6 E: DEVTYPE=disk E: DM_NAME=swap E: DM_SUSPENDED=0 E: DM_UDEV_PRIMARY_SOURCE_FLAG=1 E: DM_UDEV_RULES_VSN=2 E: DM_UUID=CRYPT-PLAIN-swap E: MAJOR=254 E: MINOR=6 E: SUBSYSTEM=block E: SYSTEMD_READY=0 E: TAGS=:systemd: E: UDISKS_DM_TARGETS_COUNT=1 E: UDISKS_DM_TARGETS_LENGTH=4194304 E: UDISKS_DM_TARGETS_START=0 E: UDISKS_DM_TARGETS_TYPE=crypt E: UDISKS_PRESENTATION_NOPOLICY=1 E: USEC_INITIALIZED=935266 Also: # dev=$(readlink -f /dev/mapper/swap) # udevadm test /block/${dev#/dev/} calling: test version 201 This program is for debugging only, it does not run any program specified by a RUN key. It may show incorrect results, because some values may be different, or not available at a simulation run. === trie on-disk === tool version: 201 file size: 5632867 bytes header size 80 bytes strings 1260755 bytes nodes 4372032 bytes load module index read rules file: /usr/lib/udev/rules.d/10-dm.rules read rules file: /usr/lib/udev/rules.d/11-dm-lvm.rules read rules file: /usr/lib/udev/rules.d/13-dm-disk.rules read rules file: /etc/udev/rules.d/20-crystalhd.rules read rules file: /usr/lib/udev/rules.d/40-libgphoto2.rules IMPORT found builtin 'usb_id --export %%p', replacing /usr/lib/udev/rules.d/40-libgphoto2.rules:11 read rules file: /usr/lib/udev/rules.d/40-usb-media-players.rules read rules file: /usr/lib/udev/rules.d/42-usb-hid-pm.rules read rules file: /usr/lib/udev/rules.d/50-firmware.rules read rules file: /usr/lib/udev/rules.d/50-udev-default.rules read rules file: /usr/lib/udev/rules.d/56-hpmud.rules read rules file: /usr/lib/udev/rules.d/60-cdrom_id.rules read rules file: /usr/lib/udev/rules.d/60-ffado.rules read rules file: /usr/lib/udev/rules.d/60-fprint-autosuspend.rules read rules file: /usr/lib/udev/rules.d/60-net.rules read rules file: /usr/lib/udev/rules.d/60-persistent-alsa.rules read rules file: /usr/lib/udev/rules.d/60-persistent-input.rules read rules file: /usr/lib/udev/rules.d/60-persistent-serial.rules read rules file: /usr/lib/udev/rules.d/60-persistent-storage-tape.rules read rules file: /usr/lib/udev/rules.d/60-persistent-storage.rules read rules file: /usr/lib/udev/rules.d/60-persistent-v4l.rules read rules file: /usr/lib/udev/rules.d/60-raw.rules read rules file: /usr/lib/udev/rules.d/61-accelerometer.rules read rules file: /usr/lib/udev/rules.d/61-gnome-bluetooth-rfkill.rules read rules file: /usr/lib/udev/rules.d/64-btrfs.rules read rules file: /usr/lib/udev/rules.d/64-md-raid.rules read rules file: /usr/lib/udev/rules.d/65-libsane.rules read rules file: /usr/lib/udev/rules.d/65-libwacom.rules read rules file: /usr/lib/udev/rules.d/65-md-incremental.rules read rules file: /usr/lib/udev/rules.d/69-cd-sensors.rules IMPORT found builtin 'usb_id --export %p', replacing /usr/lib/udev/rules.d/69-cd-sensors.rules:89 read rules file: /usr/lib/udev/rules.d/69-dm-lvm-metad.rules read rules file: /usr/lib/udev/rules.d/69-libmtp.rules read rules file: /usr/lib/udev/rules.d/69-xorg-vmmouse.rules read rules file: /etc/udev/rules.d/70-persistent-cd.rules read rules file: /etc/udev/rules.d/70-persistent-net.rules read rules file: /usr/lib/udev/rules.d/70-power-switch.rules read rules file: /usr/lib/udev/rules.d/70-printers.rules read rules file: /usr/lib/udev/rules.d/70-uaccess.rules read rules file: /usr/lib/udev/rules.d/70-wacom.rules read rules file: /usr/lib/udev/rules.d/71-seat.rules read rules file: /usr/lib/udev/rules.d/73-seat-late.rules read rules file: /usr/lib/udev/rules.d/75-net-description.rules read rules file: /usr/lib/udev/rules.d/75-probe_mtd.rules read rules file: /usr/lib/udev/rules.d/75-tty-description.rules read rules file: /usr/lib/udev/rules.d/77-mm-ericsson-mbm.rules read rules file: /usr/lib/udev/rules.d/77-mm-longcheer-port-types.rules read rules file: /usr/lib/udev/rules.d/77-mm-nokia-port-types.rules read rules file: /usr/lib/udev/rules.d/77-mm-pcmcia-device-blacklist.rules read rules file: /usr/lib/udev/rules.d/77-mm-platform-serial-whitelist.rules read rules file: /usr/lib/udev/rules.d/77-mm-simtech-port-types.rules read rules file: /usr/lib/udev/rules.d/77-mm-usb-device-blacklist.rules read rules file: /usr/lib/udev/rules.d/77-mm-x22x-port-types.rules read rules file: /usr/lib/udev/rules.d/77-mm-zte-port-types.rules read rules file: /usr/lib/udev/rules.d/77-nm-olpc-mesh.rules read rules file: /usr/lib/udev/rules.d/78-sound-card.rules read rules file: /usr/lib/udev/rules.d/80-drivers.rules read rules file: /usr/lib/udev/rules.d/80-mm-candidate.rules read rules file: /usr/lib/udev/rules.d/80-net-name-slot.rules read rules file: /usr/lib/udev/rules.d/80-udisks.rules read rules file: /usr/lib/udev/rules.d/80-udisks2.rules read rules file: /etc/udev/rules.d/85-directfb.rules specified group 'input' unknown read rules file: /usr/lib/udev/rules.d/85-regulatory.rules read rules file: /usr/lib/udev/rules.d/85-usbmuxd.rules read rules file: /usr/lib/udev/rules.d/90-alsa-restore.rules read rules file: /etc/udev/rules.d/90-hal.rules read rules file: /usr/lib/udev/rules.d/90-pulseaudio.rules read rules file: /usr/lib/udev/rules.d/91-drm-modeset.rules read rules file: /usr/lib/udev/rules.d/95-cd-devices.rules read rules file: /usr/lib/udev/rules.d/95-dm-notify.rules read rules file: /usr/lib/udev/rules.d/95-keyboard-force-release.rules read rules file: /usr/lib/udev/rules.d/95-keymap.rules read rules file: /usr/lib/udev/rules.d/95-osinfo.rules read rules file: /usr/lib/udev/rules.d/95-rtirq.rules read rules file: /usr/lib/udev/rules.d/95-udev-late.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-dell.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-fujitsu.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-gateway.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-ibm.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-lenovo.rules read rules file: /usr/lib/udev/rules.d/95-upower-battery-recall-toshiba.rules read rules file: /usr/lib/udev/rules.d/95-upower-csr.rules read rules file: /usr/lib/udev/rules.d/95-upower-hid.rules read rules file: /usr/lib/udev/rules.d/95-upower-wup.rules read rules file: /usr/lib/udev/rules.d/97-bluetooth-serial.rules read rules file: /etc/udev/rules.d/99-libftdi.rules read rules file: /usr/lib/udev/rules.d/99-systemd.rules rules contain 393216 bytes tokens (32768 * 12 bytes), 43745 bytes strings 33107 strings (275961 bytes), 29183 de-duplicated (236141 bytes), 3925 trie nodes used unable to open device '/sys/block/mapper/swap' unload module index (this is the /dev/mapper/swpa which was manually activated with swapon -a after editing fstab)
/lib/udev/rules.d/99-systemd.rules has: # Ignore encrypted devices with no identified superblock on it, since # we are probably still calling mke2fs or mkswap on it. #SUBSYSTEM=="block", KERNEL!="ram*|loop*", ENV{DM_UUID}=="CRYPT-*", ENV{ID_PART_TABLE_TYPE}=="", ENV{ID_FS_USAGE}=="", ENV{SYSTEMD_READY}="0" The comment is probably correct. But that raises the question: How and where is the mkswap/mkfs performed and what changes SYSTEMD_READY to contain "1"? It appears that THAT step is not performed but that the /dev/mapper/swap is correctly created and that mkswap is run on the device. (as we can manually activate the swap without further work on the device) The kernel contains the UEVENT parameter: # gzip -dc /proc/config.gz |grep _DM_ # CONFIG_DM_DEBUG is not set CONFIG_DM_CRYPT=y # CONFIG_DM_SNAPSHOT is not set # CONFIG_DM_THIN_PROVISIONING is not set # CONFIG_DM_CACHE is not set # CONFIG_DM_MIRROR is not set # CONFIG_DM_RAID is not set # CONFIG_DM_ZERO is not set # CONFIG_DM_MULTIPATH is not set # CONFIG_DM_DELAY is not set CONFIG_DM_UEVENT=y # CONFIG_DM_FLAKEY is not set # CONFIG_DM_VERITY is not set So that cannot be a problem. Please explain the mkswap/mkfs and SYSTEMD_READY=1 part.
We find: # cat systemd-cryptsetup # Automatically generated by systemd-cryptsetup-generator [Unit] Description=Cryptography Setup for %I Documentation=man:systemd-cryptsetup@.service(8) man:crypttab(5) SourcePath=/etc/crypttab Conflicts=umount.target DefaultDependencies=no BindsTo=dev-mapper-%i.device After=systemd-readahead-collect.service systemd-readahead-replay.service Before=cryptsetup.target After=systemd-random-seed-load.service BindsTo=dev-sda2.device After=dev-sda2.device Before=umount.target [Service] Type=oneshot RemainAfterExit=yes TimeoutSec=0 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach 'swap' '/dev/sda2' '/dev/urandom' 'swap,cipher=aes-cbc-essiv:sha256,size=256' ExecStop=/usr/lib/systemd/systemd-cryptsetup detach 'swap' ExecStartPost=/sbin/mkswap '/dev/mapper/swap' I see no direct action to manipulate SYSTEMD_READY after doing the swap creation. Is this performed in /lib/systemd/system-generators/systemd-cryptsetup-generator ? # cat dev-mapper-swap.device.d/50-job-timeout-sec-0.conf # Automatically generated by systemd-cryptsetup-generator [Unit] JobTimeoutSec=0 This is not so useful for debugging.... As the conf is created by systemd-cryptsetup-generator, how do we change the timeout?
(In reply to udo from comment #26) > How and where is the mkswap/mkfs performed and what changes SYSTEMD_READY to > contain "1"? mkswap is run by systemd-cryptsetup, as its ExecStartPost. udev has an inotify watch set on the block device (verify it with "ls -l /run/udev/watch"). When mkswap closes the device, udev gets the inotify event and synthesises a "change" event on the block device. This causes it to process the udev rules again. This time the ID_* fields are expected to be filled in, so "SYSTEMD_READY=0" will not be set.
(In reply to udo from comment #26) > # gzip -dc /proc/config.gz |grep _DM_ ... > So that cannot be a problem. Right. But you should not limit your search to the _DM_ config options. When looking for other suitable candidates, I suggest you start with the ones listed in the REQUIREMENTS section in /usr/share/doc/systemd/README
THanks for the reference to /usr/share/doc/systemd/README. All of the mandatory stuff was enabled. I made some adjustments where legacy could be in the way. Please explain where SYSTEMD_READY=0 is removed or set to SYSTEMD_READY=1.
(In reply to udo from comment #30) > I made some adjustments where legacy could be in the way. Which option(s) was it exactly? Did it fix the problem? > Please explain where SYSTEMD_READY=0 is removed or set to SYSTEMD_READY=1. At the start of the processing of each event the list of properties is clear. Therefore nothing needs to explicitly remove it. It is sufficient that no rule assigns it during the processing of the "change" event.
I just tested and now it works! So I guess (!) that it must be one of the `optional` items from the README. (hotplug?) No more SYSTEMD_READY. To be more sure about the functionality, more reboots are required to be sure it is not halfway functional like in bug 759402 but I have good hopes. Thanks for assisting!
So to make it more clear what was changed: -CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" +CONFIG_UEVENT_HELPER_PATH="" -# CONFIG_DM_UEVENT is not set +CONFIG_DM_UEVENT=y -CONFIG_EXPORTFS=m # unrelated? +CONFIG_EXPORTFS=y