Bug 1803070

Actually it is simpler to rebuild srpm from koji.
https://kojipkgs.fedoraproject.org//packages/systemd/245~rc1/2.fc32/src/systemd-245~rc1-2.fc32.src.rpm

e.g. https://copr.fedorainfracloud.org/coprs/lslebodn/test_rawhide/build/1239672/

But it does not contain commit 3666983037e56865f622c2bc1aa30399b75bd725

Yeah, it fails this way on one of my machines too. I think it's a kernel bug that
has long been fixed (I haven't seen it since F30, but it reproduced reliably with
linux-5.2.11-100.fc29.x86_64.

Unfortunately that build log gives 404 now. Do you know what kernel version was/is
running in copr?

https://github.com/systemd/systemd/pull/14970

Sorry, wrong bug.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> Yeah, it fails this way on one of my machines too. I think it's a kernel bug
> that
> has long been fixed (I haven't seen it since F30, but it reproduced reliably
> with
> linux-5.2.11-100.fc29.x86_64.
> 
> Unfortunately that build log gives 404 now. Do you know what kernel version
> was/is
> running in copr?

Here is a fresh rebuild
https://download.copr.fedorainfracloud.org/results/lslebodn/test_rawhide/fedora-rawhide-x86_64/01259489-systemd/

I briefly checked and I was not able to find anything about version of kernel in logs.
copr devs might know more details.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> Yeah, it fails this way on one of my machines too. I think it's a kernel bug
> that
> has long been fixed (I haven't seen it since F30, but it reproduced reliably
> with
> linux-5.2.11-100.fc29.x86_64.
> 

BTW, feel free to close it if it is a kernel bug. I hit this issue as part of debugging BZ1803240

There's kernel 5.0.16, we did not have to update the cloud images
for builders for some time already.

The issue appears to occur when name_to_handle_at is not available. It is possible that our fallback code
is buggy. Let's leave this open for now.

I was able to reproduce this on F32 with the latest kernel few days back.

$ fedpkg srpm  # fedora master branch
Downloading systemd-245.4.tar.gz
######################################################################## 100.0%
setting SOURCE_DATE_EPOCH=1585699200
Wrote: /tmp/systemd/systemd-245.4-1.fc33.src.rpm
$ mock -r fedora-rawhide-x86_64 /tmp/systemd/systemd-245.4-1.fc33.src.rpm
...
283/490 test-libmount                           OK      0.016834259033203125 s 
284/490 test-mount-util                         OK      0.014326333999633789 s 
285/490 test-mountpoint-util                    FAIL    0.637660026550293 s (killed by signal 6 SIGABRT)
286/490 test-exec-util                          OK      1.1383476257324219 s 
287/490 test-hexdecoct                          OK      0.014191627502441406 s 
...

$ cat ~/.config/mock.cfg
config_opts['plugin_conf']['tmpfs_enable'] = True
config_opts['plugin_conf']['tmpfs_opts']['keep_mounted'] = True
config_opts['nosync'] = True

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.6.3-300.fc32.x86_64  ...

Apparently I'm already CC'ed, but wanted to report that this happens also with the RHEL 8.2 systemd, as well as the latest rawhide (fc33) systemd.

Rawhide output:
https://copr.fedorainfracloud.org/coprs/cyberpear/systemd/build/1367589/

285/491 test-mountpoint-util                    FAIL    0.4175901412963867 s (killed by signal 6 SIGABRT)


Output for EPEL 8, RHEL 8.2 systemd:
https://copr.fedorainfracloud.org/coprs/cyberpear/systemd/build/1367207/

133/314 test-mount-util                         FAIL     0.42 s (killed by signal 6 SIGABRT)

--- command ---
SYSTEMD_KBD_MODEL_MAP='/builddir/build/BUILD/systemd-239/src/locale/kbd-model-map' PATH='/builddir/build/BUILD/systemd-239/x86_64-redhat-linux-gnu:/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin' SYSTEMD_LANGUAGE_FALLBACK_MAP='/builddir/build/BUILD/systemd-239/src/locale/language-fallback-map' /builddir/build/BUILD/systemd-239/x86_64-redhat-linux-gnu/test-mount-util
--- stderr ---
mnt id of /proc/keys is 423
mnt id of /dev/loop9 is 518
mnt id of /proc/sys/kernel/random/boot_id is 433
mnt id of /proc/acpi is 426
mnt id of /proc is 524
mnt id of /dev/loop10 is 509
mnt id of /proc/kcore is 422
mnt id of /run/systemd/nspawn/incoming is 520
mnt id of /proc/timer_list is 425
mnt id of /proc/fs is 428
mnt id of /run is 519
mnt id of /dev is 503
mnt id of /sys/fs/cgroup is 432
mnt id of /dev/pts is 505
mnt id of /dev/loop6 is 515
mnt id of /dev/loop2 is 511
mnt id of /var/cache/yum is 498
mnt id of /dev/loop5 is 514
mnt id of /sys is 501
mnt id of /proc/sys is 420
mnt id of /dev/loop1 is 508
mnt id of /etc/resolv.conf is 521
mnt id of /dev/loop-control is 506
mnt id of /dev/loop7 is 516
mnt id of /var/cache/dnf is 499
mnt id of /proc/sysrq-trigger is 424
mnt id of /dev/loop8 is 517
mnt id of /proc/filesystems is 524
Assertion 'path_equal_ptr(hashmap_get(h, INT_TO_PTR(mnt_id2)), p)' failed at ../src/test/test-mount-util.c:77, function test_mnt_id(). Aborting.
-------

This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

I managed to find a reliable reproducer on one of the fedora test machines. AFAICT, it's a kernel bug.

(gdb) !uname -r
5.6.19-200.fc31.ppc64le

(gdb) !cat /proc/self/mountinfo
697 664 253:0 /var/lib/mock/fedora-31-ppc64le/root / rw,relatime shared:298 master:1 - xfs /dev/mapper/fedora_rh--power--vm14-root rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
698 697 253:0 /var/cache/mock/fedora-31-ppc64le/yum_cache /var/cache/yum rw,relatime shared:299 master:1 - xfs /dev/mapper/fedora_rh--power--vm14-root rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
699 697 253:0 /var/cache/mock/fedora-31-ppc64le/dnf_cache /var/cache/dnf rw,relatime shared:300 master:1 - xfs /dev/mapper/fedora_rh--power--vm14-root rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
700 697 0:32 /mock-selinux-plugin.7me9bfpi /proc/filesystems rw,nosuid,nodev shared:301 master:18 - tmpfs tmpfs rw,seclabel <==========================================================
701 697 0:41 / /sys ro,nosuid,nodev,noexec,relatime shared:302 - sysfs sysfs ro,seclabel
702 701 0:21 / /sys/fs/selinux ro,nosuid,nodev,noexec,relatime shared:306 master:8 - selinuxfs selinuxfs rw
703 697 0:42 / /dev rw,nosuid shared:303 - tmpfs tmpfs rw,seclabel,mode=755
704 703 0:43 / /dev/shm rw,nosuid,nodev shared:304 - tmpfs tmpfs rw,seclabel
705 703 0:45 / /dev/pts rw,nosuid,noexec,relatime shared:307 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=666
706 703 0:6 /btrfs-control /dev/btrfs-control rw,nosuid shared:308 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
707 703 0:6 /loop-control /dev/loop-control rw,nosuid shared:309 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
708 703 0:6 /loop0 /dev/loop0 rw,nosuid shared:310 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
709 703 0:6 /loop1 /dev/loop1 rw,nosuid shared:311 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
710 703 0:6 /loop10 /dev/loop10 rw,nosuid shared:312 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
711 703 0:6 /loop11 /dev/loop11 rw,nosuid shared:313 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
712 703 0:6 /loop2 /dev/loop2 rw,nosuid shared:314 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
713 703 0:6 /loop3 /dev/loop3 rw,nosuid shared:315 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
714 703 0:6 /loop4 /dev/loop4 rw,nosuid shared:316 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
715 703 0:6 /loop5 /dev/loop5 rw,nosuid shared:317 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
716 703 0:6 /loop6 /dev/loop6 rw,nosuid shared:318 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
717 703 0:6 /loop7 /dev/loop7 rw,nosuid shared:319 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
718 703 0:6 /loop8 /dev/loop8 rw,nosuid shared:320 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
719 703 0:6 /loop9 /dev/loop9 rw,nosuid shared:321 master:9 - devtmpfs devtmpfs rw,seclabel,size=4107840k,nr_inodes=64185,mode=755
720 697 0:44 / /run rw,nosuid,nodev shared:305 - tmpfs tmpfs rw,seclabel,mode=755
721 720 0:25 /systemd/nspawn/propagate/9cc8a155d0244558b273f773d2b92142 /run/systemd/nspawn/incoming ro master:12 - tmpfs tmpfs rw,seclabel,mode=755
722 697 0:32 /mock-resolv.dvml91hp /etc/resolv.conf rw,nosuid,nodev shared:322 master:18 - tmpfs tmpfs rw,seclabel
725 697 0:47 / /proc rw,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
603 725 0:47 /sys /proc/sys ro,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
604 725 0:44 /systemd/inaccessible/reg /proc/kallsyms ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
605 725 0:44 /systemd/inaccessible/reg /proc/kcore ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
606 725 0:44 /systemd/inaccessible/reg /proc/keys ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
607 725 0:44 /systemd/inaccessible/reg /proc/sysrq-trigger ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
608 725 0:44 /systemd/inaccessible/reg /proc/timer_list ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
609 725 0:47 /bus /proc/bus ro,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
610 725 0:47 /fs /proc/fs ro,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
611 725 0:47 /irq /proc/irq ro,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
612 725 0:47 /scsi /proc/scsi ro,nosuid,nodev,noexec,relatime shared:323 - proc proc rw
613 703 0:46 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:324 - mqueue mqueue rw,seclabel
614 701 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:325 - cgroup2 cgroup rw,seclabel,nsdelegate
615 603 0:44 /.#proc-sys-kernel-random-boot-id4fbdce67af46d1c2//deleted /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec shared:305 - tmpfs tmpfs rw,seclabel,mode=755
616 725 0:44 /.#proc-sys-kernel-random-boot-id4fbdce67af46d1c2//deleted /proc/sys/kernel/random/boot_id rw,nosuid,nodev shared:305 - tmpfs tmpfs rw,seclabel,mode=755
617 725 0:44 /.#proc-kmsg5b7a8bcfe6717139//deleted /proc/kmsg rw,nosuid,nodev shared:305 - tmpfs tmpfs rw,seclabel,mode=755

The test process does
name_to_handle_at("/proc/filesystems") which returns -EOPNOTSUPP, and then
openat(AT_FDCWD, "/proc/filesystems") which returns 4, and then
read(4, ...) which gives
"pos:\t0\nflags:\t012100000\nmnt_id:\t725\n"

which is clearly inconsistent with "700" in /proc/self/mountinfo.

We could either drop the fallback path (and fail name_to_handle_at() is not avaiable) or ignore the error in the test.
Not sure what is better.

https://github.com/systemd/systemd/pull/17050

FEDORA-2020-0d29e88946 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

FEDORA-2020-dc4f0fb907 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-dc4f0fb907

FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0d29e88946`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-dc4f0fb907 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-dc4f0fb907`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-dc4f0fb907

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-0d29e88946 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0d29e88946`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0d29e88946

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-0d29e88946 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2020-dc4f0fb907 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.