Bug 2148578

Summary: Hit message: couldn't write '1' to 'vm/unprivileged_userfaultfd in the package installation terminal
Product: Red Hat Enterprise Linux 8 Reporter: chhu
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.7CC: hasuzuki, jdenemar, jsuchane, jtomko, lmen, lnykryn, msekleta, systemd-maint-list, virt-maint
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-8.0.0-14.module+el8.8.0+17806+11a519fc Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-16 08:18:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2158160    
Bug Blocks:    

Description chhu 2022-11-26 02:40:23 UTC
Description of problem:
Hit message: "Couldn't write '1' to 'vm/unprivileged_userfaultfd'" in the terminal when installing libvirt-daemon-driver-qemu-8.0.0-10.1.module+el8.7.0

Version-Release number of selected component (if applicable):
systemd-libs-239-68.el8_7.2.x86_64
systemd-udev-239-68.el8_7.2.x86_64
systemd-pam-239-68.el8_7.2.x86_64
systemd-container-239-68.el8_7.2.x86_64
systemd-239-68.el8_7.2.x86_64
libvirt-daemon-8.0.0-10.1.module+el8.7.0+17192+cbc2449b.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Uninstall the libvirt-daemon-driver-qemu package
# rpm -qa|grep libvirt-daemon-driver-qemu
libvirt-daemon-driver-qemu-8.0.0-10.1.module+el8.7.0+17192+cbc2449b.x86_64
# rpm -e libvirt-daemon-driver-qemu-8.0.0-10.1.module+el8.7.0+17192+cbc2449b.x86_64

2. Try to install libvirt-daemon-driver-qemu package, hit message in the terminal:"Couldn't write '1' to 'vm/unprivileged_userfaultfd', ignoring: No such file or directory". 

# rpm -ivh libvirt-daemon-driver-qemu-8.0.0-10.1.module+el8.7.0+17192+cbc2449b.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-daemon-driver-qemu-8.0.0-################################# [100%]
Couldn't write '1' to 'vm/unprivileged_userfaultfd', ignoring: No such file or directory

Actual results:
In step2: Hit the message in the terminal, which confuses customers and think there is something wrong with the installation.

Expected results:
In step2: No such message in the terminal, just logging it

Additional info:

Comment 1 Lukáš Nykrýn 2022-11-28 13:43:16 UTC
It seems that we don't have unprivileged_userfaultfd in RHEL8, so either libvirt needs to drop it from their sysctl conf or this bug should be reassigned to kernel to backport it.

This is the patch for the kernel:
https://lore.kernel.org/all/20190319030722.12441-1-peterx@redhat.com/

Comment 2 Jiri Denemark 2022-11-28 14:36:14 UTC
Well, the problem is not missing vm/unprivileged_userfaultfd because the
postcopy-migration.sysctl file contains

    -vm.unprivileged_userfaultfd = 1

In other words, we explicitly request write failures to be ignored. However
systemd still logs the error. Logging it is fine, after all, the log message
has an info (or notice, I don't remember now) level so it's ok. But the
question is whether printing it on stderr makes sense.

Anyway, this is not a bug from libvirt POV as we mark the settings with "-".
We could remove the file completely from RHEL-8, but it seems keeping it and
marking as "ignore failures" is more future proof as there would be nothing to
worry about should the parameter be backported to RHEL-8.

If you think this is not an issue from systemd POV, please explain it and
close the bug.

Comment 3 Ján Tomko 2022-11-28 14:46:51 UTC
Looks like RHEL8 libvirt is missing the change that ignored the failures:
commit 558f00397a0d46ad22bf53a22a40ed6fc4fdb5eb

    qemu: Ignore missing vm.unprivileged_userfaultfd sysctl

git describe: v8.0.0-328-g558f00397a contains: v8.1.0~154

Comment 4 Jiri Denemark 2022-11-28 15:20:45 UTC
Heh. I wonder how this happened. I thought I was adding the "-" prefix specifically because of RHEL-8 :-(

Anyway, this clearly is libvirt bug then.

Comment 5 Jiri Denemark 2022-11-28 15:27:30 UTC
However, would the dash actually help with systemd (as it is ignoring failures
even without it)?

Comment 6 Michal Sekletar 2023-01-04 11:38:01 UTC
There is a bug in systemd-sysctl on RHEL-8, we print the error message even in the case when option name is prefixed with "-". This is inconsistent with the current upstream behavior which was changed in, https://github.com/systemd/systemd/commit/32458cc9687c1b60ff0f22c0e71da93ce78b1534 . As for ignoring failures, recently there has been a strict mode (sysctl exits with error if write wasn't successful) introduced for systemd-sysctl in https://github.com/systemd/systemd/commit/e88748c17e58aad6818e64fd3071de011808165e . However, it is not used by default and we wouldn't be able to used it by default as well.

Comment 7 Michal Sekletar 2023-01-04 11:51:41 UTC
(In reply to Michal Sekletar from comment #6)

> There is a bug in systemd-sysctl on RHEL-8, we print the error message even
> in the case when option name is prefixed with "-".

I have now filled https://bugzilla.redhat.com/show_bug.cgi?id=2158160.

Comment 9 yalzhang@redhat.com 2023-01-12 05:40:22 UTC
Reproduce the bug on libvirt-8.0.0-13 with systemd-239-69.el8.x86_64:

1. install the libvirt related packages;

2. remove libvirt-daemon-driver-qemu package:
# rpm -e libvirt-daemon-driver-qemu  libvirt-daemon-kvm libguestfs

3. download the package, and install by rpm:
# rpm -ivh libvirt-daemon-driver-qemu-8.0.0-13.module+el8.8.0+17719+f18c2d1b.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-daemon-driver-qemu-8.0.0-################################# [100%]
Couldn't write '1' to 'vm/unprivileged_userfaultfd', ignoring: No such file or directory

Test on libvirt-8.0.0-14 with systemd-239-69.el8.x86_64:
1. install all libvirt related packages;

2. remove libvirt-daemon-driver-qemu package:
# rpm -e libvirt-daemon-driver-qemu  libvirt-daemon-kvm  libvirt

3. install libvirt-daemon-driver-qemu by rpm:
# rpm -ivh libvirt-daemon-driver-qemu-8.0.0-14.module+el8.8.0+17806+11a519fc.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-daemon-driver-qemu-8.0.0-################################# [100%]
Couldn't write '1' to 'vm/unprivileged_userfaultfd', ignoring: No such file or directory

It fails because the systemd bug is not fixed. I will retest it after bug 2158160 fixed.

Comment 12 yalzhang@redhat.com 2023-01-28 04:06:11 UTC
Test on libvirt-8.0.0-14.module+el8.8.0 with systemd-239-70.el8, the result is as expected:

1. install all libvirt related packages;
2. remove libvirt-daemon-driver-qemu;
# rpm -e libvirt-daemon-driver-qemu  libvirt-daemon-kvm  libvirt  libguestfs
3. install libvirt-daemon-driver-qemu by rpm command:
# rpm -ivh ./libvirt-daemon-driver-qemu-8.0.0-14.module+el8.8.0+17806+11a519fc.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libvirt-daemon-driver-qemu-8.0.0-################################# [100%]

no error occurs, the issue is fixed.

Comment 14 errata-xmlrpc 2023-05-16 08:18:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2757