Bug 1904008
| Summary: | livemedia-creator aborts on /mnt/sysimage/run unmount failed | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Gilles Buloz <gilles.buloz> | ||||||
| Component: | lorax | Assignee: | Brian Lane <bcl> | ||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 33 | CC: | anaconda-maint-list, bcl, fedoraproject, filbranden, flepied, kasong, lnykryn, msekleta, reallylongword, ssahani, s, systemd-maint, yuwatana, zbyszek | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2021-10-01 22:57:47 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
From the end of livemedia.log it looks like systemd has grabbed /mnt/sysimage/run/user/1001 for some reason. Could you attach a copy of your kickstart?
These are from Anaconda trying to unmount run:
2020-11-12 11:01:30,964 INFO pylorax: Error in atexit._run_exitfuncs:
2020-11-12 11:01:30,966 INFO pylorax: Traceback (most recent call last):
2020-11-12 11:01:30,967 INFO pylorax: File "/usr/lib/python3.9/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply
2020-11-12 11:01:30,987 INFO pylorax: return self._handle_method_error(error)
2020-11-12 11:01:30,989 INFO pylorax: File "/usr/lib/python3.9/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error
2020-11-12 11:01:30,992 INFO pylorax: raise exception from None
2020-11-12 11:01:30,993 INFO pylorax: dasbus.error.DBusError: umount of /mnt/sysimage/run failed (32)
This is livemedia-creator trying to clean up after Anaconda:
2020-11-12 11:03:33,092 INFO livemedia-creator: Shutting down log processing
2020-11-12 11:03:33,127 WARNING pylorax.imgutils: failed to unmount /mnt/sysimage/run/user/1001. retrying (1/3)...
2020-11-12 11:03:33,155 DEBUG pylorax.imgutils: fuser -vm:
USER PID ACCESS COMMAND
/mnt/sysimage/run/user/1001:
root kernel mount /run/user/1001
gbl 28218 F.... systemd
2020-11-12 11:03:34,163 WARNING pylorax.imgutils: failed to unmount /mnt/sysimage/run/user/1001. retrying (2/3)...
2020-11-12 11:03:34,176 DEBUG pylorax.imgutils: fuser -vm:
USER PID ACCESS COMMAND
/mnt/sysimage/run/user/1001:
root kernel mount /run/user/1001
gbl 28218 F.... systemd
2020-11-12 11:03:35,183 ERROR pylorax: Cleanup of /mnt/sysimage/run/user/1001 failed. See program.log for details
2020-11-12 11:03:35,186 WARNING pylorax.imgutils: failed to unmount /mnt/sysimage/run. retrying (1/3)...
2020-11-12 11:03:35,199 DEBUG pylorax.imgutils: fuser -vm:
USER PID ACCESS COMMAND
/mnt/sysimage/run: root kernel mount /run
root 1 F.... systemd
root 430 F...m systemd-journal
root 440 F.... systemd-udevd
root 531 F.... NetworkManager
root 534 F.... systemd-homed
chrony 537 F.... chronyd
dbus 539 F.... dbus-broker-lau
dbus 541 F.... dbus-broker
root 559 f.... systemd-logind
root 595 F.... systemd-userdbd
root 920 F.... login
root 28213 F.... sshd
gbl 28230 F.... sshd
root 28444 F.... systemd-userwor
root 100293 F.... systemd-userwor
root 100294 F.... systemd-userwor
Created attachment 1736391 [details]
The custom kickstart passed to livemedia-creator
This is a kickstart file based on official Xfce spin kickstart, but with changes for our remix live image (changes are marked with the word "TRACe").
Two additional %post scripts are added at the end of file to build and install extra custom software to the image.
It looks like your %post is doing quite a bit with rpmbuild and rpm which is unusual for a live build. I would try simplifying your setup to see if you can narrow it down. But in general, changes and packages should be added to a repo and installed by anaconda, not from a %post, and not built while running inside anaconda. It looks like this is happening because of user 1001 inside the install, which is odd because there shouldn't be any extra users until the live iso is booted. Figuring out what user 1001 is may shed some light on this. I'm going to move this to systemd, since the problem is really that it is holding onto the mount. There's nothing livemedia-creator or anaconda can do about that. Maybe they can help you debug it in more detail. Thanks Brian, I'll try the Xfce spin kickstart without my addons. Additional info : - the UID 1001 is the one of user "gbl" of by VM from which I run "sudo livemedia-creator ..." - I haven't seen the problem again with my VM configured with 4 CPUs cores. Only saw it with only one CPU. This looks like a timing related issue. - I build some packages from %post before some contain extra kernel modules and must be built for the live kernel, so it's easier for me to build and install then from %post instead of building then separately and adding them to my local repo. - One of these packages has some "systemctl enable xxx" in its RPM postinstall script, so this systemctl is run from the %post. But I see the original Xfce kickstart also has a "systemctl enable tmp.mount" run from a %post, so this should not be a problem. The user manager will open sockets under /run/user/UID/. This causes 'fuser -vm' to show 'F systemd' in the listing.
So it is expected that systemd will block /run/user/UID/ while it is running. It is not clear why it is running
at all though. But I think this is something to be figured out from the livemedia-creator side, because this
is a very special environment.
> One of these packages has some "systemctl enable xxx" in its RPM postinstall script, so this systemctl is run from the %post.
This systemctl will talk to the system instance. So even if it had some effect, I don't see how it could cause
the *user* manager to be started.
We had some bug about systemd doing too much from scriptlets executed in a chroot, that might be related.
But I can't find it now. I'll post a link if I do.
Sorry, I don't think I can help with this until there's more info. *Something* causes the user instance to
start, but until we know what and why, I can't say whether systemd is behaving correctly.
|
Created attachment 1736036 [details] log files produced by livemedia-creator Description of problem: The live media generation fails with the following messages : 2020-11-12 11:01:30,964: Error in atexit._run_exitfuncs: 2020-11-12 11:01:30,966: Traceback (most recent call last): 2020-11-12 11:01:30,967: File "/usr/lib/python3.9/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply 2020-11-12 11:01:30,987: return self._handle_method_error(error) 2020-11-12 11:01:30,989: File "/usr/lib/python3.9/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error 2020-11-12 11:01:30,992: raise exception from None 2020-11-12 11:01:30,993: dasbus.error.DBusError: umount of /mnt/sysimage/run failed (32) Version-Release number of selected component (if applicable): livemedia-creator v33.10-1 How reproducible: I get this error when running livemedia-creator from a Fedora 33 VM run by qemu-kvm on a x86_64 host. This error is not systematic but SEEMS TO OCCUR ONLY WHEN RUNNING WITH ONE CPU (-smp 1), and no more when using more CPUs (testd with -smp 4) Steps to Reproduce: 1. Run a Fedora 33 VM from qemu-kvm using one CPU I use qemu-kvm -m 2048 -smp 1 -cpu host -hda hda.qcow2 -net nic -net user,hostfwd=tcp::5556-:22 -nographic # need console=ttyS0 for all messages, or at least ttyS0 login enabled 2. Generate a kickstart file from fedora-kickstarts I use Xfce spin kickstart 3. Run livemedia-creator I use : sudo livemedia-creator --ks trace_livecd.ks.tmp --no-virt --resultdir ${PWD}/iso --tmp=${CREATOR_TMP_DIR} --project "TRACe-Live-${FEDORA_RELEASEVER}" --releasever ${BSSID} --make-iso --volid TRACe-Live_FC${FEDORA_RELEASEVER}_ID${BSSID} --iso-only --iso-name TRACe-Live_FC${FEDORA_RELEASEVER}_ID${BSSID}.iso --macboot --extra-boot-args "rhgb $CMDLINE_EXTRAS" Actual results: The error message above Expected results: No such error and live image created Additional info: The log files produced by livemedia-creator when the error occurs : see anaconda_err.log.tgz