Bug 2007045

Summary: Can not build live media with livecd-creator because of "fsck after resize" error
Product: [Fedora] Fedora Reporter: Scott Dowdle <dowdle>
Component: livecd-toolsAssignee: Neal Gompa <ngompa13>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 36CC: adam.stokes, awilliam, bcl, bruno, dhuff, katzj, lersek, metherid, ngompa13, sergey, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedFreezeException
Fixed In Version: livecd-tools-30.0-1.fc36 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-05 18:37:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1953786    

Description Scott Dowdle 2021-09-22 23:53:16 UTC
Trying to get up-to-speed with building remixes of F35 as it isn't too far away from release.

I am getting these errors near the end of the build process:

- - - - -
Unmounting directory /var/tmp/imgcreate-{random}/install_root failed, using lazy umount
Unmounting directory /var/tmp/imgcreate-{random}/install_root failed, using lazy umount
lazy umount succeeded on /var/tmp/imgcreate-{random}/install_root
Error creating Live CD : fsck after resize returned an error (1)!
- - - - -

I build media for 7 different spins and they are yielding the same thing... but once I've been able to it to complete and actually output an .iso which I used to do a fresh install with (in a KVM VM) and use as my build host going forward.  Still getting that error.

I doubt it is actually livecd-creator... but most likely some underlying tool.  We have run into this issue in the past (I think the last time I remember was 2012?) and perhaps it is a timing issue.

Any idea what might be going on?

I'm pretty sure it isn't my kickstarts because I haven't really modified the stock Fedora ones other than to change the repos (local mirror) and package list.  All of the %post stuff is stock.

Comment 1 Sergey 2022-02-13 12:03:54 UTC
I am getting the same error on up-to-date F35.

Actually the bug is very old (see https://bugzilla.redhat.com/show_bug.cgi?id=1602046). 

My system is equipped with two drives, one HDD and one SSD. The whole filesystem but /home resides on SSD. I noted that when I invoke livecd-creator with `--tmpdir=/home/livecdtmp` option then the fsck error happens. When I omit the `--tmpdir` option, i.e. livecd-creator uses the default `/var/tmp`, then fsck error does not happen. I used `--cache=/home/livecdcache` in both cases.

So the bug is timing-related: the kernel does not catch to flush the caches (?) before the fsck invocation in case of slowest drive.

Neal Gompa wrote in https://bugzilla.redhat.com/show_bug.cgi?id=1602046#c8:
> (In reply to Scott Dowdle from comment 7)
> > I'm pretty sure prior to the update it was giving the same lazy umount
> > message.  I have no idea what is causing that but I don't think that is a
> > new issue.
> >
> 
> Yes, we always had those messages. We just ignore them.

I suspect that clean unmounting fails due to the presence of virtual file systems mounts (/sys, /proc, e.t.c.) inside build root.

Though, the cd image produced by successful livecd-creator invocation using fedora-live-xfce.ks does not boot into X interface, but this is another problem which I will dig into after I (or someone else) defeat `lazy unmount` issue.

Comment 2 Sergey 2022-02-13 15:33:56 UTC
(In reply to Sergey from comment #1)

> I suspect that clean unmounting fails due to the presence of virtual file
> systems mounts (/sys, /proc, e.t.c.) inside build root.

I was not correct:

<--cut-->
121 man subdirectories contained newer manual pages.
7344 manual pages were added.
0 stray cats were added.
0 old database entries were purged.
error reading information on service network: No such file or directory
/ 100.0%

Unmounting directory /var/tmp/imgcreate-fct3glbx/install_root
bash-5.1# lsof /dev/loop0
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
COMMAND      PID USER  FD   TYPE DEVICE SIZE/OFF  NODE NAME
livecd-cr 333994 root mem    REG    7,0   357880 21299 /var/tmp/imgcreate-fct3glbx/install_root/usr/lib64/libnss_systemd.so.2
bash-5.1#
<--cut-->

Neal, do you have an idea what may cause livecd-creator to load that library from inside install_root?

Comment 3 Sergey 2022-02-15 04:00:04 UTC
Neal, I found that the similar problem has been already solved for libnss_sss (see #1127103)
PR submitted.

Comment 4 Laszlo Ersek 2022-04-24 13:46:30 UTC
https://github.com/livecd-tools/livecd-tools/pull/223

upstream commit 4afde9352e82

Neal, can you please build new RPMs? Thanks!

Comment 5 Fedora Update System 2022-04-24 15:18:03 UTC
FEDORA-2022-794d75ddf5 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-794d75ddf5

Comment 6 Fedora Blocker Bugs Application 2022-04-24 15:19:43 UTC
Proposed as a Freeze Exception for 36-final by Fedora user ngompa using the blocker tracking app because:

 Fixes the ability to produce working live ISOs for F35+ (including F36).

Comment 7 Fedora Update System 2022-04-24 15:20:19 UTC
FEDORA-2022-193031a230 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-193031a230

Comment 8 Fedora Update System 2022-04-24 15:20:35 UTC
FEDORA-2022-d515f28d09 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-d515f28d09

Comment 9 Fedora Update System 2022-04-24 15:20:52 UTC
FEDORA-EPEL-2022-dabf0b8100 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-dabf0b8100

Comment 10 Fedora Update System 2022-04-24 15:21:05 UTC
FEDORA-EPEL-2022-20b2f34120 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-20b2f34120

Comment 11 Fedora Update System 2022-04-24 18:44:17 UTC
FEDORA-2022-794d75ddf5 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-794d75ddf5`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-794d75ddf5

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2022-04-24 19:48:47 UTC
FEDORA-2022-d515f28d09 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-d515f28d09`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-d515f28d09

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2022-04-24 20:09:20 UTC
FEDORA-EPEL-2022-20b2f34120 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2022-04-24 20:36:25 UTC
FEDORA-EPEL-2022-dabf0b8100 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Fedora Update System 2022-04-24 20:43:28 UTC
FEDORA-2022-193031a230 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-193031a230`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-193031a230

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2022-04-26 22:01:51 UTC
FEDORA-2022-30c0b0af10 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-30c0b0af10`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-30c0b0af10

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2022-04-27 09:14:35 UTC
FEDORA-2022-095ac0abfb has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-095ac0abfb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-095ac0abfb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2022-04-27 09:31:47 UTC
FEDORA-2022-e4a46d0bd0 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-e4a46d0bd0`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-e4a46d0bd0

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Adam Williamson 2022-04-28 21:33:57 UTC
+6 in https://pagure.io/fedora-qa/blocker-review/issue/774 , marking accepted.

Comment 20 Fedora Update System 2022-05-03 14:54:56 UTC
FEDORA-2022-e4a46d0bd0 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 21 Fedora Update System 2022-05-05 18:37:11 UTC
FEDORA-2022-30c0b0af10 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Fedora Update System 2022-05-12 01:46:47 UTC
FEDORA-2022-095ac0abfb has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.