Bug 2021340

Summary: OSTree Repo based builds fail installation
Product: [Fedora] Fedora Reporter: Chris <christoph>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 35CC: anaconda-maint-list, bcl, jonathan, kellin, vanmeeuwen+fedora, vponcova, vslavik, walters, w
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-15 19:56:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
A short script that is used to automate the build process
none
Lorax build log when using the attached script
none
Anaconda log with Error (at row 2965)
none
Anaconda log when using official Silverblue image. Does not try to copy /mnt/sysroot/usr/lib/ostree-boot/loader none

Description Chris 2021-11-08 22:18:13 UTC 
Created attachment 1840761 [details]
A short script that is used to automate the build process

Description of problem:
Using Lorax to create an OSTree installer (Silverblue) .iso file based on a .yaml file works with no errors reported in the log, yet installing via the newly created .iso file does not work due to an unusual error (“The command ‘cp -r -r /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot’ exited with the code 1.”). This happens regardless of version, or Silverblue spin when using the official .yaml files hosted at https://pagure.io/workstation-ostree-config.git.

Version-Release number of selected component (if applicable):
Fedora Workstation 35
rpm-ostree-2021.11-3.fc35
lorax-35.7-2.fc35

How reproducible:
Every single time, regardless of the .yaml file employed, Fedora version of the build system, whether the install is attempted in a hypervisor or on real hardware.

Steps to Reproduce:
I have attached a script I use to automate the described steps below.
Steps to reproduce the behavior:
1. Install Lorax and RPM-OSTree via dnf or rpm on a fresh installation of Fedora 35 Workstation
2. Clone https://pagure.io/workstation-ostree-config.git and https://pagure.io/fedora-lorax-templates.git into a local folder
3. Initialize and compose a new repository based on one a .yaml files inside the workstation-ostree-config folder
4. Execute Lorax with ostree_install_repo and ostree_update_repo pointing to the created local repository
5. After Lorax has finished the build process, try installing the created .iso in a hypervisor or physical device

Actual results:
Encountering a fatal Anaconda error at the end of the install process (“The command ‘cp -r -r /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot’ exited with the code 1.”)

Expected results:
The .iso should install as expected when basing a new image on a repository generated via the official ostree-config files.
Comment 1 Chris 2021-11-09 14:48:55 UTC 
Created attachment 1840896 [details]
Lorax build log when using the attached script
Comment 2 Chris 2021-11-09 15:23:24 UTC 
Created attachment 1840905 [details]
Anaconda log with Error (at row 2965)

I have added both the Lorax build log (which finishes without an error) and the Anaconda log showcasing the error. The error can be seen at row 2965 in "anaconda-tb-all.log".
Comment 3 Vendula Poncova 2021-11-10 12:32:20 UTC 
From anaconda log:

14:53:38,538 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.modules.payloads.payload.rpm_ostree.installation:Copying bootloader data: loader
14:53:38,539 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... cp -r -p /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot
14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:cp: cannot overwrite non-directory '/mnt/sysimage/boot/loader' with directory '/mnt/sysroot/usr/lib/ostree-boot/loader'
14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:program:Return code: 1
14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.threading:Thread Failed: AnaTaskThread-CopyBootloaderDataTask-1 (139742087718464)
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-CopyBootloaderDataTask-1 has failed: Traceback (most recent call last):
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    threading.Thread.run(self)
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/threading.py", line 946, in run
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    self._target(*self._args, **self._kwargs)
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 96, in _thread_run_callback
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    self._task_run_callback()
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 109, in _task_run_callback
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    self._set_result(self.run())
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 280, in run
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    self._run()
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 314, in _run
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    safe_exec_with_redirect('cp', ['-r', '-p', srcpath, physboot])
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 51, in safe_exec_with_redirect
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:    raise PayloadInstallationError(
14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'cp -r -p /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot' exited with the code 1.

The important part is that we "cannot overwrite non-directory '/mnt/sysimage/boot/loader' with directory '/mnt/sysroot/usr/lib/ostree-boot/loader'". I think that /mnt/sysimage/boot/loader should be a directory. I am not sure why it isn't.
Comment 4 Chris 2021-11-10 13:59:29 UTC 
(In reply to Vendula Poncova from comment #3)
>I think that /mnt/sysimage/boot/loader should be a directory. I am not sure why it isn't.

Thank you for your help. Yes, that was my initial assumption as well. For testing, I have just executed an install with the official Silverblue 35 x86_64 release image in KVM and exported the Anaconda log from that successful install. I tried to find a reference to the directory that causes the installation failure (...ostree-boot/loader), yet there was none. (see newest attachment)

With official builds, only the folder /mnt/sysroot/usr/lib/ostree-boot/grub2 is being copied to /mnt/sysimage/boot, not ...ostree-boot/loader.

What is particularly weird is that, if you look at the first Anaconda log, Anaconda initially runs the same command, copying ...ostree-boot/grub2. This command returns code: 0 indicating success, yet after that Anaconda also tries to copy ...ostree-boot/loader which creates the error.

Basically, for an unknown reason builds created with repos based on the files provided by https://pagure.io/workstation-ostree-config.git seem to lead to Anaconda to both copy the correct directory and also a second directory which does not exist, causing the error.

In official builds Anaconda does not try to use this specific directory on top of the ...ostree-boot/grub2 directory. I do not know why, seeing as the official builds also rely on the data hosted at https://pagure.io/workstation-ostree-config.git

I have attached a second Anaconda log showcasing the successful installation when using the official image. As stated, there is no reference to this directory (/mnt/sysroot/usr/lib/ostree-boot/loader) in that log, yet in the first log both /mnt/sysroot/usr/lib/ostree-boot/loader and /mnt/sysroot/usr/lib/ostree-boot/grub2 are being referenced, causing the error.
Comment 5 Chris 2021-11-10 14:00:27 UTC 
Created attachment 1841078 [details]
Anaconda log when using official Silverblue image. Does not try to copy /mnt/sysroot/usr/lib/ostree-boot/loader
Comment 6 Chris 2021-12-01 13:24:28 UTC 
Sorry to bump this, but is there any information that may be helpful to solving this issue? Perhaps there is something that I haven't yet provided.

Additionally, I made a thread at Fedora Discussion where one other person reported experiencing the same issue so this issue appears to not be isolated to me: https://discussion.fedoraproject.org/t/silverblue-iso-made-via-lorax-fails-installation/34242
Comment 7 Vladimír Slávik 2021-12-01 14:15:14 UTC 
Colin, what do you think about comment 4 ? It sounds like there is some secret sauce to this.

Alternatively, where should Chris turn for advice? Should that be on https://github.com/coreos/rpm-ostree ?
Comment 8 Chris 2021-12-15 19:56:24 UTC 
I am happy to report that the cause for this issue has been found. Using the --unified-core argument while composing causes the additional folder to be included: https://github.com/coreos/rpm-ostree/issues/3270#issuecomment-995160984

I still do not know why that would be the case, though can now say with certainty that this is not an issue connected to Anaconda.