Created attachment 1840761 [details] A short script that is used to automate the build process Description of problem: Using Lorax to create an OSTree installer (Silverblue) .iso file based on a .yaml file works with no errors reported in the log, yet installing via the newly created .iso file does not work due to an unusual error (“The command ‘cp -r -r /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot’ exited with the code 1.”). This happens regardless of version, or Silverblue spin when using the official .yaml files hosted at https://pagure.io/workstation-ostree-config.git. Version-Release number of selected component (if applicable): Fedora Workstation 35 rpm-ostree-2021.11-3.fc35 lorax-35.7-2.fc35 How reproducible: Every single time, regardless of the .yaml file employed, Fedora version of the build system, whether the install is attempted in a hypervisor or on real hardware. Steps to Reproduce: I have attached a script I use to automate the described steps below. Steps to reproduce the behavior: 1. Install Lorax and RPM-OSTree via dnf or rpm on a fresh installation of Fedora 35 Workstation 2. Clone https://pagure.io/workstation-ostree-config.git and https://pagure.io/fedora-lorax-templates.git into a local folder 3. Initialize and compose a new repository based on one a .yaml files inside the workstation-ostree-config folder 4. Execute Lorax with ostree_install_repo and ostree_update_repo pointing to the created local repository 5. After Lorax has finished the build process, try installing the created .iso in a hypervisor or physical device Actual results: Encountering a fatal Anaconda error at the end of the install process (“The command ‘cp -r -r /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot’ exited with the code 1.”) Expected results: The .iso should install as expected when basing a new image on a repository generated via the official ostree-config files.
Created attachment 1840896 [details] Lorax build log when using the attached script
Created attachment 1840905 [details] Anaconda log with Error (at row 2965) I have added both the Lorax build log (which finishes without an error) and the Anaconda log showcasing the error. The error can be seen at row 2965 in "anaconda-tb-all.log".
From anaconda log: 14:53:38,538 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.modules.payloads.payload.rpm_ostree.installation:Copying bootloader data: loader 14:53:38,539 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... cp -r -p /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot 14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:cp: cannot overwrite non-directory '/mnt/sysimage/boot/loader' with directory '/mnt/sysroot/usr/lib/ostree-boot/loader' 14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:program:Return code: 1 14:53:38,550 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.threading:Thread Failed: AnaTaskThread-CopyBootloaderDataTask-1 (139742087718464) 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-CopyBootloaderDataTask-1 has failed: Traceback (most recent call last): 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: threading.Thread.run(self) 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/threading.py", line 946, in run 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: self._target(*self._args, **self._kwargs) 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 96, in _thread_run_callback 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: self._task_run_callback() 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 109, in _task_run_callback 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: self._set_result(self.run()) 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 280, in run 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: self._run() 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 314, in _run 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: safe_exec_with_redirect('cp', ['-r', '-p', srcpath, physboot]) 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 51, in safe_exec_with_redirect 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads: raise PayloadInstallationError( 14:53:38,558 WARNING org.fedoraproject.Anaconda.Modules.Payloads:pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'cp -r -p /mnt/sysroot/usr/lib/ostree-boot/loader /mnt/sysimage/boot' exited with the code 1. The important part is that we "cannot overwrite non-directory '/mnt/sysimage/boot/loader' with directory '/mnt/sysroot/usr/lib/ostree-boot/loader'". I think that /mnt/sysimage/boot/loader should be a directory. I am not sure why it isn't.
(In reply to Vendula Poncova from comment #3) >I think that /mnt/sysimage/boot/loader should be a directory. I am not sure why it isn't. Thank you for your help. Yes, that was my initial assumption as well. For testing, I have just executed an install with the official Silverblue 35 x86_64 release image in KVM and exported the Anaconda log from that successful install. I tried to find a reference to the directory that causes the installation failure (...ostree-boot/loader), yet there was none. (see newest attachment) With official builds, only the folder /mnt/sysroot/usr/lib/ostree-boot/grub2 is being copied to /mnt/sysimage/boot, not ...ostree-boot/loader. What is particularly weird is that, if you look at the first Anaconda log, Anaconda initially runs the same command, copying ...ostree-boot/grub2. This command returns code: 0 indicating success, yet after that Anaconda also tries to copy ...ostree-boot/loader which creates the error. Basically, for an unknown reason builds created with repos based on the files provided by https://pagure.io/workstation-ostree-config.git seem to lead to Anaconda to both copy the correct directory and also a second directory which does not exist, causing the error. In official builds Anaconda does not try to use this specific directory on top of the ...ostree-boot/grub2 directory. I do not know why, seeing as the official builds also rely on the data hosted at https://pagure.io/workstation-ostree-config.git I have attached a second Anaconda log showcasing the successful installation when using the official image. As stated, there is no reference to this directory (/mnt/sysroot/usr/lib/ostree-boot/loader) in that log, yet in the first log both /mnt/sysroot/usr/lib/ostree-boot/loader and /mnt/sysroot/usr/lib/ostree-boot/grub2 are being referenced, causing the error.
Created attachment 1841078 [details] Anaconda log when using official Silverblue image. Does not try to copy /mnt/sysroot/usr/lib/ostree-boot/loader
Sorry to bump this, but is there any information that may be helpful to solving this issue? Perhaps there is something that I haven't yet provided. Additionally, I made a thread at Fedora Discussion where one other person reported experiencing the same issue so this issue appears to not be isolated to me: https://discussion.fedoraproject.org/t/silverblue-iso-made-via-lorax-fails-installation/34242
Colin, what do you think about comment 4 ? It sounds like there is some secret sauce to this. Alternatively, where should Chris turn for advice? Should that be on https://github.com/coreos/rpm-ostree ?
I am happy to report that the cause for this issue has been found. Using the --unified-core argument while composing causes the additional folder to be included: https://github.com/coreos/rpm-ostree/issues/3270#issuecomment-995160984 I still do not know why that would be the case, though can now say with certainty that this is not an issue connected to Anaconda.