Description of problem: Capsule LEAPP upgrade fails during actor target_userspace_creator with: [Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca' Version-Release number of selected component (if applicable): 7.0.0 Snap13 leapp-0.13.0-1.el7_9.noarch leapp-upgrade-el7toel8-0.15.0-6.el7_9.noarch How reproducible: deterministic Steps to Reproduce: 1. Have 7.0 Capsule on RHEL7 (yet enable Capsule 7.0 el8 compose repo) 2. Install LEAPP and resolve upgrade inhibitors (unload old kernel modules, sshd options) 3. Run LEAPP preupgrade # LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE="8.5" leapp preupgrade --enablerepo capsule-el8 ... Process Process-417: Traceback (most recent call last): File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run actor_instance.run(*args, **kwargs) File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 335, in run self.process(*args) File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 52, in process userspacegen.perform() File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper return target_item(*args, **kwargs) File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 681, in perform _create_target_userspace(context, indata.packages, indata.files, target_repoids) File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 655, in _create_target_userspace _prep_repository_access(context, target_path) File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 238, in _prep_repository_access context.copytree_from('/etc/pki', os.path.join(target_etc, 'pki')) File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 221, in copytree_from shutil.copytree(self.full_path(src), dst) File "/usr/lib64/python2.7/shutil.py", line 208, in copytree raise Error, errors Error: [('/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca', '/var/lib/leapp/el8userspace/etc/pki/pulp/content/pulp-global-repo.ca', "[Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'")] ========================================================================================================= Actor target_userspace_creator unexpectedly terminated with exit code: 1 - Please check the above details ========================================================================================================= Actual results: preupgrade failed Expected results: preupgrade/upgrade is successful Additional info:
>>> source tree # ll /var/lib/leapp/scratch/ ls: cannot access /var/lib/leapp/scratch/: No such file or directory >>> target tree # ll /var/lib/leapp/el8userspace/etc/pki/pulp total 28 drwxr-xr-x. 2 root root 34 Oct 1 11:09 content drwxr-x---. 2 root root 38 Nov 3 2019 qpid -rw-r--r--. 1 root root 5632 Dec 1 2016 server_ca.crt -rw-r--r--. 1 root root 5775 Dec 1 2016 ssl_apache.crt -r--------. 1 root root 1679 Dec 1 2016 ssl_apache.key -rw-r--r--. 1 root root 5632 Dec 1 2016 ssl_ca.crt # ll /var/lib/leapp/el8userspace/etc/pki/pulp/content/ total 0 -rw-r--r--. 1 root root 0 Dec 12 22:00 pulp-protected-repos
While this is certainly the wrong leapp version (it's not the one containing the Satellite patches), can you please show the output of `ls -lahR /etc/pki/pulp/` and/or give me SSH access to the machine?
Okay, so the problem is that the global repo ca file is a dangling symlink: [root@dhcp-2-184 ~]# ls -alh /etc/pki/pulp/content/pulp-global-repo.ca lrwxrwxrwx. 1 root root 20 Oct 29 2020 /etc/pki/pulp/content/pulp-global-repo.ca -> /etc/pki/pulp/ca.crt [root@dhcp-2-184 ~]# ls -alh /etc/pki/pulp/ca.crt ls: cannot access /etc/pki/pulp/ca.crt: No such file or directory And LEAPP tries to copy all of /etc/pki (and /etc/rhsm) from the EL7 system to the EL8 target: https://github.com/oamg/leapp-repository/blob/a098738a2d248d78a3c07b6cdf5317a4d1ffbb54/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py#L231-L232 It does so by using shutil.copytree: https://github.com/oamg/leapp-repository/blob/a098738a2d248d78a3c07b6cdf5317a4d1ffbb54/repos/system_upgrade/common/libraries/mounting.py#L213-L221 And shutil.copytree doesn't like to copy dangling symlinks (TIL!) by default: shutil.copytree(src, dst, symlinks=False, ignore=None) If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree, but the metadata of the original links is NOT copied; if false or omitted, the contents and metadata of the linked files are copied to the new tree. https://docs.python.org/2.7/library/shutil.html#shutil.copytree (same-ish also in Python 3: https://docs.python.org/3/library/shutil.html#shutil.copytree). We can probably add a cleanup step to the installer/maintain before the upgrade, but I'd also argue LEAPP should catch this error better (and will talk to the LEAPP team).
Ad cleanup, we can probably add /etc/pki/pulp to the pulp_data_dirs of the "foreman-maintain content remove-pulp2" command: https://github.com/theforeman/foreman_maintain/blob/dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.rb#L18-L27
This LEAPP failure happens only on setups upgraded since 6.9 (and older) for both Satellite and Capsule
Moved the leapp part of this to https://github.com/oamg/leapp-repository/issues/859 Making this now a FM BZ to clean up that path during the "remove pulp2" procedure
Created redmine issue https://projects.theforeman.org/issues/34618 from this bug
Adding a "blocked by" on BZ#2054182 as without that the cleanup won't happen during an upgrade of a capsule.
VERIFIED. @Satellite 6.11.0 Snap16 rubygem-foreman_maintain-1.0.7-1.el7sat.noarch by the following manual reproducer: 1) Have a Sat 6.9.9 and Capsule 6.9.9 associated 2) Upgrade both Satellite and Capsule to 6.10.4 (symlink /etc/pki/pulp/content/pulp-global-repo.ca -> /etc/pki/pulp/ca.crt is present) 3) Upgrade both Satellite and Capsule to 6.11.0 and check for broken symlink # ls -l /etc/pki/pulp/content/pulp-global-repo.ca ls: cannot access /etc/pki/pulp/content/pulp-global-repo.ca: No such file or directory >>> automatic pulp2 removal removes also the broken symlink >>> LEAPP upgrade is going to be successful as there is no pulp2 broken symlink
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498