2063910 – LEAPP upgrade fails with [Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2063910 - LEAPP upgrade fails with [Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'

Summary: LEAPP upgrade fails with [Errno 2] No such file or directory: '/var/lib/leapp...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Satellite Maintain
Sub Component:
Version:	6.11.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	6.11.0
Assignee:	satellite6-bugs
QA Contact:	Lukas Pramuk
Docs Contact:
URL:
Whiteboard:
Depends On:	2054182
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-14 15:50 UTC by Lukas Pramuk
Modified:	2022-07-19 11:02 UTC (History)
CC List:	6 users (show)
Fixed In Version:	rubygem-foreman_maintain-1.0.5
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-07-05 14:34:27 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Foreman Issue Tracker	34618	0	Normal	Closed	pulp2 removal doesn't clean up /etc/pki/pulp/content, leading to errors during upgrades	2022-03-31 14:55:58 UTC
Red Hat Product Errata	RHSA-2022:5498	0	None	None	None	2022-07-05 14:34:38 UTC

Description Lukas Pramuk 2022-03-14 15:50:00 UTC

Description of problem:
Capsule LEAPP upgrade fails during actor target_userspace_creator with:
[Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'

Version-Release number of selected component (if applicable):
7.0.0 Snap13
leapp-0.13.0-1.el7_9.noarch
leapp-upgrade-el7toel8-0.15.0-6.el7_9.noarch


How reproducible:
deterministic

Steps to Reproduce:
1. Have 7.0 Capsule on RHEL7 (yet enable Capsule 7.0 el8 compose repo)
2. Install LEAPP and resolve upgrade inhibitors (unload old kernel modules, sshd options)
3. Run LEAPP preupgrade

# LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE="8.5" leapp preupgrade --enablerepo capsule-el8
...

Process Process-417:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 335, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 52, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 681, in perform
    _create_target_userspace(context, indata.packages, indata.files, target_repoids)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 655, in _create_target_userspace
    _prep_repository_access(context, target_path)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 238, in _prep_repository_access
    context.copytree_from('/etc/pki', os.path.join(target_etc, 'pki'))
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 221, in copytree_from
    shutil.copytree(self.full_path(src), dst)
  File "/usr/lib64/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca', '/var/lib/leapp/el8userspace/etc/pki/pulp/content/pulp-global-repo.ca', "[Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'")]


=========================================================================================================
Actor target_userspace_creator unexpectedly terminated with exit code: 1 - Please check the above details
=========================================================================================================

Actual results:
preupgrade failed


Expected results:
preupgrade/upgrade is successful

Additional info:

Comment 1 Lukas Pramuk 2022-03-14 15:57:03 UTC

>>> source tree
# ll /var/lib/leapp/scratch/
ls: cannot access /var/lib/leapp/scratch/: No such file or directory


>>> target tree
# ll /var/lib/leapp/el8userspace/etc/pki/pulp
total 28
drwxr-xr-x. 2 root root   34 Oct  1 11:09 content
drwxr-x---. 2 root root   38 Nov  3  2019 qpid
-rw-r--r--. 1 root root 5632 Dec  1  2016 server_ca.crt
-rw-r--r--. 1 root root 5775 Dec  1  2016 ssl_apache.crt
-r--------. 1 root root 1679 Dec  1  2016 ssl_apache.key
-rw-r--r--. 1 root root 5632 Dec  1  2016 ssl_ca.crt

# ll /var/lib/leapp/el8userspace/etc/pki/pulp/content/
total 0
-rw-r--r--. 1 root root 0 Dec 12 22:00 pulp-protected-repos

Comment 2 Evgeni Golov 2022-03-14 17:53:09 UTC

While this is certainly the wrong leapp version (it's not the one containing the Satellite patches),

can you please show the output of `ls -lahR /etc/pki/pulp/` and/or give me SSH access to the machine?

Comment 3 Evgeni Golov 2022-03-15 07:35:01 UTC

Okay, so the problem is that the global repo ca file is a dangling symlink:

[root@dhcp-2-184 ~]# ls -alh /etc/pki/pulp/content/pulp-global-repo.ca 
lrwxrwxrwx. 1 root root 20 Oct 29  2020 /etc/pki/pulp/content/pulp-global-repo.ca -> /etc/pki/pulp/ca.crt
[root@dhcp-2-184 ~]# ls -alh /etc/pki/pulp/ca.crt
ls: cannot access /etc/pki/pulp/ca.crt: No such file or directory

And LEAPP tries to copy all of /etc/pki (and /etc/rhsm) from the EL7 system to the EL8 target:
https://github.com/oamg/leapp-repository/blob/a098738a2d248d78a3c07b6cdf5317a4d1ffbb54/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py#L231-L232

It does so by using shutil.copytree:
https://github.com/oamg/leapp-repository/blob/a098738a2d248d78a3c07b6cdf5317a4d1ffbb54/repos/system_upgrade/common/libraries/mounting.py#L213-L221

And shutil.copytree doesn't like to copy dangling symlinks (TIL!) by default:
 shutil.copytree(src, dst, symlinks=False, ignore=None)
    If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree, but the metadata of the original links is NOT copied; if false or omitted, the contents and metadata of the linked files are copied to the new tree.

https://docs.python.org/2.7/library/shutil.html#shutil.copytree (same-ish also in Python 3: https://docs.python.org/3/library/shutil.html#shutil.copytree).

We can probably add a cleanup step to the installer/maintain before the upgrade, but I'd also argue LEAPP should catch this error better (and will talk to the LEAPP team).

Comment 4 Evgeni Golov 2022-03-15 07:50:14 UTC

Ad cleanup, we can probably add /etc/pki/pulp to the pulp_data_dirs of the "foreman-maintain content remove-pulp2" command:
https://github.com/theforeman/foreman_maintain/blob/dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.rb#L18-L27

Comment 5 Lukas Pramuk 2022-03-15 13:38:00 UTC

This LEAPP failure happens only on setups upgraded since 6.9 (and older) for both Satellite and Capsule

Comment 6 Evgeni Golov 2022-03-15 13:59:39 UTC

Moved the leapp part of this to https://github.com/oamg/leapp-repository/issues/859

Making this now a FM BZ to clean up that path during the "remove pulp2" procedure

Comment 7 Evgeni Golov 2022-03-15 14:07:33 UTC

Created redmine issue https://projects.theforeman.org/issues/34618 from this bug

Comment 8 Evgeni Golov 2022-03-16 08:34:18 UTC

Adding a "blocked by" on BZ#2054182 as without that the cleanup won't happen during an upgrade of a capsule.

Comment 11 Lukas Pramuk 2022-04-20 10:04:29 UTC

VERIFIED.

@Satellite 6.11.0 Snap16
rubygem-foreman_maintain-1.0.7-1.el7sat.noarch

by the following manual reproducer:

1) Have a Sat 6.9.9 and Capsule 6.9.9 associated

2) Upgrade both Satellite and Capsule to 6.10.4 (symlink /etc/pki/pulp/content/pulp-global-repo.ca -> /etc/pki/pulp/ca.crt is present)

3) Upgrade both Satellite and Capsule to 6.11.0 and check for broken symlink

# ls -l /etc/pki/pulp/content/pulp-global-repo.ca 
ls: cannot access /etc/pki/pulp/content/pulp-global-repo.ca: No such file or directory

>>> automatic pulp2 removal removes also the broken symlink

>>> LEAPP upgrade is going to be successful as there is no pulp2 broken symlink

Comment 14 errata-xmlrpc 2022-07-05 14:34:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498

Note You need to log in before you can comment on or make changes to this bug.