Bug 1911802
| Summary: | Leapp upgrades fails when /var is XFS with ftype=0 and rootfs is a different FS type | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Christophe Besson <cbesson> |
| Component: | leapp-repository | Assignee: | Petr Stodulka <pstodulk> |
| Status: | CLOSED ERRATA | QA Contact: | Martin KlusoĊ <mkluson> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.9 | CC: | fkrska, mkluson, podvody, pstodulk, saydas |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | leapp-repository-0.15.0-1.el7_9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-10 09:13:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1818077, 1818088 | ||
Hi Chris, thank you for the report. As a workaround, please suggest customer to copy the same file from /etc/pki/product-default/69.pem: # cp /etc/pki/product-default/69.pem /etc/pki/product/69.pem This should fix the problem for now. Just curious, the machine is clean RHEL 7 installation or it is upgraded from RHEL 6? Hi Petr, thanks for the feedback. From the anaconda logs, it was a RHEL 7.1 install. Hi guys, I looked at it and realize that I saw something completely different in my mind than you reported. My apology. Probably I mixed it with some reports when I read emails. You are right. I could imagine why this could happen, but only in older versions of leapp-repository (which is not case of this issue), which had some issues with workaround in case of XFS without ftype (ftype=0), which is present on this system, but only for /var and /opt, not for rootfs or /etc... I will take a better look in upcoming days. Thank you for your patiance. Just an update. I am able to reproduce the issue on my VM. To reproduce the issue: 1. install clean RHEL 7.1 with the same partitioning / - ext4 /var - xfs /boot - ext3 /opt - xfs (probaby the first two should be enough, I believe the problem is with XFS under overlayfs). 2. update to RHEL 7.9 3. install & run leapp (+ download leapp data) Hi, so we finished the inspection and prepared a fix [0] - that doesn't have to be the final one, but fixes this particular reported issue for the rootfs. In case customers are willing to test it, they can install the devel builds available on our public COPR repository [1], installing the "leapp-repository*.PR633.*" rpm, e.g.:
# yum install "leapp-repository*.PR633.*"
if the repository is enabled on the machine for testing (next to other usualy required repositories for the installation of leapp).
Note the build is temporary - iow, the build is going to be removed automatically after 1-3weeks. In such a case, we can create the build again if needed (if the PR is still opened)
Keep in mind, we do not know when the fix will be released (delivered) in the official builds right now.
[0] https://github.com/oamg/leapp-repository/pull/633
[1] https://copr.fedorainfracloud.org/coprs/g/oamg/leapp/repo/epel-7/group_oamg-leapp-epel-7.repo
Hi guys, we had quite rush times here and forgot about this bug completely. I removed the original git branch by mistake, so the PR solving the issue is here: https://github.com/oamg/leapp-repository/pull/674 I will raise the topic on our team mtgs. Regarding the number of comments, putting here the summary: - The minimal reproducer is here https://bugzilla.redhat.com/show_bug.cgi?id=1911802#c11 - tl;dr; problem happens when /var/lib/leapp is on a partition formated with XFS without ftype and rootfs has whatever different format that is not limited by the problem (e.g. XFS with ftype, EXT4, ...) - The fix has been verified by customer - The fix solves this particular BZ, but the problem (with different symptomps) could be possibly still hit on a different setups - the proper solution would require currently a lot of disk space until we start to manage the required disk space for the XFS ftype=0 workaround per partition (planned in future). Regarding the current reported BZs, I believe that other such problems would be super rare, so the current fix could be enough for now (even this case seems rare) The PR has been merged so the fix mentioned above will be part of the next release:
https://github.com/oamg/leapp-repository/pull/674
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (leapp, leapp-repository, and cockpit-leapp bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4603 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |
Description of problem: Customer is unable to upgrade to RHEL 8. While doing a `yum clean all`, we can see the error below: ~~~ Error loading certificate: [Errno 2] No such file or directory: '/etc/pki/product/69.pem' ~~~ The file is present on the host, the system is properly registered and Satellite repositories are supposedly correct. Version-Release number of selected component (if applicable): leapp-repository-0.12.0-2.el7_9.noarch How reproducible: 100% for the customer but unable to reproduce internally. Actual results: From the report: ~~~ 2020-12-24 10:37:32.537658 [ERROR] Actor: target_userspace_creator Message: There are no enabled target repositories for the upgrade process to proceed. ~~~ From the log: ~~~ 2020-12-24 10:37:29.221 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['systemd-nspawn', '--register=no', '--quiet', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--bind=/etc/hosts:/etc/hosts', '--setenv=LEAPP_NO_RHSM=0', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/el7toel8/files', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_EXECUTION_ID=db17bbc2-6e7d-43ef-854b-b2ba705fcd05', '--setenv=LEAPP_HOSTNAME=self072-chef.davis.sensus.lab', 'yum', 'clean', 'all']\n 2020-12-24 10:37:29.288 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Failed to create directory /var/lib/leapp/scratch/mounts/root_/system_overlay//sys/fs/selinux: Read-only file system 2020-12-24 10:37:29.312 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Failed to create directory /var/lib/leapp/scratch/mounts/root_/system_overlay//sys/fs/selinux: Read-only filesystem 2020-12-24 10:37:29.332 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Host and machine ids are equal (dcf7371fed3740ddbb84b8ed8867f2e4): refusing to link journals 2020-12-24 10:37:30.395 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id, 2020-12-24 10:37:30.411 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: : search-disabled-repos, subscription-manager 2020-12-24 10:37:30.423 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Error loading certificate: [Errno 2] No such file or directory: '/etc/pki/product/69.pem' 2020-12-24 10:37:30.449 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Cleaning repos: rhel-7-server-extras-rpms rhel-7-server-rpms 2020-12-24 10:37:30.481 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Uploading Enabled Repositories Report 2020-12-24 10:37:30.510 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Loaded plugins: langpacks, product-id, subscription-manager 2020-12-24 10:37:30.526 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Error loading certificate: [Errno 2] No such file or directory: '/etc/pki/product/69.pem' ~~~ Additional info: Strace analysis ~~~ 7808 10:37:29.201564 unlink("/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/product/69.pem") = 0 <0.008360> 7808 10:37:29.209999 unlink("/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/product-default/69.pem") = 0 <0.001437> : : 7875 10:37:29.251009 execve("/usr/bin/systemd-nspawn", ["systemd-nspawn", "--register=no", "--quiet", "-D", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "--bind=/etc/hosts:/etc/hosts", "--setenv=LEAPP_NO_RHSM=0", "--setenv=LEAPP_EXPERIMENTAL=0", "--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/tools", "--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/el7toel8/files", "--setenv=LEAPP_UNSUPPORTED=0", "--setenv=LEAPP_EXECUTION_ID=db17bbc2-6e7d-43ef-854b-b2ba705fcd05", "--setenv=LEAPP_HOSTNAME=***********", "yum", "clean", "all"], ["LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/tools", "LEAPP_CURRENT_ACTOR=target_userspace_creator", "LEAPP_UNSUPPORTED=0", "LESSOPEN=||/usr/bin/lesspipe.sh %s", "SSH_CLIENT=*********** 50530 22", "LOGNAME=root", "USER=root", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/etc/leapp/repos.d/system_upgrade/el7toel8/tools", "HOME=/root", "LANG=en_US.UTF-8", "TERM=xterm", "SHELL=/bin/bash", "LEAPP_EXPERIMENTAL=0", "SHLVL=1", "HISTSIZE=1000", "LEAPP_HOSTNAME=*********", "XDG_RUNTIME_DIR=/run/user/0", "LEAPP_VERBOSE=1", "LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/el7toel8/files", "LEAPP_EXECUTION_ID=db17bbc2-6e7d-43ef-854b-b2ba705fcd05", [...]) = 0 <0.000883> : 7875 10:37:29.264938 clone(child_stack=NULL, flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|SIGCHLD) = 7876 <0.000202> : : 7876 10:37:30.412275 stat("/etc/pki/product/69.pem", <unfinished ...> 7876 10:37:30.412323 <... stat resumed>0x7ffc6868f7c0) = -1 ENOENT (No such file or directory) <0.000038> : 7876 10:37:30.412499 open("/etc/pki/product/69.pem", O_RDONLY <unfinished ...> 7876 10:37:30.412559 <... open resumed>) = -1 ENOENT (No such file or directory) <0.000040> : 7876 10:37:30.414056 write(2<pipe:[311323]>, "Error loading certificate: [Errno 2] No such file or directory: '/etc/pki/product/69.pem'\n", 90 <unfinished ...> : 7808 10:37:30.423646 write(3</var/log/leapp/leapp-preupgrade.log>, "2020-12-24 10:37:30.423 DEBUG PID: 7808 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Error loading certificate: [Errno 2] No such file or directory: '/etc/pki/product/69.pem'\n", 207) = 207 <0.000015> ~~~ => PID 7808 is a leapp thread (apparently the target_userspace_creator actor). It removed the cert in question in the container spawned for the upgrade just before yum needs it. => PID 7875 being the "yum clean all" command running within the container. => The removal of 69.pem is expected during the process, but I'm not able to tell why the submgr yum's plugin still needs it at this stage. Reviewing a 2nd time that strace, I noticed another strange behaviour. The `yum clean all` launched by systemd-nspawn within the container still sees the cert at 10:37:30 (1s after its removal), as per the getdents() call, but the file is already gone less than 1 ms later as per the stat() call which got an ENOENT. ~~~ 7876 10:37:30.411951 getdents(4</etc/pki/product> <unfinished ...> 7876 10:37:30.412040 <... getdents resumed>, [{d_ino=671088772, d_off=1, d_reclen=24, d_name="..", d_type=DT_DIR}, {d_ino=268435837, d_off=2, d_reclen=32, d_name="69.pem", d_type=DT_UNKNOWN }, {d_ino=704643201, d_off=3, d_reclen=24, d_name=".", d_type=DT_DIR}, {d_ino=704643219, d_off=4, d_reclen=32, d_name="479.pem", d_type=DT_UNKNOWN}], 32768) = 112 <0.000069> 7876 10:37:30.412090 getdents(4</etc/pki/product>, [], 32768) = 0 <0.000019> 7876 10:37:30.412173 close(4</etc/pki/product> <unfinished ...> 7876 10:37:30.412221 <... close resumed>) = 0 <0.000031> 7808 10:37:30.412246 <... fstat resumed>{st_dev=makedev(253, 1), st_ino=1241514125, st_mode=S_IFREG|0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=29936, st_size=15325184, st_atime=1608824250 /* 2020-12-24T10:37:30.395858646-0500 */, st_atime_nsec=395858646, st_mtime=1608824250 /* 2020-12-24T10:37:30.406858599-0500 */, st_mtime_nsec=406858599, st_ctime=160882 4250 /* 2020-12-24T10:37:30.406858599-0500 */, st_ctime_nsec=406858599}) = 0 <0.000044> 7876 10:37:30.412275 stat("/etc/pki/product/69.pem", <unfinished ...> 7876 10:37:30.412323 <... stat resumed>0x7ffc6868f7c0) = -1 ENOENT (No such file or directory) <0.000038> ~~~