Bug 2161969
| Summary: | [leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Christophe Besson <cbesson> |
| Component: | leapp-repository | Assignee: | Petr Stodulka <pstodulk> |
| Status: | CLOSED MIGRATED | QA Contact: | upgrades-and-conversions |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 7.9 | CC: | amepatil, pstodulk, rmetrich |
| Target Milestone: | rc | Keywords: | MigratedToJIRA, WorkAround |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-12 12:39:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I'm not sure we'll be able to reproduce such an issue. As we rarely observed this issue, in my opinion one sufficient remediation could be to have the ability to use the same workaround as for XFS without ftype. Something like LEAPP_ENABLE_EXT4_UPPER=<part1,part2,part3> Thank you for the report and initial investigation. Regarding additional problems we have reported related to use of overlayfs, it seems that we will anyway start to create virtual fss/images for all partitions always, which would resolve this problem also. Just we need to figure out which technology to use to handle it right, without consumption of too much space (we have our horses in game already). We will update the ticket when we have additional info. Adding a new case in this BZ, the root cause being very likely *similar*.
No EINVAL errno this time, but an invisible /usr in the installroot leading to the below error:
STDERR:
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm.
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm.
Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm.
Could not open: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm
The system is very standard, upgraded, without noticeable 3rd party known to cause an issue.
The rootfs is presented as an overlayfs in /var/lib/leapp/scratch/mounts/root_/system_overlay (lower layer is a ro / and upper layer is rw from /var).
It is then bind-mounted into /var/lib/leapp/el8userspace/installroot:
2023-01-24 15:32:32.516 DEBUG PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_/system_overlay', u'/var/lib/leapp/el8userspace/installroot']
And then, any subsequent mount into /var/lib/leapp/scratch/mounts/root_/system_overlay _should_ be visible into /var/lib/leapp/el8userspace/installroot as it is a bind-mount.
2023-01-24 15:32:33.190 DEBUG PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr']
:
2023-01-24 15:32:33.205 DEBUG PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['rm', '-rf', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/usr']
:
2023-01-24 15:32:33.224 DEBUG PID: 37375 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/usr']
From the strace of another execution we have the below relevant calls:
63742 16:29:41.450384 mkdir("/var/lib/leapp/scratch/mounts", 0777) = -1 EEXIST (File exists) <0.000011>
63742 16:29:41.604071 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0 <0.000064>
63742 16:29:41.604372 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0 <0.000045>
63742 16:29:41.604662 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0 <0.000045>
63754 16:29:41.640372 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000010>
63754 16:29:41.641234 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/v
ar/lib/leapp/scratch/mounts/root_/work") = 0 <0.000191>
63742 16:29:41.713580 mkdir("/var/lib/leapp/el8userspace/installroot", 0777) = 0 <0.000074>
63756 16:29:41.741549 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000010>
63756 16:29:41.742459 mount("/var/lib/leapp/scratch/mounts/root_/system_overlay", "/var/lib/leapp/el8userspace/installroot", 0x55e64fdfd240, MS_MGC_VAL|MS_BIND, NULL) = 0 <0.000027>
:
63742 16:29:43.659714 mkdir("/var/lib/leapp/scratch/mounts/root_usr/upper", 0777) = 0 <0.000096>
63742 16:29:43.660080 mkdir("/var/lib/leapp/scratch/mounts/root_usr/work", 0777) = 0 <0.000042>
63742 16:29:43.660339 mkdir("/var/lib/leapp/scratch/mounts/root_usr/root_usr", 0777) = 0 <0.000080>
63795 16:29:43.696451 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000009>
63795 16:29:43.697315 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_usr/root_usr", "overlay", MS_MGC_VAL, "lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work") = 0 <0.000191>
63742 16:29:43.770861 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", 0777) = 0 <0.000105>
63797 16:29:43.802181 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) <0.000009>
63797 16:29:43.803206 mount("/var/lib/leapp/scratch/mounts/root_usr/root_usr", "/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", 0x564737735240, MS_MGC_VAL|MS_BIND, NULL) = 0 <0.000024>
63797 16:29:43.803267 access("/var/lib/leapp/scratch/mounts/root_/system_overlay/usr", W_OK) = 0 <0.000011>
:
63742 16:29:43.812295 write(4</var/log/leapp/leapp-preupgrade.log>, "2023-01-24 16:29:43.812 DEBUG PID: 63742 leapp.workflow.TargetTransactionCheck.dnf_transaction_check: External command has finished: ['mount', '-o', 'bind', u'/var/lib/leapp/scratch/mounts/root_usr/root_usr', u'/var/lib/leapp/scratch/mounts/root_/system"..., 271) = 271 <0.000018>
:
:
63896 16:30:02.306331 write(3</var/log/dnf.log>, "2023-01-24T16:30:02+0100 DEBUG Excludes in dnf.conf: leapp, leapp-upgrade-el7toel8, python2-leapp, snactor\n", 107) = 107 <0.000019>
63896 16:30:02.306946 stat("/installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.
rpm", 0x7ffdacbfbc10) = -1 ENOENT (No such file or directory) <0.000023>
63896 16:30:02.307034 access("/installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarc
h.rpm", R_OK) = -1 ENOENT (No such file or directory) <0.000013>
63896 16:30:02.307102 write(7</var/log/hawkey.log>, "2023-01-24T16:30:02+0100 WARN not a readable RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files
/bundled-rpms/leapp-deps-el8-5.0.8-100.202208231727Z.29a77886.master.el8.noarch.rpm, skipping\n", 231) = 231 <0.000016>
The customer was not afraid to think outside the box and used a workaround a little bit rough, but claiming it worked for them:
mount -o bind,ro /var/lib/leapp/scratch/mounts/root_/system_overlay/usr /var/lib/leapp/el8userspace/installroot/usr
at the right instant.
I guess the workaround suggested in the description of this BZ is more difficult to implement but it could work and would be safer.
Another EINVAL on mkdir()
9465 13:11:09.956191 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work") = 0 <0.000281>
9453 13:11:10.199485 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/var/cache/dnf", 0777) = -1 EINVAL (Invalid argument) <0.000096>
Not supported as it requires to patch the code, but it's the easier workaround: # sed -i 's/def is_xfs_without_ftype(mp):/def is_xfs_without_ftype(mp):\n return True/' /usr/share/leapp-repository/repositories/system_upgrade/common/actors/xfsinfoscanner/libraries/xfsinfoscanner.py It makes leapp believes it has to deal with XFS ftype=0 cases Possibly adjust $LEAPP_OVL_SIZE as explained in https://access.redhat.com/solutions/5057391 (worked for the last case) Thank you Chris for the investigation! The suggested workaround supports our expectation the issue will be fixed by the redesigned solution we are working on - as we are going to create a diskimage for each partition listed in FSTAB for temporary overlay filesystems we create during the upgrade process. Should be fixed by upstream PR: * https://github.com/oamg/leapp-repository/pull/1097 If it is ever discovered again on any machine, we need to test it with the particular fix. I believe the current fix should work right also with XFS hopefully. If not, it will be possible to switch to the ext4 FS for upper directories using: LEAPP_OVL_IMG_FS_EXT4=1 leapp ... It's possible the issue is caused also by an older XFS format or some exotic XFS params. Currently used disk images (for the upgrade purposes) will be formatted by XFS with up-to-date (and same) params. The possibility to switch between XFS and Ext4 will tell us whether there is hidden more deep issue between XFS and overlayfs. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: Leapp failed at the preupgrade step due to a directory that cannot be created on top of <overlayfs>/tmp. The issue was not with this specific partition, if this one is not used it fails on the next one. * They use the latest el7 kernel and the latest leapp packages * They have no 3rd party tainting the kernel. * All their XFS have an ftype (hence it supposedly supports overlays). * The strace shows the first mkdir() on the mounted overlay fails with an EINVAL errno (excerpt of relevant calls): 24271 11:46:12.656828 mkdir("/var/lib/leapp/scratch", 0777) = 0 24271 11:46:12.656885 mkdir("/var/lib/leapp/scratch/mounts", 0777) = 0 24271 11:46:12.776710 mkdir("/var/lib/leapp/scratch/mounts/root_", 0777) = 0 24271 11:46:12.776772 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0 24271 11:46:12.776866 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0 24271 11:46:12.776954 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0 24282 11:46:12.801320 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) 24282 11:46:12.801935 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/lea"...) = 0 24271 11:46:12.930815 mkdir("/var/lib/leapp/scratch/mounts/root_tmp", 0777) = 0 24271 11:46:12.930881 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/upper", 0777) = 0 24271 11:46:12.930973 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/work", 0777) = 0 24271 11:46:12.931066 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", 0777) = 0 24288 11:46:12.956517 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) 24288 11:46:12.957128 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", "overlay", MS_MGC_VAL, "lowerdir=/tmp,upperdir=/var/lib/"...) = 0 24290 11:46:12.998509 newfstatat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", {st_mode=S_IFDIR|0755, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0 24290 11:46:12.998848 unlinkat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", AT_REMOVEDIR) = 0 24271 11:46:13.015461 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", 0777) = -1 EINVAL (Invalid argument) My current assumption is a problem with overlay and xfs, in very rare conditions. Attaching the leapp.db they shared. Version-Release number of selected component (if applicable): leapp-upgrade-el7toel8-0.17.0-1.el7_9 How reproducible: Reproducible at will on customer site. Not able to reproduce internally. Actual results: Upgrade is not done. Additional info: * A similar (but not identical) issue has been reported in the past (rhbz#1870668). It has been closed due to insufficient data, and the KCS (#5339021) clearly does not apply for them. * Workaround was to make Leapp believes it has to create ext4 images as upper layers (as it is done for XFS without ftype) Edit /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py and insert one line (52) to define an array in _prepare_required_mounts(): 46 def _prepare_required_mounts(scratch_dir, mounts_dir, mount_points, xfs_info): 47 result = { 48 mount_point.fs_file: mounting.NullMount( 49 _mount_dir(mounts_dir, mount_point.fs_file)) for mount_point in mount_points 50 } 51 52 xfs_info.mountpoints_without_ftype = [ "/", "/var", "/usr", "/boot" ] # <<< 53 if not xfs_info.mountpoints_without_ftype: 54 return result Additionally, it has been suggested comment out the partitions which are not required during the in-place upgrade and use a larger LEAPP_OVL_SIZE (8192). * A simple reproducer involving the same set of syscalls has been suggested but the customer didn't spend the time to do so. yum install strace trace-cmd strace -fTttyyvs 1024 -o /root/reproducer.strace -p $$ & rm -rf /var/lib/leapp/scratch mkdir -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work} mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp rm -rf /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp trace-cmd record -o /root/mkdir.trc -b 10000 -F -p function_graph -O hex -- mkdir -m 777 /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay pkill strace umount -fl /var/lib/leapp/scratch/mounts/root_/system_overlay trace-cmd report /root/mkdir.trc > /root/mkdir.trc.txt tar zcf /tmp/debug.tgz /root/mkdir.trc /root/mkdir.trc.txt /root/reproducer.strace /var/lib/leapp/leapp.db * From mkdir() man page: EINVAL The final component ("basename") of the new directory's pathname is invalid (e.g., it contains characters not permitted by the underlying filesystem). * lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 150G 0 disk |-sda1 8:1 0 953M 0 part /boot `-sda2 8:2 0 139.7G 0 part |-rhel-root 253:0 0 23.3G 0 lvm / |-rhel-usr 253:1 0 18.6G 0 lvm /usr |-rhel-tmp 253:2 0 9.3G 0 lvm /tmp |-rhel-swap 253:3 0 9.3G 0 lvm /swap |-rhel-home 253:4 0 28G 0 lvm /home |-rhel-opt 253:5 0 28G 0 lvm /opt `-rhel-var 253:6 0 23.3G 0 lvm /var sr0 11:0 1 1024M 0 rom * /etc/fstab /dev/mapper/rhel-root / xfs defaults 0 0 UUID=7a37d1ea-22f6-43b4-bc36-e2dcad4d8212 /boot xfs defaults 0 0 /dev/mapper/rhel-home /home xfs defaults 0 0 /dev/mapper/rhel-opt /opt xfs defaults 0 0 /dev/mapper/rhel-swap /swap xfs defaults 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults 0 0 /dev/mapper/rhel-usr /usr xfs defaults 0 0 /dev/mapper/rhel-var /var xfs defaults 0 0 * blkid /dev/sda1: UUID="7a37d1ea-22f6-43b4-bc36-e2dcad4d8212" TYPE="xfs" /dev/sda2: UUID="SwTtcI-Zs2a-acFH-MWED-Z1HO-We1q-Ln3JLg" TYPE="LVM2_member" /dev/mapper/rhel-root: UUID="961840e5-dbd0-4112-a243-043073a98092" TYPE="xfs" /dev/mapper/rhel-usr: UUID="a0f82a03-e72a-4109-abad-a0588c1ebf96" TYPE="xfs" /dev/mapper/rhel-tmp: UUID="8d569aed-4522-433f-9f59-2cf02651422b" TYPE="xfs" /dev/mapper/rhel-swap: UUID="25df4b5c-79b3-4453-a993-d1e2df3e3dde" TYPE="xfs" /dev/mapper/rhel-home: UUID="cb4b763e-d4dc-4277-bfb6-7d500b3dafd3" TYPE="xfs" /dev/mapper/rhel-opt: UUID="e547d839-3948-4638-91ca-ab0aba2d1582" TYPE="xfs" /dev/mapper/rhel-var: UUID="6765f39b-b8bc-4b83-814b-f312a522a191" TYPE="xfs" * grep -o ftype=. sos_commands/xfs/* sos_commands/xfs/xfs_info:ftype=1 sos_commands/xfs/xfs_info_.boot:ftype=1 sos_commands/xfs/xfs_info_.home:ftype=1 sos_commands/xfs/xfs_info_.opt:ftype=1 sos_commands/xfs/xfs_info_.swap:ftype=1 sos_commands/xfs/xfs_info_.tmp:ftype=1 sos_commands/xfs/xfs_info_.usr:ftype=1 sos_commands/xfs/xfs_info_.var:ftype=1 * default sysctls * cmdline BOOT_IMAGE=/vmlinuz-3.10.0-1160.80.1.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/usr rhgb quiet LANG=en_IN.UTF-8