Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1972238

Summary: [leapp] inhibit the upgrade when FSTAB contains entries in incorrect order
Product: Red Hat Enterprise Linux 7 Reporter: Christophe Besson <cbesson>
Component: leapp-repositoryAssignee: Leapp team <leapp-notifications>
Status: CLOSED ERRATA QA Contact: Upgrades and Supportability <upgrades-and-supportability>
Severity: high Docs Contact:
Priority: high    
Version: 7.9CC: fkrska, mmacura, prjagtap, pstodulk
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: leapp-repository-0.18.0-1.el7_9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-17 15:00:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1818077, 1818088    

Description Christophe Besson 2021-06-15 13:42:18 UTC 
Description of problem:
Customer experienced a critical issue during the IPU, it ends in the emergency step, unable to find some files located in /var/log/leapp (dnf-plugin-data.txt and answerfile).

Version-Release number of selected component (if applicable):
leapp-0.12.1-1.el7_9.noarch
leapp-repository-0.14.0-4.el7_9.noarch

How reproducible:
100% for the customer. Didn't try to reproduce (but for me, it was working in the past, for sure).

Actual results:
Jun 15 05:03:17 localhost dracut-initqueue[715]: Scanning devices sda2 sdb1  for LVM logical volumes rhel/root rhel/swap rhel/Fvar
Jun 15 05:03:17 localhost dracut-initqueue[715]: WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/root' [<18.63 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/swap' [22.35 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/Fhome' [20.00 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/Fvar' [5.00 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/Ftmp' [5.00 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/Fvarlog' [5.00 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: inactive '/dev/rhel/Fvartmp' [5.00 GiB] inherit
Jun 15 05:03:17 localhost dracut-initqueue[715]: WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
Jun 15 05:03:17 localhost dracut-initqueue[715]: Allowing activation with --readonly --sysinit.
Jun 15 05:03:17 localhost systemd[1]: Found device /dev/mapper/rhel-root.
Jun 15 05:03:17 localhost systemd[1]: Starting File System Check on /dev/mapper/rhel-root...
Jun 15 05:03:17 localhost systemd-fsck[772]: fsck.xfs doesn't exist, not checking file system on /dev/mapper/rhel-root.
Jun 15 05:03:17 localhost systemd[1]: Started File System Check on /dev/mapper/rhel-root.
Jun 15 05:03:17 localhost systemd[1]: Mounting /sysroot...
 :
Jun 15 05:03:17 localhost kernel: XFS (dm-0): Mounting V5 Filesystem
Jun 15 05:03:17 localhost kernel: XFS (dm-0): Ending clean mount
Jun 15 05:03:17 localhost systemd[1]: Mounted /sysroot.
 :
Jun 15 05:03:17 localhost kernel: XFS (sda1): Mounting V5 Filesystem
Jun 15 05:03:19 localhost kernel: XFS (sda1): Ending clean mount
Jun 15 05:03:19 localhost kernel: XFS (dm-3): Mounting V5 Filesystem
Jun 15 05:03:19 localhost kernel: XFS (dm-3): Ending clean mount
 :
Jun 15 05:03:19 localhost upgrade[804]: mount: mount point /var/tmp does not exist
Jun 15 05:03:19 localhost kernel: XFS (dm-5): Mounting V5 Filesystem
Jun 15 05:03:19 localhost kernel: XFS (dm-5): Ending clean mount
Jun 15 05:03:19 localhost kernel: XFS (dm-2): Mounting V5 Filesystem
Jun 15 05:03:19 localhost kernel: XFS (dm-2): Ending clean mount
Jun 15 05:03:19 localhost kernel: XFS (dm-4): Mounting V5 Filesystem
Jun 15 05:03:19 localhost kernel: XFS (dm-4): Ending clean mount
 :
Jun 15 05:03:36 localhost kernel: XFS (dm-6): Mounting V5 Filesystem
Jun 15 05:03:37 localhost kernel: XFS (dm-6): Ending clean mount
 :
 :
Jun 15 05:03:45 localhost upgrade[804]: ==> Processing phase `RPMUpgrade`
Jun 15 05:03:45 localhost upgrade[804]: ====> * dnf_upgrade_transaction
Jun 15 05:03:45 localhost upgrade[804]:         Setup and call DNF upgrade command
Jun 15 05:03:46 localhost upgrade[804]: Process Process-181:
Jun 15 05:03:46 localhost upgrade[804]: Traceback (most recent call last):
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Jun 15 05:03:46 localhost upgrade[804]:     self.run()
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
Jun 15 05:03:46 localhost upgrade[804]:     self._target(*self._args, **self._kwargs)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
Jun 15 05:03:46 localhost upgrade[804]:     actor_instance.run(*args, **kwargs)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 335, in run
Jun 15 05:03:46 localhost upgrade[804]:     self.process(*args)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/dnfupgradetransaction/actor.py", line 52, in process
Jun 15 05:03:46 localhost upgrade[804]:     plugin_info=plugin_info
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/dnfplugin.py", line 226, in perform_transaction_install
Jun 15 05:03:46 localhost upgrade[804]:     cmd_prefix=cmd_prefix
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/dnfplugin.py", line 114, in _transaction
Jun 15 05:03:46 localhost upgrade[804]:     backup_config(context=context)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/dnfplugin.py", line 84, in backup_config
Jun 15 05:03:46 localhost upgrade[804]:     context.copy_from(DNF_PLUGIN_DATA_PATH, DNF_PLUGIN_DATA_LOG_PATH)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/mounting.py", line 222, in copy_from
Jun 15 05:03:46 localhost upgrade[804]:     shutil.copy2(self.full_path(src), dst)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
Jun 15 05:03:46 localhost upgrade[804]:     copyfile(src, dst)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
Jun 15 05:03:46 localhost upgrade[804]:     with open(dst, 'wb') as fdst:
Jun 15 05:03:46 localhost upgrade[804]: IOError: [Errno 2] No such file or directory: '/var/log/leapp/dnf-plugin-data.txt'
Jun 15 05:03:46 localhost upgrade[804]: ========================================================================================================
Jun 15 05:03:46 localhost upgrade[804]: Actor dnf_upgrade_transaction unexpectedly terminated with exit code: 1 - Please check the above details
Jun 15 05:03:46 localhost upgrade[804]: ========================================================================================================
Jun 15 05:03:46 localhost upgrade[804]: Traceback (most recent call last):
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/bin/leapp", line 9, in <module>
Jun 15 05:03:46 localhost upgrade[804]:     load_entry_point('leapp==0.12.1', 'console_scripts', 'leapp')()
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/cli/__init__.py", line 18, in main
Jun 15 05:03:46 localhost upgrade[804]:     cli.command.execute('leapp version {}'.format(VERSION))
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 105, in execute
Jun 15 05:03:46 localhost upgrade[804]:     args.func(args)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 127, in called
Jun 15 05:03:46 localhost upgrade[804]:     self.target(args)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/cli/upgrade/__init__.py", line 321, in upgrade
Jun 15 05:03:46 localhost upgrade[804]:     workflow.save_answers(answerfile_path, userchoices_path)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/workflows/__init__.py", line 117, in save_answers
Jun 15 05:03:46 localhost upgrade[804]:     self._answer_store.generate(self._dialogs, answerfile_path)
Jun 15 05:03:46 localhost upgrade[804]:   File "/usr/lib/python2.7/site-packages/leapp/messaging/answerstore.py", line 136, in generate
Jun 15 05:03:46 localhost upgrade[804]:     with open(answer_file_path, 'w') as f:
Jun 15 05:03:46 localhost upgrade[804]: IOError: [Errno 2] No such file or directory: '/var/log/leapp/answerfile'
Jun 15 05:03:46 localhost kernel: XFS (dm-6): Unmounting Filesystem
Jun 15 05:03:46 localhost kernel: XFS (dm-2): Unmounting Filesystem
Jun 15 05:03:46 localhost kernel: XFS (dm-5): Unmounting Filesystem
Jun 15 05:03:46 localhost kernel: XFS (dm-3): Unmounting Filesystem
Jun 15 05:03:46 localhost kernel: XFS (sda1): Unmounting Filesystem
Jun 15 05:03:46 localhost kernel: XFS (dm-4): Unmounting Filesystem

Additional info:
* I don't see any workaround except merging the /var/log partition into /var.

* Here is their lsblk output, to correlate with the dm-X devices:
NAME             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                2:0    1     4K  0 disk 
sda                8:0    0   450G  0 disk 
|-sda1             8:1    0   488M  0 part /boot
`-sda2             8:2    0 413.5G  0 part 
  |-rhel-root    253:0    0  18.6G  0 lvm  /
  |-rhel-swap    253:1    0  22.4G  0 lvm  [SWAP]
  |-rhel-Fvar    253:2    0     5G  0 lvm  /var
  |-rhel-Fhome   253:3    0    20G  0 lvm  /home
  |-rhel-Ftmp    253:4    0     5G  0 lvm  /tmp
  |-rhel-Fvarlog 253:5    0     5G  0 lvm  /var/log
  `-rhel-Fvartmp 253:6    0     5G  0 lvm  /var/tmp
sdb                8:16   0   250G  0 disk 
`-sdb1             8:17   0   250G  0 part
Comment 4 Christophe Besson 2022-09-01 13:36:17 UTC 
Looks like we have a similar error with another case.

As per the boot logs, it seems /var/log (dm-1) is mounted 1 sec before /var (sda7):
~~~
[    4.852936] localhost kernel: EXT4-fs (dm-1): recovery complete
[    4.853231] localhost kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
 :
[    5.747177] localhost kernel: EXT4-fs (sda7): recovery complete
[    5.748109] localhost kernel: EXT4-fs (sda7): mounted filesystem with ordered data mode. Opts: (null)
 :
[   33.446512] localhost upgrade[660]: IOError: [Errno 2] No such file or directory: '/var/log/leapp/leapp-upgrade.log'
[   33.491105] localhost upgrade[646]: Container sysroot failed with error code 1.
[   33.499459] localhost upgrade[613]: writing logs to disk and rebooting
~~~

That would explain why /var/log/leapp* is not found, that path being hidden by the 2nd mount.

/var/log is defined before /var in /etc/fstab, asked for the customer to reorder it to see how it behaves.
Comment 5 Christophe Besson 2022-09-01 14:33:19 UTC 
Just to confirm reordering fstab fixed the issue.
I already suggested this workaround in another case but I cannot find the number.
And I also saw that in the past with a separate /var/log/audit (that we just recommended to comment out from fstab).
I guess checking the order from the fstab entries would help to prevent this.
Comment 6 Petr Stodulka 2022-09-02 10:02:57 UTC 
Yes, regardin fstab:
~~~
The order of records in fstab is important because fsck(8), mount(8), and umount(8) sequentially iterate through fstab doing their thing.
~~~

So from this point, if a customer has in fstab e.g.:
~~~
/var/log ....
/var     ....
~~~

It's an 'invalid' order since the beginning. However, since RHEL 7, the mounting is handled by systemd which ignores the order of lines in fstab. That's probably the difference as in our case we use `mount -a` in scripts.

1) We could start to check the order of lines in /etc/fstab so users will be required to fix the order prior the upgrade.
2) Or we will investigate the possibility of mounting using systemd

Regarding current implementation, option 1) seems more feasible To have this covered sooner. The second option would possibly require additional bigger changes.
Comment 7 Christophe Besson 2022-09-02 11:32:03 UTC 
I would prefer the first solution, as it brings less complexity :)
Comment 10 Petr Stodulka 2023-03-07 19:57:07 UTC 
The fix is part of the new upstream release and will be present in the next update:
    https://github.com/oamg/leapp-repository/pull/1009
Comment 15 errata-xmlrpc 2023-05-17 15:00:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (leapp and leapp-repository bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3187