Bug 1691155

Summary: Upgrade failure due to missing leapp/dnf-plugin-data.txt with separate /var
Product: Red Hat Enterprise Linux 7 Reporter: Bernie Hoefer <bhoefer>
Component: leapp-repositoryAssignee: Leapp team <leapp-notifications>
Status: CLOSED ERRATA QA Contact: Alois Mahdal <amahdal>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: alexandre.chanu, amkulkar, aromito, asanders, hannsj_uhl, jcastran, jentrena, jotodd, kkinge, kupadhya, leapp-notifications, mbliss, mbocek, mgazdik, mskinner, pdwyer, pravisha, pstodulk, sprestma, swolfe, syangsao, tbowling
Target Milestone: rcKeywords: Upgrades
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: leapp-repository-0.9.0-3.el7 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-05 06:59:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Output of "leapp upgrade --debug" when run on a server with a separate /var file system. none

Description Bernie Hoefer 2019-03-21 02:02:47 UTC
Description of problem:
On a RHEL server that uses a separate /var file system, the "leapp upgrade --debug" command fails with this Python traceback:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 59, in _do_run
    target_actor(logger=logger, messaging=messaging).run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 228, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/prepareupgradetransaction/actor.py", line 245, in process
    error = self.dnf_plugin_rpm_download(ofs_info)
  File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/prepareupgradetransaction/actor.py", line 182, in dnf_plugin_rpm_download
    with open(os.path.join(overlayfs_info.merged, 'var', 'lib', 'leapp', 'dnf-plugin-data.txt'), 'w+') as data:
IOError: [Errno 2] No such file or directory: '/tmp/leapp-overlay/merged/var/lib/leapp/dnf-plugin-data.txt'
============================================================================================================
Actor prepare_upgrade_transaction unexpectedly terminated with exit code: 1 - Please check the above details
============================================================================================================


Version-Release number of selected component (if applicable):
leapp-0.5.0-1.el7_6.noarch
leapp-repository-0.5.0-1.el7_6.noarch


Steps to Reproduce:
1. Install a RHEL 7.6 system with a separate /var file system.  (I made mine a
   4 GB XFS one.)
2. Follow the leapp upgrade instructions provided to high touch beta
   (HTB) customers via:
   <https://access.redhat.com/groups/3614781/announcements/3746941>


Actual results:
Leapp upgrade fails.


Expected results:
Leapp upgrade succeeds.


Additional info:
This problem was brought to my attention by a HTB customer that was testing the leapp upgrade on a RHEL 7.6 server with:
  12 GB ext4 mounted to /,
   2 GB ext4 mounted to /tmp, and 
   4 GB ext4 mounted to /var.

After finding:
https://github.com/oamg/leapp-repository/issues/83
...I reproduced the customer's problem on a RHEL 7.6 server that I set up to have a separate 4 GB XFS /var file system.

Please see the attached FailedLeappUpgradeUsingXFS_var_FileSystem.txt for the debug output to STDOUT.  While the traceback doesn't appear until line 2544 in that file, the upgrade process appears to start to have problems at line 187 with the subscription-manager INFO lines... and then at line 195 a WARNING.  I have not seen lines like that in past, successful, leapp upgrades.

Comment 2 Bernie Hoefer 2019-03-21 02:07:49 UTC
Created attachment 1546280 [details]
Output of "leapp upgrade --debug" when run on a server with a separate /var file system.

Comment 3 Petr Stodulka 2019-03-22 11:05:50 UTC
Yes, it's known problem. We will update the BZ when we have new info.

Comment 7 Michal Bocek 2019-05-14 13:51:51 UTC
To clarify, the issue still persist even for the upgrades to RHEL 8.0 GA. I.e. it's not limited to RHEL 8 HTB only.

Steps to Reproduce:
1. Install a RHEL 7.6 system with a separate /var file system.
2. Follow the official instructions for upgrading to RHEL 8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/upgrading_to_rhel_8/index

Comment 8 Mason Loring Bliss 2019-06-27 15:23:52 UTC
DISA STIG specifies that /home, /var, /var/log/audit, and /tmp be on separate partitions. It seems reasonable that Leapp accomodate this fairly common customer configuration.

This bug speaks to /var, but I wonder if we might also include /tmp in the same bug, rather than opening a new bug for Leapp's handling to separate /tmp. If desired, I'll open a new bug targeted at Leapp under DISA STIG separately. Thanks in advance for feedback.

Comment 9 Michal Bocek 2019-06-27 15:34:50 UTC
Hi Mason, we plan to tackle not only /var but all the other system directories as well. We know that having various system directories on separate partitions is quite common and allowing that is on our roadmap.

Comment 11 Petr Stodulka 2019-07-19 08:22:37 UTC
*** Bug 1731249 has been marked as a duplicate of this bug. ***

Comment 18 Petr Stodulka 2019-10-24 21:40:05 UTC
Related upstream PR: https://github.com/oamg/leapp-repository/pull/344

Comment 22 Alois Mahdal 2019-11-05 05:15:18 UTC
Verifed with leapp-repository-0.9.0-4.el7 as part of regression testing:

https://projects.engineering.redhat.com/browse/OAMG-667

Comment 24 errata-xmlrpc 2019-11-05 06:59:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3306