Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1486962

Summary: The redhat-upgrade-tool does not complete with rhel 7.4
Product: Red Hat Enterprise Linux 6 Reporter: Devon <dshumake>
Component: redhat-upgrade-toolAssignee: Michal Bocek <mbocek>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact: Jiri Hornicek <jhornice>
Priority: urgent    
Version: 6.9CC: cdonnell, dconsoli, dwojewod, fkrska, jcastran, jentrena, jhornice, jstodola, ktordeur, mateusz.bacal, mbocek, mmatsuya, ovasik, phracek, pkotvan, release-test-team-automation, snavale, srandhaw
Target Milestone: rcKeywords: Extras, PrioBumpGSS, Regression
Target Release: 6.9   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: redhat-upgrade-tool-0.7.50-1.el6 Doc Type: Bug Fix
Doc Text:
The introduction of kernel modules compression in RHEL 7.4 caused that the upgrade from RHEL 6.9 to RHEL 7.4 was not possible. Now, the Red Hat Upgrade Tool decompresses the modules before starting to upgrade the system, and the upgrade works as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-03 13:36:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1461138    

Description Devon 2017-08-30 22:28:59 UTC
Description of problem:

Using the 7.4 binary with the redhat-upgrade-tool causes the upgrade process to fail with the following errors after rebooting twice after the redhat-upgrade-tool completes.

==========================================
Red Hat Enterprise Linux Server release 6.9 (Santiago)
Kernel 2.6.32-696.10.1.el6.x86_64 on an x86_64

upgrade.example.com login: root
Password: 
login: no shell: Permission denied.
init: serial (ttyS0) main process ended, respawning
==========================================

In the redhat-upgrade-tool logs we see many repeating desolve errors which look like the ones below.

==========================================
 → libstdc++.so.6
[    63.095] (DD) redhat_upgrade_tool.yum:_requiringFromTransaction() TSINFO: Ma
rking libstdc++-4.8.5-16.el7.i686 as install for pcre-8.32-17.el7.i686
[    63.095] (DD) redhat_upgrade_tool.depsolve:procReqPo() pcre-8.32-17.el7.i686
 → libstdc++.so.6(CXXABI_1.3)
[    63.096] (DD) redhat_upgrade_tool.yum:_quickWhatProvides() Quick matched lib
stdc++-4.8.5-16.el7.i686 to require for libstdc++.so.6(CXXABI_1.3)
[    63.096] (DD) redhat_upgrade_tool.depsolve:procReqPo() pcre-8.32-17.el7.i686
 → libstdc++.so.6(GLIBCXX_3.4)
[    63.096] (DD) redhat_upgrade_tool.yum:_quickWhatProvides() Quick matched lib
stdc++-4.8.5-16.el7.i686 to require for libstdc++.so.6(GLIBCXX_3.4)
[    63.096] (DD) redhat_upgrade_tool.depsolve:procReqPo() pcre-8.32-17.el7.i686
 → libstdc++.so.6(GLIBCXX_3.4.9)
[    63.096] (DD) redhat_upgrade_tool.yum:_quickWhatProvides() Quick matched lib
stdc++-4.8.5-16.el7.i686 to require for libstdc++.so.6(GLIBCXX_3.4.9)
[    63.096] (DD) redhat_upgrade_tool.depsolve:pkgAdded() 4:perl-5.10.1-144.el6.
x86_64: remove (updated)
[    63.210] (DD) redhat_upgrade_tool.depsolve:procReqPo() 2:vim-enhanced-7.4.62
9-5.el6_8.1.x86_64 → perl(:MODULE_COMPAT_5.10.1)
[    63.210] (DD) redhat_upgrade_tool.depsolve:format_missing_requires() MISSING
 REQ: 2:vim-enhanced-7.4.629-5.el6_8.1.x86_64 requires perl(:MODULE_COMPAT_5.10.
1)
[    63.210] (DD) redhat_upgrade_tool.depsolve:format_missing_requires() MISSING REQ: 2:vim-enhanced-7.4.629-5.el6_8.1.x86_64 requires perl(:MODULE_COMPAT_5.10.1)
[    63.211] (DD) redhat_upgrade_tool.depsolve:pkgAdded() perl-Pod-Perldoc-3.20-4.el7.noarch: install (install)
[    63.227] (DD) redhat_upgrade_tool.depsolve:procReqPo() perl-Pod-Perldoc-3.20-4.el7.noarch → perl(HTTP::Tiny)
[    63.228] (DD) redhat_upgrade_tool.yum:_requiringFromTransaction() TSINFO: Marking perl-HTTP-Tiny-0.033-3.el7.noarch as install for perl-Pod-Perldoc-3.20-4.el7.noarch
[    63.229] (DD) redhat_upgrade_tool.depsolve:procReqPo() perl-Pod-Perldoc-3.20-4.el7.noarch → perl(parent)
[    63.230] (DD) redhat_upgrade_tool.yum:_requiringFromTransaction() TSINFO: Marking 1:perl-parent-0.225-244.el7.noarch as install for perl-Pod-Perldoc-3.20-4.el7.noarch
[    63.231] (DD) redhat_upgrade_tool.depsolve:pkgAdded() perl-podlators-2.5.1-3.el7.noarch: install (install)
[    63.242] (DD) redhat_upgrade_tool.depsolve:pkgAdded() python-2.6.6-66.el6_8.x86_64: remove (updated)
[    63.246] (DD) redhat_upgrade_tool.depsolve:procReqPo() python-argparse-1.2.1-2.1.el6.noarch → python(abi) = 2.6
[    63.246] (DD) redhat_upgrade_tool.depsolve:format_missing_requires() MISSING REQ: python-argparse-1.2.1-2.1.el6.noarch requires python(abi)
:

==========================================

Version-Release number of selected component (if applicable):

7.4

How reproducible:

Every time

Steps to Reproduce:
1. Use the redhat-upgrade-tool to upgrade a rhel 6.9 system to rhel 7.4 ( the only supported option ).
2. Reboot the system (which would typically complete the upgrade process and install 7.4 packages on the system, replacing the rhel 6 packages.
3. Then we are only able to boot the system into single-user mode and is still rhel 6 packages.

Actual results:

The system would be upgraded to rhel 7.4

Expected results:

The system does not get upgraded and is then unable to login unless booting to single-user.

Additional info:

Comment 9 Michal Bocek 2017-09-06 12:54:09 UTC
I was able to reproduce this and will work on a solution.

Comment 10 Michal Bocek 2017-09-18 12:15:01 UTC
*** Bug 1492609 has been marked as a duplicate of this bug. ***

Comment 15 Michal Bocek 2017-09-22 18:44:27 UTC
With the help of pcahyna there's been some progress.
The key problem is that the kernel init script [1], which is provided by the Red Hat Upgrade Tool, is being executed under the RHEL 6.9 root so it uses the RHEL 6.9 modprobe (module-init-tools-3.9-26.el6) which can handle only zlib-compressed modules, not xz-compressed ones.
Note: The RHEL 7.4 modules [2] are mounted to the RHEL 6.9 root by the Red Hat Upgrade Dracut-provided initramfs, specifically by [3].

One option would be to decompress the kernel modules first - at the beginning of the kernel init script [1]. We're working on this option and will report results.

[1] <redhat-upgrade-tool-repo>/systemd/upgrade-init (during the upgrade it resides in /usr/libexec/)
[2] /lib/modules/3.10.0-693.el7.x86_64
[3] <redhat-upgrade-dracut-repo>/85system-upgrade-redhat/keep-initramfs.sh->bind_into_newroot()

Comment 16 Michal Bocek 2017-09-22 19:52:06 UTC
Adding the line below to the upgrade-init (see comment 15) script fixes the issue. I'll create new version of Red Hat Upgrade Tool with this fix as soon as possible.

find /lib/modules/$(uname -r) -name '*.ko.xz' | while read x; do xz -d $x; ln ${x%%.xz} $x; done

Comment 17 msi_bacalm 2017-09-25 07:25:30 UTC
That's great to hear ;)


What is the estimated date of new version of redhat upgrade tool available for download from redhat website ?

Comment 18 Michal Bocek 2017-09-25 09:28:05 UTC
It depends on the quality assurance team capacity to test the fix. Once it's tested it will be out for download within a day.

Comment 20 Peter Kotvan 2017-09-25 12:43:57 UTC
I was able to verify this fix with redhat-upgrade-tool-0.7.50-1.

Comment 21 msi_bacalm 2017-09-25 12:51:50 UTC
Hi,

Great to hear. When it will be available to donwload on your website ?

Comment 25 msi_bacalm 2017-09-27 18:53:05 UTC
We have tested the upgrade and now it is triggering properly but we faced next issue.

We did comparison between upgrade 6.9 -> 7.3 vs 6.9 -> 7.4.
As the outcome of it we observed that grub2-pc and it dependency are not installed as it was happening in previous version of redhat-upgrade-tool.
In the end after the last reboot when we shall end up on 7.4 we ending up in grub command line without possibility to boot forward.

Is it expected behavior ?
If there is something that we do not know about?

Reproduction:
1. Start Redhat 6.9
2. Install hotfix version of redhat upgrade tool with needed dependencies
3. Start upgrade
    a) first reboot trigger upgrade to rhel 7.4
    b) next rhel 7 packages are installed
    c) final reboot is triggered to switch to redhat 7.4
4. End of reproduction.

Comment 26 msi_bacalm 2017-09-27 18:56:45 UTC
For this test we used hotfix version of redhat-upgrade-tool rpm package...

Version of all packages related to upgrade:
redhat-upgrade-tool-0.7.50-1
preupgrade-assistant-2.4.4-1.el6
preupgrade-assistant-el6toel7-data-0.20170712-1.el6
preupgrade-assistant-tools-2.4.4-1.el6
preupgrade-assistant-el6toel7-0.6.71-2.el6

Comment 27 msi_bacalm 2017-09-27 19:03:43 UTC
Does it somehow related to one of the releases for the package preupgrade-assistant-el6toel7
2017-06-21 Petr Stodulka <pstodulk> - 0.6.69-1
- bind-chroot: backup only relevant config files 
- use correct risk level for downgraded packages 
- modified yaboot module to print correct info and do not create 
the /etc/default/grub.conf file automatically, but keep it on user 
to be sure that content of the file is really valid 
- text polishing 
- grub: added instructions how to migrate GRUB to GRUB2 
- Resolves: #1416324 #1414873 #1296658 #1447705 #1447706 #1458956


Especially this section about grub???

Comment 28 msi_bacalm 2017-09-28 06:27:26 UTC
Hi All

After digging into more details it appears that there is let's say "small" change which was not mention about in grub2 package.

I did comparison of grub2-2.02-0.44 vs grub2-2.02-0.64 and yum transaction logs from redhat-upgrade-tool and it appears that in 2.02-0.64 grub2 rpm is not longer obsoletes grub rpm.

For comparison grub2 - 2.02-0.44 was doing that that is why redhat upgrade tool would have grub2 installed after upgrade.


Here is I think next clue..... - does this change was intentional, with the grub/grub2?  

According to document https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-upgrading_from_grub_legacy_to_grub2 there is statement which clearly says:

"After finishing the in-place upgrade from RHEL 6 to RHEL 7, the grub2 package should be installed automatically"

How we can workaround this ?

Comment 29 Peter Kotvan 2017-10-02 14:22:42 UTC
Hello,

actually the grub and grub2 packages behave differently and are not upgraded. This does not necessarily depend on an obsolete clause in the package.

In the documentation you mention there is also "If the package is not on the system, you can install it by running...". Moreover in the Preupgrade Assistant analysis report there is this statement:

"During the upgrade process, GRUB will be preserved in the Master Boot Record, and a temporary GRUB configuration will be deployed, so that you can boot into the upgraded system.
Because it is not possible to support the temporary configuration, you need to rewrite it, and install GRUB 2 to the Master Boot Record manually after the upgrade."

I agree that this is confusing and I will investigate the problem further with my colleagues and possibly file another bug report.

However this is a different issue not related to the original bug so I'll close this one as verified. The problem with grub2 package can be tracked in bug 1497731.

Comment 31 msi_bacalm 2017-10-03 05:51:18 UTC
Hi,

I agree that this is separate bug.

Based on your comment 
"actually the grub and grub2 packages behave differently and are not upgraded. This does not necessarily depend on an obsolete clause in the package." - that is not fully true - redhat upgrade tool when I turned on debug logs I noticed that he is performing dummy yum transaction check based on which he is discovering what packages need to be installed on upgrade - at least this is my opinion.

When I compared the same output I noticed that on upgrades 6.9 -> 7.3 transaction showing that grub is obsoleted by grub2. In that scenario I have installed grub2. 

Unfortunately on upgrade 6.9 -> 7.4 I do not have one....

For now I was able to workaround it by adding postupgrade script which will remove legacy grub and install grub2. I have separated script which is handling migration.

But I will copy this message to original bugzilla to not make additional confusion on that one ;)

Comment 32 errata-xmlrpc 2017-10-03 13:36:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2856