Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Upgrading with Leapp from RHEL 7.6 to 8.0, after reboot when the RPMs are being upgraded, the machine hangs.
Tested with Leapp both from Copr and from subscription content:
leapp-0.7.0-2.el7_6
leapp-repository-0.7.0-5.el7_6
and
leapp-0.7.0-1.201904300837Z.f766a84.master.el7_6
leapp-repository-0.7.0-1.201905141319Z.e83b797.master.el7_6
How reproducible:
Always, but every time a different RPM is reported as upgrading when the hang happens. It doesn't seem to be related to a particular RPM being upgraded.
Steps to Reproduce:
1. Started with an OpenStack controller node.
2. Stopped OpenStack, removed OpenStack packages.
3. Installed leapp, ran leapp upgrade --debug, rebooted.
Actual results:
Upgrade of RPMs starts but hangs indefinitely after 1-2 minutes.
Expected results:
Upgrade of RPMs should not hang :).
I would also welcome knowing some debugging technique for this phase of upgrade, how can i get more information that could shed some light on what is happening. At this point the node is obviously not reachable via ssh, and the TTY i'm getting via virsh-console is non-interactive. If i can pass something to `leapp upgrade` or set something to make the upgrade print some useful info, please let me know.
Here is how the last output looks:
[ 90.226146] upgrade[319]: Upgrading : device-mapper-multipath-libs-0.7.8-7.el8.x86_6 509/1532
[ 90.227339] upgrade[319]: Running scriptlet: device-mapper-multipath-libs-0.7.8-7.el8.x86_6 509/1532
[ 90.228502] upgrade[319]: Upgrading : device-mapper-event-8:1.02.155-6.el8.x86_64 510/1532
[ 90.229685] upgrade[319]: Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 510/1532
[ 90.230916] upgrade[319]: Upgrading : lvm2-libs-8:2.03.02-6.el8.x86_64 511/1532
[ 90.232097] upgrade[319]: Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64 511/1532
[ 90.233332] upgrade[319]: Upgrading : lvm2-8:2.03.02-6.el8.x86_64 512/1532
[ 90.234512] upgrade[319]: Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 512/1532
[ 90.235703] upgrade[319]: Upgrading : samba-common-libs-4.9.1-8.el8.x86_64 513/1532
[ 90.236892] upgrade[319]: Running scriptlet: samba-common-libs-4.9.1-8.el8.x86_64 513/1532
[ 90.238110] upgrade[319]: Upgrading : libwbclient-4.9.1-8.el8.x86_64 514/1532
[ 90.239290] upgrade[319]: Upgrading : samba-client-libs-4.9.1-8.el8.x86_64 515/1532
[ 90.240565] upgrade[319]: Running scriptlet: samba-client-libs-4.9.1-8.el8.x86_64 515/1532
[ 90.241800] upgrade[319]: Upgrading : libsmbclient-4.9.1-8.el8.x86_64 516/1532
[ 90.242996] upgrade[319]: Running scriptlet: libsmbclient-4.9.1-8.el8.x86_64 516/1532
[ 90.244199] upgrade[319]: Installing : network-scripts-10.00.1-1.el8.x86_64 517/1532
[ 90.245442] upgrade[319]: Running scriptlet: network-scripts-10.00.1-1.el8.x86_64 517/1532
[ 90.246640] upgrade[319]: Installing : dhcp-client-12:4.3.6-30.el8.x86_64 518/1532
[ 90.247829] upgrade[319]: Upgrading : gnutls-dane-3.6.5-2.el8.x86_64 519/1532
[ 94.430563] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 224.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 294.526160] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 404.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 474.526173] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 584.526150] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 654.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 764.526172] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 834.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 944.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1014.526147] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1124.526153] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1194.526149] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1304.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1374.526145] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1484.526178] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1554.526188] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1664.526176] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1734.526244] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1844.526156] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1914.526161] systemd-journald[200]: Sent WATCHDOG=1 notification.
When taking a look at "leapp-upgrade.log" I see:
...
...
...
Upgrading : grubby-8.40-34.el8.x86_64
...
...
...
Sep 23 10:13:27 localhost upgrade[509]: 2019-09-23 10:12:56.091374 [ERROR] Actor: kernelcmdlineconfig Message: Failed to append extra arguments to kernel command line.
Could you please try with latest el8 "grubby" (should be 8.40-37) ? How is the machine registered?
I am now commenting on dropping into emergency mode.
Turns out the PR has been verified as OAMG-681 using a seemingly unrelated method:
* have an iso9660 partition (not necessarily mounted) present during upgrade.
This hasn't been automated, ie. we can't re-run it with Brew build easily. It looks like it could be pretty easy to automate though. OR at least re-run manually. Creating task in Jira.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2020:1959