Hide Forgot
Description of problem: Upgrading with Leapp from RHEL 7.6 to 8.0, after reboot when the RPMs are being upgraded, the machine hangs. Tested with Leapp both from Copr and from subscription content: leapp-0.7.0-2.el7_6 leapp-repository-0.7.0-5.el7_6 and leapp-0.7.0-1.201904300837Z.f766a84.master.el7_6 leapp-repository-0.7.0-1.201905141319Z.e83b797.master.el7_6 How reproducible: Always, but every time a different RPM is reported as upgrading when the hang happens. It doesn't seem to be related to a particular RPM being upgraded. Steps to Reproduce: 1. Started with an OpenStack controller node. 2. Stopped OpenStack, removed OpenStack packages. 3. Installed leapp, ran leapp upgrade --debug, rebooted. Actual results: Upgrade of RPMs starts but hangs indefinitely after 1-2 minutes. Expected results: Upgrade of RPMs should not hang :). I would also welcome knowing some debugging technique for this phase of upgrade, how can i get more information that could shed some light on what is happening. At this point the node is obviously not reachable via ssh, and the TTY i'm getting via virsh-console is non-interactive. If i can pass something to `leapp upgrade` or set something to make the upgrade print some useful info, please let me know. Here is how the last output looks: [ 90.226146] upgrade[319]: Upgrading : device-mapper-multipath-libs-0.7.8-7.el8.x86_6 509/1532 [ 90.227339] upgrade[319]: Running scriptlet: device-mapper-multipath-libs-0.7.8-7.el8.x86_6 509/1532 [ 90.228502] upgrade[319]: Upgrading : device-mapper-event-8:1.02.155-6.el8.x86_64 510/1532 [ 90.229685] upgrade[319]: Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 510/1532 [ 90.230916] upgrade[319]: Upgrading : lvm2-libs-8:2.03.02-6.el8.x86_64 511/1532 [ 90.232097] upgrade[319]: Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64 511/1532 [ 90.233332] upgrade[319]: Upgrading : lvm2-8:2.03.02-6.el8.x86_64 512/1532 [ 90.234512] upgrade[319]: Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 512/1532 [ 90.235703] upgrade[319]: Upgrading : samba-common-libs-4.9.1-8.el8.x86_64 513/1532 [ 90.236892] upgrade[319]: Running scriptlet: samba-common-libs-4.9.1-8.el8.x86_64 513/1532 [ 90.238110] upgrade[319]: Upgrading : libwbclient-4.9.1-8.el8.x86_64 514/1532 [ 90.239290] upgrade[319]: Upgrading : samba-client-libs-4.9.1-8.el8.x86_64 515/1532 [ 90.240565] upgrade[319]: Running scriptlet: samba-client-libs-4.9.1-8.el8.x86_64 515/1532 [ 90.241800] upgrade[319]: Upgrading : libsmbclient-4.9.1-8.el8.x86_64 516/1532 [ 90.242996] upgrade[319]: Running scriptlet: libsmbclient-4.9.1-8.el8.x86_64 516/1532 [ 90.244199] upgrade[319]: Installing : network-scripts-10.00.1-1.el8.x86_64 517/1532 [ 90.245442] upgrade[319]: Running scriptlet: network-scripts-10.00.1-1.el8.x86_64 517/1532 [ 90.246640] upgrade[319]: Installing : dhcp-client-12:4.3.6-30.el8.x86_64 518/1532 [ 90.247829] upgrade[319]: Upgrading : gnutls-dane-3.6.5-2.el8.x86_64 519/1532 [ 94.430563] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 224.526152] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 294.526160] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 404.526152] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 474.526173] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 584.526150] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 654.526180] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 764.526172] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 834.526180] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 944.526152] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1014.526147] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1124.526153] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1194.526149] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1304.526152] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1374.526145] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1484.526178] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1554.526188] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1664.526176] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1734.526244] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1844.526156] systemd-journald[200]: Sent WATCHDOG=1 notification. [ 1914.526161] systemd-journald[200]: Sent WATCHDOG=1 notification.
Output from other attempts is also at bug 1708241, but it looks always pretty much the same.
When taking a look at "leapp-upgrade.log" I see: ... ... ... Upgrading : grubby-8.40-34.el8.x86_64 ... ... ... Sep 23 10:13:27 localhost upgrade[509]: 2019-09-23 10:12:56.091374 [ERROR] Actor: kernelcmdlineconfig Message: Failed to append extra arguments to kernel command line. Could you please try with latest el8 "grubby" (should be 8.40-37) ? How is the machine registered? I am now commenting on dropping into emergency mode.
A link for completeness -- the underlying symptom has been discussed and investigated at bz1712456, now closed/notabug.
Fixed in upstream: https://github.com/oamg/leapp-repository/commit/0d1668597bdf16316bb10220525d26b5c3a0f0ed
Turns out the PR has been verified as OAMG-681 using a seemingly unrelated method: * have an iso9660 partition (not necessarily mounted) present during upgrade. This hasn't been automated, ie. we can't re-run it with Brew build easily. It looks like it could be pretty easy to automate though. OR at least re-run manually. Creating task in Jira.
VERIFIED on all platforms with: - leapp-0.10.0-2.el7_8 - leapp-repository-0.10.0-2.el7_8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1959