Bug 1709890

Summary:	Leapp upgrade hangs when upgrading RPMs
Product:	Red Hat Enterprise Linux 7	Reporter:	Jiri Stransky <jstransk>
Component:	leapp-repository	Assignee:	Michal Reznik <mreznik>
Status:	CLOSED ERRATA	QA Contact:	Alois Mahdal <amahdal>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.6	CC:	amarecek, ddelcian, dmacpher, jfrancoa, lbezdick, mbocek, michele, mreznik, pdwyer, pstodulk
Target Milestone:	rc	Keywords:	Reopened, Upgrades
Target Release:	---	Flags:	amahdal: needinfo-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	leapp-repository-0.10.0-2.el7_8	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-04-29 01:45:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1708241, 1727807

Description Jiri Stransky 2019-05-14 13:39:41 UTC

Description of problem:

Upgrading with Leapp from RHEL 7.6 to 8.0, after reboot when the RPMs are being upgraded, the machine hangs.

Tested with Leapp both from Copr and from subscription content:

leapp-0.7.0-2.el7_6
leapp-repository-0.7.0-5.el7_6

and

leapp-0.7.0-1.201904300837Z.f766a84.master.el7_6
leapp-repository-0.7.0-1.201905141319Z.e83b797.master.el7_6

How reproducible:

Always, but every time a different RPM is reported as upgrading when the hang happens. It doesn't seem to be related to a particular RPM being upgraded.

Steps to Reproduce:
1. Started with an OpenStack controller node.
2. Stopped OpenStack, removed OpenStack packages.
3. Installed leapp, ran leapp upgrade --debug, rebooted.

Actual results:

Upgrade of RPMs starts but hangs indefinitely after 1-2 minutes.

Expected results:

Upgrade of RPMs should not hang :).

I would also welcome knowing some debugging technique for this phase of upgrade, how can i get more information that could shed some light on what is happening. At this point the node is obviously not reachable via ssh, and the TTY i'm getting via virsh-console is non-interactive. If i can pass something to `leapp upgrade` or set something to make the upgrade print some useful info, please let me know.


Here is how the last output looks:

[   90.226146] upgrade[319]:   Upgrading        : device-mapper-multipath-libs-0.7.8-7.el8.x86_6    509/1532
[   90.227339] upgrade[319]:   Running scriptlet: device-mapper-multipath-libs-0.7.8-7.el8.x86_6    509/1532
[   90.228502] upgrade[319]:   Upgrading        : device-mapper-event-8:1.02.155-6.el8.x86_64       510/1532
[   90.229685] upgrade[319]:   Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64       510/1532
[   90.230916] upgrade[319]:   Upgrading        : lvm2-libs-8:2.03.02-6.el8.x86_64                  511/1532
[   90.232097] upgrade[319]:   Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64                  511/1532
[   90.233332] upgrade[319]:   Upgrading        : lvm2-8:2.03.02-6.el8.x86_64                       512/1532
[   90.234512] upgrade[319]:   Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64                       512/1532
[   90.235703] upgrade[319]:   Upgrading        : samba-common-libs-4.9.1-8.el8.x86_64              513/1532
[   90.236892] upgrade[319]:   Running scriptlet: samba-common-libs-4.9.1-8.el8.x86_64              513/1532
[   90.238110] upgrade[319]:   Upgrading        : libwbclient-4.9.1-8.el8.x86_64                    514/1532
[   90.239290] upgrade[319]:   Upgrading        : samba-client-libs-4.9.1-8.el8.x86_64              515/1532
[   90.240565] upgrade[319]:   Running scriptlet: samba-client-libs-4.9.1-8.el8.x86_64              515/1532
[   90.241800] upgrade[319]:   Upgrading        : libsmbclient-4.9.1-8.el8.x86_64                   516/1532
[   90.242996] upgrade[319]:   Running scriptlet: libsmbclient-4.9.1-8.el8.x86_64                   516/1532
[   90.244199] upgrade[319]:   Installing       : network-scripts-10.00.1-1.el8.x86_64              517/1532
[   90.245442] upgrade[319]:   Running scriptlet: network-scripts-10.00.1-1.el8.x86_64              517/1532
[   90.246640] upgrade[319]:   Installing       : dhcp-client-12:4.3.6-30.el8.x86_64                518/1532
[   90.247829] upgrade[319]:   Upgrading        : gnutls-dane-3.6.5-2.el8.x86_64                    519/1532
[   94.430563] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  224.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  294.526160] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  404.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  474.526173] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  584.526150] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  654.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  764.526172] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  834.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  944.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1014.526147] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1124.526153] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1194.526149] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1304.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1374.526145] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1484.526178] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1554.526188] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1664.526176] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1734.526244] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1844.526156] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1914.526161] systemd-journald[200]: Sent WATCHDOG=1 notification.

Comment 2 Jiri Stransky 2019-05-14 13:40:51 UTC

Output from other attempts is also at bug 1708241, but it looks always pretty much the same.

Comment 9 Michal Reznik 2019-10-22 15:33:14 UTC

When taking a look at "leapp-upgrade.log" I see:

...
...
...
Upgrading        : grubby-8.40-34.el8.x86_64
...
...
...
Sep 23 10:13:27 localhost upgrade[509]: 2019-09-23 10:12:56.091374 [ERROR] Actor: kernelcmdlineconfig Message: Failed to append extra arguments to kernel command line.

Could you please try with latest el8 "grubby" (should be 8.40-37) ? How is the machine registered?

I am now commenting on dropping into emergency mode.

Comment 12 Jiri Stransky 2019-10-29 12:45:37 UTC

A link for completeness -- the underlying symptom has been discussed and investigated at bz1712456, now closed/notabug.

Comment 16 Michal Reznik 2019-11-13 11:26:48 UTC

Fixed in upstream:

https://github.com/oamg/leapp-repository/commit/0d1668597bdf16316bb10220525d26b5c3a0f0ed

Comment 31 Alois Mahdal 2020-04-22 08:46:23 UTC

Turns out the PR has been verified as OAMG-681 using a seemingly unrelated method:

 *  have an iso9660 partition (not necessarily mounted) present during upgrade.

This hasn't been automated, ie. we can't re-run it with Brew build easily.  It looks like it could be pretty easy to automate though.  OR at least re-run manually.  Creating task in Jira.

Comment 32 Alois Mahdal 2020-04-27 11:22:23 UTC

VERIFIED on all platforms with:

      - leapp-0.10.0-2.el7_8
      - leapp-repository-0.10.0-2.el7_8

Comment 34 errata-xmlrpc 2020-04-29 01:45:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1959