RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1709890 - Leapp upgrade hangs when upgrading RPMs
Summary: Leapp upgrade hangs when upgrading RPMs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: leapp-repository
Version: 7.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Michal Reznik
QA Contact: Alois Mahdal
URL:
Whiteboard:
Depends On:
Blocks: 1708241 1727807
TreeView+ depends on / blocked
 
Reported: 2019-05-14 13:39 UTC by Jiri Stransky
Modified: 2023-10-06 18:18 UTC (History)
10 users (show)

Fixed In Version: leapp-repository-0.10.0-2.el7_8
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-29 01:45:58 UTC
Target Upstream Version:
Embargoed:
amahdal: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github oamg leapp-repository pull 398 0 None closed Call "nsenter" with dnf transaction install 2021-02-15 09:53:07 UTC
Red Hat Product Errata RHBA-2020:1959 0 None None None 2020-04-29 01:46:15 UTC

Description Jiri Stransky 2019-05-14 13:39:41 UTC 
Description of problem:

Upgrading with Leapp from RHEL 7.6 to 8.0, after reboot when the RPMs are being upgraded, the machine hangs.

Tested with Leapp both from Copr and from subscription content:

leapp-0.7.0-2.el7_6
leapp-repository-0.7.0-5.el7_6

and

leapp-0.7.0-1.201904300837Z.f766a84.master.el7_6
leapp-repository-0.7.0-1.201905141319Z.e83b797.master.el7_6

How reproducible:

Always, but every time a different RPM is reported as upgrading when the hang happens. It doesn't seem to be related to a particular RPM being upgraded.

Steps to Reproduce:
1. Started with an OpenStack controller node.
2. Stopped OpenStack, removed OpenStack packages.
3. Installed leapp, ran leapp upgrade --debug, rebooted.

Actual results:

Upgrade of RPMs starts but hangs indefinitely after 1-2 minutes.

Expected results:

Upgrade of RPMs should not hang :).

I would also welcome knowing some debugging technique for this phase of upgrade, how can i get more information that could shed some light on what is happening. At this point the node is obviously not reachable via ssh, and the TTY i'm getting via virsh-console is non-interactive. If i can pass something to `leapp upgrade` or set something to make the upgrade print some useful info, please let me know.


Here is how the last output looks:

[   90.226146] upgrade[319]:   Upgrading        : device-mapper-multipath-libs-0.7.8-7.el8.x86_6    509/1532
[   90.227339] upgrade[319]:   Running scriptlet: device-mapper-multipath-libs-0.7.8-7.el8.x86_6    509/1532
[   90.228502] upgrade[319]:   Upgrading        : device-mapper-event-8:1.02.155-6.el8.x86_64       510/1532
[   90.229685] upgrade[319]:   Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64       510/1532
[   90.230916] upgrade[319]:   Upgrading        : lvm2-libs-8:2.03.02-6.el8.x86_64                  511/1532
[   90.232097] upgrade[319]:   Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64                  511/1532
[   90.233332] upgrade[319]:   Upgrading        : lvm2-8:2.03.02-6.el8.x86_64                       512/1532
[   90.234512] upgrade[319]:   Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64                       512/1532
[   90.235703] upgrade[319]:   Upgrading        : samba-common-libs-4.9.1-8.el8.x86_64              513/1532
[   90.236892] upgrade[319]:   Running scriptlet: samba-common-libs-4.9.1-8.el8.x86_64              513/1532
[   90.238110] upgrade[319]:   Upgrading        : libwbclient-4.9.1-8.el8.x86_64                    514/1532
[   90.239290] upgrade[319]:   Upgrading        : samba-client-libs-4.9.1-8.el8.x86_64              515/1532
[   90.240565] upgrade[319]:   Running scriptlet: samba-client-libs-4.9.1-8.el8.x86_64              515/1532
[   90.241800] upgrade[319]:   Upgrading        : libsmbclient-4.9.1-8.el8.x86_64                   516/1532
[   90.242996] upgrade[319]:   Running scriptlet: libsmbclient-4.9.1-8.el8.x86_64                   516/1532
[   90.244199] upgrade[319]:   Installing       : network-scripts-10.00.1-1.el8.x86_64              517/1532
[   90.245442] upgrade[319]:   Running scriptlet: network-scripts-10.00.1-1.el8.x86_64              517/1532
[   90.246640] upgrade[319]:   Installing       : dhcp-client-12:4.3.6-30.el8.x86_64                518/1532
[   90.247829] upgrade[319]:   Upgrading        : gnutls-dane-3.6.5-2.el8.x86_64                    519/1532
[   94.430563] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  224.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  294.526160] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  404.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  474.526173] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  584.526150] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  654.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  764.526172] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  834.526180] systemd-journald[200]: Sent WATCHDOG=1 notification.
[  944.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1014.526147] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1124.526153] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1194.526149] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1304.526152] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1374.526145] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1484.526178] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1554.526188] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1664.526176] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1734.526244] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1844.526156] systemd-journald[200]: Sent WATCHDOG=1 notification.
[ 1914.526161] systemd-journald[200]: Sent WATCHDOG=1 notification.
Comment 2 Jiri Stransky 2019-05-14 13:40:51 UTC 
Output from other attempts is also at bug 1708241, but it looks always pretty much the same.
Comment 9 Michal Reznik 2019-10-22 15:33:14 UTC 
When taking a look at "leapp-upgrade.log" I see:

...
...
...
Upgrading        : grubby-8.40-34.el8.x86_64
...
...
...
Sep 23 10:13:27 localhost upgrade[509]: 2019-09-23 10:12:56.091374 [ERROR] Actor: kernelcmdlineconfig Message: Failed to append extra arguments to kernel command line.

Could you please try with latest el8 "grubby" (should be 8.40-37) ? How is the machine registered?

I am now commenting on dropping into emergency mode.
Comment 12 Jiri Stransky 2019-10-29 12:45:37 UTC 
A link for completeness -- the underlying symptom has been discussed and investigated at bz1712456, now closed/notabug.
Comment 16 Michal Reznik 2019-11-13 11:26:48 UTC 
Fixed in upstream:

https://github.com/oamg/leapp-repository/commit/0d1668597bdf16316bb10220525d26b5c3a0f0ed
Comment 31 Alois Mahdal 2020-04-22 08:46:23 UTC 
Turns out the PR has been verified as OAMG-681 using a seemingly unrelated method:

 *  have an iso9660 partition (not necessarily mounted) present during upgrade.

This hasn't been automated, ie. we can't re-run it with Brew build easily.  It looks like it could be pretty easy to automate though.  OR at least re-run manually.  Creating task in Jira.
Comment 32 Alois Mahdal 2020-04-27 11:22:23 UTC 
VERIFIED on all platforms with:

      - leapp-0.10.0-2.el7_8
      - leapp-repository-0.10.0-2.el7_8
Comment 34 errata-xmlrpc 2020-04-29 01:45:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1959
Note You need to log in before you can comment on or make changes to this bug.