Bug 1947003 - Fails to reboot after failed rollback with Oracle Linux 7.9
Summary: Fails to reboot after failed rollback with Oracle Linux 7.9
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: convert2rhel
Version: 7.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Eric Gustavsson
QA Contact: Upgrades and Supportability
Miriam Portman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-07 13:34 UTC by Terry Bowling
Modified: 2021-05-03 13:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-03 13:43:01 UTC
Target Upstream Version:


Attachments (Terms of Use)
output of journalctl -b (44.79 KB, text/plain)
2021-04-07 15:56 UTC, Terry Bowling
no flags Details
output of /run/initramfs/rdsosreport.txt (53.77 KB, text/plain)
2021-04-07 15:57 UTC, Terry Bowling
no flags Details
convert2rhel logs (15.25 KB, application/x-xz)
2021-04-26 17:00 UTC, Terry Bowling
no flags Details

Description Terry Bowling 2021-04-07 13:34:02 UTC
Description of problem:

Oracle Linux 7.9 does not reboot after failed rollback.

After installing a default, minimal install of Oracle Linux 7.9, convert2rhel aborts conversion as expected as tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1944815

However, the rollback process fails rollback when trying to reinstall removed packages.  

/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/wireless/mediatek/mt76/mt76x0/mt76x0-common.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/sound/firewire/dice/snd-dice.ko.xz
Uninstall or disable them and run convert2rhel again to continue with the conversion.
WARNING - Abnormal exit! Performing rollback ...

[04/07/2021 08:28:51] TASK - [Rollback: Unregistering the system from RHSM] *********************
System unregistered successfully
Removing RHEL subscription-manager packages.

[04/07/2021 08:28:54] TASK - [Rollback: Removing installed packages] ****************************
No package to remove

[04/07/2021 08:28:54] TASK - [Rollback: Installing removed packages] ****************************
Installing packages:
	/var/lib/convert2rhel/backup/oracle-logos-70.7.0-1.0.7.el7.noarch.rpm
	/var/lib/convert2rhel/backup/rhn-check-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhn-client-tools-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhn-setup-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhnlib-2.5.65-8.0.3.el7.noarch.rpm
	/var/lib/convert2rhel/backup/rhnsd-5.0.13-10.0.1.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/yum-plugin-ulninfo-0.2-13.el7.noarch.rpm
	/var/lib/convert2rhel/backup/yum-rhn-plugin-2.0.1-10.0.1.el7.noarch.rpm
	/var/lib/convert2rhel/backup/oraclelinux-release-el7-1.0-13.1.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/oraclelinux-release-7.9-1.0.9.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/redhat-release-server-7.9-3.0.1.el7.x86_64.rpm
WARNING - Couldn't install /var/lib/convert2rhel/backup/oracle-logos-70.7.0-1.0.7.el7.noarch.rpm /var/lib/convert2rhel/backup/rhn-check-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-client-tools-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-setup-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhnlib-2.5.65-8.0.3.el7.noarch.rpm /var/lib/convert2rhel/backup/rhnsd-5.0.13-10.0.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/yum-plugin-ulninfo-0.2-13.el7.noarch.rpm /var/lib/convert2rhel/backup/yum-rhn-plugin-2.0.1-10.0.1.el7.noarch.rpm /var/lib/convert2rhel/backup/oraclelinux-release-el7-1.0-13.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/oraclelinux-release-7.9-1.0.9.el7.x86_64.rpm /var/lib/convert2rhel/backup/redhat-release-server-7.9-3.0.1.el7.x86_64.rpm packages.

[04/07/2021 08:28:54] TASK - [Rollback: Restore .repo files to /etc/yum.repos.d/] ***************
Restored repo: oracle-linux-ol7.repo
Restored repo: uek-ol7.repo
Restored repo: virt-ol7.repo
Restored repo: convert2rhel.repo

[04/07/2021 08:28:54] TASK - [Rollback: Restoring /etc/system-release from backup] **************
File /etc/system-release restored

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Terry Bowling 2021-04-07 15:56:17 UTC
Created attachment 1769968 [details]
output of journalctl -b

Note that while testing, I did *not* perform a "yum update -y" nor reboot to the RHEL compatible kernel as directed in the documentation.

After reboot, it fails to emergency mode.

Attaching error log files:
  - journal.txt
  - rdsosreport.txt

Comment 3 Terry Bowling 2021-04-07 15:57:59 UTC
Created attachment 1769970 [details]
output of /run/initramfs/rdsosreport.txt

emergency mode logs to /run/initramfs/rdsosreport.txt

Comment 4 Terry Bowling 2021-04-07 19:04:58 UTC
I made copies of my VM disk image.  I restored the original disk image, updated to latest 7.9 errata, rebooted into RHEL compatible kernel and conversion completed successfully without triggering the foreign kmod detection.

So stressing the importance of updating is important here.  Not sure how the tool could accurately detect or enforce as "latest" is always a moving target.

However, it may be worthwhile to understand the root cause to help ensure we do not harm if the user forgets to update first.  I have the failed disk image if you need me to collect more data or make it accessible to you.

Comment 5 Michal Bocek 2021-04-12 18:55:45 UTC
Hi Terry, I need the /var/log/convert2rhel.log which contains debug info. Are you still able to obtain it?
I suspect it might be the same or very similar issue as https://issues.redhat.com/browse/OAMG-4696, fixed by https://github.com/oamg/convert2rhel/pull/238.

Comment 6 Eric Gustavsson 2021-04-23 11:31:05 UTC
Haven't been successful in reproducing this. log file with debug info would be helpful, failed disk image wouldn't hurt either but log file might be enough

Comment 7 Terry Bowling 2021-04-26 17:00:39 UTC
Created attachment 1775651 [details]
convert2rhel logs

I reproduced this issue from a fresh, minimal installation of Oracle Linux 7.9 with no errata updates.

Comment 8 Terry Bowling 2021-04-26 17:03:06 UTC
Here are the relevant console messages that go with the logs in comment 7.

...
snip
...
[04/26/2021 11:41:22] TASK - [Convert: Patch yum configuration file] ****************************
[04/26/2021 11:41:22] DEBUG - /etc/yum.conf patched.

[04/26/2021 11:41:22] TASK - [Convert: Final system checks before main conversion] **************
[04/26/2021 11:41:22] DEBUG - Calling command 'uname -r'
5.4.17-2011.6.2.el7uek.x86_64
[04/26/2021 11:41:22] DEBUG - Calling command 'find /lib/modules/5.4.17-2011.6.2.el7uek.x86_64 -name "*.ko*" -type f'
[04/26/2021 11:41:22] DEBUG - Calling command 'repoquery --releasever=7Server  --repoid rhel-7-server-rpms -f /lib/modules/*.ko*'
[04/26/2021 11:43:50] DEBUG - Calling command 'repoquery --releasever=7Server  --repoid rhel-7-server-rpms -l kernel-0:3.10.0-1160.24.1.el7.x86_64 kernel-debug-0:3.10.0-1160.24.1.el7.x86_64 kmod-ahci-0:327_RH1-1.el7_2.x86_64 kmod-bnx2x-0:1.712.30-1.el7_2.x86_64 kmod-bnxt_en-0:1.2.0-2.el7_2.x86_64 kmod-hpsa-0:3.4.4_1_RH1-1.el7_0.x86_64 kmod-i40e-0:1.5.10_k-2.el7_2.x86_64 kmod-i40evf-0:1.5.10_k-2.el7_2.x86_64 kmod-igb-0:5.2.15_k-1.el7_2.x86_64 kmod-ixgbe-0:4.4.0_k-1.el7_2.x86_64 kmod-kvdo-0:6.1.3.23-5.el7.x86_64 kmod-lpfc-0:11.1.0.2-1.el7_2.x86_64 kmod-megaraid_sas-0:06.811.02.00-1.el7_2.x86_64 kmod-mpt3sas-0:13.100.00.00-1.el7_2.x86_64 kmod-oracleasm-0:2.0.8-28.el7.x86_64 kmod-qed-0:8.7.1.20-1.el7_2.x86_64 kmod-qede-0:8.7.1.20-1.el7_2.x86_64 kmod-qla2xxx-0:8.07.00.33.07.3_k-1.el7_2.x86_64 kmod-redhat-atlantic-0:2.0.2.1_kern_dup7.5-1.el7_5.x86_64 kmod-redhat-bnxt_en-0:1.10.0_dup7.7-1.el7.x86_64 kmod-redhat-ena-0:2.0.2K_dup7.6-2.el7_6.x86_64 kmod-redhat-i40e-0:2.8.10_k_dup7.7-2.el7_7.x86_64 kmod-redhat-i40evf-0:3.0.1_k_dup7.4-2.1.el7_4.x86_64 kmod-redhat-ixgbe-0:4.4.0_k_rh7.4_z-2.el7_3.x86_64 kmod-redhat-ixgbevf-0:3.2.2_k_rh7.4_z-1.el7_3.x86_64 kmod-redhat-lpfc-0:12.0.0.5_dup7.6-1.el7_6.x86_64 kmod-redhat-megaraid_sas-0:07.702.06.00_rh2_dup7.5-1.el7_5.x86_64 kmod-redhat-mgag200-0:4.11.0_dup7.3-5.el7_3.x86_64 kmod-redhat-mpt3sas-0:26.100.01.00_dup7.6-3.el7_6.x86_64 kmod-redhat-nfit-0:3.10.0_957-1.el7_6.x86_64 kmod-redhat-qed-0:8.33.0.20_dup7.6-1.el7_6.x86_64 kmod-redhat-qede-0:8.33.0.20_dup7.6-1.el7_6.x86_64 kmod-redhat-qedf-0:8.33.0.20_dup7.6-1.el7_6.x86_64 kmod-redhat-qedi-0:8.33.0.20_dup7.6-1.el7_6.x86_64 kmod-redhat-qedr-0:8.10.10.0_dup7.4-2.el7_4.x86_64 kmod-redhat-qedr-2:3.10.0_957_dup7.6-3.el7_6.x86_64 kmod-redhat-qla2xxx-0:10.01.00.20.07.8_k_dup7.7-2.el7_7.x86_64 kmod-rtsx_usb-0:0.1_rh1-1.el7_2.x86_64 kmod-rtsx_usb_sdmmc-0:0.1_rh1-1.el7_2.x86_64 kmod-sfc-0:4.0-1.el7_2.x86_64'
[04/26/2021 11:43:55] DEBUG - Calling command 'uname -r'
5.4.17-2011.6.2.el7uek.x86_64
CRITICAL - The following kernel modules are not supported in RHEL:
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/power/supply/lp8727_charger.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/watchdog/pc87413_wdt.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/usb/typec/tcpm/tcpm.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/hid/hid-betopff.ko.xz
...
snip
...
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/eql.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/wireless/mediatek/mt76/mt76x0/mt76x0-common.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/sound/firewire/dice/snd-dice.ko.xz
Uninstall or disable them and run convert2rhel again to continue with the conversion.
[04/26/2021 11:43:55] DEBUG - Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/convert2rhel/main.py", line 88, in main
    pre_ponr_conversion()
  File "/usr/lib/python2.7/site-packages/convert2rhel/main.py", line 208, in pre_ponr_conversion
    checks.perform_pre_ponr_checks()
  File "/usr/lib/python2.7/site-packages/convert2rhel/checks.py", line 83, in perform_pre_ponr_checks
    ensure_compatibility_of_kmods()
  File "/usr/lib/python2.7/site-packages/convert2rhel/checks.py", line 109, in ensure_compatibility_of_kmods
    ).format(kmods=not_supported_kmods)
  File "/usr/lib/python2.7/site-packages/convert2rhel/logger.py", line 103, in _critical
    sys.exit(msg)
SystemExit: The following kernel modules are not supported in RHEL:
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/power/supply/lp8727_charger.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/watchdog/pc87413_wdt.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/usb/typec/tcpm/tcpm.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/hid/hid-betopff.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/staging/uwb/uwb.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/block/drbd/drbd.ko.xz
...
snip
...
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/usb/serial/upd78f0730.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/hwmon/dell-smm-hwmon.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/media/rc/keymaps/rc-hisi-tv-demo.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/usb/serial/mxuport.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/input/mouse/elan_i2c.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/wireless/zydas/zd1201.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/media/dvb-frontends/cxd2841er.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/can/m_can/m_can.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/misc/ad525x_dpot.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/crypto/xxhash_generic.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/eql.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/drivers/net/wireless/mediatek/mt76/mt76x0/mt76x0-common.ko.xz
/lib/modules/5.4.17-2011.6.2.el7uek.x86_64/kernel/sound/firewire/dice/snd-dice.ko.xz
Uninstall or disable them and run convert2rhel again to continue with the conversion.

WARNING - Abnormal exit! Performing rollback ...

[04/26/2021 11:43:55] TASK - [Rollback: Unregistering the system from RHSM] *********************
[04/26/2021 11:43:55] DEBUG - Calling command 'subscription-manager unregister'
System unregistered successfully
Removing RHEL subscription-manager packages.
[04/26/2021 11:44:01] DEBUG - Calling command 'yum remove -y --disablerepo=* --releasever=7Server --enablerepo=rhel-7-server-rpms subscription-manager-rhsm subscription-manager subscription-manager-rhsm-certificates'

[04/26/2021 11:44:01] TASK - [Rollback: Removing installed packages] ****************************
No package to remove

[04/26/2021 11:44:01] TASK - [Rollback: Installing removed packages] ****************************
Installing packages:
	/var/lib/convert2rhel/backup/oracle-logos-70.7.0-1.0.7.el7.noarch.rpm
	/var/lib/convert2rhel/backup/rhn-check-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhn-client-tools-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhn-setup-2.0.2-24.0.7.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/rhnlib-2.5.65-8.0.3.el7.noarch.rpm
	/var/lib/convert2rhel/backup/rhnsd-5.0.13-10.0.1.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/yum-plugin-ulninfo-0.2-13.el7.noarch.rpm
	/var/lib/convert2rhel/backup/yum-rhn-plugin-2.0.1-10.0.1.el7.noarch.rpm
	/var/lib/convert2rhel/backup/oraclelinux-release-el7-1.0-13.1.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/oraclelinux-release-7.9-1.0.9.el7.x86_64.rpm
	/var/lib/convert2rhel/backup/redhat-release-server-7.9-3.0.1.el7.x86_64.rpm
[04/26/2021 11:44:01] DEBUG - Calling command 'rpm -i --replacepkgs /var/lib/convert2rhel/backup/oracle-logos-70.7.0-1.0.7.el7.noarch.rpm /var/lib/convert2rhel/backup/rhn-check-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-client-tools-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-setup-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhnlib-2.5.65-8.0.3.el7.noarch.rpm /var/lib/convert2rhel/backup/rhnsd-5.0.13-10.0.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/yum-plugin-ulninfo-0.2-13.el7.noarch.rpm /var/lib/convert2rhel/backup/yum-rhn-plugin-2.0.1-10.0.1.el7.noarch.rpm /var/lib/convert2rhel/backup/oraclelinux-release-el7-1.0-13.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/oraclelinux-release-7.9-1.0.9.el7.x86_64.rpm /var/lib/convert2rhel/backup/redhat-release-server-7.9-3.0.1.el7.x86_64.rpm'
[04/26/2021 11:44:01] DEBUG - error: Failed dependencies:
	subscription-manager is obsoleted by rhn-client-tools-2.0.2-24.0.7.el7.x86_64
WARNING - Couldn't install /var/lib/convert2rhel/backup/oracle-logos-70.7.0-1.0.7.el7.noarch.rpm /var/lib/convert2rhel/backup/rhn-check-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-client-tools-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhn-setup-2.0.2-24.0.7.el7.x86_64.rpm /var/lib/convert2rhel/backup/rhnlib-2.5.65-8.0.3.el7.noarch.rpm /var/lib/convert2rhel/backup/rhnsd-5.0.13-10.0.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/yum-plugin-ulninfo-0.2-13.el7.noarch.rpm /var/lib/convert2rhel/backup/yum-rhn-plugin-2.0.1-10.0.1.el7.noarch.rpm /var/lib/convert2rhel/backup/oraclelinux-release-el7-1.0-13.1.el7.x86_64.rpm /var/lib/convert2rhel/backup/oraclelinux-release-7.9-1.0.9.el7.x86_64.rpm /var/lib/convert2rhel/backup/redhat-release-server-7.9-3.0.1.el7.x86_64.rpm packages.

[04/26/2021 11:44:01] TASK - [Rollback: Restore .repo files to /etc/yum.repos.d/] ***************
Restored repo: oracle-linux-ol7.repo
Restored repo: uek-ol7.repo
Restored repo: virt-ol7.repo
Restored repo: convert2rhel.repo

[04/26/2021 11:44:01] TASK - [Rollback: Restoring /etc/system-release from backup] **************
File /etc/system-release restored

[04/26/2021 11:44:01] TASK - [Rollback: Restoring /etc/yum.conf from backup] ********************
File /etc/yum.conf restored

[04/26/2021 11:44:01] TASK - [Rollback: Restoring /etc/yum/pluginconf.d/versionlock.list from backup] 
WARNING - /etc/yum/pluginconf.d/versionlock.list hasn't been backed up
[root@o2r79 ~]# ls /var/logs

Comment 9 Michal Bocek 2021-04-28 14:07:43 UTC
I also couldn't reproduce it. But it seems it has something to do with not removing the subscription-manager packages properly.
If you can, Terry, you can try this build from the PR https://github.com/oamg/convert2rhel/pull/238, which should give us more info about why's the subscription-manager not being removed, or possibly even fixing this issue: https://download.copr.fedorainfracloud.org/results/@oamg/convert2rhel/epel-7-x86_64/02107851-convert2rhel/convert2rhel-0.20-1.20210329220954359488.pr238.1.g0c91bf4.el7.noarch.rpm

Comment 10 Michal Bocek 2021-05-03 13:43:01 UTC
Since we can't reproduce it, I'm tilting towards closing this BZ. If we get our hands on another reproducer, we can definitely reopen it.


Note You need to log in before you can comment on or make changes to this bug.