Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionChristophe Besson
2020-12-08 14:19:11 UTC
Description of problem:
After an in-place upgrade from RHEL 6 to the latest RHEL 7.9, a leapp upgrade fails when persistent net udev rules are present (no net.ifnames or biosdevname in cmdline). The first step works as expected but it comes to the emergency mode during the reboot step after a python traceback (in biosdevname.py, unable to import pyudev). It happened very lately in the upgrade process, after the package upgrade and the grub install (likely done by the leapp_resume service).
Version-Release number of selected component (if applicable):
leapp-repository-0.12.0-2.el7_9.noarch
How reproducible:
100%
Steps to Reproduce:
1. Create a RHEL 7.9 VM with 2 interfaces
2. Setup persistent udev rules as it was usually done on RHEL 6
~~~
# cat << EOF >/etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:41:b3:32", NAME="net0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:9b:75:d5", NAME="net1"
EOF
~~~
3. Adapt ifcfg config files accordingly (comment out UUID, rename NAME/DEVICE vars, rename the ifcfg file)
Actual results:
Dec 08 12:29:10 localhost upgrade[19164]: 2020-12-08 13:29:10.369 DEBUG PID: 3 leapp.repository.system_upgrade_el7toel8: Starting actor discovery in actors/biosdevname
Dec 08 12:29:10 localhost upgrade[19164]: Process Process-5:
Dec 08 12:29:10 localhost upgrade[19164]: Traceback (most recent call last):
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
Dec 08 12:29:10 localhost upgrade[19164]: self.run()
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
Dec 08 12:29:10 localhost upgrade[19164]: self._target(*self._args, **self._kwargs)
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 30, in inspect_actor
Dec 08 12:29:10 localhost upgrade[19164]: definition.load()
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 179, in load
Dec 08 12:29:10 localhost upgrade[19164]: self._module = importer.find_module(name).load_module(name)
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/pkgutil.py", line 243, in load_module
Dec 08 12:29:10 localhost upgrade[19164]: mod = imp.load_module(fullname, self.file, self.filename, self.etc)
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/biosdevname/actor.py", line 2, in <module>
Dec 08 12:29:10 localhost upgrade[19164]: from leapp.libraries.actor.biosdevname import check_biosdevname
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/pkgutil.py", line 243, in load_module
Dec 08 12:29:10 localhost upgrade[19164]: mod = imp.load_module(fullname, self.file, self.filename, self.etc)
Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/biosdevname/libraries/biosdevname.py", line 3, in <module>
Dec 08 12:29:10 localhost upgrade[19164]: import pyudev
Dec 08 12:29:10 localhost upgrade[19164]: ImportError: No module named pyudev
Dec 08 12:29:10 localhost kernel: XFS (vda1): Unmounting Filesystem
Dec 08 12:29:10 localhost upgrade[19164]: 2020-12-08 13:29:10.388 ERROR PID: 3 leapp.repository.system_upgrade_el7toel8: Process inspecting actor in actors/biosdevname failed with 1
Dec 08 12:29:10 localhost upgrade[19164]: Error: Inspection of actor in actors/biosdevname failed
Dec 08 12:29:10 localhost upgrade[19164]: Container sysroot failed with error code 2.
Dec 08 12:29:10 localhost upgrade[19164]: writing logs to disk and rebooting
Additional info:
* The package `python3-pyudev` replaced `python-pyudev` whereas the latter is still useful for leapp which works with python2 libs.
* Tested the following workaround: keep the python-pyudev package and apply the patch below. The original issue is resolved but another issue is raised.
# echo python-pyudev >> /etc/leapp/transaction/to_keep
--- /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/preparepythonworkround/libraries/workaround.py.orig 2020-12-08 14:43:10.424139601 +0100
+++ /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/preparepythonworkround/libraries/workaround.py 2020-12-08 14:44:18.609023039 +0100
@@ -9,8 +9,11 @@
py3_leapp = os.path.join(LEAPP_HOME, 'leapp3')
makedirs(LEAPP_HOME)
leapp_lib_symlink_path = os.path.join(LEAPP_HOME, 'leapp')
+ leapp_pyudev_symlink_path = os.path.join(LEAPP_HOME, 'pyudev')
if not os.path.exists(leapp_lib_symlink_path):
os.symlink('/usr/lib/python2.7/site-packages/leapp', leapp_lib_symlink_path)
+ if not os.path.exists(leapp_pyudev_symlink_path):
+ os.symlink('/usr/lib/python2.7/site-packages/pyudev', leapp_pyudev_symlink_path)
with open(py3_leapp, 'w') as f:
f_content = [
'#!/usr/bin/python3',
But got:
~~~
Dec 08 14:07:41 localhost upgrade[618]: 2020-12-08 15:07:41.427 INFO PID: 3 leapp.workflow.Applications: Executing actor persistentnetnamesconfig
Dec 08 14:07:41 localhost upgrade[618]: > /etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py(27)process()
Dec 08 14:07:41 localhost upgrade[618]: -> rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces
Dec 08 14:07:41 localhost upgrade[618]: (Pdb)
Dec 08 14:07:41 localhost upgrade[618]: Process Process-166:
Dec 08 14:07:41 localhost upgrade[618]: Traceback (most recent call last):
Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Dec 08 14:07:41 localhost upgrade[618]: self.run()
Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
Dec 08 14:07:41 localhost upgrade[618]: self._target(*self._args, **self._kwargs)
Dec 08 14:07:41 localhost upgrade[618]: File "/root/tmp_leapp_py3/leapp/repository/actor_definition.py", line 72, in _do_run
Dec 08 14:07:41 localhost upgrade[618]: actor_instance.run(*args, **kwargs)
Dec 08 14:07:41 localhost upgrade[618]: File "/root/tmp_leapp_py3/leapp/actors/__init__.py", line 335, in run
Dec 08 14:07:41 localhost upgrade[618]: self.process(*args)
Dec 08 14:07:41 localhost upgrade[618]: File "/etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py", line 27, in process
Dec 08 14:07:41 localhost upgrade[618]: rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces
Dec 08 14:07:41 localhost upgrade[618]: File "/etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py", line 27, in process
Dec 08 14:07:41 localhost upgrade[618]: rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces
Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/bdb.py", line 51, in trace_dispatch
Dec 08 14:07:41 localhost upgrade[618]: return self.dispatch_line(frame)
Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/bdb.py", line 70, in dispatch_line
Dec 08 14:07:41 localhost upgrade[618]: if self.quitting: raise BdbQuit
Dec 08 14:07:41 localhost upgrade[618]: bdb.BdbQuit
Dec 08 14:07:41 localhost upgrade[618]: =========================================================================================================
Dec 08 14:07:41 localhost upgrade[618]: Actor persistentnetnamesconfig unexpectedly terminated with exit code: 1 - Please check the above details
Dec 08 14:07:41 localhost upgrade[618]: =========================================================================================================
~~~
Comment 2Christophe Besson
2020-12-08 14:23:27 UTC
Hmm, the error I got after my workaround was due to the pdb.set_trace() I added in the persistentnetnamesconfig actor...
Comment 3Christophe Besson
2020-12-08 15:13:14 UTC
Just to confirm the workaround works perfectly well.
Will write a KCS and suggest it to the customer.
Updating priority to Urgent. The location for the workaround has also changed. We can resolve this by linking the pyudev python module.
/usr/share/leapp-repository/repositories/system_upgrade/common/actors/preparepythonworkround/libraries/
--- a/libraries/workaround.py 2022-09-22 10:47:15.150354412 -0400
+++ b/libraries/workaround.py 2022-09-22 10:50:00.470673091 -0400
@@ -20,8 +20,11 @@
py3_leapp = os.path.join(LEAPP_HOME, 'leapp3')
makedirs(LEAPP_HOME)
leapp_lib_symlink_path = os.path.join(LEAPP_HOME, 'leapp')
+ leapp_pyudev_symlink_path = os.path.join(LEAPP_HOME, 'pyudev')
if not os.path.exists(leapp_lib_symlink_path):
os.symlink(_get_orig_leapp_path(), leapp_lib_symlink_path)
+ if not os.path.exists(leapp_pyudev_symlink_path):
+ os.symlink('/usr/lib/python2.7/site-packages/pyudev', leapp_pyudev_symlink_path)
with open(py3_leapp, 'w') as f:
f_content = [
'#!/usr/bin/python3',
Comment 15arunmozhi_ramasamy
2023-01-17 22:42:39 UTC
I got similar issue during leapp upgrade
# rpm -q leapp --last
leapp-0.15.0-2.el7_9.noarch Tue 17 Jan 2023 09:22:59 AM MST
No errors during the upgrade. However, after reboot server is not coming up and entering into emergency mode.
:/# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● upgrade.service loaded failed failed System Upgrade
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
:/# journalctl -u upgrade.service | grep -A 12 Traceback
ImportError: No module named pyudev
upgrade[19164]: Error: Inspection of actor in actors/persistentnetnames failed
Comment 16arunmozhi_ramasamy
2023-01-17 22:44:46 UTC
We have fixed a symptom in upstream that causes the final confusing error, so the original problem should be now better visible:
* https://github.com/oamg/leapp-repository/pull/996
The issues can be hit if the previous run of leapp ended with an error and the upgrade.target in the upgrade initramfs is re-run - e.g. when returning from an emergency mode. So the bug with the missing pyudev module should not be visible anymore. This fix however does not fix the original issue(s) that caused the previous leapp failure. I am considering it as partial fix right now, so the real issues can be revealed.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (leapp and leapp-repository bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:3187
Description of problem: After an in-place upgrade from RHEL 6 to the latest RHEL 7.9, a leapp upgrade fails when persistent net udev rules are present (no net.ifnames or biosdevname in cmdline). The first step works as expected but it comes to the emergency mode during the reboot step after a python traceback (in biosdevname.py, unable to import pyudev). It happened very lately in the upgrade process, after the package upgrade and the grub install (likely done by the leapp_resume service). Version-Release number of selected component (if applicable): leapp-repository-0.12.0-2.el7_9.noarch How reproducible: 100% Steps to Reproduce: 1. Create a RHEL 7.9 VM with 2 interfaces 2. Setup persistent udev rules as it was usually done on RHEL 6 ~~~ # cat << EOF >/etc/udev/rules.d/70-persistent-net.rules SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:41:b3:32", NAME="net0" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:9b:75:d5", NAME="net1" EOF ~~~ 3. Adapt ifcfg config files accordingly (comment out UUID, rename NAME/DEVICE vars, rename the ifcfg file) Actual results: Dec 08 12:29:10 localhost upgrade[19164]: 2020-12-08 13:29:10.369 DEBUG PID: 3 leapp.repository.system_upgrade_el7toel8: Starting actor discovery in actors/biosdevname Dec 08 12:29:10 localhost upgrade[19164]: Process Process-5: Dec 08 12:29:10 localhost upgrade[19164]: Traceback (most recent call last): Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap Dec 08 12:29:10 localhost upgrade[19164]: self.run() Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run Dec 08 12:29:10 localhost upgrade[19164]: self._target(*self._args, **self._kwargs) Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 30, in inspect_actor Dec 08 12:29:10 localhost upgrade[19164]: definition.load() Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 179, in load Dec 08 12:29:10 localhost upgrade[19164]: self._module = importer.find_module(name).load_module(name) Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/pkgutil.py", line 243, in load_module Dec 08 12:29:10 localhost upgrade[19164]: mod = imp.load_module(fullname, self.file, self.filename, self.etc) Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/biosdevname/actor.py", line 2, in <module> Dec 08 12:29:10 localhost upgrade[19164]: from leapp.libraries.actor.biosdevname import check_biosdevname Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/lib64/python2.7/pkgutil.py", line 243, in load_module Dec 08 12:29:10 localhost upgrade[19164]: mod = imp.load_module(fullname, self.file, self.filename, self.etc) Dec 08 12:29:10 localhost upgrade[19164]: File "/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/biosdevname/libraries/biosdevname.py", line 3, in <module> Dec 08 12:29:10 localhost upgrade[19164]: import pyudev Dec 08 12:29:10 localhost upgrade[19164]: ImportError: No module named pyudev Dec 08 12:29:10 localhost kernel: XFS (vda1): Unmounting Filesystem Dec 08 12:29:10 localhost upgrade[19164]: 2020-12-08 13:29:10.388 ERROR PID: 3 leapp.repository.system_upgrade_el7toel8: Process inspecting actor in actors/biosdevname failed with 1 Dec 08 12:29:10 localhost upgrade[19164]: Error: Inspection of actor in actors/biosdevname failed Dec 08 12:29:10 localhost upgrade[19164]: Container sysroot failed with error code 2. Dec 08 12:29:10 localhost upgrade[19164]: writing logs to disk and rebooting Additional info: * The package `python3-pyudev` replaced `python-pyudev` whereas the latter is still useful for leapp which works with python2 libs. * Tested the following workaround: keep the python-pyudev package and apply the patch below. The original issue is resolved but another issue is raised. # echo python-pyudev >> /etc/leapp/transaction/to_keep --- /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/preparepythonworkround/libraries/workaround.py.orig 2020-12-08 14:43:10.424139601 +0100 +++ /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/preparepythonworkround/libraries/workaround.py 2020-12-08 14:44:18.609023039 +0100 @@ -9,8 +9,11 @@ py3_leapp = os.path.join(LEAPP_HOME, 'leapp3') makedirs(LEAPP_HOME) leapp_lib_symlink_path = os.path.join(LEAPP_HOME, 'leapp') + leapp_pyudev_symlink_path = os.path.join(LEAPP_HOME, 'pyudev') if not os.path.exists(leapp_lib_symlink_path): os.symlink('/usr/lib/python2.7/site-packages/leapp', leapp_lib_symlink_path) + if not os.path.exists(leapp_pyudev_symlink_path): + os.symlink('/usr/lib/python2.7/site-packages/pyudev', leapp_pyudev_symlink_path) with open(py3_leapp, 'w') as f: f_content = [ '#!/usr/bin/python3', But got: ~~~ Dec 08 14:07:41 localhost upgrade[618]: 2020-12-08 15:07:41.427 INFO PID: 3 leapp.workflow.Applications: Executing actor persistentnetnamesconfig Dec 08 14:07:41 localhost upgrade[618]: > /etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py(27)process() Dec 08 14:07:41 localhost upgrade[618]: -> rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces Dec 08 14:07:41 localhost upgrade[618]: (Pdb) Dec 08 14:07:41 localhost upgrade[618]: Process Process-166: Dec 08 14:07:41 localhost upgrade[618]: Traceback (most recent call last): Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap Dec 08 14:07:41 localhost upgrade[618]: self.run() Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run Dec 08 14:07:41 localhost upgrade[618]: self._target(*self._args, **self._kwargs) Dec 08 14:07:41 localhost upgrade[618]: File "/root/tmp_leapp_py3/leapp/repository/actor_definition.py", line 72, in _do_run Dec 08 14:07:41 localhost upgrade[618]: actor_instance.run(*args, **kwargs) Dec 08 14:07:41 localhost upgrade[618]: File "/root/tmp_leapp_py3/leapp/actors/__init__.py", line 335, in run Dec 08 14:07:41 localhost upgrade[618]: self.process(*args) Dec 08 14:07:41 localhost upgrade[618]: File "/etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py", line 27, in process Dec 08 14:07:41 localhost upgrade[618]: rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces Dec 08 14:07:41 localhost upgrade[618]: File "/etc/leapp/repos.d/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py", line 27, in process Dec 08 14:07:41 localhost upgrade[618]: rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/bdb.py", line 51, in trace_dispatch Dec 08 14:07:41 localhost upgrade[618]: return self.dispatch_line(frame) Dec 08 14:07:41 localhost upgrade[618]: File "/usr/lib64/python3.6/bdb.py", line 70, in dispatch_line Dec 08 14:07:41 localhost upgrade[618]: if self.quitting: raise BdbQuit Dec 08 14:07:41 localhost upgrade[618]: bdb.BdbQuit Dec 08 14:07:41 localhost upgrade[618]: ========================================================================================================= Dec 08 14:07:41 localhost upgrade[618]: Actor persistentnetnamesconfig unexpectedly terminated with exit code: 1 - Please check the above details Dec 08 14:07:41 localhost upgrade[618]: ========================================================================================================= ~~~