RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1968081 - leapp.workflow.Download.dnf_package_download: DNF execution failed:
Summary: leapp.workflow.Download.dnf_package_download: DNF execution failed:
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: leapp-repository
Version: 7.6-Alt
Hardware: ppc64le
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Leapp team
QA Contact: Upgrades and Supportability
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-04 21:49 UTC by Kim Tran
Modified: 2022-06-20 08:02 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-17 12:30:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Kim Tran 2021-06-04 21:49:29 UTC 
Description of problem:
I am trying to do an inplace upgrade of my system at RHEL 7.6 ALT to RHEL 8.x by using leapp upgrade and got the error message: 
2021-06-04 12:04:44.736 ERROR    PID: 106393 leapp.workflow.Download.dnf_package_download: DNF execution failed:

============================================================
                           ERRORS
============================================================

2021-06-04 12:05:17.365905 [ERROR] Actor: dnf_package_download
Message: DNF execution failed with non zero exit code.
...
STDERR:
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/local/lib64/bpf conflicts between attempted installs of filesystem-3.8-3.el8.ppc64le and filesystem-3.8-3.el8.ppc64le



============================================================
                       END OF ERRORS
============================================================


Debug output written to /var/log/leapp/leapp-upgrade.log

Version-Release number of selected component (if applicable):


How reproducible:
leapp upgrade

Steps to Reproduce:
1.AC922 
2.]$ uname -r
4.14.0-115.26.1.el7a.ppc64le

3.leapp-data14.tar.gz
$ rpm -qa | grep leapp
leapp-repository-deps-0.14.0-4.el7_9.noarch
python2-leapp-0.12.1-1.el7_9.noarch
leapp-deps-0.12.1-1.el7_9.noarch
leapp-0.12.1-1.el7_9.noarch
leapp-repository-0.14.0-4.el7_9.noarch


Actual results:
STDERR:
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/local/lib64/bpf conflicts between attempted installs of filesystem-3.8-3.el8.ppc64le and filesystem-3.8-3.el8.ppc64le



============================================================
                       END OF ERRORS
============================================================


Debug output written to /var/log/leapp/leapp-upgrade.log

============================================================
                           REPORT
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT
============================================================

Answerfile has been generated at /var/log/leapp/answerfile

Expected results:

No error and reboot to upgrade the node to RHEL 8.x
Additional info:
Here is a snippet of the output of the upgrade
(943/944): kernel-tools-4.18.0-305.3.1.el8_4.pp 4.1 MB/s | 6.0 MB     00:01
(944/944): linux-firmware-20201218-102.git05789 3.1 MB/s | 123 MB     00:39
--------------------------------------------------------------------------------
Total                                           2.5 MB/s | 686 MB     04:33
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
2021-06-04 12:04:44.736 ERROR    PID: 106393 leapp.workflow.Download.dnf_package_download: DNF execution failed:

============================================================
                           ERRORS
============================================================

2021-06-04 12:05:17.365905 [ERROR] Actor: dnf_package_download
Message: DNF execution failed with non zero exit code.
STDOUT:
Last metadata expiration check: 0:08:48 ago on Fri Jun  4 11:51:03 2021.
Dependencies resolved.
==============================================================================================================================

(941/944): kernel-tools-libs-4.18.0-305.3.1.el8 4.7 MB/s | 5.9 MB     00:01
(942/944): polkit-0.115-11.el8_4.1.ppc64le.rpm  188 kB/s | 157 kB     00:00
(943/944): kernel-tools-4.18.0-305.3.1.el8_4.pp 4.1 MB/s | 6.0 MB     00:01
(944/944): linux-firmware-20201218-102.git05789 3.1 MB/s | 123 MB     00:39
--------------------------------------------------------------------------------
Total                                           2.5 MB/s | 686 MB     04:33
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

STDERR:
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
Warning: Packages marked by Leapp for upgrade not found in repositories metadata: gpg-pubkey
RPM: warning: Generating 6 missing index(es), please wait...
Error: Transaction test error:
  file /usr/local/lib64/bpf conflicts between attempted installs of filesystem-3.8-3.el8.ppc64le and filesystem-3.8-3.el8.ppc64le



============================================================
                       END OF ERRORS
============================================================


Debug output written to /var/log/leapp/leapp-upgrade.log

============================================================
                           REPORT
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT
============================================================

Answerfile has been generated at /var/log/leapp/answerfile
Comment 2 Kim Tran 2021-06-04 21:53:25 UTC 
this is a P9 system
Comment 3 Petr Stodulka 2021-06-08 13:52:28 UTC 
Thank you for the report. Unfortunately, we are missing any data to be able to do any investigation.
 - Is it sure the filesystem-3.8-3.el8.ppc64le rpm is provided only from one, official, RHEL repository? IOW, are any other custom repositories used for the upgrade or all repositories are obtained just via subscription-manager from CDN? (custom repo could be used e.g. via the --enablerepo option for the leapp) 
 - what is the output of these commands on rhel-7 system?
   $ ls -ld /usr/local/lib64/bpf
   $ rpm -qf /usr/local/lib64/bpf
   $ rpm -qi $(rpm -qf /usr/local/lib64/bpf)

Based on the output of these commands, we will or won't need additional data for the investigation.

Note: Please keep in mind that RHEL-ALT is officialy EOL.
Comment 4 Kim Tran 2021-06-08 15:39:01 UTC 
I understand that rhel-alt is EOL.  This is the reason why I want to upgrade the node to RHEL 8.2 using upgrade in place tool: leapp.

Here is the requested info. It did not appear that that package was on the system.
$ ls -ld /usr/local/lib64/bpf
ls: cannot access /usr/local/lib64/bpf: No such file or directory
$ rpm -qf /usr/local/lib64/bpf
error: file /usr/local/lib64/bpf: No such file or directory
$ rpm -qi $(rpm -qf /usr/local/lib64/bpf)
error: file /usr/local/lib64/bpf: No such file or directory


I don't see any other repositories besides the based repositories.
$ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-extras-rpms                                                                      | 3.8 kB  00:00:00
rhel-7-for-power-9-optional-rpms                                                                    | 3.8 kB  00:00:00
rhel-7-for-power-9-rpms                                                                             | 4.0 kB  00:00:00
rhel-7-for-power-le-rpms                                                                            | 4.1 kB  00:00:00
(1/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/updateinfo                                   |  83 kB  00:00:00
(2/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/group                                        |  124 B  00:00:00
(3/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/primary_db                                   | 329 kB  00:00:00
(4/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/updateinfo                                 | 647 kB  00:00:00
(5/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/group                                      |  26 kB  00:00:00
(6/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/primary_db                                 | 4.4 MB  00:00:00
(7/12): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo                                          | 906 kB  00:00:00
(8/12): rhel-7-for-power-9-rpms/7Server/ppc64le/group                                               | 744 kB  00:00:00
(9/12): rhel-7-for-power-le-rpms/7Server/ppc64le/group                                              | 599 kB  00:00:00
(10/12): rhel-7-for-power-le-rpms/7Server/ppc64le/updateinfo                                        | 2.9 MB  00:00:00
(11/12): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db                                         |  30 MB  00:00:01
(12/12): rhel-7-for-power-le-rpms/7Server/ppc64le/primary_db                                        |  63 MB  00:00:02
repo id                                                repo name                                                     status
rhel-7-for-power-9-extras-rpms/7Server/ppc64le         Red Hat Enterprise Linux 7 for POWER9 - Extras (RPMs)            650
rhel-7-for-power-9-optional-rpms/7Server/ppc64le       Red Hat Enterprise Linux 7 for POWER9 - Optional (RPMs)        7,598
rhel-7-for-power-9-rpms/7Server/ppc64le                Red Hat Enterprise Linux 7 for POWER9 (RPMs)                   9,587
rhel-7-for-power-le-rpms/7Server/ppc64le               Red Hat Enterprise Linux 7 for IBM Power LE (RPMs)            23,464
repolist: 41,299
Comment 5 Petr Stodulka 2021-06-08 17:43:32 UTC 
This looks pretty weird. From the obtained data and original report, we think that it's highly possible the rpmdb is corrupted:
  a) RPM: warning: Generating 6 missing index(es), please wait...
  b) file /usr/local/lib64/bpf conflicts between attempted installs of filesystem-3.8-3.el8.ppc64le and filesystem-3.8-3.el8.ppc64le
     - the package conflicts with itself but the file is not installed

I suggest to check the integrity of the rpmdb and possibly to rebuild it:
   1. verification of the rpmdb integrity: https://access.redhat.com/articles/3763
      (doesn't have to detect all possible issues; but if detects any, the rpmdb integrity is broken for sure)
   2. rebuild rpmdb: https://access.redhat.com/solutions/6903
      (definitely backup all rpmdb related files before the rpmdb rebuild!)


On our power9 machines we are not able to reproduce the issue. If the problem persists after the steps above, we will need more data.
  - leapp log & data files:
      $ tar -czf leapp-logs.tgz $(find /var/log/leapp | grep -v archive) /var/lib/leapp/leapp.db
  - sosreport - it could be problem as bugzilla doesn't allow to attach files bigger than ..10 MBs? maybe the leapp-logs.tgz will be enough, but not sure
Comment 6 Kim Tran 2021-06-09 18:35:21 UTC 
it did not appear that rpmdb is corrupted. I am in the process of rebooting the node.  I will try the upgrade process again and will collect data as requested.  Sorry for the delay.
Comment 7 Kim Tran 2021-06-11 15:42:21 UTC 
I rebooted and tried again. Same error messages.  I collected the leapp log and data files per your instruction.  However I have not found how to attach the tar file with the ticket. Any hints for me?  Thanks.
Comment 9 Petr Stodulka 2022-05-04 09:11:17 UTC 
Hi Kim,
I am sorry for very late response. We've been completely overloaded by the work and this ticket slipped from our view. I guess you are probably on RHEL 8 already, either via the in-place upgrade with leapp or clean installation. However, have you been successful in the end? If so, what was the problem? So far, we have not seen this issue anywhere else and the issue is stil mysterious to us, leaving us in a dark alley without more data.

In case you are still on RHEL-ALT 7, attach please the required data. I will have a time to give it a look after the RHEL 9.0 GA release.
Possibility to attach files is above the description of the ticket. Search for  "Add an attachment".
Comment 10 Kim Tran 2022-06-17 12:30:18 UTC 
I had reinstalled the node. So I do not have any system to collect the data.  Thanks.
Comment 11 Petr Stodulka 2022-06-20 08:02:05 UTC 
Thanks for the info. As we have no idea what really happened, I am keeping it closed for insufficient data so the bug could be reopened if someone else hit this in future.
Note You need to log in before you can comment on or make changes to this bug.