Bug 1968081

this is a P9 system

Thank you for the report. Unfortunately, we are missing any data to be able to do any investigation.
 - Is it sure the filesystem-3.8-3.el8.ppc64le rpm is provided only from one, official, RHEL repository? IOW, are any other custom repositories used for the upgrade or all repositories are obtained just via subscription-manager from CDN? (custom repo could be used e.g. via the --enablerepo option for the leapp) 
 - what is the output of these commands on rhel-7 system?
   $ ls -ld /usr/local/lib64/bpf
   $ rpm -qf /usr/local/lib64/bpf
   $ rpm -qi $(rpm -qf /usr/local/lib64/bpf)

Based on the output of these commands, we will or won't need additional data for the investigation.

Note: Please keep in mind that RHEL-ALT is officialy EOL.

I understand that rhel-alt is EOL.  This is the reason why I want to upgrade the node to RHEL 8.2 using upgrade in place tool: leapp.

Here is the requested info. It did not appear that that package was on the system.
$ ls -ld /usr/local/lib64/bpf
ls: cannot access /usr/local/lib64/bpf: No such file or directory
$ rpm -qf /usr/local/lib64/bpf
error: file /usr/local/lib64/bpf: No such file or directory
$ rpm -qi $(rpm -qf /usr/local/lib64/bpf)
error: file /usr/local/lib64/bpf: No such file or directory


I don't see any other repositories besides the based repositories.
$ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-extras-rpms                                                                      | 3.8 kB  00:00:00
rhel-7-for-power-9-optional-rpms                                                                    | 3.8 kB  00:00:00
rhel-7-for-power-9-rpms                                                                             | 4.0 kB  00:00:00
rhel-7-for-power-le-rpms                                                                            | 4.1 kB  00:00:00
(1/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/updateinfo                                   |  83 kB  00:00:00
(2/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/group                                        |  124 B  00:00:00
(3/12): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/primary_db                                   | 329 kB  00:00:00
(4/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/updateinfo                                 | 647 kB  00:00:00
(5/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/group                                      |  26 kB  00:00:00
(6/12): rhel-7-for-power-9-optional-rpms/7Server/ppc64le/primary_db                                 | 4.4 MB  00:00:00
(7/12): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo                                          | 906 kB  00:00:00
(8/12): rhel-7-for-power-9-rpms/7Server/ppc64le/group                                               | 744 kB  00:00:00
(9/12): rhel-7-for-power-le-rpms/7Server/ppc64le/group                                              | 599 kB  00:00:00
(10/12): rhel-7-for-power-le-rpms/7Server/ppc64le/updateinfo                                        | 2.9 MB  00:00:00
(11/12): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db                                         |  30 MB  00:00:01
(12/12): rhel-7-for-power-le-rpms/7Server/ppc64le/primary_db                                        |  63 MB  00:00:02
repo id                                                repo name                                                     status
rhel-7-for-power-9-extras-rpms/7Server/ppc64le         Red Hat Enterprise Linux 7 for POWER9 - Extras (RPMs)            650
rhel-7-for-power-9-optional-rpms/7Server/ppc64le       Red Hat Enterprise Linux 7 for POWER9 - Optional (RPMs)        7,598
rhel-7-for-power-9-rpms/7Server/ppc64le                Red Hat Enterprise Linux 7 for POWER9 (RPMs)                   9,587
rhel-7-for-power-le-rpms/7Server/ppc64le               Red Hat Enterprise Linux 7 for IBM Power LE (RPMs)            23,464
repolist: 41,299

This looks pretty weird. From the obtained data and original report, we think that it's highly possible the rpmdb is corrupted:
  a) RPM: warning: Generating 6 missing index(es), please wait...
  b) file /usr/local/lib64/bpf conflicts between attempted installs of filesystem-3.8-3.el8.ppc64le and filesystem-3.8-3.el8.ppc64le
     - the package conflicts with itself but the file is not installed

I suggest to check the integrity of the rpmdb and possibly to rebuild it:
   1. verification of the rpmdb integrity: https://access.redhat.com/articles/3763
      (doesn't have to detect all possible issues; but if detects any, the rpmdb integrity is broken for sure)
   2. rebuild rpmdb: https://access.redhat.com/solutions/6903
      (definitely backup all rpmdb related files before the rpmdb rebuild!)


On our power9 machines we are not able to reproduce the issue. If the problem persists after the steps above, we will need more data.
  - leapp log & data files:
      $ tar -czf leapp-logs.tgz $(find /var/log/leapp | grep -v archive) /var/lib/leapp/leapp.db
  - sosreport - it could be problem as bugzilla doesn't allow to attach files bigger than ..10 MBs? maybe the leapp-logs.tgz will be enough, but not sure

it did not appear that rpmdb is corrupted. I am in the process of rebooting the node.  I will try the upgrade process again and will collect data as requested.  Sorry for the delay.

I rebooted and tried again. Same error messages.  I collected the leapp log and data files per your instruction.  However I have not found how to attach the tar file with the ticket. Any hints for me?  Thanks.

Hi Kim,
I am sorry for very late response. We've been completely overloaded by the work and this ticket slipped from our view. I guess you are probably on RHEL 8 already, either via the in-place upgrade with leapp or clean installation. However, have you been successful in the end? If so, what was the problem? So far, we have not seen this issue anywhere else and the issue is stil mysterious to us, leaving us in a dark alley without more data.

In case you are still on RHEL-ALT 7, attach please the required data. I will have a time to give it a look after the RHEL 9.0 GA release.
Possibility to attach files is above the description of the ticket. Search for  "Add an attachment".

I had reinstalled the node. So I do not have any system to collect the data.  Thanks.

Thanks for the info. As we have no idea what really happened, I am keeping it closed for insufficient data so the bug could be reopened if someone else hit this in future.