RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1660411 - Many rpm processes are running for very long time if we have a corrupt db
Summary: Many rpm processes are running for very long time if we have a corrupt db
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: rpm
Version: 8.0
Hardware: All
OS: All
unspecified
medium
Target Milestone: rc
: 8.0
Assignee: Packaging Maintenance Team
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-18 09:56 UTC by sangamesh
Modified: 2023-09-14 04:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-12 08:36:13 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description sangamesh 2018-12-18 09:56:06 UTC
I have a scenario where rpm processes are being run for very long time if we have a corrupt database which eventually causing performance issue.

If their is a corruption, and any new rpm process will try to acquire the lock and then calls select with timeout value.

So, before the previous process timeout expires and release the lock, if another rpm process is started it will be waiting the for the lock.
Sequence here is, acquire the lock (after first process timeout expires and released the lock) and then wait for sometime using timeout value as we have corrupted rpm database.
If we run more rpm processes (through cron etc..), original process completion gets delayed and more and more processes keep on running.

This continues until we don't fix the database manually.
But, if in case of automated environment where some cron jobs would be running really can cause a performance issue.

Here we can see that process 13107464 initially was waiting for the lock and then later goes to wait till the timeout expires.

bash-4.4# ps -aef | grep rpm | cut -d' ' -f6 | xargs procstack
13107464: /usr/opt/freeware/bin/rpm -qa
0xd0121694 __fcntl(??, ??, ??) + 0x1d4
0xd0121930 fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dcc00, 0x0) + 0x30
0xd08c6e80 serialize_env(??) + 0xc0
0xd08c60bc db_init(??, ??) + 0x47c
0xd08c4d10 db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38 dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534 pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4 openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8 rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4 loadKeyringFromDB(??) + 0x54
0xd08cfd90 loadKeyring(??) + 0x50
0xd08d27b0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c initFilterIterator(??, ??) + 0x3c
0xd092d6bc rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c rpmcliQuery(??, ??, ??) + 0xcc
0x100008ac main() + 0x4ac
0x10000190 __start() + 0x68
15401230: /usr/opt/freeware/bin/rpm -qa
0xd0121694 __fcntl(??, ??, ??) + 0x1d4
0xd0121930 fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dd800, 0x0) + 0x30
0xd08c6e80 serialize_env(??) + 0xc0
0xd08c60bc db_init(??, ??) + 0x47c
0xd08c4d10 db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38 dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534 pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4 openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8 rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4 loadKeyringFromDB(??) + 0x54
0xd08cfd90 loadKeyring(??) + 0x50
0xd08d27b0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c initFilterIterator(??, ??) + 0x3c
0xd092d6bc rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c rpmcliQuery(??, ??, ??) + 0xcc
0x10000190 __start() + 0x68
22806956: /usr/opt/freeware/bin/rpm -qa
0xd027582c __fd_select(??, ??, ??, ??, ??) + 0xcc
0xd08c5fa8 db_init(??, ??) + 0x368
0xd08c4d10 db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38 dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534 pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4 openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8 rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd092c13c initFilterIterator(??, ??) + 0x3c
0xd092d6bc rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c rpmcliQuery(??, ??, ??) + 0xcc
0x10000190 __start() + 0x68

bash-4.4# ps -aef | grep rpm | cut -d' ' -f6 | xargs procstack
13107464: /usr/opt/freeware/bin/rpm -qa
0xd027582c __fd_select(??, ??, ??, ??, ??) + 0xcc
0xd0f0a40c select() + 0xc
0xd0f0a2b0 __os_sleep() + 0x30
0xd0f87c68 __env_attach() + 0x228
0xd0f849f4 __env_attach_regions() + 0x134
0xd0f86088 __env_open() + 0x2a8
0xd0f86260 __env_open_pp() + 0x120
0xd08c5fa8 db_init(??, ??) + 0x368
0xd08c4d10 db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38 dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534 pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4 openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8 rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4 loadKeyringFromDB(??) + 0x54
0xd08cfd90 loadKeyring(??) + 0x50
0xd08d27b0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c initFilterIterator(??, ??) + 0x3c
0xd092d6bc rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c rpmcliQuery(??, ??, ??) + 0xcc
0x100008ac main() + 0x4ac
0x10000190 __start() + 0x68
15401230: /usr/opt/freeware/bin/rpm -qa
0xd0121694 __fcntl(??, ??, ??) + 0x1d4
0xd0121930 fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dd800, 0x0) + 0x30
0xd08c6e80 serialize_env(??) + 0xc0
0xd08c60bc db_init(??, ??) + 0x47c
0xd08c4d10 db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38 dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534 pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4 openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8 rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4 loadKeyringFromDB(??) + 0x54
0xd08cfd90 loadKeyring(??) + 0x50
0xd08d27b0 rpmtsInitIterator@AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c initFilterIterator(??, ??) + 0x3c
0xd092d6bc rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c rpmcliQuery(??, ??, ??) + 0xcc
0x10000190 __start() + 0x68
bash-4.4#

Please let me know if this is a know issue or bug in rpm or db package.

I could see that same issue can happen in multiple distros like linux, AIX etc..

Comment 1 Panu Matilainen 2019-02-11 10:15:35 UTC
> 15401230: /usr/opt/freeware/bin/rpm -qa

What exactly is this, and on what system? The rpm provided by RHEL certainly does not sit in such a path.

Comment 2 Daniel Mach 2019-03-12 08:36:13 UTC
Closing INSUFFICIENT_DATA.
Please reopen the bug if you have more details that would help us to understand nature of the issue and make code changes accordingly.

Comment 4 Red Hat Bugzilla 2023-09-14 04:43:58 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.