RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1935846 - insights-client kills rpm/yum processes leaving stale lock files in the rpm database
Summary: insights-client kills rpm/yum processes leaving stale lock files in the rpm d...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: insights-client
Version: 7.9
Hardware: All
OS: All
urgent
urgent
Target Milestone: rc
: ---
Assignee: Bob Fahr
QA Contact: Pavol Kotvan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-05 16:57 UTC by jcastran
Modified: 2024-12-20 19:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-24 18:36:53 UTC
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1761941 1 high CLOSED insights-client kills rpm/yum processes leaving stale lock files in the rpm database [rhel-7.9.z] 2024-10-01 16:21:51 UTC

Description jcastran 2021-03-05 16:57:24 UTC
Description of problem:
Previously marked resolved in https://bugzilla.redhat.com/show_bug.cgi?id=1761941

We are still seeing insights-client run rpm/yum commands with timeout -s 9 which will kill the process when it takes too long. This can leave stale lock files behind, preventing rpm/yum from functioning until those are removed manually.

Version-Release number of selected component (if applicable):
insights-client-3.1.1-1.el7_9.noarch

How reproducible:
Everytime

Steps to Reproduce:
1. insights-client --register 
2. insights-client --verbose |& grep -E 'timeout.*rpm|timeout.*yum'

Actual results:
2021-03-05 08:46:53,830    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/bin/rpm', '-qa', '--qf', '\\{"name":"%{NAME}","epoch":"%{EPOCH}","version":"%{VERSION}","release":"%{RELEASE}","arch":"%{ARCH}","installtime":"%{INSTALLTIME:date}","buildtime":"%{BUILDTIME}","vendor":"%{VENDOR}","buildhost":"%{BUILDHOST}","sigpgp":"%{SIGPGP:pgpsig}"\\}\n']]
2021-03-05 08:46:55,785    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/usr/bin/yum', '-C', '--noplugins', 'repolist']]
2021-03-05 08:46:59,625    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/bin/rpm', '-qa', '--qf', '\\{"name":"%{NAME}","epoch":"%{EPOCH}","version":"%{VERSION}","release":"%{RELEASE}","arch":"%{ARCH}","installtime":"%{INSTALLTIME:date}","buildtime":"%{BUILDTIME}","vendor":"%{VENDOR}","buildhost":"%{BUILDHOST}","sigpgp":"%{SIGPGP:pgpsig}"\\}\n']]
2021-03-05 08:47:01,544    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/usr/bin/yum', '-C', '--noplugins', 'repolist']]
2021-03-05 08:47:10,643    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/bin/rpm', '-V', 'coreutils', 'procps', 'procps-ng', 'shadow-utils', 'passwd', 'sudo', 'chrony']]
2021-03-05 08:47:12,722    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', 'yum', '-C', '--noplugins', 'list', 'available']]
2021-03-05 08:47:22,281    DEBUG insights.util.subproc Executing: [['timeout', '-s', '9', '120', '/usr/bin/yum', '-C', 'updateinfo', 'list']]

Expected results:
insights-client should not be running SIGKILLS against rpm or yum commands.


/etc/insights-client/insights/client/insights_spec.py

        # use TERM for rpm/yum commands, KILL for everything else
        if (self.command.startswith('/bin/rpm') or
           self.command.startswith('yum') or
           self.command.startswith('/usr/bin/yum')):
            signal = 'TERM'
        else:
            signal = 'KILL'

        timeout_command = 'timeout -s %s %s %s' % (
            signal, self.config.cmd_timeout, self.command)

Additional info:

Comment 3 Kyle Walker 2021-03-05 20:31:54 UTC
Okay,

The fix in the previous bug didn't actually fix the problem. Specifically, the following commit being pulled in:

    $ git show 5122e0d51f8d392bbeebff27f5a8fbab4bed8ba4
    commit 5122e0d51f8d392bbeebff27f5a8fbab4bed8ba4
    Author: Jeremy Crafts <gravitypriest.github.com>
    Date:   Thu Jul 16 10:24:15 2020 -0400
    
        use SIGTERM for rpm and yum commands (#2630)
        
        Signed-off-by: Jeremy Crafts <jcrafts>
    
    diff --git a/insights/client/insights_spec.py b/insights/client/insights_spec.py
    index 1e2de068..d0b9037b 100644
    --- a/insights/client/insights_spec.py
    +++ b/insights/client/insights_spec.py
    @@ -47,8 +47,17 @@ class InsightsCommand(InsightsSpec):
     
             # all commands should timeout after a long interval so the client does not hang
             # prepend native nix 'timeout' implementation
    -        timeout_command = 'timeout -s KILL %s %s' % (
    -            self.config.cmd_timeout, self.command)
    +
    +        # use TERM for rpm/yum commands, KILL for everything else
    +        if (self.command.startswith('/bin/rpm') or
    +           self.command.startswith('yum') or
    +           self.command.startswith('/usr/bin/yum')):
    +            signal = 'TERM'
    +        else:
    +            signal = 'KILL'
    +
    +        timeout_command = 'timeout -s %s %s %s' % (
    +            signal, self.config.cmd_timeout, self.command)
     
             # ensure consistent locale for collected command output
             cmd_env = {'LC_ALL': 'C',

The problem here is that there are a number of problematic commands that are in insights/specs/default.py:

$ grep -e 'simple_command.*yum' -e 'simple_command.*/rpm' insights/specs/default.py 
    rpm_V_packages = simple_command("/bin/rpm -V coreutils procps procps-ng shadow-utils passwd sudo chrony", keep_rc=True)
    yum_list_available = simple_command("yum -C --noplugins list available")
    yum_repolist = simple_command("/usr/bin/yum -C --noplugins repolist")
    yum_updateinfo = simple_command("/usr/bin/yum -C updateinfo list")
    installed_rpms = simple_command("/bin/rpm -qa --qf '%s'" % rpm_format, context=HostContext)



That isn't fixed at all by the InsightsCommand() class modifications (AFAICT). Unless I'm badly mistaken, we need to provide a way for the simple_command() definitions to indicate the specific signal to be delivered... Something akin to:

$ grep -e 'simple_command.*yum' -e 'simple_command.*/rpm' insights/specs/default.py 
    rpm_V_packages = simple_command("/bin/rpm -V coreutils procps procps-ng shadow-utils passwd sudo chrony", keep_rc=True, signum=signal.SIGTERM)
    yum_list_available = simple_command("yum -C --noplugins list available", signum=signal.SIGTERM)
    yum_repolist = simple_command("/usr/bin/yum -C --noplugins repolist", signum=signal.SIGTERM)
    yum_updateinfo = simple_command("/usr/bin/yum -C updateinfo list", signum=signal.SIGTERM)
    installed_rpms = simple_command("/bin/rpm -qa --qf '%s'" % rpm_format, context=HostContext, signum=signal.SIGTERM)


This doesn't look impossible, just a matter of plumbing through:

  simple_command()
  -> CommandOutputProvider()
   -> shell_out()

Comment 4 jcrafts 2021-03-08 14:34:41 UTC
(In reply to Kyle Walker from comment #3)

That's correct. We've since switched the collection code to the insights-core collection lib, and so the previous fix will need to be ported.

Comment 6 Bob Fahr 2021-04-01 13:48:13 UTC
The change to insights-core [1] will go out release today for testing.  The release will be generally available to customers on Monday morning if no problems are identified. The change uses SIGTERM instead of SIGKILL for all yum and rpm collection commands.

[1] https://github.com/RedHatInsights/insights-core/pull/2974

Comment 7 Bob Fahr 2021-04-28 21:24:53 UTC
@jcastran are you still seeing this problem?


Note You need to log in before you can comment on or make changes to this bug.