Bug 1803794

Upstream commit: a71d800985e10de71717b0aaa032ac7a8cf4caac

Is this really not going to be fixed until 7.9 or is an async errata coming for 7.8 hosts? I updated to RHEL 7.8 and am seeing this error now.  Thank you.

Affects all our newly upgraded RHEL7.8 systems

*** Bug 1820635 has been marked as a duplicate of this bug. ***

*** Bug 1819613 has been marked as a duplicate of this bug. ***

*** Bug 1821325 has been marked as a duplicate of this bug. ***

Hi,

I confirm what Vadym Chepkov says in comment #6. We see this on all RHEL7.8 systems that do not have a "processTimeout" setting in /etc/rhsm/rhsm.conf. The systems have subscription-manager-1.24.26-1.el7.x86_64 RPM installed.

Sorry, I don't know how to see the Upstream commit: a71d800985e10de71717b0aaa032ac7a8cf4caac mentioned in comment #2 and the fix. In which got repository is it?

I think the code snippet from /etc/cron.daily/rhsmd:

if [ -n $config ]; then
  rhsmd_timeout=$config
else
  rhsmd_timeout=300
fi

is missing double quotes. It should be:

if [ -n "$config" ]; then
  rhsmd_timeout=$config
else
  rhsmd_timeout=300
fi

In order for it to work correctly in the case where "processTimeout" is not defined.

I would say that an update to the subscription-manager RPM is needed.

(In reply to Preslav from comment #12)
> Hi,
> 
> I confirm what Vadym Chepkov says in comment #6. We see this on all RHEL7.8
> systems that do not have a "processTimeout" setting in /etc/rhsm/rhsm.conf.
> The systems have subscription-manager-1.24.26-1.el7.x86_64 RPM installed.
> 
> Sorry, I don't know how to see the Upstream commit:
> a71d800985e10de71717b0aaa032ac7a8cf4caac mentioned in comment #2 and the
> fix. In which got repository is it?
> 
> I think the code snippet from /etc/cron.daily/rhsmd:
> 
> if [ -n $config ]; then
>   rhsmd_timeout=$config
> else
>   rhsmd_timeout=300
> fi
> 
> is missing double quotes. It should be:
> 
> if [ -n "$config" ]; then
>   rhsmd_timeout=$config
> else
>   rhsmd_timeout=300
> fi
> 
> In order for it to work correctly in the case where "processTimeout" is not
> defined.
> 
> I would say that an update to the subscription-manager RPM is needed.

Hi Preslav, 

Thanks for reaching out to us, link to referred commit is here https://github.com/candlepin/subscription-manager/pull/2218/commits/a71d800985e10de71717b0aaa032ac7a8cf4caac . We are looking in to options for making this fix available in 7.8 . 

Sorry for the inconvenience and thanks for all your valuable feed back.

thanks,
Rehana

Ahhh!  I totally agree with Sayan's concern in comment 18 and comment 22.
There is a second coding fix that must be made to /etc/cron.daily/rhsmd.

Please add an "--ignore-case" option to the first line of code in /etc/cron.daily/rhsmd
Change this line of code...

> config=$(grep -E "^processTimeout" /etc/rhsm/rhsm.conf | grep -Po "[0-9]+")

to this...

> config=$(grep -iE "^processTimeout" /etc/rhsm/rhsm.conf | grep -Po "[0-9]+")

Moving back to NEW/FailedQA

(In reply to John Sefler from comment #23)
> Ahhh!  I totally agree with Sayan's concern in comment 18 and comment 22.
> There is a second coding fix that must be made to /etc/cron.daily/rhsmd.
> 
> Please add an "--ignore-case" option to the first line of code in
> /etc/cron.daily/rhsmd
> Change this line of code...
> 
> > config=$(grep -E "^processTimeout" /etc/rhsm/rhsm.conf | grep -Po "[0-9]+")
> 
> to this...
> 
> > config=$(grep -iE "^processTimeout" /etc/rhsm/rhsm.conf | grep -Po "[0-9]+")
> 
> Moving back to NEW/FailedQA


Thanks, John. 

I should have pointed out in my first comment only but I am glad that we are on the same page now.

Let's handle the fix for comment 18 thru comment 24 in NEW Bug 1827708 and clones thereof.

Verifying Version...

[root@intel-s3e8132-01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 Beta (Maipo)
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# rpm -q --whatprovides /etc/cron.daily/rhsmd
subscription-manager-1.24.37-1.el7.x86_64
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# rpm --verify subscription-manager
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# rpm -q subscription-manager --changelog | egrep "1803794|1794653|1823523"
- 1823523: Detect rhsm-icon running without psutil (csnyder)
- 1794653: corrected missing quotes for config check; ENT-2010
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# head /etc/cron.daily/rhsmd
#!/bin/sh
# nightly run of rhsmd to log entitlement expiration/validity errors to syslog
# this is a cron job because it doesn't need to 'phone home'. should that
# change, look into calling the dbus interface from rhsmcertd instead.
config=$(grep -iE "^processTimeout" /etc/rhsm/rhsm.conf | grep -Po "[0-9]+")
if [ -n "$config" ]; then               <================ VERIFIED: $config is now double quoted
  rhsmd_timeout=$config
else
  rhsmd_timeout=300
fi
[root@intel-s3e8132-01 ~]# 

VERIFIED: the logic error in the original if clause will now evaluate as intended with the double quoted "$config" when $config is an empty string.

Note: an empty string is possible in the following malicious config setting...

[root@intel-s3e8132-01 ~]# subscription-manager config --rhsmd.processtimeout=""
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# grep -i processtimeout /etc/rhsm/rhsm.conf 
processTimeout = 
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# time /etc/cron.daily/rhsmd

real	5m0.036s
user	0m0.329s
sys	0m0.139s
[root@intel-s3e8132-01 ~]# 

VERIFIED: even when there is no processTimeout value configured, the default (300 second) value of 5 minutes will be used.

Note: when a customer upgrades from an older version of RHEL-7.y to RHEL-7.9, it is also possible that the /etc/rhsm/rhsm.conf has no configuration setting at all for processTimeout, as simulated here...

[root@intel-s3e8132-01 ~]# sed -i s/^\\[rhsmd\\]//g /etc/rhsm/rhsm.conf
[root@intel-s3e8132-01 ~]# sed -i s/^process.*//g /etc/rhsm/rhsm.conf
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# grep -i "\[rhsmd\]" /etc/rhsm/rhsm.conf 
[root@intel-s3e8132-01 ~]# grep -i processtimeout /etc/rhsm/rhsm.conf 
[root@intel-s3e8132-01 ~]# 
[root@intel-s3e8132-01 ~]# time /etc/cron.daily/rhsmd

real	5m0.026s
user	0m0.338s
sys	0m0.121s
[root@intel-s3e8132-01 ~]# 

VERIFIED: and when there is no processTimeout configuration at all after an upgrade, the default (300 second) value of 5 minutes will be used.
VERIFIED: best of all... no more "sleep: missing operand" when /etc/cron.daily/rhsmd is run

There is still a problem with the /etc/cron.daily/rhsmd that needs to be corrected.  The processTimeout issue is correct however the script is now killing off an rhsmd process it should not kill.  The script kicks off the rhsmd process and then sleeps the timeout value.  When the script begins to execute again it blindly checks for a /usr/libexec/rhsmd process and kills it when it should be looking specifically for the rhsmd process that it kicked off.  

When this script runs we are now seeing the following every morning in the logs because the rhsmd script is killing the rhsmd process that was started by dbus rather than checking if what it is killing is the proper process:

Jun  5 03:26:02 testhost1 rhsmd: rhsmd process exceeded runtime and was killed.
Jun  5 03:26:05 testhost1 dbus[778]: [system] Activating service name='com.redhat.SubscriptionManager' (using servicehelper)
Jun  5 03:26:05 testhost1 dbus[778]: [system] Successfully activated service 'com.redhat.SubscriptionManager'

(In reply to Kurt Bechstein from comment #28)
> There is still a problem with the /etc/cron.daily/rhsmd that needs to be
> corrected.  The processTimeout issue is correct however the script is now
> killing off an rhsmd process it should not kill.  The script kicks off the
> rhsmd process and then sleeps the timeout value.  When the script begins to
> execute again it blindly checks for a /usr/libexec/rhsmd process and kills
> it when it should be looking specifically for the rhsmd process that it
> kicked off.  
> 
> When this script runs we are now seeing the following every morning in the
> logs because the rhsmd script is killing the rhsmd process that was started
> by dbus rather than checking if what it is killing is the proper process:
> 
> Jun  5 03:26:02 testhost1 rhsmd: rhsmd process exceeded runtime and was
> killed.
> Jun  5 03:26:05 testhost1 dbus[778]: [system] Activating service
> name='com.redhat.SubscriptionManager' (using servicehelper)
> Jun  5 03:26:05 testhost1 dbus[778]: [system] Successfully activated service
> 'com.redhat.SubscriptionManager'

Hi Kurt, 

Thanks for the observation, Can you please open a separate bug for this particular scenario ?. Also Can you please help us understand if this is only a log issue or Is there any functional impact ?   

thanks,
Rehana

Rehana, I will go ahead and open a new bug on this one as requested.  We haven't noticed any actual functional impact from this bug, but I'm assuming dbus is restarting this process for a reason.  I don't necessarily know what that exact reason is, but it would seem reasonable that a little more care can be taken when killing off system processes as root.

Bug 1846665 has been created.

Kurt has created Bug 1846665 and I am adding to that one.

But is seams no one is working on this one.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3866