Bug 740675

Summary:	rhsmcertd does not automatically restart when upgrading subscription-manager
Product:	Red Hat Enterprise Linux 6	Reporter:	John Sefler <jsefler>
Component:	subscription-manager	Assignee:	Chris Duryee <cduryee>
Status:	CLOSED ERRATA	QA Contact:	IDM QE LIST <seceng-idm-qe-list>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.2	CC:	arfernan, cduryee, ddumas, skallesh, spandey
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-12-06 17:24:33 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	682238

Description John Sefler 2011-09-22 21:55:39 UTC

Description of problem:
When upgrading subscription-manager packages, an already existing rhsmcertd process is not restarted automatically.  The reason this is a problem is because rhel61 customers are running an rhsmcertd process that is only aware of the certFrequency.  The upgrade to rhel62 subscription-manager packages includes a new healFrequency that is also run by rhsmcertd.  If the customer doesn't know to manually restart the rhsmcertd service, then he will not pick up the new auto heal feature. 

See bug 739227 for the recreate scenario


Version-Release number of selected component (if applicable):
subscription-manager.x86_64 0:0.96.11-1.el6

Comment 1 John Sefler 2011-09-22 22:30:33 UTC

There is another problem with the new healFrequency configuration during the upgrade process....  It gets lost in /etc/rhsm/rhsm.conf.rpmnew

Because it is not in /etc/rhsm/rhsm, the user does not know it is settable and the config tools do not see list it or show it in the help...

[root@jsefler-stage-6workstation tmp]# subscription-manager config --list | grep -i frequency
   certfrequency = 120
[root@jsefler-stage-6workstation tmp]# subscription-manager config --help | grep -i frequency
  --rhsmcertd.certfrequency=RHSMCERTD.CERTFREQUENCY
                        Section: rhsmcertd, Name: certfrequency
[root@jsefler-stage-6workstation tmp]# 




Only after replacing the rhsm.conf with the new rhsm.conf.rpmnew does the healFrequency show up....


[root@jsefler-stage-6workstation tmp]# mv /etc/rhsm/rhsm.conf /etc/rhsm/rhsm.conf.old
[root@jsefler-stage-6workstation tmp]# cp /etc/rhsm/rhsm.conf.rpmnew /etc/rhsm/rhsm.conf
[root@jsefler-stage-6workstation tmp]# subscription-manager config --list | grep -i frequency
   certfrequency = 240
   healfrequency = 1440
[root@jsefler-stage-6workstation tmp]# subscription-manager config --help | grep -i frequency
  --rhsmcertd.certfrequency=RHSMCERTD.CERTFREQUENCY
                        Section: rhsmcertd, Name: certfrequency
  --rhsmcertd.healfrequency=RHSMCERTD.HEALFREQUENCY
                        Section: rhsmcertd, Name: healfrequency
[root@jsefler-stage-6workstation tmp]# 


This is also demonstrated in bug 739227

Comment 2 Chris Duryee 2011-09-23 15:25:05 UTC

per http://fedoraproject.org/wiki/Packaging:SysVInitScript#Initscripts_in_spec_file_scriptlets, looks like we should be bouncing on upgrade. Working on this now.

Comment 4 Chris Duryee 2011-09-27 15:18:52 UTC

21c378d RHEL6.2 0.96.13+
396f882 master 0.97.2+

Denise, thanks for the heads up re: exception flags

Comment 5 John Sefler 2011-09-30 16:22:04 UTC

Verification steps...

STEP 1:  START WITH THE LATEST RHEL61 ERRATA FOR SUBSCRIPTION_MANAGER INSTALLED
https://errata.devel.redhat.com/errata/stateview/11273

[root@jsefler-onprem-62server tmp]# rpm -i http://download.devel.redhat.com/brewroot/packages/subscription-manager/0.95.17/1.el6_1/x86_64/subscription-manager-0.95.17-1.el6_1.x86_64.rpm http://download.devel.redhat.com/brewroot/packages/subscription-manager/0.95.17/1.el6_1/x86_64/subscription-manager-firstboot-0.95.17-1.el6_1.x86_64.rpm http://download.devel.redhat.com/brewroot/packages/subscription-manager/0.95.17/1.el6_1/x86_64/subscription-manager-gnome-0.95.17-1.el6_1.x86_64.rpm
[root@jsefler-onprem-62server tmp]# rpm -qa | grep subscription-manager
subscription-manager-gnome-0.95.17-1.el6_1.x86_64
subscription-manager-firstboot-0.95.17-1.el6_1.x86_64
subscription-manager-0.95.17-1.el6_1.x86_64


STEP 2: MAKE SURE THE CONFIG FILE /etc/rhsm/rhsm.conf IS ALTERED

[root@jsefler-onprem-62server tmp]# s/certFrequency.*/certFrequency=120/g /etc/rhsm/rhsm.conf^C
[root@jsefler-onprem-62server tmp]# sed -i s/certFrequency.*/certFrequency=120/g /etc/rhsm/rhsm.conf
[root@jsefler-onprem-62server tmp]# grep -i frequency /etc/rhsm/rhsm.conf
# Frequency of certificate refresh (in minutes):
certFrequency=120

^^^ NOTE THAT THE RHEL61 PACKAGE DOES NOT EMPLOY A healFrequency CONFIGURATION


STEP 3: MAKE SURE rhsmcertd IS RUNNING AND NOTE IT'S PID

[root@jsefler-onprem-62server tmp]# service rhsmcertd start
Starting rhsmcertd 120                                     [  OK  ]
[root@jsefler-onprem-62server tmp]# service rhsmcertd status
rhsmcertd (pid 4883) is running...


STEP 4: UPGRADE TO NEW SUBSCRIPTION-MANGER RPMS CONTAINING THIS BUG FIX (I AM USING RPMS FROM A PRE-BREW/CONTINUOUS INTEGRATION GIT BUILD)

[root@jsefler-onprem-62server tmp]# ls subscription-manager*
subscription-manager-firstboot.x86_64.rpm  subscription-manager.x86_64.rpm
subscription-manager-gnome.x86_64.rpm
[root@jsefler-onprem-62server tmp]# yum update -q subscription-manager*
This system is not registered with RHN.
RHN Satellite or RHN Classic support will be disabled.

=======================================================================================================
 Package              Arch   Version                      Repository                              Size
=======================================================================================================
Updating:
 subscription-manager x86_64 0.96.11-1.git.20.199bf0e.el6 /subscription-manager.x86_64           1.6 M
 subscription-manager-firstboot
                      x86_64 0.96.11-1.git.20.199bf0e.el6 /subscription-manager-firstboot.x86_64  21 k
 subscription-manager-gnome
                      x86_64 0.96.11-1.git.20.199bf0e.el6 /subscription-manager-gnome.x86_64     550 k

Transaction Summary
=======================================================================================================
Upgrade       3 Package(s)

Is this ok [y/N]: y
Warning: RPMDB altered outside of yum.
warning: /etc/rhsm/rhsm.conf created as /etc/rhsm/rhsm.conf.rpmnew
duration: 262(ms)
Installed products updated.
[root@jsefler-onprem-62server tmp]# 


^^^ NOTICE THAT /etc/rhsm/rhsm.conf.rpmnew WAS CREATED DURING THE UPGRADE

STEP 5: NOW THAT WE HAVE UPGRADED, LET'S VERIFY THAT rhsmcertd STATUS WAS RESTARTED

[root@jsefler-onprem-62server tmp]# service rhsmcertd status
rhsmcertd (pid 4958 4956) is running...
[root@jsefler-onprem-62server tmp]# 

^^^ VERIFIED, THE ORIGINAL PID 4883 THAT REPRESENTS THE certFrequency JOB IS GONE AND REPLACED BY TWO NEW PIDs 4958 4956) REPRESENTING THE certFrequency AND NEW healFrequency.

[root@jsefler-onprem-62server tmp]# tail /var/log/rhsm/rhsmcertd.log
Fri Sep 30 11:41:16 2011: started: interval = 120 minutes
Fri Sep 30 11:41:16 2011: update failed (255), retry in 10 minutes
Fri Sep 30 11:46:23 2011: started: interval = 1440 minutes
Fri Sep 30 11:46:23 2011: started: interval = 120 minutes
Fri Sep 30 11:46:23 2011: certificates updated
Fri Sep 30 11:46:23 2011: certificates updated

^^^ DOUBLE VERIFIED THAT /var/log/rhsm/rhsmcertd.log ALSO SHOWS THE START OF INTERVAL 1440 WHICH REPRESENTS THE NEW healFrquency JOB

Comment 6 John Sefler 2011-09-30 19:26:21 UTC

marking this bug as VERIFIED and opening bug 742620 to address comment 1

Comment 7 errata-xmlrpc 2011-12-06 17:24:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1695.html

Comment 8 linferna 2012-04-17 04:27:19 UTC

After upgrading to "2.6.32-220.7.1.el6.x86_64" system hangs during reboot at the following line:

< Starting rhsmcertd 240 1440 >

So I have no choice other than booting back to "2.6.32-220.el6.x86_64".

Here is the grub info after the update:
<snip>
title Red Hat Enterprise Linux Workstation (2.6.32-220.7.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-220.7.1.el6.x86_64 ro root=UUID=a3cfe83c-84fc-4440-b72d-66d96d467fed rd_NO_LUKS  KEYBOA
RDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_LVM rd_NO_
DM rhgb quiet
        initrd /initramfs-2.6.32-220.7.1.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-220.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-220.el6.x86_64 ro root=UUID=a3cfe83c-84fc-4440-b72d-66d96d467fed rd_NO_LUKS  KEYBOARDTY
PE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_LVM rd_NO_DM r
hgb quiet
        initrd /initramfs-2.6.32-220.el6.x86_64.img
<snip>

Here is the 'tail /var/log/rhsm/rhsmcertd.log' output:
Mon Apr 16 15:09:33 2012: started: interval = 1440 minutes
Mon Apr 16 15:09:33 2012: started: interval = 240 minutes
Mon Apr 16 15:09:35 2012: certificates updated
Mon Apr 16 15:09:35 2012: certificates updated
Mon Apr 16 15:14:27 2012: started: interval = 1440 minutes
Mon Apr 16 15:14:27 2012: started: interval = 240 minutes
Mon Apr 16 15:14:30 2012: certificates updated
Mon Apr 16 15:14:30 2012: certificates updated
Mon Apr 16 19:14:29 2012: certificates updated
Mon Apr 16 23:14:30 2012: certificates updated

It appears this problem has been fixed but I still experience the original problem. Appreciate a feedback.

Comment 9 linferna 2012-04-17 16:09:52 UTC

Since I get hang up during reboot (kernel 2.6.32-220.7.1.el6.x86_64), I tried stopping (chkconfig rhsmcertd off). Then the system hangs up at RHN Daemon (Starting Red Hat Network Daemon  [OK] ). 

However, I was able to boot back to 2.6.32-220.el6.x86_64 kernel from grub, while my 'rhsmcertd off' state. I am surprised why this issue hasn't reopen. Thanks for any feedback.

Comment 10 John Sefler 2012-04-17 16:19:50 UTC

(In reply to comment #8)
> After upgrading to "2.6.32-220.7.1.el6.x86_64" system hangs during reboot at
> the following line:
> 
...
> 
> It appears this problem has been fixed but I still experience the original
> problem. Appreciate a feedback.

The problem described in comment #0 is a simpler scenario than reported in comment #8.  In comment #0 only subscription-manager packages were being upgraded.  In comment #8, a full yum update appears to include a kernel upgrade which is leading to a new problem.  Comment #8 should be reported as a separate bug.