Bug 1252571

Summary: daemon and/or listener logs disappear
Product: Red Hat Certification Program Reporter: Brian Brock <bbrock>
Component: redhat-certificationAssignee: Greg Nichols <gnichols>
Status: CLOSED ERRATA QA Contact: Brian Brock <bbrock>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.0CC: brose
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: redhat-certification-1.0-20150813 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 16:35:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
RedHatCertDaemon.log captured with tail none

Description Brian Brock 2015-08-11 17:56:51 UTC 
After starting the daemon & listener, and then running a test, the log files /var/log/rhcert/RedHatCertificationListener.log and RedHatCertDaemon.log are no longer present.

redhat-certification-1.0-20150723.el6
redhat-certification-hardware-1.6.6-20150810.2.el6

How reproducible:
no reproducible case yet, but occurs regularly in testing.

Steps to Reproduce:
1. Start the daemon with `rhcert-backend server start`
2. verify that the servers (daemon & listener) started.  retry, if needed.
3. run any tests
4. check for the log files in /var/log/rhcert

Actual results:
/var/log/RedHatCertDaemon.log and /var/log/RedHatCertificationListener.log do not exist.  Before the most recent test run, with no server restart, those log files existed.

Expected results:
persistent log files

Additional info:
seen while running solely the cd-rw test, during test execution (debug option not selected).  LTS is the SUT (server is also client).  The cd-rw test completed and passed.  /var/log/rhcert/runs/ appears complete.  There's no obvious effect from the web ui.  /var/log/messages doesn't show anything related.

The daemon and listener are still running. /proc/$daemonpid/fd shows that stdout and stderr are pointed at "/var/log/rhcert/RedHatCertDaemon.log (deleted)" or "/var/log/rhcert/RedHatCertificationListener.log (deleted)"

dmesg has some non-fatal errors (the burn completed successfully) over sr0
Comment 1 Brian Brock 2015-08-11 17:59:29 UTC 
I've previously (recently) seen similar behavior when running other tests, but have so far been unable to reproduce the failure at will.
Comment 2 Brian Brock 2015-08-11 19:21:38 UTC 
Saw this on a fresh install on combined LTS / SUT system.

Triggered by attaching a host to a certification, immediately after registering that host to the LTS.

The directory /var/log/rhcert/runs is created after the log files disappear.

All of /var/log/rhcert disappears.  The listener log seems to disappear first, but that could be an artifact of how I'm monitoring the logs.  A tail watching either log file continues to log new information.

The message "Checking for continuation" is printed to RedHatCertDaemon.log at roughly the same time as the log files disappear.  I'll paste the log in another message.
Comment 3 Brian Brock 2015-08-11 19:27:45 UTC 
Created attachment 1061658 [details]
RedHatCertDaemon.log captured with tail

captured via tail.  output includes output after the file is unlinked from /var/log/rhcert
Comment 4 Greg Nichols 2015-08-11 20:59:46 UTC 
(In reply to Brian Brock from comment #2)
> Saw this on a fresh install on combined LTS / SUT system.
> 
> Triggered by attaching a host to a certification, immediately after
> registering that host to the LTS.
> 

In this case, "clean" is called to remove any existing test results, including /var/log/rhcert.   It should probably just remove /var/log/rhcert/runs (the test run logging directories) so that the process logs are preserved.

Could this also explain the Summary case?  Did step 3) include associating the SUT with a certification?
Comment 7 Brian Brock 2015-09-16 21:18:25 UTC 
The original case occurred when a fresh system started the daemon, stopped, and then started again.  Associating the SUT with a cert happened (normally) without any problems.
Comment 8 Brian Brock 2015-10-01 04:59:57 UTC 
verified in:
redhat-certification-2.0-20150930.el7.noarch
redhat-certification-hardware-1.7.1-20150901.el7.noarch

Logs are no longer disappearing.
Comment 10 errata-xmlrpc 2015-11-19 16:35:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2479.html