Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1176781

Summary: Unable to collect sosreport from OpenStack nodes using 'rhos-log-collector'
Product: Red Hat OpenStack Reporter: Udayendu Sekhar Kar <ukar>
Component: sos-plugins-openstackAssignee: Alan Pevec <apevec>
Status: CLOSED ERRATA QA Contact: Ofer Blaut <oblaut>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0 (RHEL 7)CC: ahirshbe, apevec, bmr, chorn, jraju, jschluet, lhh, lnatapov, lyarwood, nbarcet, oblaut, pbrady, rharwood, rpai, swatson, yeylon
Target Milestone: z5Keywords: ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: sos-plugins-openstack-2013.2-6.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1209515 (view as bug list) Environment:
Last Closed: 2015-09-10 11:46:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1209515    
Attachments:
Description Flags
Proposed patch for r-l-c none

Description Udayendu Sekhar Kar 2014-12-23 07:40:38 UTC
Description of problem:
Unable to collect sosreport from OpenStack nodes using "rhos-log-collector"

Version-Release number of selected component (if applicable):
rhos-log-collector-2013.2-5.el6ost.noarch

How reproducible:
100%

Steps to Reproduce:
While trying to run "rhos-log-collector" from one of the controller node, got the below error message:

==--==
# rhos-log-collector --hosts=host.txt
[1] 16:20:35 [SUCCESS] root.0.250
Stderr: temporary directory /root/sos-be53be74-885b-11e4-bcc0-005056a5280f does not exist or is not writable
[2] 16:20:35 [SUCCESS] root.0.251
Stderr: temporary directory /root/sos-be53be74-885b-11e4-bcc0-005056a5280f does not exist or is not writable
[3] 16:20:35 [SUCCESS] root.0.252
Stderr: temporary directory /root/sos-be53be74-885b-11e4-bcc0-005056a5280f does not exist or is not writable
[4] 16:20:35 [SUCCESS] root.0.253
Stderr: temporary directory /root/sos-be53be74-885b-11e4-bcc0-005056a5280f does not exist or is not writable
Remote path sos-be53be74-885b-11e4-bcc0-005056a5280f must be an absolute path
Support reports from all hosts are saved in sos-be53be74-885b-11e4-bcc0-005056a5280f
==--==


Actual results:
"rhos-log-collector" failed to collect the sosreport from all the OpenStack nodes.

Expected results:
"rhos-log-collector" should collect the sosreport from all the OpenStack nodes.

Additional Info:
Followed the steps as mentioned in https://access.redhat.com/solutions/704893

Comment 2 Robbie Harwood 2014-12-23 17:36:38 UTC
I am unable to reproduce this with a single node installation of RHOS5 on RHEL6.  I will try again with a second node.

Comment 3 Robbie Harwood 2014-12-23 17:59:13 UTC
I am unable to reproduce this in a two-node setup.  Since you say this is 100%, what's the topology you're using?

Comment 4 Robbie Harwood 2014-12-23 18:02:41 UTC
Also, if possible, please provide an exact console log of the commands being run and the permissions on the /root directories on all machines.

Comment 5 Udayendu Sekhar Kar 2014-12-24 05:30:28 UTC
Hi Robbie,

Customer tried it with 4 nodes. And in my test setup I tried it with 3 nodes. I did the installation through foreman. Then on the foreman system I install all the three required packages abd tried to collect the sosreport. from foreman installer passwordless ssh was enable to all the nodes. But it was unable to get the sosreport with the error as mentioned in the case description.

One of our engineer from NA also experienced the same. Then I have opened it.

If you need I will make complete reproducer and will let you know the system credentials for the verification.

Let me know how you want me to proceed further ?

Cheers !
Uday

Comment 6 Udayendu Sekhar Kar 2014-12-24 05:31:59 UTC
I tried with /root permission 755 and with 550(default) too.

Comment 8 Robbie Harwood 2015-01-02 16:20:32 UTC
I'd been checking on rhel6.  Could you confirm which RHEL version this affects?

Comment 9 Robbie Harwood 2015-01-02 21:46:11 UTC
Okay, this is rhel7 only.  I have replicated the issue there.

Comment 10 Robbie Harwood 2015-01-05 18:10:51 UTC
Alright, minimizing this, it occurs on a single-node rhel7 deployment (doesn't matter what deployed it).  The reason here is that rhos-log-collector calls out to pssh, in a manner like so:

pssh -i -t 0 -h host.txt sosreport --tmp-dir ~/sos-d6c6796a-9503-11e4-9562-525400d94c36

(the --tmp-dir argument is created from a uuid).  So on a single node system, pssh connects to localhost, and runs:

sosreport --tmp-dir ~/sos-d6c6796a-9503-11e4-9562-525400d94c36

(keeping the uuid).  This then fails, because the directory does not exist.

The issue is therefore either in sosreport or in rhos-log-collector, and I'm not sure which; the documentation for sosreport doesn't specify whether it plans to create the directory that is pased through --tmp-dir or not (though clearly it does not).

Comment 11 Bryn M. Reeves 2015-01-08 15:48:52 UTC
> The issue is therefore either in sosreport

This is not an sos bug. The directory passed to --tmp-dir must exist; the log collector either needs to create the directory itself, use a predictable path, or provide some wrapper that will create the directory before executing sosreport.

Comment 13 Bryn M. Reeves 2015-03-31 11:55:02 UTC
It would also work to just specify "--tmp-dir /root"; sos always creates unique directory names anyway (if you need to know the name you can always parse the output).

Comment 14 Alan Pevec 2015-04-07 15:39:30 UTC
Created attachment 1011846 [details]
Proposed patch for r-l-c

Comment 16 Dave Maley 2015-04-29 17:15:09 UTC
re-targeting based on comment 9

Comment 26 Jon Schlueter 2015-08-27 18:56:45 UTC
retest with sos-plugins-openstack-2013.2-6.el7ost which will be in OSP 5z5 puddle

Comment 29 Leonid Natapov 2015-09-01 11:10:47 UTC
sos-3.2-15.el7_1.5.noarch

[root@mac848f69fbc49f ~]# rhos-log-collector --hosts=hosts
[1] 14:09:06 [SUCCESS] root.0.3
[1] 14:09:07 [SUCCESS] root.0.3

sosreport (version 3.2)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/root/sos-dc7b6bac-5099-11e5-bff6-848f69fbc49f and may be provided to a
Red Hat support representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Remote path sos-dc7b6bac-5099-11e5-bff6-848f69fbc49f must be an absolute path
Support reports from all hosts are saved in sos-dc7b6bac-5099-11e5-bff6-848f69fbc49f
[root@mac848f69fbc49f ~]#

Comment 33 errata-xmlrpc 2015-09-10 11:46:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1762.html