Bug 1415470

Summary: foreman-debug to collect /var/log/httpd/pulp* access and error logs
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Foreman DebugAssignee: Lukas Zapletal <lzap>
Status: CLOSED ERRATA QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.8CC: bbuckingham, dlobatog, jcallaha, ktordeur, oshtaier
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:49:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2017-01-22 10:14:29 UTC 
Description of problem:
For investigation of pulp issues (esp. 404 not found in a repo), it is usefull to check /var/log/httpd/foreman-ssl_access_ssl.log on Satellite - it logs each and every HTTP(S) request on the server, including return code.

However, Capsule logs the same to /var/log/httpd/pulp-https_access_ssl.log that is not collected by either way. This file is requested to be collected as primary one.

Additionally, it might make sense to collect also /var/log/httpd/pulp-https_error_ssl.log - but so far the only "error" I saw there is "Failed to set r->user to 'SSL_CLIENT_S_DN_CN'". So not sure if something valuable can be there.

Additionally, when adding httpd logs, it is questionable if puppet logs can be valuable for investigation of a problem, i.e. shall we collect:

/var/log/httpd/puppet_access_ssl.log
/var/log/httpd/puppet_error_ssl.log

?


Version-Release number of selected component (if applicable):
Sat 6.2.6


How reproducible:
100%


Steps to Reproduce:
1. Take foreman-debug on Satellite _and_ also on a Capsule
2. Check for presence of these files in the tarball:
(only on Caps)
/var/log/httpd/pulp-https_access_ssl.log*
/var/log/httpd/pulp-http_access_ssl.log*
/var/log/httpd/pulp-https_error_ssl.log*
/var/log/httpd/pulp-http_error_ssl.log*
(on both Caps and Sat)
/var/log/httpd/puppet-https_access_ssl.log*
/var/log/httpd/puppet-http_access_ssl.log*
/var/log/httpd/puppet-https_error_ssl.log*
/var/log/httpd/puppet-http_error_ssl.log*


Actual results:
Neither file collected.


Expected results:
All the files (that we agree to collect from the list) to be collected.


Additional info:
Comment 2 Pavel Moravec 2017-01-22 10:16:17 UTC 
Kenny, do you see puppet httpd logs valuable to collect? Would they be used for investigation of a customer issue, or just make a foreman-debug redundantly bigger?
Comment 3 Kenny Tordeurs 2017-01-23 08:41:30 UTC 
CAPSULE:
~~~
/var/log/httpd/pulp-http_access_ssl.log*
/var/log/httpd/pulp-http_error_ssl.log*
~~~
=> Empty mostly

Contains information about tasks, we should be able to view this from task-export:
~~~
/var/log/httpd/pulp-https_access_ssl.log*
~~~

Contains only lines like Fails to set r->user to:
~~~
/var/log/httpd/pulp-https_error_ssl.log*
~~~

For the puppet logs:
~~~
192.168.25.3 - - [23/Jan/2017:10:34:55 +0100] "GET /production/node/ktordeur-capsule-provisioning.pipopopo.lan?transaction_uuid=107eb9e1-f4f7-49e7-bae4-da5623adf59d&fail_on_404=true HTTP/1.1" 200 7234 "-" "Ruby"
192.168.25.3 - - [23/Jan/2017:10:34:56 +0100] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 278 "-" "Ruby"
192.168.25.3 - - [23/Jan/2017:10:34:58 +0100] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 36682 "-" "Ruby"
192.168.25.3 - - [23/Jan/2017:10:35:02 +0100] "POST /production/catalog/ktordeur-capsule-provisioning.pipopopo.lan HTTP/1.1" 200 603 "-" "Ruby"
192.168.25.3 - - [23/Jan/2017:10:35:03 +0100] "PUT /production/report/ktordeur-capsule-provisioning.pipopopo.lan HTTP/1.1" 200 11 "-" "Ruby"
~~~
Not sure how useful that information would be.
Comment 4 Pavel Moravec 2017-01-23 09:33:40 UTC 
(In reply to Kenny Tordeurs from comment #3)
> CAPSULE:
> ~~~
> /var/log/httpd/pulp-http_access_ssl.log*
> /var/log/httpd/pulp-http_error_ssl.log*
> ~~~
> => Empty mostly
> 
> Contains information about tasks, we should be able to view this from
> task-export:
> ~~~
> /var/log/httpd/pulp-https_access_ssl.log*
> ~~~
> 
> Contains only lines like Fails to set r->user to:
> ~~~
> /var/log/httpd/pulp-https_error_ssl.log*
> ~~~
> 
> For the puppet logs:
> ~~~
> 192.168.25.3 - - [23/Jan/2017:10:34:55 +0100] "GET
> /production/node/ktordeur-capsule-provisioning.pipopopo.
> lan?transaction_uuid=107eb9e1-f4f7-49e7-bae4-da5623adf59d&fail_on_404=true
> HTTP/1.1" 200 7234 "-" "Ruby"
> 192.168.25.3 - - [23/Jan/2017:10:34:56 +0100] "GET
> /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.
> svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 278 "-" "Ruby"
> 192.168.25.3 - - [23/Jan/2017:10:34:58 +0100] "GET
> /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.
> svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 36682 "-" "Ruby"
> 192.168.25.3 - - [23/Jan/2017:10:35:02 +0100] "POST
> /production/catalog/ktordeur-capsule-provisioning.pipopopo.lan HTTP/1.1" 200
> 603 "-" "Ruby"
> 192.168.25.3 - - [23/Jan/2017:10:35:03 +0100] "PUT
> /production/report/ktordeur-capsule-provisioning.pipopopo.lan HTTP/1.1" 200
> 11 "-" "Ruby"
> ~~~
> Not sure how useful that information would be.

If I do e.g. "yum reinstall sos" on a client registered to a Capsule, the Capsule's /var/log/httpd/pulp-https_access_ssl.log has:

10.34.85.74 - 8aa2d50d59a64a8f0159a6fd9ff80070 [23/Jan/2017:10:25:12 +0100] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1" 200 2024 "-" "urlgrabber/3.10 yum/3.4.3"
10.34.85.74 - 8aa2d50d59a64a8f0159a6fd9ff80070 [23/Jan/2017:10:25:13 +0100] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.2/os/repodata/repomd.xml HTTP/1.1" 200 2162 "-" "urlgrabber/3.10 yum/3.4.3"
10.34.85.74 - 8aa2d50d59a64a8f0159a6fd9ff80070 [23/Jan/2017:10:25:13 +0100] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.2/os/repodata/fad9e1b3dd1338d465e48520b07e5ff909a9d3583b714d4a46f83b29837a6350-updateinfo.xml.gz HTTP/1.1" 200 7819 "-" "urlgrabber/3.10 yum/3.4.3"
10.34.85.74 - 8aa2d50d59a64a8f0159a6fd9ff80070 [23/Jan/2017:10:25:13 +0100] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.2/os/repodata/e5d50cafe40f267e98c5cd649633cff1a9ab58f46242b9f251c4bfd59041dd02-primary.xml.gz HTTP/1.1" 200 18595 "-" "urlgrabber/3.10 yum/3.4.3"
10.34.85.74 - 8aa2d50d59a64a8f0159a6fd9ff80070 [23/Jan/2017:10:25:27 +0100] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/os/sos-3.3-5.el7_3.noarch.rpm HTTP/1.1" 200 365060 "-" "urlgrabber/3.10 yum/3.4.3"

that is what I meant in #c0.

If puppet logs dont contain anything useful to collect, let skip them.

So I suggest fix:

--- /usr/share/foreman/script/foreman-debug.d/katello-debug.sh.orig	2017-01-23 10:30:04.523839128 +0100
+++ /usr/share/foreman/script/foreman-debug.d/katello-debug.sh	2017-01-23 10:31:02.144900457 +0100
@@ -56,6 +56,8 @@ add_files /etc/tomcat/server.xml
 add_files /etc/pulp/*.conf
 add_files /etc/httpd/conf.d/pulp.conf
 add_files /etc/pulp/server/plugins.conf.d/nodes/distributor/*
+add_files /var/log/httpd/pulp-http{s,}_access_ssl.log*
+add_files /var/log/httpd/pulp-http{s,}_error_ssl.log*
 
 # MongoDB (*)
 if [ $NOGENERIC -eq 0 ]; then


(it can be shortened to:

+add_files /var/log/httpd/pulp-http{s,}_{access,error}_ssl.log*

but due to bz1372884, I prefer to ensure collecting some (newest) access and some (newest) error log as well)
Comment 5 Satellite Program 2017-01-24 13:02:01 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18198 has been resolved.
Comment 6 Daniel Lobato Garcia 2017-08-09 10:32:39 UTC 
Verified. 

Tested on:

satellite-6.3.0-16.0.beta.el7sat.noarch
foreman-debug-1.15.2-1.el7sat.noarch


All files matching:

add_files /var/log/httpd/pulp-http{s,}_access_ssl.log*              
add_files /var/log/httpd/pulp-http{s,}_error_ssl.log*               
add_files /etc/default/pulp*                                        

are added to the debug tarball.
Comment 7 Satellite Program 2018-02-21 16:49:54 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336