Bug 1133993

Summary:	postgres data is not collected
Product:	[Retired] oVirt	Reporter:	Petr Beňas <pbenas>
Component:	ovirt-engine-logcollector	Assignee:	Sandro Bonazzola <sbonazzo>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Pavel Novotny <pnovotny>
Severity:	low	Docs Contact:
Priority:	medium
Version:	3.5	CC:	bmr, bugs, ecohen, gklein, iheim, jbelka, pstehlik, rbalakri, sbonazzo, tdosek, yeylon
Target Milestone:	---	Keywords:	Reopened
Target Release:	3.5.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	integration
Fixed In Version:	ovirt-3.5.1_rc1	Doc Type:	Bug Fix
Doc Text:	Cause: sosreport command line parser converted numerical passwords to integer so posgtgresql sos plugin crashed while setting the PG_PASSWORD env variable Consequence: postgres sos plugin did not work if a numerical password was specified Fix: the result of the command line parsing is now correctly converted to string Result: postgres sos plugin now works with numerical passwords	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-01-21 16:04:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1192945

Description Petr Beňas 2014-08-26 15:22:43 UTC

Description of problem:
14:24:14 2014-08-26 14:24:14,750 - MainThread - util - ERROR - number of occurrence is out of limit: 0 not in <1:1>; .*/var/lib/pgsql
14:24:14 2014-08-26 14:24:14,751 - MainThread - util - ERROR - number of occurrence is out of limit: 0 not in <1:1>; .*/var/lib/pgsql/pgstartup.log
14:24:14 2014-08-26 14:24:14,752 - MainThread - util - ERROR - number of occurrence is out of limit: 0 not in <1:1>; .*/var/lib/pgsql/data
14:24:14 2014-08-26 14:24:14,753 - MainThread - util - ERROR - number of occurrence is out of limit: 0 not in <3:3>; .*/var/lib/pgsql/data/.*[.]conf
14:24:14 2014-08-26 14:24:14,753 - MainThread - util - ERROR - number of occurrence is out of limit: 0 not in <1:1>; .*/var/lib/pgsql/data/pg_log

Version-Release number of selected component (if applicable):
ovirt-log-collector-3.5.0-0.1.master.el6.noarch

How reproducible:


Steps to Reproduce:
1. engine-log-collector collect
2. inspect collected archive
3.

Actual results:
/var/lib/pgsql not collected

Expected results:
/var/lib/pgsql collected

Additional info:
Discovered by automated test

Comment 1 Jiri Belka 2014-10-21 14:54:59 UTC

[root@jb-rhevm34 ~]# rpm -qf `which engine-log-collector`
rhevm-log-collector-3.4.5-1.el6ev.noarch
[root@jb-rhevm34 ~]# tar -xJOf /tmp/sosreport-LogCollector-20141021164943.tar.xz './log-collector-data/postgresql-sosreport-qaredhat.com-20141021164924-e358.tar.xz' | tar tJvf -
drwx------ root/root         0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/
drwxr-xr-x root/root         0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/var/
drwxr-xr-x root/root         0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/var/lib/
drwxr-xr-x root/root         0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/var/lib/pgsql/
-rw------- postgres/postgres 1152 2013-02-22 13:02 jb-rhevm34-2014102116491413902959/var/lib/pgsql/pgstartup.log
drwxr-xr-x root/root            0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/
-rw------- postgres/postgres    4 2013-02-22 13:02 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/PG_VERSION
-rw------- postgres/postgres   57 2014-10-19 17:31 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/postmaster.opts
drwxr-xr-x root/root            0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/
-rw------- postgres/postgres    0 2014-10-21 00:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Tue.log
-rw------- postgres/postgres    0 2014-10-20 00:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Mon.log
-rw------- postgres/postgres  171 2014-10-17 17:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Fri.log
-rw------- postgres/postgres    0 2014-10-16 00:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Thu.log
-rw------- postgres/postgres    0 2014-10-15 00:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Wed.log
-rw------- postgres/postgres    0 2014-10-11 00:00 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Sat.log
-rw------- postgres/postgres  151 2014-10-19 17:31 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_log/postgresql-Sun.log
-rw------- postgres/postgres 16970 2014-03-04 17:02 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/postgresql.conf
-rw------- postgres/postgres  1631 2013-02-22 13:02 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_ident.conf
-rw------- postgres/postgres  4563 2014-04-25 10:41 jb-rhevm34-2014102116491413902959/var/lib/pgsql/data/pg_hba.conf
drwxr-xr-x root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/sos_logs/
-rw-r--r-- root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/sos_logs/sos.log
drwxr-xr-x root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/sos_reports/
-rw-r--r-- root/root          2151 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/sos_reports/sosreport.html
drwxr-xr-x root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/sos_commands/
drwxr-xr-x root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/tmp/
drwxr-xr-x root/root             0 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/tmp/tmpP6ZJxZ/
-rw-r--r-- root/root       4565504 2014-10-21 16:49 jb-rhevm34-2014102116491413902959/tmp/tmpP6ZJxZ/sos_pgdump.tar

Comment 2 Petr Beňas 2014-10-22 11:13:18 UTC

Pretty weird, got the same version of collector, but the DB wasn't collected. 
[root@pb-rh34 ~]#  tar -xJOf /tmp/sosreport-LogCollector-20141022104209.tar.xz './log-collector-data/postgresql-sosreport-admin-20141022104030-53ba.tar.xz' | tar tJvf -
drwx------ root/root         0 2014-10-22 10:40 pb-rh34-2014102210401413967229/
drwxr-xr-x root/root         0 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_logs/
-rw-r--r-- root/root         0 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_logs/sos.log
-rw-r--r-- root/root       527 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_logs/sosreport-plugin-errors.txt
drwxr-xr-x root/root         0 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_reports/
-rw-r--r-- root/root       666 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_reports/sosreport.html
drwxr-xr-x root/root         0 2014-10-22 10:40 pb-rh34-2014102210401413967229/sos_commands/
[root@pb-rh34 ~]# rpm -qa | grep collector
rhevm-log-collector-3.4.5-1.el6ev.noarch

Comment 3 Simone Tiraboschi 2014-10-30 09:50:11 UTC

I tried to reproduce with rhevm-log-collector-3.4.5-1.el6ev.noarch on RHEL6 but it works as expected.

Petr, do you have any hint about how to reproduce?

Comment 4 Petr Beňas 2014-10-31 09:15:36 UTC

Still reproducible in vt8 with our automation job. Feel free to contact me directly for the test code details. 
http://jenkins.qa.lab.tlv.redhat.com:8080/view/RhevmCore/view/3.5-ALL/job/3.5-git-rhevmCore-infra_tools_log_collector_nfs/51/consoleFull

Comment 5 Sandro Bonazzola 2014-11-14 11:55:58 UTC

(In reply to Petr Beňas from comment #4)
> Still reproducible in vt8 with our automation job. Feel free to contact me
> directly for the test code details. 
> http://jenkins.qa.lab.tlv.redhat.com:8080/view/RhevmCore/view/3.5-ALL/job/3.
> 5-git-rhevmCore-infra_tools_log_collector_nfs/51/consoleFull

Petr, can you change the job at this line:
23:21:34 2014-10-30 23:21:34,129 - MainThread - rhevm_tools - INFO - executed command: ['echo', 'Y', '|', '/usr/bin/rhevm-log-collector', '--data-center=newdc', 'collect'] ; rc: 0, out: This command will collect system configuration and diagnostic

for adding --verbose to command line?
without that I can't debug what's wrong there and I can't reproduce on my system.

Comment 6 Petr Beňas 2014-11-14 12:09:45 UTC

Hi Sandro, there already is a test case running with the --verbose option. Search for "LogCollectorSingleDC.test_log_collector_collect_verbosity". I see there an error "err: ERROR: Failed to collect logs from: 10.35.160.33; Could not parse sosreport output to determine filename".

Comment 7 Sandro Bonazzola 2014-11-14 12:39:48 UTC

23:27:59 Traceback (most recent call last):
23:27:59   File "/usr/lib/python2.6/site-packages/sos/sosreport.py", line 736, in sosreport
23:27:59     plug.setup()
23:27:59   File "/usr/lib/python2.6/site-packages/sos/plugins/postgresql.py", line 98, in setup
23:27:59     self.pg_dump()
23:27:59 TypeError: putenv() argument 2 must be string, not int
23:27:59   Running plugins. Please wait 

Opened a bug on sos report: https://github.com/sosreport/sos/issues/433

will solve adding a str

Comment 8 Sandro Bonazzola 2014-11-14 12:41:34 UTC

it affects only numerical passwords

Comment 9 Bryn M. Reeves 2014-11-14 13:16:52 UTC

Adding a coercion to string in the plugin is a workaround; unless this is highly urgent I'd suggest holding out for a fix from upstream.

Comment 10 Sandro Bonazzola 2014-11-14 13:37:43 UTC

Bryn, just let me know when the plugin is fixed upstream and I'll be happy to sync. Also because this change is on el6 only on log-collector side: sent a patch to sos-devel with the same fix used here.

Comment 11 Bryn M. Reeves 2014-11-14 14:27:48 UTC

Will do - most likely it'll require a fix in sosreport.py::_set_tunables() so it'll need to be fixed in the base sos package (if that's going to cause scheduling problems then the workaround of coercing to str in the plugin is an acceptable short-term fix).

Comment 13 Sandro Bonazzola 2015-01-15 15:09:21 UTC

This is an automated message: 
This bug should be fixed in oVirt 3.5.1 RC1, moving to QA

Comment 14 Sandro Bonazzola 2015-01-21 16:04:28 UTC

oVirt 3.5.1 has been released. If problems still persist, please make note of it in this bug report.