Bug 1539027

Summary: 4.2.3-1 analyzer cannot still find pg dump in archive collected on 7.4/4.2 env
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: ovirt-log-collectorAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED CURRENTRELEASE QA Contact: David Necpal <dnecpal>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: jbelka, lsvaty, ylavi
Target Milestone: ovirt-4.2.2   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-log-collector-4.2.4-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-16 07:43:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jiri Belka 2018-01-26 12:58:00 UTC 
Description of problem:

$ ovirt-log-collector-analyzer /tmp/sos.tar.xz 
Preparing environment:
======================
Temporary working directory is /tmp/tmp.BV65ksQwby
Unpacking postgres data. This can take up to several minutes.
tar: sosreport-10-37-137-41.rhev.lab.eng.brq.redhat.com-20180126134346/sos_commands/postgresql/pgdump.tar\nsosreport-10-37-137-41.rhev.lab.eng.brq.redhat.com-20180126134346/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar: Not found in archive
tar: Exiting with failure status due to previous errors
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
-bash-4.2$ rpm -q sos
sos-3.4-13.el7_4.noarch
-bash-4.2$ logout
[root@10-37-137-41 tmp]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)

# rpm -q ovirt-log-collector-analyzer sos
ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch
sos-3.4-13.el7_4.noarch

Version-Release number of selected component (if applicable):
ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. run analyzer with sos report collected on 4.2/el 7.4
2.
3.

Actual results:
failure

Expected results:
should work

Additional info:
previous version was failing with:

...
++ tar tf /tmp/tmp.DP8d20dmUC/unpacked_sosreport/sosreport-LogCollector-20180126134351/log-collector-data/postgresql-sosreport-10-37-137-41.rhev.lab.eng.brq.redhat.com-20180126134346.tar.xz
++ grep sos_pgdump.tar
+ PG_DUMP_TAR=
+ :
+ '[' '!' '' ']'
+ echo 'Unable to detect sos_pgdump.tar from sosreport, aborting..'
/usr/share/ovirt-log-collector/analyzer/unpackAndPrepareDump.sh: line 87: /dev/stderr: Permission denied
-bash-4.2$ rpm -q ovirt-log-collector-analyzer
ovirt-log-collector-analyzer-4.2.0-1.el7ev.noarch
Comment 2 Douglas Schilling Landgraf 2018-02-15 20:51:37 UTC 
Hi Jiri,

I cannot reproduce your report using the last downstream packages available.

ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch.rpm  sos-3.4-13.el7_4.noarch.rpm or sos-3.5-1.el7.noarch


> /usr/share/ovirt-log-collector/analyzer/unpackAndPrepareDump.sh: line 87: 
> /dev/stderr: Permission denied
> -bash-4.2$ rpm -q ovirt-log-collector-analyzer
> ovirt-log-collector-analyzer-4.2.0-1.el7ev.noarch

  ^^^^ Just to confirm, are you trying to use upstream package here? Why not 
ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch  as you are opening a bug against it?
Comment 3 Jiri Belka 2018-02-20 08:24:56 UTC 
(In reply to Douglas Schilling Landgraf from comment #2)
> Hi Jiri,
> 
> I cannot reproduce your report using the last downstream packages available.
> 
> ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch.rpm 
> sos-3.4-13.el7_4.noarch.rpm or sos-3.5-1.el7.noarch
> 
> 
> > /usr/share/ovirt-log-collector/analyzer/unpackAndPrepareDump.sh: line 87: 
> > /dev/stderr: Permission denied
> > -bash-4.2$ rpm -q ovirt-log-collector-analyzer
> > ovirt-log-collector-analyzer-4.2.0-1.el7ev.noarch
> 
>   ^^^^ Just to confirm, are you trying to use upstream package here? Why not 
> ovirt-log-collector-analyzer-4.2.3-1.el7ev.noarch  as you are opening a bug
> against it?

No, why would I?

# rpm -qf /etc/redhat-release 
redhat-release-server-7.4-18.el7.x86_64

# rpm -q sos
sos-3.4-13.el7_4.noarch

# rpm -qi ovirt-log-collector-analyzer
Name        : ovirt-log-collector-analyzer
Version     : 4.2.3
Release     : 1.el7ev
Architecture: noarch
Install Date: Wed 07 Feb 2018 03:40:43 PM CET
Group       : Unspecified
Size        : 208908
License     : ASL 2.0 and GPLv2+
Signature   : (none)
Source RPM  : ovirt-log-collector-4.2.3-1.el7ev.src.rpm
Build Date  : Fri 19 Jan 2018 07:36:46 PM CET
Build Host  : x86-040.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.ovirt.org
Summary     : Log Collector for oVirt Engine - Analysis tools
Description :
This package contains tools to help analyze oVirt Log Collector archives

^^  this is build in RH

# rpm -q ovirt-engine
ovirt-engine-4.2.1.3-0.1.el7.noarch

# ovirt-log-collector --no-hypervisors                                                                             
This command will collect system configuration and diagnostic
information from this system.
The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before
being passed to any third party.
No changes will be made to system configuration.
INFO: Gathering oVirt Engine information...
INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost...
INFO: Skipping hypervisor collection...
Creating compressed archive...
INFO: Log files have been collected and placed in /tmp/sosreport-LogCollector-20180220090555.tar.xz
The MD5 for this file is 95c7c6dc0bb9a2e4434d1d6add07f3db and its size is 44.2M
# chown postgres:postgres /tmp/sosreport-LogCollector-20180220090555.tar.xz

# sudo su postgres
$ cd /tmp/
$ ovirt-log-collector-analyzer sosreport-LogCollector-20180220090555.tar.xz
Preparing environment:
======================
Temporary working directory is /tmp/tmp.G8KwStbCI6
Unpacking postgres data. This can take up to several minutes.
tar: sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar\nsosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar: Not found in archive
tar: Exiting with failure status due to previous errors
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Comment 4 Jiri Belka 2018-02-20 08:35:54 UTC 
and ovirt-log-collector itself:

$ rpm -q ovirt-log-collector
ovirt-log-collector-4.2.3-1.el7ev.noarch
Comment 5 Jiri Belka 2018-02-20 15:06:45 UTC 
$ ovirt-log-collector-analyzer sosreport-LogCollector-20180220090555.tar.xz
Preparing environment:
======================
Temporary working directory is /tmp/tmp.gRM2KRp1ja
Unpacking postgres data. This can take up to several minutes.
tar: sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar\nsosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar: Not found in archive
tar: Exiting with failure status due to previous errors
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors


bash-4.2$ rpm -q ovirt-log-collector-analyzer
ovirt-log-collector-analyzer-4.2.4-1.el7ev.noarch
Comment 6 Jiri Belka 2018-02-20 15:07:22 UTC 
(In reply to Jiri Belka from comment #5)
> $ ovirt-log-collector-analyzer sosreport-LogCollector-20180220090555.tar.xz
> Preparing environment:
> ======================
> Temporary working directory is /tmp/tmp.gRM2KRp1ja
> Unpacking postgres data. This can take up to several minutes.
> tar:
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump.tar\nsosreport-brq-setup.rhev.lab.eng.brq.redhat.com-
> 20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar: Not
> found in archive
> tar: Exiting with failure status due to previous errors
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> 
> 
> bash-4.2$ rpm -q ovirt-log-collector-analyzer
> ovirt-log-collector-analyzer-4.2.4-1.el7ev.noarch

see '\n' in path. that's the issue.
Comment 7 Jiri Belka 2018-02-20 15:32:14 UTC 
the problem is on line 84 where you get two lines in the output:

  84 PG_DUMP_TAR=$(tar tf "$TAR_WITH_POSTGRES_SOSREPORT" | grep ".*pgdump.*tar.*") || :
  85 if [ ! "${PG_DUMP_TAR}" ]; then
  86     echo "Unable to detect the database dump from sosreport, aborting.." 1>&2
  87     rm -rf "${UNPACKED_SOSREPORT} ${PG_DUMP_DIR}"
  88     exit -1
  89 fi

and what does grep filters out?

$ tar --exclude='sosreport*/proc' --exclude='sosreport*/dev' --exclude='sosreport*/sys' -xOJf sosreport-LogCollector-20180220090555.tar.xz sosreport-LogCollector-20180220090555/log-collector-data/postgresql-sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar -tJf - | grep ".*pgdump.*tar.*" # grep from line 84


sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar
sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar

anyway, even if you would removed double quoting around PG_DUMP_TAR and tar would on line 93 interpret this variable as two paths, it seems you cannot pipe 2 tars into next tar.

$ echo $OUTPUT; tar --exclude='sosreport*/proc' --exclude='sosreport*/dev' --exclude='sosreport*/sys' -xOJf sosreport-LogCollector-20180220090555.tar.xz sosreport-LogCollector-20180220090555/log-collector-data/postgresql-sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar -xOvJf - ${OUTPUT} | tar -C /tmp/out -xf -

sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar

sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar
sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
tar: This does not look like a tar archive
tar: Skipping to next header

thus, we have to double check we detect valid tar for PG_DUMP_TAR. following is a test with one value:

bash-4.2$ PG_DUMP_TAR=sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar

bash-4.2$ echo $PG_DUMP_TAR; tar --exclude='sosreport*/proc' --exclude='sosreport*/dev' --exclude='sosreport*/sys' -xOJf sosreport-LogCollector-20180220090555.tar.xz sosreport-LogCollector-20180220090555/log-collector-data/postgresql-sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar -xOvJf - ${PG_DUMP_TAR} | tar -C /tmp/out -xf -
sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar

bash-4.2$ find /tmp/out | wc -l
175

bash-4.2$ find /tmp/out | head
/tmp/out
/tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546
/tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands
/tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql
/tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump.tar
/tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
/tmp/out/toc.dat
/tmp/out/7428.dat
/tmp/out/7430.dat
/tmp/out/7432.dat
Comment 8 Sandro Bonazzola 2018-02-20 17:53:31 UTC 
Please recheck with RHEL 7.5
Comment 9 Douglas Schilling Landgraf 2018-02-21 03:09:18 UTC 
(In reply to Jiri Belka from comment #7)
> the problem is on line 84 where you get two lines in the output:
> 
>   84 PG_DUMP_TAR=$(tar tf "$TAR_WITH_POSTGRES_SOSREPORT" | grep
> ".*pgdump.*tar.*") || :
>   85 if [ ! "${PG_DUMP_TAR}" ]; then
>   86     echo "Unable to detect the database dump from sosreport,
> aborting.." 1>&2
>   87     rm -rf "${UNPACKED_SOSREPORT} ${PG_DUMP_DIR}"
>   88     exit -1
>   89 fi
> 
> and what does grep filters out?
> 
> $ tar --exclude='sosreport*/proc' --exclude='sosreport*/dev'
> --exclude='sosreport*/sys' -xOJf
> sosreport-LogCollector-20180220090555.tar.xz
> sosreport-LogCollector-20180220090555/log-collector-data/postgresql-
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar
> -tJf - | grep ".*pgdump.*tar.*" # grep from line 84
> 
> 
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump.tar
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump-scl-rh-postgresql95.tar
> 
> anyway, even if you would removed double quoting around PG_DUMP_TAR and tar
> would on line 93 interpret this variable as two paths, it seems you cannot
> pipe 2 tars into next tar.
> 
> $ echo $OUTPUT; tar --exclude='sosreport*/proc' --exclude='sosreport*/dev'
> --exclude='sosreport*/sys' -xOJf
> sosreport-LogCollector-20180220090555.tar.xz
> sosreport-LogCollector-20180220090555/log-collector-data/postgresql-
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar
> -xOvJf - ${OUTPUT} | tar -C /tmp/out -xf -
> 
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump.tar
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump-scl-rh-postgresql95.tar
> 
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump.tar
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump-scl-rh-postgresql95.tar
> tar: This does not look like a tar archive
> tar: Skipping to next header
> 
> thus, we have to double check we detect valid tar for PG_DUMP_TAR. following
> is a test with one value:
> 
> bash-4.2$
> PG_DUMP_TAR=sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/
> sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
> 
> bash-4.2$ echo $PG_DUMP_TAR; tar --exclude='sosreport*/proc'
> --exclude='sosreport*/dev' --exclude='sosreport*/sys' -xOJf
> sosreport-LogCollector-20180220090555.tar.xz
> sosreport-LogCollector-20180220090555/log-collector-data/postgresql-
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546.tar.xz | tar
> -xOvJf - ${PG_DUMP_TAR} | tar -C /tmp/out -xf -
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump-scl-rh-postgresql95.tar
> sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/sos_commands/
> postgresql/pgdump-scl-rh-postgresql95.tar
> 
> bash-4.2$ find /tmp/out | wc -l
> 175
> 
> bash-4.2$ find /tmp/out | head
> /tmp/out
> /tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546
> /tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/
> sos_commands
> /tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/
> sos_commands/postgresql
> /tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/
> sos_commands/postgresql/pgdump.tar
> /tmp/out/sosreport-brq-setup.rhev.lab.eng.brq.redhat.com-20180220090546/
> sos_commands/postgresql/pgdump-scl-rh-postgresql95.tar
> /tmp/out/toc.dat
> /tmp/out/7428.dat
> /tmp/out/7430.dat
> /tmp/out/7432.dat

Thanks Jiri for the detailed debug, according with previous discussion with sos devel guys, we were looking for one single sql dump and not two sql dumps. Patch sent for review. 

Reference: https://github.com/oVirt/ovirt-log-collector/commit/a0dc1084ecac39043b8a5ea7c1681f985f9d73d6
Comment 10 Jiri Belka 2018-02-21 08:02:05 UTC 
> Thanks Jiri for the detailed debug, according with previous discussion with
> sos devel guys, we were looking for one single sql dump and not two sql
> dumps. Patch sent for review. 
> 
> Reference:
> https://github.com/oVirt/ovirt-log-collector/commit/
> a0dc1084ecac39043b8a5ea7c1681f985f9d73d6

false expectations have always lead humanity to disasters... you trust input, iiuc this has always been wrong approach in programming.
Comment 11 Douglas Schilling Landgraf 2018-02-21 14:50:44 UTC 
*** Bug 1539029 has been marked as a duplicate of this bug. ***
Comment 12 David Necpal 2018-02-23 16:29:50 UTC 
Verified on version ovirt-log-collector-analyzer-4.2.4-2.el7ev.noarch
Comment 13 Franta Kust 2019-05-16 13:07:57 UTC 
BZ<2>Jira Resync