Bug 726121

More on this

Jul 27 17:13:20 dhcp-25-35 abrtd: Your sosreport has been generated and saved in:
Jul 27 17:13:20 dhcp-25-35 abrtd: /var/spool/abrt/ccpp-2011-07-27-17:13:13-913 sosreport-dhcp-25-35-20110727171320-1025.tar.xz

is confusing since sosreport's tarball is (within abrt) always saved to $_PATH/sosreport.tar.xz.

Does this happen when you uncomment this in abrt_event.conf?

# Example: if you want to save sosreport immediately at the moment of a crash:
# (alternatively, you can add similar command to EVENT=analyze_foo's,
# if you would rather perform this collection later):
#EVENT=post-create
        nice sosreport --tmp-dir "$DUMP_DIR" --batch \
                --only=anaconda --only=bootloader --only=devicemapper \
                --only=filesys --only=hardware --only=kernel --only=libraries \
                --only=memory --only=networking --only=nfsserver --only=pam \
                --only=process --only=rpm -k rpm.rpmva=off --only=ssh \
                --only=startup --only=yum --only=general --only=x11 \
        && {
                rm sosreport*.md5
                mv sosreport*.tar.bz2 sosreport.tar.bz2
                mv sosreport*.tar.xz sosreport.tar.xz
                true
        } 2>/dev/null

If yes, try this instead:

EVENT=post-create
        nice sosreport --tmp-dir "$DUMP_DIR" --batch \
                --only=anaconda --only=bootloader --only=devicemapper \
                --only=filesys --only=hardware --only=kernel --only=libraries \
                --only=memory --only=networking --only=nfsserver --only=pam \
                --only=process --only=rpm -k rpm.rpmva=off --only=ssh \
                --only=startup --only=yum --only=general --only=x11 \
                >sosreport.log 2>&1
        && {
                rm sosreport.log
                rm sosreport*.md5
                mv sosreport*.tar.bz2 sosreport.tar.bz2
                mv sosreport*.tar.xz sosreport.tar.xz
                exit 0
        } 2>/dev/null
        # Error in sosreport run. Let user see the problem.
        echo "sosreport run failed, log follows:"
        # sosreport prints many useless empty lines, nuke them:
        # it looks awful in syslog otherwise.
        cat sosreport.log | sed 's/  *$//' | grep -v '^$'
        rm sosreport.log
        exit 1

(In reply to comment #4)
> If yes, try this instead:
> 
> EVENT=post-create
>         nice sosreport --tmp-dir "$DUMP_DIR" --batch \
[...]
>                 >sosreport.log 2>&1
[...]

Don't forget to add '\' at the end of the line, otherwise it fails with:

Aug 15 17:04:30 dhcp-25-35 abrtd: /bin/sh: -c: line 7: syntax error near unexpected token `&&'
Aug 15 17:04:30 dhcp-25-35 abrtd: /bin/sh: -c: line 7: `        && {'


Second problem is that, when I kill `sosreport' it spits the sosreport.log but each line is there twice:

Aug 15 17:12:12 dhcp-25-35 abrt[16887]: saved core dump of pid 16886 (/bin/sleep) to /var/spool/abrt/ccpp-2011-08-15-17:12:11-16886.new/coredump (319488 bytes)
Aug 15 17:12:12 dhcp-25-35 abrtd: Directory 'ccpp-2011-08-15-17:12:11-16886' creation detected
Aug 15 17:12:13 dhcp-25-35 abrtd: /bin/sh: line 13: 16895 Killed                  nice sosreport --tmp-dir "$DUMP_DIR" --batch --only=anaconda --only=bootloader --only=devicemapper --only=filesys --only=hardware --only=kernel --only=libraries --only=memory --only=networking --only=nfsserver --only=pam --only=process --only=rpm -k rpm.rpmva=off --only=ssh --only=startup --only=yum --only=general --only=x11 > sosreport.log 2>&1
Aug 15 17:12:13 dhcp-25-35 abrtd: sosreport run failed, log follows:
Aug 15 17:12:13 dhcp-25-35 abrtd: sosreport (version 2.2)
Aug 15 17:12:13 dhcp-25-35 abrtd: This utility will collect some detailed  information about the
Aug 15 17:12:13 dhcp-25-35 abrtd: This utility will collect some detailed  information about the
Aug 15 17:12:13 dhcp-25-35 abrtd: hardware and setup of your Red Hat Enterprise Linux system.
Aug 15 17:12:13 dhcp-25-35 abrtd: The information is collected and an archive is  packaged under
Aug 15 17:12:13 dhcp-25-35 abrtd: The information is collected and an archive is  packaged under
Aug 15 17:12:13 dhcp-25-35 abrtd: /tmp, which you can send to a support representative.
Aug 15 17:12:13 dhcp-25-35 abrtd: /tmp, which you can send to a support representative.
Aug 15 17:12:13 dhcp-25-35 abrtd: Red Hat Enterprise Linux will use this information for diagnostic purposes ONLY
Aug 15 17:12:13 dhcp-25-35 abrtd: and it will be considered confidential information.
Aug 15 17:12:13 dhcp-25-35 abrtd: This process may take a while to complete.
Aug 15 17:12:13 dhcp-25-35 abrtd: This process may take a while to complete.
Aug 15 17:12:13 dhcp-25-35 abrtd: No changes will be made to your system.
Aug 15 17:12:13 dhcp-25-35 abrtd:   Running plugins. Please wait ...
Aug 15 17:12:13 dhcp-25-35 abrtd: #015  Completed [1/18] ...      #015  Completed [2/18] ...      #015  Completed [3/18] ...
Aug 15 17:12:13 dhcp-25-35 abrtd:   Running plugins. Please wait ...
Aug 15 17:12:13 dhcp-25-35 abrtd: #015  Completed [1/18] ...      #015  Completed [2/18] ...      #015  Completed [3/18] ...
Aug 15 17:12:13 dhcp-25-35 abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-08-15-17:12:11-16886 (res:2), deleting


Not sure why is that so.



As a small RFE: since sosreport run may take a while, I'd ask you to echo that sosreport is being started, e.g.:

EVENT=post-create
        echo "Starting sosreport..."
        nice sosreport --tmp-dir "$DUMP_DIR" --batch \

Should be 
       >sosreport.log 2>&1 \

(a slash should be added at the end of that line).

Are these lines duplicated in sosreport.log, or only in the syslog?

(In reply to comment #7)
> Are these lines duplicated in sosreport.log, or only in the syslog?

I had `tailf' on /var/log/messages, so syslog. When I backup sosreport.log by that hook to /tmp I can see it's OK -- no double lines.

Can't reproduce it. I uncommented sosreport event in abrt_event.conf and amde it artificially "fail" by adding a line with "false" cmd here:

                >sosreport.log 2>&1 \
                && false \
        && {
                rm sosreport.log

and my syslog looks like this:

abrtd: sosreport run failed with exit code 1, log follows:
abrtd: sosreport (version 2.1)
abrtd: This utility will collect some detailed  information about the
abrtd: hardware and setup of your Fedora system.
abrtd: The information is collected and an archive is  packaged under
abrtd: /tmp, which you can send to a support representative.
abrtd: Fedora will use this information for diagnostic purposes ONLY
abrtd: and it will be considered confidential information.
abrtd: This process may take a while to complete.
abrtd: No changes will be made to your system.
abrtd:   Running plugins. Please wait ...
abrtd: ^M  Completed [1/18] ...      ^M  Completed [2/18] ...      ^M  Completed [3/18] ...      ^M  Completed [4/18] ...      ^M  Completed [5/18] ...      ^M  Completed [6/
abrtd: Creating compressed archive...
abrtd: Your sosreport has been generated and saved in:
abrtd:   /var/spool/abrt/ccpp-2011-08-26-13:42:52-21543/sosreport-dhcp-25-63-20110826134256-697c.tar.xz
abrtd: The md5sum is: dfce63c777d3194e4bad2c24bc65697c
abrtd: Please send this file to your support representative.
abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2011-08-26-13:42:52-21543 (res:2), deleting
abrtd: Directory 'ccpp-2011-08-26-13:42:52-21563' creation detected

IOW: no duplicate lines.


One theory I have is that you killed sosreport via "killall sosreport", but you might have _two_ sosreports running, they got killed simultaneously, and got their log lines interspersed in syslog. This might explain what you see, but it's not very likely...

Michal, if you are able to reliably reproduce this, ping me on IRC and we can debug it in real time.

Unable to reproduce. Core issue should be fixed. Thanks. Add it to erratum, please.

Created attachment 528221 [details]
Proposed patch

Indeed, this fix didn't make it to RHEL6 build.
See attached patch with the fix.

Yeah, that's the patch we are interested in.

Applied to git rhel6 branch:

commit 3f6e9a1161fd2fa21a6018bd617fb0c9d2957926
Author: Denys Vlasenko <dvlasenk>
Date:   Tue Oct 18 17:47:14 2011 +0200

    Fix sosreport example to not emit gobs of output

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1598.html