Bug 619733 - fedpkg local holds rpmbuild log
fedpkg local holds rpmbuild log
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: fedora-packager (Show other bugs)
13
All Linux
low Severity medium
: ---
: ---
Assigned To: Dennis Gilmore
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-30 06:56 EDT by Petr Pisar
Modified: 2013-01-10 01:06 EST (History)
6 users (show)

See Also:
Fixed In Version: fedora-packager-0.5.1.2-2.fc14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-17 01:36:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Do not redirect local rpmbuild output (1.38 KB, patch)
2010-07-30 09:28 EDT, Petr Pisar
no flags Details | Diff
Alternate patch, which redirects/captures stdout/stderr as desired (3.96 KB, patch)
2010-08-05 14:23 EDT, Dave Malcolm
no flags Details | Diff

  None (edit)
Description Petr Pisar 2010-07-30 06:56:11 EDT
fedora-packager-0.5.1.0-1.fc13.noarch

The `fedpkg local' command does not pass rpmbuild stdout and stderr to terminal. This is regression against the dist-cvs `make local' command.

In addition, the output is saved to .build*.log after rpmbuild termination. Thus there is no way how to watch rpmbuild progress.

More ever the stdout and stderr redirection is handled in wrong way:

        try:
            proc = subprocess.Popen(' '.join(cmd), stderr=subprocess.PIPE,
                                    stdout=subprocess.PIPE, shell=True)
            output, error = proc.communicate()
        except OSError, e:
            raise FedpkgError(e)
        outfile = open(os.path.join(self.path, '.build-%s-%s.log' % (self.ver,
                       self.rel)), 'w')
        outfile.writelines(output)
        log.info(output)
        if error:
            outfile.writelines(error)
            log.error(error)
        outfile.close()

As you can see, the stderr and stdout is saved sequentially making rpmbuild output splitted into stdout and stderr part. The log should be multiplex both streams in line order as they come from rpmbuild. Otherwise user cannot tell which command failed.

Other problem is the log file is created ex post. Let's say rpmbuild hangs, user interrupts the process (^C) to find reason in log file, and there is no log file because Python terminates before creating the file (due to SIGINT).

Proposed solution:

Read rpmbuild output as soon as possible and multiplex the stream lines into log file asynchronously. Also flushing the log file after each line or at least on python exit should be forced to preserve content on abort.

Also copying the streams to terminal (by default or when special fedpkg option given) should be implemented.

Apparently, one need to operate the log in separate thread or process.
Comment 1 Petr Pisar 2010-07-30 09:28:41 EDT
Created attachment 435553 [details]
Do not redirect local rpmbuild output

This patch removes annoying rpmbuild output redirect.
Comment 2 Dave Malcolm 2010-08-05 14:23:29 EDT
Created attachment 436934 [details]
Alternate patch, which redirects/captures stdout/stderr as desired

I wrote an implementation of this which used "tee" to capture to the logfile, and also emit to both stdout and stderr.

This merged rpmbuild's stdout and stderr as seen from fedpkg's (as one stdout), and didn't preserve the "log" usage, so I tried again.

The attached patch uses helper threads to process rpmbuild's stdout and stderr separately, redirecting them to a list of file-like objects.

It sends each through to the corresponding fedpkg stream, but also captures both into the logfile.

I also added a ToLog() method, and at first I added a ToLog(log.info) to the tee for stdout, and a ToLog(log.error) to the one for stderr.  However, each of these log streams seems to be set up to go to stdout/stderr of fedpkg, so it led to duplication of the data.

This (essentially) eliminates the annoying buffering, whilst capturing the logs, and allowing separate redirection from fedpkg if desired.

Tested lightly, letting builds run to completion, and hitting Ctrl-C.  Sometimes need to hit Ctrl-C several times to actually halt a build.  Ctrl-C-K in emacs kills it first time.

There may still be some race conditions in error handling.
Comment 3 Jesse Keating 2010-08-12 02:14:36 EDT
Hrm, so I fixed this a slightly different way, before I saw any of your code.  I think it does what you want, and it still has the log file, and ^c still seems to work.  It'll be in the next build.
Comment 4 Fedora Update System 2010-08-12 15:52:49 EDT
python-kitchen-0.2-0.1.a2.el5,fedora-packager-0.5.1.2-1.el5 has been submitted as an update for Fedora EPEL 5.
http://admin.fedoraproject.org/updates/python-kitchen-0.2-0.1.a2.el5,fedora-packager-0.5.1.2-1.el5
Comment 5 Fedora Update System 2010-08-12 15:56:28 EDT
fedora-packager-0.5.1.2-1.fc14 has been submitted as an update for Fedora 14.
http://admin.fedoraproject.org/updates/fedora-packager-0.5.1.2-1.fc14
Comment 6 Fedora Update System 2010-08-12 16:00:57 EDT
fedora-packager-0.5.1.2-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/fedora-packager-0.5.1.2-1.fc13
Comment 7 Fedora Update System 2010-08-12 16:06:50 EDT
fedora-packager-0.5.1.2-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/fedora-packager-0.5.1.2-1.fc12
Comment 8 Fedora Update System 2010-08-13 17:05:49 EDT
fedora-packager-0.5.1.2-2.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update fedora-packager'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/fedora-packager-0.5.1.2-2.fc13
Comment 9 Fedora Update System 2010-08-13 18:38:25 EDT
fedora-packager-0.5.1.2-2.el5, python-kitchen-0.2-0.1.a2.el5 has been pushed to the Fedora EPEL 5 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update fedora-packager python-kitchen'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/python-kitchen-0.2-0.1.a2.el5,fedora-packager-0.5.1.2-2.el5
Comment 10 Fedora Update System 2010-08-17 01:35:18 EDT
fedora-packager-0.5.1.2-2.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 11 Fedora Update System 2010-08-24 17:08:34 EDT
fedora-packager-0.5.1.2-2.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 12 Fedora Update System 2010-08-24 19:03:03 EDT
fedora-packager-0.5.1.2-2.el5, python-kitchen-0.2-0.1.a2.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 13 Fedora Update System 2010-08-24 21:18:49 EDT
fedora-packager-0.5.1.2-2.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.