Bug 1403391

Summary:	[RFE] Have rpmbuild create the rpm files in parallel rather than one at a time
Product:	[Fedora] Fedora	Reporter:	William Cohen <wcohen>
Component:	rpm	Assignee:	Packaging Maintenance Team <packaging-team-maint>
Status:	CLOSED UPSTREAM	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	25	CC:	ignatenko, kardos.lubos, packaging-team-maint, pknirsch, pmatilai
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-05-23 09:58:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description William Cohen 2016-12-09 22:43:06 UTC

Description of problem:

When doing an "rpmbuild -ba" of a package the %make process takes advantage of the multiple processors on the machine.  However, towards the end of the rpmbuild when the rpmbuild starts writing out the rpms, it appears that only one processor is being used.  The lzma compression is computationally intensive and it looks like there would be some performance improvement by using multiple processor work on generating a separate rpm file in parallel.


Version-Release number of selected component (if applicable):

rpm-build-4.13.0-6.fc25.x86_64


How reproducible:

Every time


Steps to Reproduce:
1. yumdownloader --source systemtap
2. sudo yum-builddep ./systemtap*.src.rpm -y
3. in one window run "rpmbuild -ra ./systemtap*.src.rpm" and in another run "top"

Actual results:

Notice when generating the rpms that only one cpu is being used very heavily.

Expected results:

The  rpmbuild rpm packaging process uses multiple processors to complete the compress task in less wallclock time.


Additional info:

The code that cause the serial processing appears to be in the packageBinaries function in rpm/build/pack.c

There might be other parts of the rpmbuild process such as the strip of the debuginfo sections from binary files that could be also run in parallel.

Comment 1 Panu Matilainen 2017-05-23 09:58:20 UTC

Parallel XZ compression has been added upstream, however it's not going to be enabled by default as deltarpm requires bit-per-bit equal compression which parallel compression breaks, and there's a considerable tradeoff between speed and compression ratio involved so its not at all a clear-cut win-win situation even for that, especially for official distro builds.

Upstream find-debuginfo.sh also supports parallel processing now. Beyond that there are no immediate plans.

Closing UPSTREAM as that's where any further activity needs to happen, and parallel compression is unlikely to happen in Fedora because of the other side-effects.