Bug 1987408

Summary: community-mysql: FTBFS in Fedora-all: armv7hl KOJI builders out of resources
Product: [Fedora] Fedora Reporter: Fedora Release Engineering <releng>
Component: community-mysqlAssignee: Michal Schorm <mschorm>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: hhorak, ljavorsk, mmuzila, mschorm
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-04 18:46:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1927309, 1983926    
Attachments:
Description Flags
build.log
none
root.log
none
state.log none

Description Fedora Release Engineering 2021-07-29 14:44:36 UTC
community-mysql failed to build from source in Fedora rawhide/f35

https://koji.fedoraproject.org/koji/taskinfo?taskID=72333691


For details on the mass rebuild see:

https://fedoraproject.org/wiki/Fedora_35_Mass_Rebuild
Please fix community-mysql at your earliest convenience and set the bug's status to
ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks,
community-mysql will be orphaned. Before branching of Fedora 36,
community-mysql will be retired, if it still fails to build.

For more details on the FTBFS policy, please visit:
https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/

Comment 1 Fedora Release Engineering 2021-07-29 14:44:45 UTC
Created attachment 1807270 [details]
build.log

file build.log too big, will only attach last 32768 bytes

Comment 2 Fedora Release Engineering 2021-07-29 14:44:49 UTC
Created attachment 1807271 [details]
root.log

file root.log too big, will only attach last 32768 bytes

Comment 3 Fedora Release Engineering 2021-07-29 14:44:55 UTC
Created attachment 1807272 [details]
state.log

Comment 4 Michal Schorm 2021-07-30 10:48:27 UTC
From the Build log:
  " ... No space left on device ..."

---

Looks like KOJI builder was out of resources (either disk or memory)

That happens very rarely, though the more load on a builder, the bigger chance for it to happen.
The mass rebuild probably caused a high load.

I will re-submit the build and watch it and I'll post updates.

Comment 5 Lukas Javorsky 2021-07-30 11:23:13 UTC
I've tried the scratch-build to see if this problem is reproducible, and it looks like it is.
https://koji.fedoraproject.org/koji/taskinfo?taskID=72957080

It fails with the exact same error, which is kinda strange.
Also, it fails only on the 'armv7hl' arch only.

Comment 6 Michal Schorm 2021-08-03 06:15:15 UTC
I've tried few more builds.
The crash is not 100% in the same place.

One way or another, the package has been built successfully for F33..Rawhide before, so it is improbable that it would be caused by the 'community-mysql' package itself.

I've raised a question on the mailing list, whether the armv7hl KOJI builders configuration / resources limits changed recently.
  https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/5SCJR3NQYBKSSAADSWFTMMAERUGPNEHB/

Comment 7 Honza Horak 2021-08-04 06:27:33 UTC
This is a work-around used in case of RHSCL build of mariadb 10.5:

# to safe disk space, do not use ccache
export CCACHE_DISABLE=1

# do not use %%{?_smp_mflags} to safe memory and avoid build failure due to not enough resources
make -j2 VERBOSE=1

Comment 8 Michal Schorm 2021-08-04 08:07:07 UTC
(In reply to Honza Horak from comment #7)
> This is a work-around used in case of ...

I've also tried to reduce the debug info, which resulted in a successful build.

However from the reply to the mail thread by Kevin Fenzi, I understand that I was correct that the issue is on the side of the ARM builders.
I'll wait for his fix.

Comment 9 Michal Schorm 2021-08-04 18:46:47 UTC
> I've raised a question on the mailing list, whether the armv7hl KOJI
> builders configuration / resources limits changed recently.
>  
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> thread/5SCJR3NQYBKSSAADSWFTMMAERUGPNEHB/

Kevin Fenzi fixed and rebooted the ARM KOJI builders.
I made a successful Rawhide build.

https://koji.fedoraproject.org/koji/taskinfo?taskID=73279974