Bug 2045255

Summary: cjdns: FTBFS in Fedora rawhide/f36
Product: [Fedora] Fedora Reporter: Fedora Release Engineering <releng>
Component: cjdnsAssignee: Stuart D Gathman <stuart>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 36CC: dan, decathorpe, fzatlouk, mhroncok, stuart, thrnciar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: cjdns-21.1-8.fc37 cjdns-21.1-8.fc36 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-25 03:42:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2121490, 1991164    
Bug Blocks: 2117176, 2168842, 1992484, 2009540, 2016048, 2045102, 2045109    
Attachments:
Description Flags
build.log
none
root.log
none
state.log none

Description Fedora Release Engineering 2022-01-25 16:15:07 UTC
cjdns failed to build from source in Fedora rawhide/f36

https://koji.fedoraproject.org/koji/taskinfo?taskID=81770164


For details on the mass rebuild see:

https://fedoraproject.org/wiki/Fedora_36_Mass_Rebuild
Please fix cjdns at your earliest convenience and set the bug's status to
ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks,
cjdns will be orphaned. Before branching of Fedora 37,
cjdns will be retired, if it still fails to build.

For more details on the FTBFS policy, please visit:
https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/

Comment 1 Fedora Release Engineering 2022-01-25 16:15:11 UTC
Created attachment 1853797 [details]
build.log

file build.log too big, will only attach last 32768 bytes

Comment 2 Fedora Release Engineering 2022-01-25 16:15:13 UTC
Created attachment 1853798 [details]
root.log

file root.log too big, will only attach last 32768 bytes

Comment 3 Fedora Release Engineering 2022-01-25 16:15:15 UTC
Created attachment 1853799 [details]
state.log

Comment 4 Ben Cotton 2022-02-08 20:43:14 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 5 Tomáš Hrnčiar 2022-04-21 14:14:42 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 37.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2022-03-22).

A week before the mass branching of Fedora 38 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 36 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-38/f-38-key-tasks.html

Comment 6 Stuart D Gathman 2022-04-25 23:13:04 UTC
I think I can look at this now.  Hopefully just new compiler versions.

Comment 7 Miro Hrončok 2022-06-20 10:14:19 UTC
*** Bug 2098683 has been marked as a duplicate of this bug. ***

Comment 8 Miro Hrončok 2022-08-25 09:41:53 UTC
This Bugzilla was ASSIGNED on 2022-04-26 but nothing happened. We are past Fedora 37 Beta Freeze and this package does not even install. What is the plan wrt this package? Can it please be removed from the distribution until this is solved?

Comment 9 Stuart D Gathman 2022-08-25 19:34:53 UTC
The plan was to upgrade to the new releases - which depend on rust-sodiumoxide. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2121490 I have been working on the dependencies for that.  The new releases build and install on my dev system - which has the rust packages built and in my private repo.

I could work a little on why the old release no longer builds to unstick the works.  Is that a better idea?

Comment 10 Miro Hrončok 2022-08-29 10:50:58 UTC
> Is that a better idea?

I don't know. I guess either works. Thanks for the update.

Comment 11 Stuart D Gathman 2022-09-14 14:50:53 UTC
I have some backported patches for the 21.1 C code.  It now builds running build steps in mock container from terminal, BUT not from rpmbuild -bc.  I think I should go ahead and push those changes for f37 (but not rawhide).  I can always merge from rawhide if the rust stuff for 21.2 becomes available.

What about vendoring the rust-sodiumoxide  packages in cjdns until they are ready?

Comment 12 Miro Hrončok 2022-09-14 14:57:50 UTC
Thanks. I'd just push the workaround to f37 and rawhide both and revert it later when rust-sodiumoxide is available.

Comment 13 Stuart D Gathman 2022-09-27 02:34:35 UTC
Sigh.  Now the bundle libuv won't build with gcc-12.  I could try to patch for current libuv again, but that didn't work well last time (compiled, but the event handling doesn't seem to be compatible).

Comment 14 Miro Hrončok 2022-09-29 18:18:14 UTC
The Fedora Engineering Steering Committee has approved [1] to retire the remaining non-installable packages still depending on Python 3.10 from Fedora 37 and Rawhide 1-2 days before the final freeze. We plan to retire this package this Sunday: 2022-10-02. If you wish to fix this package for time for Fedora 37 GA, please request a freeze exception [2].

[1] https://pagure.io/fesco/issue/2876
[2] https://qa.fedoraproject.org/blockerbugs/propose_bug

Comment 15 Stuart D Gathman 2022-10-02 00:37:01 UTC
I have Caleb, the upstream author, working on the build problem.  I don't want to hold up the freeze, but if there is anyway to avoid retiring cjdns, I would appreciate it.  I am getting close, I promise!

Comment 16 Fedora Blocker Bugs Application 2022-10-02 00:48:44 UTC
Proposed as a Freeze Exception for 38-beta by Fedora user sdgathman using the blocker tracking app because:

 I have Caleb, the upsteam author, help me debug the build problem.  We are making progress.

Comment 17 Miro Hrončok 2022-10-03 09:20:14 UTC
No info needed from me.

Comment 18 Miro Hrončok 2022-10-03 09:21:15 UTC
But changing the exception request from Feodra 38 Beta to Fedora 37 Final.

Comment 19 František Zatloukal 2022-10-03 17:28:21 UTC
Discussed during the 2022-10-03 blocker review meeting: [1]

The decision to classify this bug as an AcceptedFreezeException was made:

"The decision to classify this bug as an "AcceptedFreezeException (Final)" was made as it is a noticeable issue that cannot be fixed with an update."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-10-03/f37-blocker-review.2022-10-03-16.00.log.txt

Comment 20 Miro Hrončok 2022-10-04 11:10:08 UTC
Do you have a work-in-progress state of the spec file I can try poking?

Comment 21 Stuart D Gathman 2022-10-06 22:58:22 UTC
Pushed to rawhide.  You'll need the rust dependencies, which are all straight rust2rpm - but I have review requests open for them.  I am using locally built rust packages at https://gathman.org/linux/f37/  The gathmnan-release package has repo.  (OT: I am not sure if a local repo package should stick keys in /usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-gathman - so you'll have to copy/link it there for use with mock.)  I am building for f37.

Comment 22 Stuart D Gathman 2022-10-06 23:00:24 UTC
I have a f37 vm set up for Caleb - I'm sure you have your own, but can supply if needed.

Comment 23 Miro Hrončok 2022-10-07 09:40:16 UTC
How can I help to move the package reviews forward? Fabio seems to be on top of things there.

Comment 24 Stuart D Gathman 2022-10-10 15:14:19 UTC
Fabio says I've somehow "opted out" of %rpmautospec for rust-libsodium-sys.  I will rerun rust2rpm now and see if it works any different.

Comment 25 Fabio Valentini 2022-10-11 09:21:41 UTC
I have approved rust-libsodium-sys and will push some fixes with a follow-up commit once it's imported.
The other three pending review requests are still waiting for input from their submitters.

Comment 26 Miro Hrončok 2022-10-11 11:31:48 UTC
At this point, I'd feel much safer if we retired the package for now before F37 repos are frozen, and reintroduced it with a bodhi update once ready. Is that OK? It creates a bit of a mess in dist git history, but other than that should be quite safe.

Comment 27 Fabio Valentini 2022-10-11 11:44:49 UTC
Note that F37 repos are already frozen since last week, and that you will need releng intervention to retire packages.

Comment 28 Miro Hrončok 2022-10-11 11:46:57 UTC
Yes, but this has a freeze exception. The problem is we don't know what is the real deadline.

Comment 29 Stuart D Gathman 2022-10-11 16:12:20 UTC
The f35 package installs and runs in f37 and f38.  The python subpackages can be built separately as noarch with Latest Python.  Does that help?

Comment 30 Stuart D Gathman 2022-10-11 16:17:32 UTC
Recent findings from Caleb:

     Only happens in your build env
     And only happens if it's built BY the Rust build-script, invoking nodejs directly is fine
     Happens with or without your CFLAGS
     But if your CFLAGS are duplicated many times, the error message is also duplicated

We are trying clang.  It looks like I have to master Rust enough to modify build.rs

Comment 31 Miro Hrončok 2022-10-11 18:21:43 UTC
(In reply to Stuart D Gathman from comment #29)
> The f35 package installs and runs in f37 and f38.  The python subpackages
> can be built separately as noarch with Latest Python.  Does that help?

Not really, as long as the old Python subpackages are in the repository (and we cannot remove them without removing all of them).

Comment 32 Miro Hrončok 2022-10-27 18:27:28 UTC
The target date for Fedora 37 final shifted to 15 November.

Comment 33 Miro Hrončok 2022-11-08 16:39:34 UTC
Do you still anticipate getting the rust dependencies in Fedora in time for Fedora 37 release?

Comment 34 Stuart D Gathman 2022-11-09 22:12:54 UTC
(In reply to Miro Hrončok from comment #33)
> Do you still anticipate getting the rust dependencies in Fedora in time for
> Fedora 37 release?

Doesn't look like it.  If package gets retired - what happens to the EPEL packages (which still build)?
I am going to make one more try with same upstream release as f35 which doesn't have the Rust dependencies.  I can't remember if I tried just removing -Werror from build scripts.

Comment 35 Miro Hrončok 2022-11-10 11:12:28 UTC
It would only be retired in f37 and f38. If it is in EPEL, it stays there.

Comment 36 Stuart D Gathman 2023-01-14 22:50:30 UTC
I can build cjdns in f38,f37,f36 by this procedure:
mock -r fedora-3X-x86_64 cjdns-21.1-7.fc38.src.rpm
... fails on a randomly selected Linking step.  Build script output is "eaten" by rpmbuild and sent somewhere, maybe /dev/null

mock -r fedora-3X-x86_64 --shell
su - mockbuild
cd build/BUILD/cjdns-21.1
NO_TEST=1  CJDNS_RELEASE_VERSION="cjdns=21.1-7.fc38" ./do
... builds correctly with build script output visible

cd ../../SPECS
rpmbuild -bi --short-circuit cjdns.spec
rpmbuild -bb --short-circuit cjdns.spec

Now I can add the rpms to my personal repo.  But that doesn't fix the FTBFS for Fedora.  What is going on with rpmbuild?  Why did it start eating the output of the build script in f36?  I suspect it has something to do with nodejs, as that changed with f36 and the build script is nodejs.

Comment 37 Dan Horák 2023-01-19 16:51:05 UTC
- one part of the problem is missing BuildRequires: gcc-c++
- also for debugging purposes it would be useful to switch off "parallel build", so it's easier to correlate possible error messages to what is going on

Comment 38 Stuart D Gathman 2023-01-27 18:31:11 UTC
Had BR g++.  And changing to gcc-c++ doesn't change anything.  Yes, the issue seems to be related to parallel build - but I have no clue how to turn off for make.js.

Comment 39 Dan Horák 2023-01-29 15:00:53 UTC
https://github.com/cjdelisle/cjdns/blob/master/node_build/builder.js#L786 contains the logic to set the parallelism. I would try replacing it with "1" ...

Comment 40 Stuart D Gathman 2023-01-30 15:54:50 UTC
Thanks.  With that patch, it now consistently fails on cjdroute2.c.  This is much more helpful.  The other reported failing files were red herrings, arguably an error reporting bug.  I am now trying the lastest version on github.

Comment 41 Stuart D Gathman 2023-01-31 01:21:40 UTC
Latest version adds at least 9 more rust crypto dependencies not packaged for Fedora.  :-(

Comment 42 Fedora Fails To Install 2023-02-08 09:40:24 UTC
Hello,

Please note that this comment was generated automatically by https://pagure.io/releng/blob/main/f/scripts/ftbfs-fti/follow-policy.py
If you feel that this output has mistakes, please open an issue at https://pagure.io/releng/

All subpackages of a package against which this bug was filled are now installable or removed from Fedora 38.

Thanks for taking care of it!

Comment 43 Fedora Update System 2023-02-16 00:47:47 UTC
FEDORA-2023-a36f5521d0 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a36f5521d0

Comment 44 Fedora Update System 2023-02-17 02:50:42 UTC
FEDORA-2023-a36f5521d0 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a36f5521d0`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a36f5521d0

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 45 Fedora Update System 2023-02-17 03:03:10 UTC
FEDORA-2023-353bcc3d35 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-353bcc3d35`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-353bcc3d35

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 46 Fedora Update System 2023-02-25 03:42:28 UTC
FEDORA-2023-353bcc3d35 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 47 Fedora Update System 2023-02-25 04:00:39 UTC
FEDORA-2023-a36f5521d0 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.