Bug 2000138

Summary: Generated filenames depend on undefined behaviour - doxygen output not reproducible
Product: [Fedora] Fedora Reporter: Mattias Ellert <mattias.ellert>
Component: doxygenAssignee: Than Ngo <than>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 36CC: bojan, dan.cermak, fweimer, than
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: doxygen-1.9.4-0.20220217gite18f715.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-02-17 13:57:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1482815, 1996429    

Description Mattias Ellert 2021-09-01 13:39:55 UTC
Description of problem:

When building documentation with doxygen 1.9.2 the filenames for the directory dependencies generated here:

https://github.com/doxygen/doxygen/blob/8bbce64cdbaa155499fbabedf5343e9e1d34d378/src/dotdirdeps.cpp#L165

are not reproducible. The dirCount() value depends on the order in which the files are scanned, which depends on the std::filesystem::directory_iterator. This order is not deterministic.

https://en.cppreference.com/w/cpp/filesystem/directory_iterator

says: "directory_iterator is a LegacyInputIterator that iterates over the directory_entry elements of a directory (but does not visit the subdirectories). **The iteration order is unspecified**, except that each directory entry is visited only once. The special pathnames dot and dot-dot are skipped."

Since the dirCount() values are not guaranteed to be constant between runs, they can not be used in the generated documentation or in the filenames used to save it, or the result is not-reproducible.

This causes problems when building packages for Fedora, since packages are built for multiple architectures, and the build system (koji) rejects builds where the set of filenames in a non-architecture dependent (noarch) package is different when built on different architectures.

Here is an example: https://koji.fedoraproject.org/koji/taskinfo?taskID=74850762

The build is rejected because:

BuildError: The following noarch package built differently on different architectures: root-doc-6.24.04-1.fc36.noarch.rpm
rpmdiff output was:
removed     /usr/share/doc/root/html/dir_000586_000029.html
removed     /usr/share/doc/root/html/dir_000586_000034.html
removed     /usr/share/doc/root/html/dir_000586_000043.html
removed     /usr/share/doc/root/html/dir_000586_000056.html
removed     /usr/share/doc/root/html/dir_000586_000063.html
removed     /usr/share/doc/root/html/dir_000586_000066.html
removed     /usr/share/doc/root/html/dir_000586_000075.html
removed     /usr/share/doc/root/html/dir_000586_000344.html
added       /usr/share/doc/root/html/dir_000587_000029.html
added       /usr/share/doc/root/html/dir_000587_000034.html
added       /usr/share/doc/root/html/dir_000587_000043.html
added       /usr/share/doc/root/html/dir_000587_000056.html
added       /usr/share/doc/root/html/dir_000587_000063.html
added       /usr/share/doc/root/html/dir_000587_000066.html
added       /usr/share/doc/root/html/dir_000587_000075.html
added       /usr/share/doc/root/html/dir_000587_000344.html

Version-Release number of selected component (if applicable):

doxygen-1.9.2-1.fc36

How reproducible:

Randomly. But more common with large packages. s390x seems to be more likely to generate files with different filenames than the other architectures.

Steps to Reproduce:
1. Build a package what builds documentation using doxygen.
2. Does not happen always. More likely with large packages with many directories in the source tree.

Actual results:

non-reproducible builds depending on the undefined order of the directory scan.

Expected results:

Reproducible builds not depending on the undefined order of the directory scan.

Comment 1 Mattias Ellert 2021-09-13 14:54:22 UTC
This issue is fixed upstream.

A PR with the fix is available:
https://src.fedoraproject.org/rpms/doxygen/pull-request/10

Comment 2 Bojan Smojver 2021-09-22 18:31:40 UTC
I am guessing the new build didn't make it yet into koji builds or the fix doesn't work:

https://koji.fedoraproject.org/koji/taskinfo?taskID=76099482

I will check logs for more details.

Comment 3 Bojan Smojver 2021-09-22 18:55:42 UTC
Reopening, because it seems that even the latest version of doxygen in Rawhide has this problem:

https://kojipkgs.fedoraproject.org//work/tasks/9694/76099694/root.log
-----
DEBUG util.py:446:   doxygen                                x86_64  1:1.9.2-2.fc36                  build  4.6 M
-----

Comment 4 Bojan Smojver 2021-10-24 01:36:39 UTC
Just tried another build of libspf2 and that failed again:

https://koji.fedoraproject.org/koji/taskinfo?taskID=77732452

Do we know what in doxygen is causing this? Is there are fix for it?

Comment 5 Dan Čermák 2021-10-26 21:20:23 UTC
I also observe this issue with i3: http://koji.fedoraproject.org/koji/taskinfo?taskID=77529403

Could someone please look into this? I am currently blocked by this issue to update i3 in Rawhide.

Comment 6 Than Ngo 2021-10-27 08:20:52 UTC
I am working on this problem right now. It seems that the cause comes from package built on s390x,

You could perhaps remove "BuildArch: noarch" as a workaround until we find a proper solution.

Comment 7 Ben Cotton 2022-02-08 21:36:17 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 8 Than Ngo 2022-02-17 13:57:16 UTC
it is fixed in rawhide