Bug 2000138 - Generated filenames depend on undefined behaviour - doxygen output not reproducible
Summary: Generated filenames depend on undefined behaviour - doxygen output not reprod...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: doxygen
Version: 36
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Than Ngo
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1482815 1996429
TreeView+ depends on / blocked
 
Reported: 2021-09-01 13:39 UTC by Mattias Ellert
Modified: 2022-02-17 13:58 UTC (History)
4 users (show)

Fixed In Version: doxygen-1.9.4-0.20220217gite18f715.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-17 13:57:16 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github doxygen doxygen issues 8752 0 None None None 2021-09-01 13:39:55 UTC

Description Mattias Ellert 2021-09-01 13:39:55 UTC
Description of problem:

When building documentation with doxygen 1.9.2 the filenames for the directory dependencies generated here:

https://github.com/doxygen/doxygen/blob/8bbce64cdbaa155499fbabedf5343e9e1d34d378/src/dotdirdeps.cpp#L165

are not reproducible. The dirCount() value depends on the order in which the files are scanned, which depends on the std::filesystem::directory_iterator. This order is not deterministic.

https://en.cppreference.com/w/cpp/filesystem/directory_iterator

says: "directory_iterator is a LegacyInputIterator that iterates over the directory_entry elements of a directory (but does not visit the subdirectories). **The iteration order is unspecified**, except that each directory entry is visited only once. The special pathnames dot and dot-dot are skipped."

Since the dirCount() values are not guaranteed to be constant between runs, they can not be used in the generated documentation or in the filenames used to save it, or the result is not-reproducible.

This causes problems when building packages for Fedora, since packages are built for multiple architectures, and the build system (koji) rejects builds where the set of filenames in a non-architecture dependent (noarch) package is different when built on different architectures.

Here is an example: https://koji.fedoraproject.org/koji/taskinfo?taskID=74850762

The build is rejected because:

BuildError: The following noarch package built differently on different architectures: root-doc-6.24.04-1.fc36.noarch.rpm
rpmdiff output was:
removed     /usr/share/doc/root/html/dir_000586_000029.html
removed     /usr/share/doc/root/html/dir_000586_000034.html
removed     /usr/share/doc/root/html/dir_000586_000043.html
removed     /usr/share/doc/root/html/dir_000586_000056.html
removed     /usr/share/doc/root/html/dir_000586_000063.html
removed     /usr/share/doc/root/html/dir_000586_000066.html
removed     /usr/share/doc/root/html/dir_000586_000075.html
removed     /usr/share/doc/root/html/dir_000586_000344.html
added       /usr/share/doc/root/html/dir_000587_000029.html
added       /usr/share/doc/root/html/dir_000587_000034.html
added       /usr/share/doc/root/html/dir_000587_000043.html
added       /usr/share/doc/root/html/dir_000587_000056.html
added       /usr/share/doc/root/html/dir_000587_000063.html
added       /usr/share/doc/root/html/dir_000587_000066.html
added       /usr/share/doc/root/html/dir_000587_000075.html
added       /usr/share/doc/root/html/dir_000587_000344.html

Version-Release number of selected component (if applicable):

doxygen-1.9.2-1.fc36

How reproducible:

Randomly. But more common with large packages. s390x seems to be more likely to generate files with different filenames than the other architectures.

Steps to Reproduce:
1. Build a package what builds documentation using doxygen.
2. Does not happen always. More likely with large packages with many directories in the source tree.

Actual results:

non-reproducible builds depending on the undefined order of the directory scan.

Expected results:

Reproducible builds not depending on the undefined order of the directory scan.

Comment 1 Mattias Ellert 2021-09-13 14:54:22 UTC
This issue is fixed upstream.

A PR with the fix is available:
https://src.fedoraproject.org/rpms/doxygen/pull-request/10

Comment 2 Bojan Smojver 2021-09-22 18:31:40 UTC
I am guessing the new build didn't make it yet into koji builds or the fix doesn't work:

https://koji.fedoraproject.org/koji/taskinfo?taskID=76099482

I will check logs for more details.

Comment 3 Bojan Smojver 2021-09-22 18:55:42 UTC
Reopening, because it seems that even the latest version of doxygen in Rawhide has this problem:

https://kojipkgs.fedoraproject.org//work/tasks/9694/76099694/root.log
-----
DEBUG util.py:446:   doxygen                                x86_64  1:1.9.2-2.fc36                  build  4.6 M
-----

Comment 4 Bojan Smojver 2021-10-24 01:36:39 UTC
Just tried another build of libspf2 and that failed again:

https://koji.fedoraproject.org/koji/taskinfo?taskID=77732452

Do we know what in doxygen is causing this? Is there are fix for it?

Comment 5 Dan Čermák 2021-10-26 21:20:23 UTC
I also observe this issue with i3: http://koji.fedoraproject.org/koji/taskinfo?taskID=77529403

Could someone please look into this? I am currently blocked by this issue to update i3 in Rawhide.

Comment 6 Than Ngo 2021-10-27 08:20:52 UTC
I am working on this problem right now. It seems that the cause comes from package built on s390x,

You could perhaps remove "BuildArch: noarch" as a workaround until we find a proper solution.

Comment 7 Ben Cotton 2022-02-08 21:36:17 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 8 Than Ngo 2022-02-17 13:57:16 UTC
it is fixed in rawhide


Note You need to log in before you can comment on or make changes to this bug.