Bug 1729652

This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Created attachment 1683641 [details]
simple cmake file that loads fftw

This bug also exists in Fedora 31. I'll test Fedora 32 later.

# install the fftw packages
> sudo dnf install fftw fftw-static fftw-devel

> cat /etc/fedora-release
Fedora release 31 (Thirty One)

# run cmake on the attached example
> cmake .
CMake Error at /usr/lib64/cmake/fftw3/FFTW3Config.cmake:13 (include):
  include could not find load file:

    /usr/lib64/cmake/fftw3/FFTW3LibraryDepends.cmake
Call Stack (most recent call first):
  CMakeLists.txt:5 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/XY/Documents/fedora-fftw-bug/CMakeFiles/CMakeOutput.log".

# the file is obviously missing
> ls /usr/lib64/cmake/fftw3
FFTW3Config.cmake	  FFTW3fConfigVersion.cmake  FFTW3qConfig.cmake
FFTW3ConfigVersion.cmake  FFTW3lConfig.cmake	     FFTW3qConfigVersion.cmake
FFTW3fConfig.cmake	  FFTW3lConfigVersion.cmake

# we are using  this file:
> rpm -qf /usr/lib64/cmake/fftw3/FFTW3Config.cmake
fftw-devel-3.3.8-6.fc31.x86_64

# and it is missing a dependency:
> cat /usr/lib64/cmake/fftw3/FFTW3Config.cmake
[...]
include ("${CMAKE_CURRENT_LIST_DIR}/FFTW3LibraryDepends.cmake")
[...]

I just confirmed that the same issue exists in Fedora 32:

# install the fftw packages
> sudo dnf install fftw fftw-devel

> cat /etc/fedora-release
Fedora release 32 (Thirty Two)

# run cmake on the attached example
> cmake .
[...]
CMake Error at /usr/lib64/cmake/fftw3/FFTW3Config.cmake:13 (include):
  include could not find load file:

    /usr/lib64/cmake/fftw3/FFTW3LibraryDepends.cmake
Call Stack (most recent call first):
  CMakeLists.txt:5 (find_package)


-- Configuring incomplete, errors occurred!
# the file is obviously missing
> ls /usr/lib64/cmake/fftw3
FFTW3Config.cmake	  FFTW3fConfigVersion.cmake  FFTW3qConfig.cmake
FFTW3ConfigVersion.cmake  FFTW3lConfig.cmake	     FFTW3qConfigVersion.cmake
FFTW3fConfig.cmake	  FFTW3lConfigVersion.cmake

# we are using  this file:
> rpm -qf /usr/lib64/cmake/fftw3/FFTW3Config.cmake
fftw-devel-3.3.8-7.fc32.x86_64

# and it is missing a dependency:
> cat /usr/lib64/cmake/fftw3/FFTW3Config.cmake
[...]
include ("${CMAKE_CURRENT_LIST_DIR}/FFTW3LibraryDepends.cmake")
[...]

This issue still exists in Fedora 33.

There is an upstream issue here:

"The file FFTW3LibraryDepends.cmake is missing"
https://github.com/FFTW/fftw3/issues/130

The problem is due to the fact that fftw can be compiled with either autotools or cmake, and but hen autotools is used (as in the fedora package), the CMake support files generated by the build are missing some files, namely FFTW3LibraryDepends.cmake. On the other hand, usable pkg-config files *are* generated by autotools.

Some packages which use cmake to find fftw3 as a dependency (indicated by the presence of FindFFTW3.cmake, the problem doesn't seem to occur for eigen with FindFFTW.cmake), see that a cmake module us available, 
ignore the pkg-config files, and proceed to fail with an error. 

= Possible solutions

- Change the fedora package to compile with cmake
- or, staying with autotools and including only the pkg-config files. This means omitting the
broken module in /usr/lib64/cmake/fftw3.

= Workaround for users

I've had success with the one library I needed to compile by simply deleting the /usr/lib64/cmake/fftw3 directory. Then the cmake  modules uses the pkg-config files instead and all is well.

Hi,

I'm not an expert in autotools, cmake, nor fftw.  Can someone provide actual repro steps so I can confirm something fixes the issue?

(In reply to andre.j.brand from comment #0)
> Steps to Reproduce:
> 1. Use the line find_package (FFTW3) in *any* CMakeLists.txt

Is "echo 'find_package (FFTW3)' > CMakeLists.txt" sufficient?  How should I invoke cmake?

Hi Conrad,

the repro steps with a minimal example are described in comment #3.

The attached CMakeLists.txt contains:

cmake_minimum_required(VERSION 3.15)
project(cmake-test-fftw)
find_package(FFTW3)

Then run "cmake ." in the folder where CMakeLists.txt is located.

Hi Conrad,

Regrading possible fixes, there are actually 3 scenarios:
1) autotools (gnu make) based project wishing to use fftw3 and relying on pkg-config - no issue

2) CMake based projects which include their own find module such as `cmake/FindFFTW3.cmake` - currently broken. But if `/usr/lib64/cmake/fftw3/` is removed (provided by fftw-devel), the module falls back to using pkg-config and cmake generation succeeds.

3) CMake based project relying on `find_package(FFTW3)` - currently broken, and requires the missing file(s) to fix (no fall-back to pkg-config).

Because of 3, the proper solution is probably to switch to a CMake build.

Thanks, Andre and Larry.

Unfortunately, I don't see an obvious way to build the MPI support we currently have with the FFTW CMake build.

We could still delete the broken /usr/lib64/cmake/fftw3 metadata.  But I also wonder how hard it would be to generate a correct version of the missing files.

The missing files are generated during CMake configuration -- not during the compilation step.
You could simply run cmake with each of the various configurations and store a copy of the files,
though I can't say how brittle this would be. 


Here's the resulting files of running various configurations:

CMakeFiles/Export/lib64/cmake/fftw3/FFTW3LibraryDepends-release.cmake
CMakeFiles/Export/lib64/cmake/fftw3/FFTW3LibraryDepends.cmake
CMakeFiles/Export/lib64/cmake/fftw3f/FFTW3LibraryDepends-release.cmake
CMakeFiles/Export/lib64/cmake/fftw3f/FFTW3LibraryDepends.cmake
CMakeFiles/Export/lib64/cmake/fftw3l/FFTW3LibraryDepends-release.cmake
CMakeFiles/Export/lib64/cmake/fftw3l/FFTW3LibraryDepends.cmake
FFTW3Config.cmake
FFTW3ConfigVersion.cmake
FFTW3LibraryDepends.cmake
FFTW3fConfig.cmake
FFTW3fConfigVersion.cmake
FFTW3lConfig.cmake
FFTW3lConfigVersion.cmake
FFTW3qConfig.cmake
FFTW3qConfigVersion.cmake

For now, simply dropping /usr/lib64/cmake/fftw3 should be trivial, and would improve matters for some (and perhaps quite a few) projects, without making things any worse for the others.

I would urge against postponing this until "one true fix" is achieved. There's no reason not to improve things later.

You mentioned you're not familiar with cmake. Be aware that it has an ncurses interface which you may find more convenient for experimentation - simply use "mkcd build; ccmake .." instead of "mkcd build; cmake ..".

This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This bug is still present in fc34. 

Conrad, would you kindly bump the bug version to prevent it from being auto-closed?

Created attachment 1779444 [details]
Updated spec file

Conrad, could you please review the updated spec file I've attached?

Created attachment 1779474 [details]
updated spec file v2

v2 of patch which uses directory names that (hopefully) follow CMake's conventions.

Conrad, Have you had time to look at the proposed patch? I think it's about time to resolve this issue.

Larry, I have not.

Conrad, it's been three weeks since I submitted a fix. What's the holdup?

Larry, it's a large patch and I haven't had time to review it.

> Larry, it's a large patch and I haven't had time to review it.

Can you estimate when this will be fixed? This has already been ignored for 
18 months, then, explaining the issue didn't produce any progress,
and now even providing a fix isn't helping. It's very frustrating.

Conrad, any news?

Hi,

We discussed this item in today's FESCo meeting to try to come up with some ideas to help resolve this issue.  I have read through the attached patch and comments above and have some questions:

1) The core issue seems to be the Fedora fftw package is missing .cmake files necessary for projects that use cmake *and* that need to use fftw.  Is that correct?

2) The upstream issue noted is that if you build fftw with cmake you get .cmake files and no pkg-config files, but if you build with autotools you get pkg-config files and no .cmake files.  Has this been fixed upstream yet or acknowledged upstream yet?

3) The patch includes a large refactoring of the Fedora fftw package build and how it is packaged, but that seems out of scope for the initial reported issue.  Is the refactoring work extra in this case?

Having hit issues like this before with source build systems, I propose the following:

* Convert the package over to build with cmake.  Fedora has plenty of spec file macros that make this easy.
* Handle the pkg-config scripts part manually, unless there is an available patch (or new release) upstream.  This ensures the Fedora fftw package retains the pkg-config scripts.
* Ensure the required .cmake files are included in the new package build.

I am happy to help with this work if that would be useful.

Thanks,
David Cantrell, representing myself and FESCo

Hi David,

Re: (1), to elaborate a little -- I think it actually includes *broken* cmake files.  Theoretically, some cmake-using projects could successfully fall back to pkgconfig if the broken cmake files were absent.  But we would rather just ship correct cmake files.

(2) I haven't followed the upstream issue closely, but it is still open and unresolved, suggesting to me that upstream has not fixed it yet.  I did not see if it was acknowledged officially or not.

(3) Yeah.

> Convert the package over to build with cmake.  Fedora has plenty of spec file macros that make this easy.

I'm not sure this is true.  The Fedora fftw spec build is pretty complicated and the cmake build doesn't necessarily support all of the stuff that has been added to the Fedora FFTW package.

> I am happy to help with this work if that would be useful.

Please, go ahead!

(1) It's actually that enough CMake support file are included to make CMake believe it can rely on that, and not including the files actually necessary for that support. The reason for the latter is that they are generated fftw's CMake build during a preliminary configuration stage (*not* during the full build), and the rpm build currently relies on the autotools for the build.

As a reminder:

a. I initially suggested a very easy fix. CMake has good support for using pkg-config files as a backend. The cmake-specific support files aren't strictly necessary.
We could simply drop the broken "/usr/lib64/cmake/fftw3" support files from the package and ask users to rely on CMake's support for pkg-config. This way, there's no need for disruptive change to the spec file. 

b. Alternatively, keep using autotools to build (I think CMake could do everything, but migrating would not be easy. As conrad said, it's not a simple package) but run only CMake's configuration step (per variant) to generate the missing files, then install them in the proper manner (individual directories) to the filesystem. This is friendlier towards cmake users, but requires much deeper changes to the spec file.

Either solution would be welcome.

(3) Some of it. But supporting cmake fully without actually building with cmake requires some footwork. Some refactoring to accommodate that felt appropriate.

(In reply to Conrad Meyer from comment #23)
> Hi David,
> 
> Re: (1), to elaborate a little -- I think it actually includes *broken*
> cmake files.  Theoretically, some cmake-using projects could successfully
> fall back to pkgconfig if the broken cmake files were absent.  But we would
> rather just ship correct cmake files.

Understood, thank you for the clarification.

> (2) I haven't followed the upstream issue closely, but it is still open and
> unresolved, suggesting to me that upstream has not fixed it yet.  I did not
> see if it was acknowledged officially or not.
> 
> (3) Yeah.
> 
> > Convert the package over to build with cmake.  Fedora has plenty of spec file macros that make this easy.
> 
> I'm not sure this is true.  The Fedora fftw spec build is pretty complicated
> and the cmake build doesn't necessarily support all of the stuff that has
> been added to the Fedora FFTW package.

That's reasonable.  I will defer to your and other reporters' expertise here.  I am aware of fftw and have used it as a build dependency, but do not have a deep understanding of its build options.

> > I am happy to help with this work if that would be useful.

In that case, can you add me a co-maintainer for the package?  You have administrative ownership of the package in Pagure.  My FAS account name is 'dcantrell'.

(In reply to Larry from comment #24)
> (1) It's actually that enough CMake support file are included to make CMake
> believe it can rely on that, and not including the files actually necessary
> for that support. The reason for the latter is that they are generated
> fftw's CMake build during a preliminary configuration stage (*not* during
> the full build), and the rpm build currently relies on the autotools for the
> build.
> 
> As a reminder:
> 
> a. I initially suggested a very easy fix. CMake has good support for using
> pkg-config files as a backend. The cmake-specific support files aren't
> strictly necessary.
> We could simply drop the broken "/usr/lib64/cmake/fftw3" support files from
> the package and ask users to rely on CMake's support for pkg-config. This
> way, there's no need for disruptive change to the spec file. 

This sounds like the best immediate step to fix things up for users.  I support this option as the first step.

> b. Alternatively, keep using autotools to build (I think CMake could do
> everything, but migrating would not be easy. As conrad said, it's not a
> simple package) but run only CMake's configuration step (per variant) to
> generate the missing files, then install them in the proper manner
> (individual directories) to the filesystem. This is friendlier towards cmake
> users, but requires much deeper changes to the spec file.

Given the unresolved nature of the upstream cmake work, I think it would be best to continue using autotools here rather than move to cmake now.  Unless someone wants to do the cmake work and submit a patch upstream and see it through that way.  Given that cmake supports the pkg-config files, simply removing the current cmake files from the fftw package and letting cmake rely on the pkg-config file(s) seems like the best option for us right now.

> Either solution would be welcome.
> 
> (3) Some of it. But supporting cmake fully without actually building with
> cmake requires some footwork. Some refactoring to accommodate that felt
> appropriate.

And that's reasonable.  I would like to see some improvements upstream for the this first rather than carrying that just in the Fedora package.  I do appreciate your work and help here.

So with this, I will wait to be added as a co-maintainer and then take this bug and drop the cmake files and do some local tests first and then submit it as an update.  My assumption is we need fixes for both Fedora and EPEL for this package, so I was going to do the work on all impacted branches.

Is anyone currently working to get the cmake changes upstreamed or is that an open item?  I think the best course of action for cleaning up the build process for fftw is to submit PRs upstream.  I saw https://github.com/FFTW/fftw3/pull/117 as well, which is from 2017 and adds supports for building with cmake and enabling more than one precision option at a time.  Seems like this is a direction worth pursuing to help clean up the build process.

> Given the unresolved nature of the upstream cmake work, I think it would be best to continue using 
> autotools here rather than move to cmake now. 
> [...]
> I would like to see some improvements upstream for the this first rather than carrying that just 
> in the Fedora package. 

To be clear, from their point of view, It's probaby a non-issue and arguably a strictly downstream packaging issue. 

From their point of viewe, the person who builds the library and the person who builds the application that depends on it are one and the same.  So why should they bend over backwards to support developers who insist on building fftw with autotools, but compiling against using cmake?

It's only in something like a downstream distro context that this scenario is even a thing.

> Is anyone currently working to get the cmake changes upstreamed or is that an open item?

I'm not. Again, I don't think they care. Perhaps more importantly, there's no no clear way to fix it upstream - you need files generated by cmake at configuration time. How would you do that with autotools in a way which isn't prone to breaking? I should point out that the generated files are internally defined by cmake as part of a stock macro (or somesuch), with unclear stability guarantees as to content. They are not something the fftw developers explicitly craft as part of their cmake build but failed to implement in their autotools build.  

They probably don't want the maintenance headache of having to replicate what cmake does internally in autotools, or requiring cmake to build with autotools, etc'. In that context, I can understand why they don't to deal with it. So I think it's up to downstream (or down to downstream... You know what I mean).

(In reply to David Cantrell from comment #25)
> In that case, can you add me a co-maintainer for the package?  You have
> administrative ownership of the package in Pagure.  My FAS account name is
> 'dcantrell'.

I believe I've added you now, please let me know if that didn't work.

Your proposed plan of action seems totally reasonable to me -- thank you!

> b. Alternatively, keep using autotools to build [...] but run only CMake's configuration step (per variant) 
> to generate the missing files, then install them in the proper manner (individual directories) to the 
> filesystem. This is friendlier towards cmake users, but requires much deeper changes to the spec file.

To be clear, what I described there is exactly what my patch does, and it's also the reason it is larger than you'd expect.

I'm still happy to stick with pkg-config only, but I wanted to make the patch's nature was understood.

> [Upstream] probably don't want [...] to require cmake when you build with autotools,

... but fedora packages easily can.

This message is a reminder that Fedora Linux 34 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '34'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 34 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07.

Fedora Linux 34 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release.

Thank you for reporting this bug and we are sorry it could not be fixed.