oops, the example *should* be:
$ pkg-config --libs xmu

Created attachment 146406 [details]
update to upstream patch to fixup 'make check' behavior as well

A can confirm patch works as advertised.

I'll wait a bit for Tolleef to make up his mind on the upstream bug. He said
that he would have time for pkg-config bugs in a week.

I think you are vastly underestimating the severity of this bug.

* This bug basically voids Requires.Private.

* it forces developers and packagers to pollute configure scripts, rpm.spec and
build scripts with avoidable (actually bogus) hacks.

In the example above, one will have to add xext to pkg-config checks in
configure-scripts, and will have to add "BuildRequires: libXext-devel" to specs
(otherwise mock will not work).

* it breaks previously (before Requires.Private had been introduced) functional
pkg-config deps and configure scripts.
Before Requires.Private, these deps had been Requires in *.pc, therefore they
had been automatically taken into account.


Therefore, I am making this a review blocker for fc7 also would recommend you to
fix it on older Fedoras, too.

Making this a "review blocker" is not going to force my hand. 
What do you want to do next, drop pkgconfig from fedora ?
The last thing I want is a pkgconfig in fedora that diverges from upstream in
how it treats dependencies. 

(In reply to comment #6)
> Making this a "review blocker" is not going to force my hand.
Your decision - and fault

> What do you want to do next, drop pkgconfig from fedora ?
You'd better fix the bugs.

> The last thing I want is a pkgconfig in fedora that diverges from upstream in
> how it treats dependencies. 
If upstream stuff is nonfunctional, you will not be able to avoid to apply
patches and to diverge from upstream - This is the natural movement of all
packages and should be natural to any maintainer (You already do so with other
patches in pkgconfig)

In this particular case, an alternative would be to replace all
Requires.Private: from all *.pc's - but, you don't really want this, don't you?

> Your decision - and fault


Oh great. So I was wrong to thing that it was Rex fault to tie a package review
to an issue that is totally unrelated to packaging ?

*** Bug 215942 has been marked as a duplicate of this bug. ***

Fixed in 0.21-5.fc7

I'm sorry, but this fix is broken. The vanilla pkg-config behavior for Requires.private is correct, and this just makes it act like Libs.private, which is wrong because it breaks Cflags handling.

In the example given above, rpm/yum would not allow you to remove libXext-devel if libXmu-devel requires pkgconfig(xext). It would do that if the pkg-config --print-requires patch also walked Requires.private. I opened bug 426106 that has patches to fix that. After all, you actually need the headers from libXext-devel installed even if you're not linking to libXext.a. So, just making pkg-config skip Requires.private unless --static is used is wrong.

The issue of whether the .pc file should be required to be installed under certain conditions should be handled separately. That's what the freedesktop bug is about. Incidentally, I have more patches to address that issue, but that's orthogonal to this.

It is certainly hard to argue about 'correctness' of totally unspecified and undocumented features like Requires.private. 

If upstream comes up with some specification for Requires.private and a matching implementation, that would be great.

But there is a concrete specification and implementation.

Requires: Recursively fetch Cflags and Libs from listed dependencies. Recursively fetch Libs.private if --static is specified.

Requires.private: Recursively fetch Cflags from listed dependencies. Recursively fetch Libs and Libs.private if --static is specified.

This is what vanilla pkg-config does. What pkg-config doesn't do well is handle the non-existence of .pc files when they aren't needed. In the fdo bug, the reporter is talking about --exists. We can probably agree that pkg-config shouldn't need to recursively search for listed Requires* because we only want to know about the existence of the .pc metadata for the requested package.

In this bug, the problem is that pkg-config keeps searching for .pc files for listed Requires.private even though the user has requested a case where they're not needed: --libs. It would be better if pkg-config didn't search for the listed .pc file in this case, but having it act like Libs.private is wrong because the Cflags are always needed. Another mode is needed than ignore_private_libs to handle that correctly. In the fdo bug, I have a patch which does just that, adding a ignore_requires_private flag:

https://bugs.freedesktop.org/attachment.cgi?id=13197

What I'm proposing is to punt on that issue for now by making rpm add autoreqs for packages listed in Requires and Requires.private. After all, you do need the -devel package for Requires.private packages installed to actually do anything useful because you need the headers and .so links. I.e., libXmu-devel should require libXext-devel since you actually need the headers and libXext.so/.a to compile a libXmu program. While calling `pkg-config --libs xmu' shouldn't strictly need libXext-devel to be installed, it is a lower priority failure than the failure to actually compile and link a libXmu program.

> But there is a concrete specification and implementation.

Where is that specification take from ?

man pgk-config | grep Requires.private comes up empty...

Btw, the specification that I'd really like to see written is not one that talks about the algorithm used for handling various pc file field (though that needs writing too), but one that would be titled "Software development using pkg-config". It would answer questions like

1) My program uses library x. What do I do ? 
Answer:
CFLAGS=`pkg-config --cflags x`
LIBS=`pkg-config --libs x`
(followed by a longer explanation of how this looks in autofoo)

2) My library z installs header files which include libx headers. What do I put in my z.pc file ?
...

3) My library z uses libx internally, but does not expose libx datatypes in its public api. What do I put in my z.pc file ?
...

Etc.

OK, I grant you that the specification is not there, the documentation is poor, and the upstream maintainer is not responsive. Do you agree with my description of the Requires.private implementation, though? And do you agree that this patch breaks handling of --cflags with Requires.private?

I would also be willing to write such a guide if anyone can manage to get Tollef Fog Heen on board.

Ok, I'm going to drop that patch, and you write that guide. Deal ? :-)
Just need to wait for the build system to come back...

Yay! Thanks, Matthias. I was thinking about writing a pkg-config(7) that explained pkg-config concepts. Maybe an html guide would be better, though.

What about the patch in bug 426106 for also outputting Requires.private with --print-requires? I think that'll be needed to avoid this type of issue from cropping up again.

Fixed in pkgconfig-0.23-6.fc11.

(In reply to comment #11)
> I'm sorry, but this fix is broken. The vanilla pkg-config behavior for
> Requires.private is correct, and this just makes it act like Libs.private,
I can agree that keeping the original pkg-config behavior should have been made.
But now having the patch reverted in F-10 will broke near everything. Each package maintainer try to fix thing at sight which will sooner or later have a complete rebuild of the whole distro because of bugs in chain.
(example https://bugzilla.redhat.com/show_bug.cgi?id=485667 )

I'm not sure to understand clearly what is the original pkgconfig behavior, my understanding stop at how pkgconfig creates binary linked statically versus shared library using static library as dependency.
It seems to me that requires.private should be used when building a binary and libs.private when building a library.

Anyway... Fedora doesn't use static library that much (it means static libs aren't even build at all in the majority cases). But If the need of a static library appears, then thus library should be moved to a -static subpackage which will requires the -devel one.
That's the developer responsibility to request the -static version at build time when needed. The static version will then requires every dependencies mentioned in the pkgconfig requires.private line, and that could eventually be autorequired by rpm. But the original title of this bugreport remains valid:
The "shared object only -devel sub-package" shouldn't requires.private dependency in any cases.(but this requires.private should belong to the -static one instead). It shouldn't also fail when building shared libs or against shared libs because either no static lib is present or some requires.private .pc are missing.


I'm sorry but this bug is re-opened, we (as packages maintainers) need to know how to handle the pkg-config behavior change. Either to mass-rebuild the whole F-10 distro or to revert to the ancient pkg-config.
See this case study for example: https://bugzilla.redhat.com
/show_bug.cgi?id=485667

It's not the static libraries that make the difference. It's that we need the headers and the .so regardless. In the bug you referenced, gconf2-devel doesn't have a BR on dbus-devel, so you can remove dbus-devel without removing gconf2-devel. However, you need the dbus headers (and library) to use gconf2-devel, right? I'd argue that the package was already broken. It's not like you could have compiled and linked a GConf app without dbus-devel installed.

The right way to handle this is to Requires the things that the package needs. It's broken to not require a package where you need the -devel package. I understand that can make life tough for packagers, but the previous behavior was more frustrating for developers. Imagine trying to develop software and having to consider that the tool you're using (pkg-config) works differently on one distro than another. This effectively meant that they could not use the Requires.private feature because it broke for anyone using fedora (quite a few as you can imagine).

> I'm sorry but this bug is re-opened, we (as packages maintainers) need to know
> how to handle the pkg-config behavior change. Either to mass-rebuild the whole
> F-10 distro or to revert to the ancient pkg-config.

I don't see how a mass rebuild of the whole F10 comes even remotely into play here. Yes, it is somewhat unfortunate that pkg-config changed its behaviour in the middle of stable releases, which is why I didn't push this out eagerly, but only after other people complained about behaviour difference between F10 and rawhide. 

You may have to adjust BuildRequires slightly as you build updates, that is all.

(In reply to comment #21)
> It's not the static libraries that make the difference. It's that we need the
> headers and the .so regardless.
No we don't. And this is exactly what the current bug report try to address.(look at the title).
Despite it should have been made in the pkgconfig directly instead.

In this case, using GConf2-devel doesn't requires dbus-devel at all as;
when building an application using GConf2-devel:
- The GConf2-devel headers doesn't use dbus headers explicitely. (it will use glib2-devel and popt-devel as shown by "grep include /usr/include/gconf/2/gconf/*  -R" )
- The app using GConf2-devel shouldn't explicitly use dbus-devel as a mandatory requirement in order to build (as it will not use dbus headers directly either).
- As we link to a shared GConf2, we only need to pass -lGConf2 and libs fetched from Requires: so we don't get undefined-non-weak-symbol since thoses are potentially exposed from the API.

- Since We are not using headers from Requires.private in GConf2 and dependent application, the include path and definition that can eventually be present there aren't needed also. It ends that the whole cflags from Requires.private aren't needed at all( even if using pkgconfig -static.)

- It was working fine previously, and rpmlint didn't reported undefined-non-weak-symbol on a GConf2 dependent package AFAIK).

Now I'm all for you to point me a case where the use of Requires.private is correct in a package as starting point; So I can understand how pkgconfig should handle it.

Just as a note. If a given -devel is installed, the .so and headers will be there, but not the static library. If a given -static library is installed, then it will requires the -devel (which will requires the .so and header).


>... In the bug you referenced, gconf2-devel doesn't
> have a BR on dbus-devel, so you can remove dbus-devel without removing
> gconf2-devel. However, you need the dbus headers (and library) to use
> gconf2-devel, right? 
Nope, it was working fine previously and the bugreport started with the pkg-config updates.

> like you could have compiled and linked a GConf app without dbus-devel
> installed.
That was how it behaved previously, and it worked like a charm.


Now I want you to answear few questions:

1 - Why pkg-config should look at Requires.private when not using -static ?

2 - Why either Requires.private (and Libs.private after all) aren't emptied when --disable-static is made. (static library is disabled at build time).

3 - When both shared and static version are built and only the shared version is shipped, Why the requires of the -devel package should cover the needs of the static case ? (which will be larger than the shared only accurate requirements).

4 - When should a library use -static with pkgconfig dependencies ?

5 - When should a binary use -static with pkgconfig dependencies ?

6 - What is the correct behaviour between theses two:
Whith the pkg-config GA behaviour: 
[root@kwizatz gconf]# pkg-config --cflags gconf-2.0
-I/usr/include/gconf/2 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include
With the current pkg-config updates behaviour:
[root@kwizatz gconf]# pkg-config --cflags gconf-2.0
-DORBIT2=1 -pthread -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include

7 - as you said in c#13:
>Requires: Recursively fetch Cflags and Libs from listed dependencies.
>Recursively fetch Libs.private if --static is specified.
>
>Requires.private: Recursively fetch Cflags from listed dependencies.
>Recursively fetch Libs and Libs.private if --static is specified.
Why Cflags are fetched from Requires.private if not using -static instead of just using Cflags from the given .pc ?

8 - As I'm building a package using GConf2-devel, I have:
Requires.private: ORBit-2.0 dbus-1
How can I do if I want to link to a shared version of dbus , but a static version or ORBit-2.0 ?

Now that I look at gconf, I see that you're right. This case (library a uses library b internally but does not expose its datatypes) is not handled well by pkg-config. Read this guide if you want to understand pkg-config (skip to the FAQs if you want).

http://people.freedesktop.org/~dbn/pkg-config-guide.html

A workaround would be to add -ldbus-1 to Libs.private and remove dbus-1 from Requires.private in gconf-2.0.pc. Or, since the GConf2 rpm doesn't even install libgconf-2.a, drop all mentions of dbus in the .pc file. A longer term solution would have to be added to pkg-config.

I've read the guide, but the few words about the Requires/Requires.private doesn't help me to understand the usage. (I've understood the behaviour, but I do not see the matching usage).
So what is following is what i imagine of the differences between Requires and Requires.private.

1- 
Actually, if a package use the Cflags from Requires.private for the includepath. It means it will use the headers included from Requires.private. As a consequence, it will <<have to>> "use symbols" of the related libs from Requires.private. Which, if they aren't defined at build time (linking) will need to be resolved at runtime. (rpmlint on installed packages will show undefined-non-weak-symbol).
(If you don't agree with this part you will have to give an example).
I don't see how a package could see some interest in linking to gconf while not linking to dbus directly.(if ever dbus headers was used in GConf2-devel which is not the case).
I could imagine source distribution like gentoo could see some interest in having such behaviour to avoid relinking to the new SOVERSION of a library, but it would fall under the dirty hack.

We (as precompiled distribution as a whole) want this case to fail at install time (instead of opening the possibility of missing symbol at runtime).
It could be possible to have exeption, but it is the common case. And we rely the fact that shared libraries are linked at build time with an appropriate SOVERSION.

2-
Then I've wondered if it has something to do with building software from a tree.
The main feature between tree vs installed software are that "private headers" start to be accessible. 
So it would have been unappropriate to use Requires.private from gconf-2.0.pc instead of gconf-2.0-uninstalled.pc.
But even in this case, there is a need to link to libs (more than ever actually). And fetching both Cflags and libs is exactly the same behaviour as Requires:). So maybe the Requires line of a given *-(installed).pc can be different than *-(uninstalled).pc, but it still doesn't explain the different usage of Requires.private versus Requires

3-
Now what is misleading (see http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2008-November/056017.html ) is that as Libs.private is known to do have something to do with static library, some might expect the same with Requires.private. But it is not. (that was the previously patched version of this bug, more or less).

The previous case was more or less rare cases, whereas this is a way more common. When a binary is aimed to be portable on any given GNU/Linux distro it may links against a static version of a library. But such library will only contain symbol of itself, hence there is a need for the static linker to search for its dependency also. Here is what would Requires.private aimed for.
It helps the linker to find symbols needed by a given static library. If only a shared library exist, it stop to the Libs. If it is a static library, then it pick Requires.private in the related .pc file and so on.

Then pkgconfig could look at some environment value to say when both static version and shared version exist which ones to pick. and eventually, which versioned symbol (like libresolv.so.2(GLIBC_2.9) in glibc) need to be picked from the static version, and other (ex: -lGL) from the shared version.

But it would mean pkgconfig --libs package --static would fetch the Requires.privates libs and not the cflags, which is the opposite of what the vannilla pkgconfig do.



@Matthias
As you've said previously, you were reluctant to do the update. 
Whatever your current personal thought are; I do think we need a "common decision" to be made, so we can advertise packagers about how to eventually fix things in stable release accurately. (and send the patches upstream as appropriate).

So I'm requesting a topic about "pkgconfig how to deal with Requires.private" to be discussed at friday 20th FESCo.

The common case where one library uses datatypes from another:

$ cat /usr/lib/pkgconfig/cairo.pc 
prefix=/usr
exec_prefix=/usr
libdir=/usr/lib
includedir=/usr/include

Name: cairo
Description: Multi-platform 2D graphics library
Version: 1.6.4

Requires.private: freetype2 >= 9.7.3 fontconfig libpng12 xrender >= 0.6 x11 pixman-1 >= 0.10.0 
Libs: -L${libdir} -lcairo
Libs.private: -lz -lz -lm
Cflags: -I${includedir}/cairo

I'm writing a cairo app, so I want to link it to libcairo. However, in order to compile my app, it also needs to be get the Xrender headers and link to libXrender. Now, I could directly link my app to libXrender, but since I don't use any Xrender symbols, it's extra baggage. The linker will still resolve all the symbols at build time because libXrender is marked with a a DT_NEEDED tag in libcairo.

$ readelf -d /usr/lib/libcairo.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libz.so.1]
 0x00000001 (NEEDED)                     Shared library: [libfreetype.so.6]
 0x00000001 (NEEDED)                     Shared library: [libfontconfig.so.1]
 0x00000001 (NEEDED)                     Shared library: [libpng12.so.0]
 0x00000001 (NEEDED)                     Shared library: [libXrender.so.1]
 0x00000001 (NEEDED)                     Shared library: [libX11.so.6]
 0x00000001 (NEEDED)                     Shared library: [libpixman-1.so.0]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]

So, there's no runtime lookup of undefined symbols, I just want link my app with "gcc -lcairo -o myapp myapp.c" instead of "gcc -lcairo -lXrender -lX11 ...". On the other hand, static linking myapp would require that I pull in all the libraries to resolve all symbols.

$ pkg-config --libs cairo
-lcairo
$ pkg-config --libs --static cairo
-lcairo -lfreetype -lfontconfig -lexpat -lpng12 -lz -lm -lXrender -lpixman-1 -lX11 -lpthread -lxcb-xlib -lxcb -lXau -lXdmcp

Regardless of how I want to link cairo, I need to be able to find the headers for fontconfig, Xrender and X11 since the datatypes are exposed by cairo.

That's the extremely common case where one library uses another library's datatypes and symbols. It does have something to do with static linking, but it's all part of not acquiring more DT_NEEDED libs then necessary.

http://wiki.mandriva.com/en/Overlinking

As I said before, the case you're describing with gconf and dbus is different and is not handled really well by pkg-config since it expects that you will need to access the dbus headers when compiling. Maybe there should be another pkg-config field like Requires.privatelib that does not expose the CFLAGS.

(In reply to comment #26)
...
> I'm writing a cairo app, so I want to link it to libcairo. However, in order to
> compile my app, it also needs to be get the Xrender headers and link to
> libXrender. Now, I could directly link my app to libXrender, but since I don't
> use any Xrender symbols, it's extra baggage. The linker will still resolve all
> the symbols at build time because libXrender is marked with a a DT_NEEDED tag
> in libcairo.
I Don't use cairo, so forgive me if I say weird things.
If you really do not use libXrender symbol while using cairo, why just not include cairo-xlib-xrender.h which is the only one to declare X11/extensions/Xrender.h. In the majority of the cases, cairo.pc should have Requires: xrender >= 0.6 IMO.
Now if linking with xrender is only optional when using cairo, then it should be handled in another .pc or if cairo apps aren't expecting to directly links xrender at all, then it should be best to have X11/extensions/Xrender.h not included at all from the cairo-devel (so it will not be exposed from the cairo API).


> $ readelf -d /usr/lib/libcairo.so | grep NEEDED
The internal dependency of cairo itself is not in question.
...
> So, there's no runtime lookup of undefined symbols, I just want link my app
As you have defined your fictional app not to use symbol from xrender, it is necessary true. But doesn't help with some more info either.

> with "gcc -lcairo -o myapp myapp.c" instead of "gcc -lcairo -lXrender -lX11
> ...". On the other hand, static linking myapp would require that I pull in all
> the libraries to resolve all symbols.
What if an application (compiled gcc -lcairo -o myapp myapp.c ) using cairo and xrender symbol is provided on a system where cario have been compiled without xrender and the xrender library isn't available ?

> Regardless of how I want to link cairo, I need to be able to find the headers
> for fontconfig, Xrender and X11 since the datatypes are exposed by cairo.
So why they aren't in Requires: instead of Requires.private ?
OK , there is the Overlink problem , so another workaround would be to have @LIBX11_CFLAGS@ @LIBX11_CFLAGS@ @LIBX11_CFLAGS@ added in cairo.pc.in at the Clags libs.Then the cairo maintainer would have to requires the related -devel package manually.

But in the majority of the cases, Cflags and Libs from Requires.private doesn't matches. When building your application against cario shared/static, you don't need the -I/usr/include/libpng12 for example. while you will need the -lpng while your binary will be linked statically.
This is usually the case since whenever it is possible, internal dependencies should be hidden from the API.

> That's the extremely common case where one library uses another library's
> datatypes and symbols. It does have something to do with static linking, but
> it's all part of not acquiring more DT_NEEDED libs then necessary.
A agree that there is a need to reduce linking whenever it is possible.
> http://wiki.mandriva.com/en/Overlinking

> As I said before, the case you're describing with gconf and dbus is different
> and is not handled really well by pkg-config since it expects that you will
> need to access the dbus headers when compiling. Maybe there should be another
> pkg-config field like Requires.privatelib that does not expose the CFLAGS.
To sum up, Requires.private seems to be used when: 
- compiling staticaly for the libs
- compiling shared/static for the include pathes.
This sound like a miss design for me.

If you think pkg-config is not designed correctly, please bring the discussion to the pkg-config list. Or look in the pkg-config archives since there have been many discussions on Requires.private.

http://lists.freedesktop.org/mailman/listinfo/pkg-config

The fact of the matter is that fedora can't just change the behavior of the tool because it's inconvenient. Although there is no formal specification, what I've described is the expected and stated behavior of Requires.private. It can't mean one thing on debian and another thing on fedora. That makes the tool useless.

(In reply to comment #28)
> The fact of the matter is that fedora can't just change the behavior of the
> tool because it's inconvenient. Although there is no formal specification, what
> I've described is the expected and stated behavior of Requires.private. It
> can't mean one thing on debian and another thing on fedora. That makes the tool
> useless.

This is not correct. In many applications Fedora already adds
its own implementation.

You can continue discussing this here. But if you want to see any behaviour changes, the discussion needs to happen on the pkg-config list upstream. 
Fedora is not going to deviate from upstream behaviour for basic development infrastructure.

This seems to be the nearest upstream bug-report.
http://bugs.freedesktop.org/show_bug.cgi?id=4738
I'm still searching for historic and previous reports/talks.

I wonder if an acceptable workaround wouldn't be to have pkg-config --cflags foo to remain quiet if there is missing .pc from Requires.private.

The more correct Fix would be to sort out
Requires.Cflags from Requires.private, because the same info is taken from Requires.private, but for different usage. 
I understand the Overlinking concern here, But while trying to fix it,  the problem has just moved elsewhere. (with Over requiring dependencies somehow).

This bug is re-open (agreed with mclasen) with the aim for the problem to be solved with upstream.

Fwiw, I did no such thing (agree to reopen this). I just tired to fight the close/reopen game with you.

How very unfriendly of you to take this to FESCo without even informing me.

I've just stumbled over the problem during re-compiling the current compiz package (F10).

I've just fixed it for myself by adding libXres-devel as BR to the spec file.
However, I'm not sure whether that's the correct way dealing with the problem.

Is there already an agreement what would be the best way to fix build problems in existing packages?

This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

I agree with comment #30 whole-heartedly , and since fedora is shipping upstream setting here and there is an upstream list and bug to track this now, the issue is resolved->upstream (as far as I'm concerned, the original reporter).

A new pkgconfig release is imminent, which should finally put this issue to rest.

amen! :)