Bug 1305701 - curl: enable support for Public Suffix List
curl: enable support for Public Suffix List
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: curl (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kamil Dudka
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-08 18:01 EST by Yaakov Selkowitz
Modified: 2016-04-01 06:03 EDT (History)
2 users (show)

See Also:
Fixed In Version: curl-7.47.1-2.fc24
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-10 13:24:00 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yaakov Selkowitz 2016-02-08 18:01:45 EST
curl since 7.46.0 has support for the Public Suffix List <https://publicsuffix.org/> to avoid "supercookies".  This can be enabled by BuildRequires: libpsl-devel and configuring --with-libpsl.

Note that libpsl does require libicu.  If that is too big of a dependency for curl, then libpsl can alternatively be configured to use libidn at runtime (which curl already depends upon).
Comment 1 Kamil Dudka 2016-02-09 07:10:10 EST
It sounds like it would be worth adding it.  On the other hand, as you said, it pulls in a lot of run-time dependencies, including the C++ run-time:

$ ldd /usr/lib64/libpsl.so.0.2.4
        linux-vdso.so.1 (0x00007ffdc8319000)
        libicuuc.so.54 => /lib64/libicuuc.so.54 (0x00007ffbe4aed000)
        libicudata.so.54 => /lib64/libicudata.so.54 (0x00007ffbe30c2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ffbe2d00000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffbe2ae3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007ffbe28df000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ffbe255c000)
        libm.so.6 => /lib64/libm.so.6 (0x00007ffbe225a000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffbe2043000)
        /lib64/ld-linux-x86-64.so.2 (0x00005638a0c65000)

Paul, what is your opinion on this request?
Comment 2 Paul Howarth 2016-02-09 09:45:59 EST
Looks like a classic space/functionality trade-off. I don't have a strong opinion either way really.

Regarding the dependencies, libicu rather dwarfs libcurl in terms of size, so I can see the objections to it. A libidn+libunistring build of libpsl would be considerably smaller, but libicu is upstream's default, and presumably that is for a good reason?
Comment 3 Kamil Dudka 2016-02-09 10:13:15 EST
Even if we convince libpsl maintainers to reduce their dependencies now, those dependencies can grow anytime later and we will have no control over that.  We already had requests to crop the current list of libcurl's run-time dependencies to improve the footprint of minimal installations of Fedora...

Perhaps we have reached the point where providing multiple builds of libcurl would help us?

We can provide a libcurl-minimal subpackage that would conflict with libcurl, containing a binary-compatible libcurl picked as the result of a conservatively configured build of curl.

Such a package could be used for minimal installations of Fedora, just to keep yum/dnf (and basic user-space) working.  Normal installations of Fedora would continue to use the current feature-rich packages of libcurl.
Comment 4 Paul Howarth 2016-02-09 10:31:27 EST
Wouldn't that require explicit support in the installer to pick the minimal build for minimal/cloud installs and the full-fat version for "normal" installs?

And how would a cloud/minimal user activate the full version if they needed it? Something like "dnf swap libcurl-minimal libcurl"?

I don't think it can be done as a sub-package either, as all sub-packages populate the same buildroot and so there can be only one %{_libdir}/libcurl.so.X. This could get very complicated very quickly.
Comment 5 Kamil Dudka 2016-02-09 11:18:35 EST
(In reply to Paul Howarth from comment #4)
> Wouldn't that require explicit support in the installer to pick the minimal
> build for minimal/cloud installs and the full-fat version for "normal"
> installs?

Installers and builders usually have a configurable set of packages to start with.  The set is passed to dependency solver to obtain the set of packages to be installed.  I believe that putting libcurl-minimal to the initial set of packages would do the job.

> And how would a cloud/minimal user activate the full version if they needed
> it? Something like "dnf swap libcurl-minimal libcurl"?

Yes, the above should work.

> I don't think it can be done as a sub-package either, as all sub-packages
> populate the same buildroot and so there can be only one
> %{_libdir}/libcurl.so.X. This could get very complicated very quickly.

There is definitely a way to do that with modern rpm.  Please have a look at the coreutils-single subpackage in Fedora rawhide as an example.
Comment 6 Yaakov Selkowitz 2016-02-09 11:21:52 EST
(In reply to Paul Howarth from comment #2)
> Regarding the dependencies, libicu rather dwarfs libcurl in terms of size,
> so I can see the objections to it. A libidn+libunistring build of libpsl
> would be considerably smaller, but libicu is upstream's default, and
> presumably that is for a good reason?

libicu does allow support for a newer IDN specification than libidn.  However, libpsl mostly relies on an internally compiled list, which is generated at buildtime and the support thereof requires either libidn or libicu only at buildtime; the runtime requirement is only for loading external lists, which I'm not sure that either curl nor wget support.

So, for example, libpsl could be configured --enable-builtin=libicu --enable-runtime=libidn to maximize benefit to the internal list (which is what is generally used) while minimizing runtime dependencies for the "side" feature of loading external lists.
Comment 7 Kamil Dudka 2016-02-09 11:44:59 EST
(In reply to Yaakov Selkowitz from comment #6)
> So, for example, libpsl could be configured --enable-builtin=libicu
> --enable-runtime=libidn to maximize benefit to the internal list (which is
> what is generally used) while minimizing runtime dependencies for the "side"
> feature of loading external lists.

Is there already a request to use such configuration of libpsl in Fedora?
Comment 8 Yaakov Selkowitz 2016-02-09 11:46:11 EST
(In reply to Kamil Dudka from comment #7)
> (In reply to Yaakov Selkowitz from comment #6)
> > So, for example, libpsl could be configured --enable-builtin=libicu
> > --enable-runtime=libidn to maximize benefit to the internal list (which is
> > what is generally used) while minimizing runtime dependencies for the "side"
> > feature of loading external lists.
> 
> Is there already a request to use such configuration of libpsl in Fedora?

Not AFAICS.
Comment 9 Paul Howarth 2016-02-10 03:37:48 EST
(In reply to Kamil Dudka from comment #5)
> > I don't think it can be done as a sub-package either, as all sub-packages
> > populate the same buildroot and so there can be only one
> > %{_libdir}/libcurl.so.X. This could get very complicated very quickly.
> 
> There is definitely a way to do that with modern rpm.  Please have a look at
> the coreutils-single subpackage in Fedora rawhide as an example.

Ah, RemovePathPostfixes, new in rpm 4.13. Looks like coreutils went through a few iterations getting the conflicts/obsoletes/provides right, so that's a useful template.
Comment 10 Kamil Dudka 2016-02-10 07:08:49 EST
I propose to enable the support for libpsl in libcurl now.  If somebody starts to complain that the minimal installation of Fedora is significantly bigger, we can redirect him/her to libpsl maintainers with such complaints.

Meanwhile I will experiment with the libcurl-minimal idea in a private branch.  If I ever convince myself that it was a good idea, I will open a separate bug for that and probably discuss it with a wider audience (e.g. fedora-devel).

Any objections to the above approach?
Comment 11 Paul Howarth 2016-02-10 07:14:11 EST
Works for me.
Comment 12 Kamil Dudka 2016-02-10 13:24:00 EST
fixed in curl-7.47.1-2.fc24
Comment 13 Kamil Dudka 2016-02-12 03:58:32 EST
(In reply to Kamil Dudka from comment #10)
> Meanwhile I will experiment with the libcurl-minimal idea in a private
> branch.  If I ever convince myself that it was a good idea, I will open a
> separate bug for that and probably discuss it with a wider audience (e.g.
> fedora-devel).

Just for reference, this effort is currently blocked on rpm bug #1306559.
Comment 14 Kamil Dudka 2016-02-15 13:35:48 EST
(In reply to Kamil Dudka from comment #12)
> fixed in curl-7.47.1-2.fc24

Unfortunately, this caused problems with domains like "localhost" being marked as top-level domains.  Consequently, such domains are refused for storing a cookie:

https://github.com/rockdaboot/libpsl/issues/48

... which breaks the test-suite of php-pecl-http, for instance:

https://github.com/m6w6/ext-http/issues/26
Comment 15 Fedora Update System 2016-03-31 02:01:02 EDT
libpsl-0.13.0-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffb9856f
Comment 16 Yaakov Selkowitz 2016-03-31 02:01:20 EDT
(In reply to Kamil Dudka from comment #7)
> (In reply to Yaakov Selkowitz from comment #6)
> > So, for example, libpsl could be configured --enable-builtin=libicu
> > --enable-runtime=libidn to maximize benefit to the internal list (which is
> > what is generally used) while minimizing runtime dependencies for the "side"
> > feature of loading external lists.
> 
> Is there already a request to use such configuration of libpsl in Fedora?

FYI I adopted libpsl and implemented this for rawhide and F24, update for latter:

https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffb9856f
Comment 17 Kamil Dudka 2016-04-01 06:03:14 EDT
Cool.  Thanks!

Note You need to log in before you can comment on or make changes to this bug.