Bug 1607633 - Auto-provides / requires for private perl modules should be filtered
Summary: Auto-provides / requires for private perl modules should be filtered
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: 389-ds-base
Version: 29
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: mreynolds
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-23 21:40 UTC by Adam Williamson
Modified: 2020-09-13 22:13 UTC (History)
3 users (show)

Fixed In Version: 389-ds-base-1.4.1.8-3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-27 18:14:38 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 2947 0 None None None 2020-09-13 22:13:03 UTC

Description Adam Williamson 2018-07-23 21:40:29 UTC
389-ds-base installs a number of perl modules to a private location:

[adamw@id ~]$ ls -l /usr/lib64/dirsrv/perl
total 264
-rw-r--r--. 1 root root  4394 Jun 21 10:10 DialogManager.pm
-rw-r--r--. 1 root root  7962 Jun 21 10:10 Dialog.pm
-rw-r--r--. 1 root root 54470 Jun 21 10:10 DSCreate.pm
-rw-r--r--. 1 root root  7143 Jun 21 10:10 DSDialogs.pm
-rw-r--r--. 1 root root 44895 Jun 21 10:10 DSMigration.pm
-rw-r--r--. 1 root root  4813 Jun 21 10:10 DSUpdateDialogs.pm
-rw-r--r--. 1 root root 18734 Jun 21 10:10 DSUpdate.pm
-rw-r--r--. 1 root root 57185 Jun 21 10:10 DSUtil.pm
-rw-r--r--. 1 root root 11662 Jun 21 10:10 FileConn.pm
-rw-r--r--. 1 root root  7642 Jun 21 10:10 Inf.pm
-rw-r--r--. 1 root root 11848 Jun 21 10:10 Migration.pm
-rw-r--r--. 1 root root  3530 Jun 21 10:10 Resource.pm
-rw-r--r--. 1 root root  7163 Jun 21 10:10 SetupDialogs.pm
-rw-r--r--. 1 root root  1876 Jun 21 10:10 SetupLog.pm
-rw-r--r--. 1 root root  7051 Jun 21 10:10 Setup.pm

the 389-ds-base package has auto-generated Provides for all these modules:

[adamw@id ~]$ rpm -q --provides 389-ds-base | grep perl
perl(DSCreate)
perl(DSDialogs)
perl(DSMigration)
perl(DSUpdate)
perl(DSUpdateDialogs)
perl(DSUtil)
perl(Dialog)
perl(DialogManager)
perl(DialogYesNo)
perl(FileConn)
perl(Inf)
perl(Migration)
perl(Resource)
perl(Setup)
perl(SetupDialogs)
perl(SetupLog)

(note, some/all of these scripts have been moved to 389-ds-base-legacy-tools in F28 updates-testing / Rawhide - the bug is still valid, just obviously as applied to that binary package).

per the perl packaging guidelines - https://fedoraproject.org/wiki/Packaging:AutoProvidesAndRequiresFiltering#Perl - these automatic Provides should probably be filtered out. This case is directly comparable to the Pidgin example given:

"On a x86_64 machine, the pidgin-libnotify provides pidgin-libnotify.so()(64bit) which it shouldn't as this library is not inside the paths searched by the system for libraries. It's a private, not global, "provides" and as such must not be exposed globally by RPM."

Note that there are also some automatic Requires generated for these modules - those too should be filtered out, and any dependencies between binary packages handled manually instead.

Especially as some of these Provides have quite generic names - there is already a distribution called Setup in cpan, for instance - they could actually be 'dangerous'. We don't want something that depends on a known perl module called Setup, for instance, getting 389-ds-base (or 389-ds-base-legacy-tools) installed instead of perl-Setup or whatever the 'normal' package is called.

Comment 1 mreynolds 2018-08-06 15:26:12 UTC
Upstream ticket:
https://pagure.io/389-ds-base/issue/49888

Comment 2 Jan Kurik 2018-08-14 11:23:56 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

Comment 3 Adam Williamson 2018-12-17 21:39:48 UTC
This still seems valid in F29, to me:

[adamw@id ~]$ rpm -q --provides 389-ds-base-legacy-tools | grep perl
perl(DSCreate)
perl(DSDialogs)
perl(DSMigration)
perl(DSUpdate)
perl(DSUpdateDialogs)
perl(DSUtil)
perl(Dialog)
perl(DialogManager)
perl(DialogYesNo)
perl(FileConn)
perl(Inf)
perl(Migration)
perl(Resource)
perl(Setup)
perl(SetupDialogs)
perl(SetupLog)
[adamw@id ~]$ rpm -q 389-ds-base-legacy-tools
389-ds-base-legacy-tools-1.4.0.16-1.fc29.x86_64

Comment 4 Viktor Ashirov 2019-08-01 13:12:45 UTC
Can confirm, this still happens in F30:

$ rpm -q --provides 389-ds-base-legacy-tools | grep perl
perl(DSCreate)
perl(DSDialogs)
perl(DSMigration)
perl(DSUpdate)
perl(DSUpdateDialogs)
perl(DSUtil)
perl(Dialog)
perl(DialogManager)
perl(DialogYesNo)
perl(FileConn)
perl(Inf)
perl(Migration)
perl(Resource)
perl(Setup)
perl(SetupDialogs)
perl(SetupLog)

~
$ rpm -q 389-ds-base 
389-ds-base-1.4.1.6-1.fc30.x86_64

Comment 5 mreynolds 2019-08-01 14:19:26 UTC
I had previously followed:

https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/#Perl

and added:

%{?perl_default_filter}

What else needs to be done?

From specfile:

%package          legacy-tools
Summary:          Legacy utilities for 389 Directory Server (%{variant})
Group:            System Environment/Daemons
Obsoletes:        %{name} <= 1.4.0.9
Requires:         %{name} = %{version}-%{release}
%if %{use_perl}
# for setup-ds.pl to support ipv6
%if %{use_Socket6}
Requires:         perl-Socket6
%else
Requires:         perl-Socket
%endif
Requires:         perl-NetAddr-IP
# use_openldap assumes perl-Mozilla-LDAP is built with openldap support
Requires:         perl-Mozilla-LDAP
# for setup-ds.pl
Requires:         bind-utils
%{?perl_default_filter}
%endif
# End use perl


--> note "use_perl" is set to 1

Comment 6 Viktor Ashirov 2019-08-01 15:11:34 UTC
This works for me:

%global __provides_exclude_from %{_libdir}/%{pkgname}/perl
%{?perl_default_filter}

Comment 7 Adam Williamson 2019-08-01 16:51:05 UTC
yeah, just doing perl_default_filter isn't enough. perl_default_filter is this:

%perl_default_filter %{expand: \
%global __provides_exclude_from %{?__provides_exclude_from:%__provides_exclude_from|}^%{_docdir}
%global __requires_exclude_from %{?__requires_exclude_from:%__requires_exclude_from|}^%{_docdir}
%global __provides_exclude %{?__provides_exclude:%__provides_exclude|}^perl\\\\(VMS|^perl\\\\(Win32|^perl\\\\(DB\\\\)|^perl\\\\(UNIVERSAL\\\\)
%global __requires_exclude %{?__requires_exclude:%__requires_exclude|}^perl\\\\(VMS|^perl\\\\(Win32
}

so it's excluding deps based on files in the docdir, and a few specific 'known bad' things. The docs do cover this, a bit further down, under "Private libraries".

Note, as well as __provides_exclude_from you're going to want to do a __requires_exclude too - otherwise the auto-generated Provides will be filtered out, but there will still be *Requires* for those provides included.

In openqa.spec for e.g. I do this:

%global __provides_exclude_from %{_datadir}/openqa/lib
%global __requires_exclude perl\\((OpenQA|DBIx::Class::Timestamps|MojoDebugHandle|db_helpers|db_profiler|Perl::Critic)

as the docs say, this can be a bit of a pain to maintain - especially you have to keep an eye out for new ones popping up and add them to __requires_exclude - but there isn't really a better way ATM :/ I don't know why we can't have a __requires_exclude_from that matches __provides_exclude_from behaviour, but I guess there's a reason...

Comment 8 Adam Williamson 2019-08-01 16:52:23 UTC
An alternative I guess would be to ask one of the perl packaging folks if there's a way you could tweak this stuff upstream such that the dependencies aren't generated in the first place. I dunno if there's a way to do that.

Comment 9 Fedora Update System 2019-09-26 15:53:55 UTC
FEDORA-2019-ab85eef8cc has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-ab85eef8cc

Comment 10 mreynolds 2019-09-26 16:11:01 UTC
(In reply to Viktor Ashirov from comment #6)
> This works for me:
> 
> %global __provides_exclude_from %{_libdir}/%{pkgname}/perl
> %{?perl_default_filter}

This causes rpm errors:

package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(DSCreate), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(DSMigration), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(DSUpdate), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(DSUtil), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(Dialog), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(DialogManager), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(FileConn), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(Inf), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(Migration), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(Resource), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(Setup), but none of the providers can be installed: package 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64 requires perl(SetupLog), but none of the providers can be installed: cannot install both 389-ds-base-legacy-tools-1.4.1.6-1.fc31.1.x86_64 and 389-ds-base-legacy-tools-1.4.1.8-1.fc31.x86_64: conflicting requests

Comment 11 Fedora Update System 2019-09-26 17:05:35 UTC
FEDORA-2019-d1edc3a2b8 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d1edc3a2b8

Comment 12 Adam Williamson 2019-09-26 17:08:45 UTC
As I said in comment #7, you're meant to filter *both* the provides *and* the requires :)

Comment 13 Adam Williamson 2019-09-26 18:15:54 UTC
1.4.1.8-2 dropped %{?perl_default_filter} but didn't drop %global __provides_exclude_from %{_libdir}/%{pkgname}/perl . That isn't going to fix the problem in #c10. The __provides_exclude_from filters out the *Provides* from the packages that are produced...but it does not filter out the auto-generated *Requires* for those provides. As I said in #c7 must add a corresponding %global __requires_exclude line. That should look something like this:

%global __requires_exclude perl\\((DSCreate|DSMigration|DSUpdate|DSUtil|Dialog|DialogManager|FileConn|Inf|Migration|Resource|Setup|SetupLog)

Comment 14 mreynolds 2019-09-27 16:28:42 UTC
Tested 389-ds-base & 389-ds-base-legacy-tools

$ rpm -q --provides 389-ds-base-legacy-tools | grep perl
$ rpm -q --requires 389-ds-base-legacy-tools | grep perl
/usr/bin/perl
perl-Mozilla-LDAP
perl-NetAddr-IP
perl-Socket

$ rpm -q --provides 389-ds-base | grep perl
$ rpm -q --requires 389-ds-base | grep perl
/usr/bin/perl
perl(:MODULE_COMPAT_5.28.2)
perl-Archive-Tar
perl-DB_File



None of these perl requirements are related to legacy tools or DS perlk modules, so I think we are finally good to go.  If there is another test I should do before I do a new build please let me know.

Comment 15 Adam Williamson 2019-09-27 16:43:34 UTC
that looks fine. One note, you could probably also put the default filter back, it probably wasn't causing any problems in the first place.

Comment 16 Ben Cotton 2019-10-31 18:47:52 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Adam Williamson 2019-11-04 18:25:03 UTC
so, I'm not sure where we're up to here.

The bug is against F29, but no work on this seems to have been done on F29.

On F30, F31 and master, there was a commit on 2019-09-27 (so back when we were discussing it here) which tried to fix the filters. Then on F30 - but not on F31 or master - all the filtering stuff was removed on Oct 15 on the basis that it caused some kind of unspecified 'regressions'.

Mark, what is the current status here?

Comment 18 mreynolds 2019-11-04 19:49:42 UTC
(In reply to Adam Williamson from comment #17)
> so, I'm not sure where we're up to here.
> 
> The bug is against F29, but no work on this seems to have been done on F29.
> 
> On F30, F31 and master, there was a commit on 2019-09-27 (so back when we
> were discussing it here) which tried to fix the filters. Then on F30 - but
> not on F31 or master - all the filtering stuff was removed on Oct 15 on the
> basis that it caused some kind of unspecified 'regressions'.
> 
> Mark, what is the current status here?

Sorry Adam, yeah so we have a 389-admin package that handles the frontend of the server, and it needs the perl modules "provided" by our 389-ds-base-legacy-tools subpackage.  Once we filtered out perl from 389-ds-base we could no longer install the 389-admin package:

 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(DSCreate), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(DSMigration), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(DSUpdate), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(DSUtil), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(Dialog), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(DialogManager), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(FileConn), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(Inf), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(Migration), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(Resource), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(Setup), but none of the providers can be installed
 - package 389-admin-1.1.46-3.fc30.2.x86_64 requires perl(SetupLog), but none of the providers can be installed
 - cannot install both 389-ds-base-legacy-tools-1.4.1.8-3.fc30.x86_64 and 389-ds-base-legacy-tools-1.4.1.6-1.fc30.x86_64
 - cannot install both 389-ds-base-legacy-tools-1.4.1.2-2.fc30.x86_64 and 389-ds-base-legacy-tools-1.4.1.8-3.fc30.x86_64
 - cannot install the best update candidate for package 389-ds-base-legacy-tools-1.4.1.6-1.fc30.x86_64
 - cannot install the best update candidate for package 389-admin-1.1.46-3.fc30.2.x86_64


Unfortunately, this kind of fell through the cracks as other fires popped up.  I'm not sure how to proceed in this case.  

FYI, in F32 389-admin has been removed so this then becomes a non-issue, but for now 389-admin needs them.

Comment 19 Adam Williamson 2019-11-04 20:01:01 UTC
perhaps those would go away if you rebuilt it against the 389-ds-base with the provides stripped? I'm not 100% sure.

Comment 20 Ben Cotton 2019-11-27 18:14:38 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.