Bug 489221

Summary: perl: @INC order incorrect
Product: [Fedora] Fedora Reporter: Chris Weyl <cweyl>
Component: perlAssignee: Stepan Kasal <kasal>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: rawhideCC: kasal, mmaslano, rc040203, robin.norwood, tcallawa
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 5.10.0-68.fc10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-03-23 15:57:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Weyl 2009-03-08 19:20:11 UTC
Perl has three main sets of directories it searches for extensions: site (locally built and installed modules), vendor (OS vendor (e.g. Fedora) built and supplied modules), core (modules that are provided by the Perl package itself).

In and before F-8, the @INC order of our Perl build was site, vendor, core, which allowed vendor-supplied modules to be found before core modules and local site modules to be found before both vendor and core.  This made overriding, say, a core module with a local one both easy and non-surprising :)

Sometime between F-8 and current, this order has changed to core, part of site, vendor, then the rest of site.

F-8 @INC:[1]
   /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
   /usr/lib/perl5/site_perl/5.8.8
   /usr/lib/perl5/site_perl
   /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
   /usr/lib/perl5/vendor_perl/5.8.8
   /usr/lib/perl5/vendor_perl
   /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
   /usr/lib/perl5/5.8.8
   .

From perl-5.10.0-56.fc10 @INC:
   /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
   /usr/lib/perl5/5.10.0
   /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
   /usr/local/lib/perl5/site_perl/5.10.0
   /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
   /usr/lib/perl5/vendor_perl/5.10.0
   /usr/lib/perl5/vendor_perl
   /usr/local/lib/perl5/site_perl
   /usr/local/lib64/perl5/site_perl
   /usr/lib/perl5/site_perl
   .

Can we please revert the @INC ordering to that of F-8 in at least rawhide and F-10?  Thanks :-)

Target @INC:
   /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
   /usr/local/lib/perl5/site_perl/5.10.0
   /usr/local/lib/perl5/site_perl
   /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
   /usr/lib/perl5/vendor_perl/5.10.0
   /usr/lib/perl5/vendor_perl
   /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
   /usr/lib/perl5/5.10.0
   .

[1] paths for module compatibility with 5.8.[567] removed for clarity.

Comment 1 Tom "spot" Callaway 2009-03-10 21:19:32 UTC
So, I think I can get it close to what you're asking... the problem is "/usr/local/lib/perl5/site_perl". Right now, I'm 99% sure that the only reason it shows up in @INC at all is because we're passing it in as -Dotherlibdirs.

With a patch (NOTE: I want to run it past someone like Nicholas Clark first for sanity), I can get @INC to be:

/usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.10.0
/usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl
/usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl
/usr/local/lib64/perl5/site_perl
/usr/lib/perl5/site_perl

I think if we were to overload -Dsitelib, we might be able to have it exactly match your suggestion, but I'm really wary of doing that.

Comment 2 Tom "spot" Callaway 2009-03-10 21:24:15 UTC
Whoops. Hit Commit too quickly there. What my patch does is change the internal @INC ordering from:

ARCHLIB_EXP
PRIVLIB_EXP
SITEARCH_EXP
SITELIB_EXP
PERL_VENDORLIB
PERL_OTHERLIBDIRS

to
SITEARCH_EXP
SITELIB_EXP
PERL_VENDORLIB
ARCHLIB_EXP
PRIVLIB_EXP
PERL_OTHERLIBDIRS

Comment 3 Tom "spot" Callaway 2009-03-11 13:25:29 UTC
(In reply to comment #1)

> I think if we were to overload -Dsitelib, we might be able to have it exactly
> match your suggestion, but I'm really wary of doing that.  

I tried this just for fun, and it did not work. It caused a test failure in lib/Config.t.

Comment 4 Tom "spot" Callaway 2009-03-11 14:36:02 UTC
Chris, what would you think if:

/usr/local/lib/perl5/site_perl
/usr/local/lib64/perl5/site_perl
/usr/lib/perl5/site_perl

Stayed at the bottom (because they're in otherlibs). We've added these dirs simply for convenience. Are you using any of these dirs?

Comment 5 Chris Weyl 2009-03-11 16:56:21 UTC
I think that (site, vendor, core, other) would work quite well -- from the discussion on fedora-perl-devel-list it looks like the first two are obsolete and only being retained for a transition period; and the other one is going to stick around permanently.  I'm not actively using any of the "other" directories.

Comment 6 Stepan Kasal 2009-03-11 21:00:27 UTC
(In reply to comment #1)
> [...] the problem is "/usr/local/lib/perl5/site_perl".

This directory is neither sitelib nor sitearch, and IMHO it should not appear in @INC at all.

> Right now, I'm 99% sure that the only reason
> it shows up in @INC at all is because we're passing it in as -Dotherlibdirs.

I agree, and that's the reason it appears at the end.  Being a backward compatibility cludge, it can disappear any time.

> /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
> /usr/local/lib/perl5/site_perl/5.10.0
> /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
> /usr/lib/perl5/vendor_perl/5.10.0
> /usr/lib/perl5/vendor_perl

"Hey, but why does /usr/lib/perl5/vendor_perl appear there?"
Well, IMVHO, it should not either.  This inconsistency seems like a mistake.

If I'm reading the code right, blead perl makes these two cases consistent--neither of these two directories appears in @INC.

To sum up, the following three are added via otherlibdirs:
> /usr/local/lib/perl5/site_perl
> /usr/local/lib64/perl5/site_perl
> /usr/lib/perl5/site_perl

If you want to be on the safe side, do not use them, always use sitelib (run "perl -V:sitelib").

And the first and the second are going to be removed, the third one is promised to stay as a Fedora-specific extension, see https://bugzilla.redhat.com/show_bug.cgi?id=484053#c3 .

Comment 7 Tom "spot" Callaway 2009-03-11 22:15:03 UTC
Stephan, I put a patch in F-9, F-10, and rawhide that reorders @INC. I'm doing a scratch build now, but I haven't done any official builds yet. Nicholas Clark said he would look at my patch, but I think he'll be okay with it, since it is based on his plans for 5.12.0.

Comment 8 Tom "spot" Callaway 2009-03-11 22:35:40 UTC
Scratch build of perl-5.10.0-62.fc11:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1237539

Comment 9 Chris Weyl 2009-03-16 17:03:03 UTC
From building the F-10 branch locally, @INC looks good (to me, at any rate) in perl-5.10.0-57.fc10, with the above comments about the legacy dirs in mind:

  @INC:
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/local/lib/perl5/site_perl
    /usr/local/lib64/perl5/site_perl
    /usr/lib/perl5/site_perl
    .

Current rawhide (-60) @INC looks a touch off, but -62 looks good.

[not ok] 5.10.0-60.fc11:

  @INC:
    /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/site_perl
    .

[ok] 5.10.0-62.fc11:

  @INC:
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/lib/perl5/site_perl
    .

F-9 also looks good.

[ok] 5.10.0-42.fc9:

  @INC:
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/local/lib/perl5/site_perl
    /usr/lib/perl5/site_perl
    .

Is there anything blocking us at this point from building to -62 in rawhide, and -42, -57 in F-9 & F-10, respectively?

Comment 10 Tom "spot" Callaway 2009-03-16 17:05:40 UTC
I'm waiting for:

A) skasal's opinion on the @INC patch in CVS
B) Nicholas Clark's opinion on the @INC patch in CVS

If both of them think this is a reasonable approach, we'll go with it.

Comment 11 Stepan Kasal 2009-03-20 14:53:49 UTC
(In reply to comment #10)
> I'm waiting for:
> A) skasal's opinion on the @INC patch in CVS

I apologize that it took me so long to get to this review.

Yes, I support the reorderINC patch.

More detailed comment: I planned to resurrect the incorder.patch Fedora had until the upgrade to perl 5.10: it moved ARCHLIB and PRIVLIB to the end.
Upon close look, that patch had two problems:
1) it moved dirs went *after* otherlibdirs; I would move them *before* that, because otherlibdirs is a compatibility-last-resort thing.
2) it moved ARCHLIB from outside ifdef MACOS_TRADITIONAL to its #else branch; that was not fair to poor old Mac!

And, voila, spot's reorderINC patch is the same as the incorder.patch except that it corrects these two deficiencies.  So yes, this is exactly the patch I would write, if I managed to get it right!

Comment 12 Fedora Update System 2009-03-23 15:54:45 UTC
perl-5.10.0-42.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/perl-5.10.0-42.fc9

Comment 13 Stepan Kasal 2009-03-23 15:57:05 UTC
Fix is built in perl-5.10.0-62.

Comment 14 Fedora Update System 2009-03-23 17:22:39 UTC
perl-5.10.0-63.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/perl-5.10.0-63.fc10

Comment 15 Fedora Update System 2009-04-22 00:54:39 UTC
perl-5.10.0-68.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2009-04-22 01:08:21 UTC
perl-5.10.0-68.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.