Description of problem: After upgrading a system from RedHat 5.10 to 5.11 (including perl), many perl scripts, including /usr/bin/cpan start failing with: Attempt to free unreferenced scalar: SV 0x9e01024, Perl interpreter: 0x9ce1008 at /usr/lib/perl5/5.8.8/ExtUtils/Liblist.pm line 6. Segmentation fault Version-Release number of selected component (if applicable): perl-5.8.8-42.el5 How reproducible: Very. Steps to Reproduce: 1. Start with a fresh 5.10 install, including perl and gcc 2. via /usr/bin/cpan, upgrade PathTools (cpan Cwd) 3. yum -y update perl 4. cpan Actual results: Attempt to free unreferenced scalar: SV 0x9e01024, Perl interpreter: 0x9ce1008 at /usr/lib/perl5/5.8.8/ExtUtils/Liblist.pm line 6. Segmentation fault Expected results: cpan> Additional info: The problem: On 5.10, Installing perl provides Cwd.so (version 3.12) in: /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so or /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd/Cwd.so Then if you upgrade Cwd to the CPAN version, it overwrites the Cwd.so file with the 3.47 version upgrading perl via YUM then downgrades Cwd.so back to 3.12 BUT does not downgrade the I note in Makefile.PL that they are using 'INSTALLDIRS' => 'perl'. What's problematic is that the arch files are overwritten but not the pure perl code. If both had been overwritten, I'd say this was a bug with PathTools. However, the act of upgrading the perl RPM currently replaces some but not all of the PathTools distro. This seems to be more of a design issue with where the RPM versus the CPAN distro installs INSTALLDIRS => 'perl' files to.
That's said, but that's a problem in CPAN PathTools build script. It should not overwrite Perl core and vendor files. It should install the files into site locations. You can force it by setting makepl_arg CPAN option to INSTALLDIRS=site value before installing thew PathTools from CPAN.
The isn't really an upstream issue with CPAN. In perl 5.8.8, it was necessary for Dual Life modules to install to INSTALLDIRS=perl. The reason was that in 5.8.8, the modules which came with Perl came first in the INC path. This made it impossible to upgrade these modules without installing them to that location. I can give you a list of the dual life modules and show you that they all behave this way. This was fixed in 5.12 by among other commits, b9ba2fadb18. Red Hat fixed it early in 5.8.8, however this is irrelevant to the issue at hand. NOTE: I'm using i386 in my example below but the problem is in x86_64 too. So the problem here is that when I install a module with INSTALLDIRS=perl, it's supposed to overwrite (bear with me I know you despise this part) what shipped with the perl RPM. The perl RPM installs them to: -bash-3.2# rpm -vV perl|egrep 'Cwd|File/Spec' ........ /usr/lib/perl5/5.8.8/File/Spec ........ /usr/lib/perl5/5.8.8/File/Spec.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Cygwin.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Epoc.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Functions.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Mac.pm ........ /usr/lib/perl5/5.8.8/File/Spec/OS2.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Unix.pm ........ /usr/lib/perl5/5.8.8/File/Spec/VMS.pm ........ /usr/lib/perl5/5.8.8/File/Spec/Win32.pm (These are all keying on the %Config value installprivlib) ........ /usr/lib/perl5/5.8.8/i386-linux-thread-multi/Cwd.pm ........ /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd ........ /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd/Cwd.so (These are all keying on the %Config value installarchlib) However if I install the module from CPAN, with installdirs=perl, it goes to this path because PathTools is an XS module so all of its pm files go into that path. Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Cwd/Cwd.so Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/Cwd.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Cygwin.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/VMS.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Functions.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Unix.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/OS2.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Win32.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Mac.pm Installing /usr/lib/perl5/5.8.8/i386-linux-thread-multi/File/Spec/Epoc.pm These all installed to installprivlib. In perl 5.8, this is what Dual Life modules are supposed to do. If the perl RPM that Red Hat ships, were to re-locate the File::Spec pm files to i386-linux-thread-multi, then all of the broken servers out there would magically self repair. It is my current belief (though I've not done a perlbrew to prove it definitively) that this is a bug in 5.8.8. However it only affects anyone who uses perl from a distro. Your solution to make this a upstream issue, dismisses the problem that even running /usr/bin/cpan at this point is broken on the servers in question so even if CPAN were to put a fix in for RedHat 5 only, it would be irrelevant since nobody affected could update from CPAN. The /usr/bin/cpan script breaks since requires PathTools to work. If you get past this with something like maybe cpanm, I'd also point out to you that ExtUtils::MakeMaker also breaks so you couldn't even install the new module version. I'm not objecting to the RPM downgrading Cwd.so and Cwd.pm. I'm saying that it needs to also downgrade the File::Spec pm files since the mismatch is what's causing the segfaults. This has no negative impact I can see and has the positive impact of restoring perl to a working state on your customer RHEL 5 servers.
Created attachment 945441 [details] This would correct the problem with PathTools segfaulting perl.
See perl -V for the order of the search paths: @INC: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 . It tries first site, then vendor, then core. Therefore if you install a module into site, it will take preference over the one provided by the RPM package. It works: # perl -e 'for (qw(Cwd File::Spec)) { eval "use $_; print qq{$_ \$${_}::VERSION\n}" }' Cwd 3.12 File::Spec 3.12 ## Now install Cwd module by cpan into site cpan> o conf makepl_arg INSTALLDIRS=site makepl_arg INSTALLDIRS=site cpan> o conf commit commit: wrote /usr/lib/perl5/5.8.8/CPAN/Config.pm [...] cpan> force install Cwd [...] Running make install Installing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.bs Installing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so Files found in blib/arch: installing files in blib/lib into architecture dependent library tree Installing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Cwd.pm Installing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/File/Spec.pm [...] ## Check the versions again # perl -e 'for (qw(Cwd File::Spec)) { eval "use $_; print qq{$_ \$${_}::VERSION\n}" }' Cwd 3.47 File::Spec 3.47 ## Reinstall perl RPM package # yum reinstall perl [...] ## And check the versions again # perl -e 'for (qw(Cwd File::Spec)) { eval "use $_; print qq{$_ \$${_}::VERSION\n}" }' Cwd 3.47 File::Spec 3.47 Your patch would "fix" only File::Spec name space. What about other modules? And provided RHEL-5 is almost at the end of life, we are really not going to shuffle random files from here to there. RPM really cannot deal with files overwritten by a different (cpan) packaging system.
> ## Now install Cwd module by cpan into site > cpan> o conf makepl_arg INSTALLDIRS=site > makepl_arg INSTALLDIRS=site > > cpan> o conf commit > commit: wrote /usr/lib/perl5/5.8.8/CPAN/Config.pm That's great that you can do that in the CPAN client. Did you ever document this to sysadmins? What about the ones who did "cpan Moose" or something and it required a newer PathTools automatically. They probably never even noticed that PathTools upgraded. > Your patch would "fix" only File::Spec name space. What about other modules? > And provided RHEL-5 is almost at the end of life, we are really not going to > shuffle random files from here to there. The other files are not causing a segfault of perl when common scripts are run. This package set is. At the moment, this is the only set of perl modules I see as at issue. > RPM really cannot deal with files overwritten by a different (cpan) packaging system. That gets into a very large philosophical discussion about ARCHLIB, VENDORLIB (which I don't think 5.8.8 even had), and site_perl when it comes to packaged perl. Again I would point out it distracts from the issue. The fact remains that Redhat has never shipped /usr/bin/cpan with a warning that it should never be used or a hack to ExtUtils::MakeMaker and Module::Build to never honor INSTALLDIRS=perl. Yes it's true that you guys fixed site_perl (perl-5.8.5-incorder.patch) to come first. I think that's awesome. But that's not what the rest of the world had to deal with. They had to deal with arch_lib coming first. So the commonly agreed on solution for the rest of the world was that dual life modules would have to install to the ARCHLIB path. IMO by changing the INC order, you only fixed half of the problem. I realize RHEL 5 is close to EOL but the fact remains the act of them having somehow installed a newer version of PathTools from cpan and then upgraded or reinstalled perl is that they new have a segfaulting perl. None of your solutions fix the fact that the servers' perl is now fried. It's your call if you want to fix it.
If you have a problem which you think Red Hat could fix for you, please contact Red Hat support.
We encountered this recently when upgrading 5.4 and 5.6 RHEL machines to 5.11. This bug gave some good insights. Our "fix" was a hack, but it worked: \rm -fr /usr/lib/perl5 /usr/lib64/perl5 ... removes all RPM and CPAN libs yum reinstall perl ... to replace RPM-provided libs and then rpm -a | grep perl to see other RPM-provided perl libs, and yum reinstall them as well.... Finally, use CPAN to recompile any remaining modules we needed, compiling them against current 5.11 dependencies rather than older stuff. Worked... not really an automated solution, but it worked.