Bug 379791 - perl bless/overload performance problem
perl bless/overload performance problem
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: perl (Show other bugs)
5.2
All Linux
urgent Severity high
: rc
: ---
Assigned To: Marcela Mašláňová
BaseOS QE
GSSApproved
: ZStream
: 460095 (view as bug list)
Depends On: 460308
Blocks:
  Show dependency treegraph
 
Reported: 2007-11-13 07:14 EST by Nigel Metheringham
Modified: 2010-10-22 16:21 EDT (History)
45 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 16:10:34 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Nigel Metheringham 2007-11-13 07:14:04 EST
RHEL5 perl shows the same performance issues as the Fedora 7 perl did - see
Bug #196836 and Bug #253728

This has been demonstrated in the recent perl update perl-5.8.8-10.el5_0.2

Same fix needs taking across to RHEL, ideally as a update release rather than
waiting for next major release cycle.

I do not have RHEL5.1 to test against right now, but the timing of the Fedora
fixes leads me to believe these would be much too late for the 5.1 release
cycle.
Comment 2 Martin Kutter 2007-11-30 05:24:01 EST
The issue can be observed running the benchmark script from the recent
SOAP::WSDL package.

To do so, download SOAP-WSDL-2.00_24 (and its dependencies) from CPAN, run perl
Build.PL && perl Build, cd into benchmark and run perl -I../blib/lib 01_expat.t

This is the Output from RHEL4:
perl -I../lib 01_expat.t
Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL):  4 wallclock secs ( 3.48 usr +  0.01 sys =  3.49 CPU) @
1432.66/s (n=5000)
XML::Simple (Hash):  7 wallclock secs ( 7.19 usr +  0.03 sys =  7.22 CPU) @
692.52/s (n=5000)
XSD (SOAP::WSDL):  6 wallclock secs ( 6.06 usr +  0.01 sys =  6.07 CPU) @
823.72/s (n=5000)

And this (with reduced n) is from RHEL5 (different machine, perl-5.8.8-10):

Benchmark: timing 500 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL):  1 wallclock secs ( 0.59 usr +  0.00 sys =  0.59 CPU) @
847.46/s (n=500)
XML::Simple (Hash):  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @
471.70/s (n=500)
XSD (SOAP::WSDL): 11 wallclock secs (11.34 usr +  0.01 sys = 11.35 CPU) @
44.05/s (n=500)

Increasing the number of runs shows the O(n^2) nature of the performance problem
- increasing the number of runs by a factor of 10 increases the runtime for the
XSD bench by a factor of nearly 100:

Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL):  6 wallclock secs ( 6.19 usr +  0.03 sys =  6.22 CPU) @
803.86/s (n=5000)
XML::Simple (Hash): 11 wallclock secs (11.20 usr +  0.02 sys = 11.22 CPU) @
445.63/s (n=5000)
XSD (SOAP::WSDL): 851 wallclock secs (847.36 usr +  2.28 sys = 849.64 CPU) @ 
5.88/s (n=5000)
Comment 3 RHEL Product and Program Management 2007-12-03 15:47:35 EST
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release.  This request will
be reviewed for a future Red Hat Enterprise Linux release.
Comment 4 ritz 2008-02-22 06:50:04 EST
Patch - http://cvs.fedoraproject.org/viewcvs/rpms/perl/F
-7/perl-5.8.8-U28775.patch?rev=1.1&view=markup


_TEST_ packages with the above mentioned fix can be found at -
http://people.redhat.com/rkhadgar/work/bz379791/
Comment 5 ritz 2008-02-22 06:51:25 EST
Steps to Reproduce:
1. Use the test script below

#!/usr/bin/perl
use overload q(<) => sub {};
my %h;
for (my $i=0; $i<50000; $i++) {
   $h{$i} = bless [ ] => 'main';
   print STDERR '.' if $i % 1000 == 0;
}
---------------------------------------------------

as an executable file "stour-test.pl".

2. Type

$ time ./stour-test.pl

and the program will print 50 dots.

The program should complete in less than 1 second, but takes much longer under
RHEL5.
Comment 6 Marcela Mašláňová 2008-04-28 09:28:58 EDT
This issue was solved with patch perl-5.8.8-U28775.patch in F-8. It's also
working fix for RHEL-5.
Comment 7 RHEL Product and Program Management 2008-06-02 16:28:35 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 8 Nigel Metheringham 2008-06-05 06:32:08 EDT
Confirmation that the RHEL5.2 perl package (perl-5.8.8-10.el5_0.2) has not
been fixed so RHEL still has a perl package which is effectively unusable
for many applications (ie anything using DBIx::Class, SOAP etc).

Comment 10 Amos Shapira 2008-07-16 07:26:04 EDT
Can anyone point to an alternative perl package which can be used on RHEL5.2
instead of the default one?

Thanks.
Comment 11 Marcela Mašláňová 2008-07-16 07:40:00 EDT
What should this alternative do?

I'll fix this problem for 5.3, so it will be in next update.
Comment 13 Peter 2008-08-07 12:38:26 EDT
Will the new package be backported to RHEL5U2 ?
I would not like to be forced to upgrade 60 servers in order to get perl bug fixed.

Any timescale estimations ?
It has really big impact many critical applications.
Comment 14 Marcela Mašláňová 2008-08-08 02:42:52 EDT
This bug will be solved in RHEL5U3.
Comment 15 Peter 2008-08-08 03:43:01 EDT
Well, it does not answer any of my questions.
Comment 16 Ludek Smid 2008-08-08 03:54:45 EDT
(In reply to comment #13)
> Will the new package be backported to RHEL5U2 ?
> Any timescale estimations ?
> It has really big impact many critical applications.

Since Bugzilla is not a customer support tool, it's unlikely that you can receive response in this way.

I recommend to contact Global Support Services [https://www.redhat.com/apps/support/] to escalate this request.
Comment 17 Peter 2008-08-08 04:50:29 EDT
Well, it does not answer any of my questions.
Comment 18 Peter 2008-08-08 04:55:19 EDT
 
> Since Bugzilla is not a customer support tool, it's unlikely that you can
> receive response in this way.
> 
> I recommend to contact Global Support Services
> [https://www.redhat.com/apps/support/] to escalate this request.


I did it 3 days ago (many of my friends over a week ago) 
and RH does not even bother to comment on it.
RH Support is not the best place to ask for support.
Comment 26 Peter 2008-08-21 07:07:02 EDT
I just received hotfix from RedHat, the test script above runs 50 times faster on
patched perl comparing to default on RHEL4.5 but i still see slowness when using perl::DBI with Mysql
I will hopefully post some more detailed examples soon
Comment 27 Peter 2008-08-21 07:10:23 EDT
"comparing to default on RHEL5.2" - of course
Comment 28 Ask Bjørn Hansen 2008-08-25 15:20:07 EDT
Marcela,

What is/was RedHat trying to fix with whatever patch to Perl 5.8.8 that introduced this problem?


    - ask
Comment 29 Florin Andrei 2008-08-25 16:01:56 EDT
Congrats, you're on Reddit now:

http://www.reddit.com/comments/6xxaj/massive_redhat_perl_performance_issue/
Comment 30 Dag Wieers 2008-08-25 22:21:35 EDT
And on Steve Ballmer's blog as well:

http://linuxhaters.blogspot.com/2008/08/rants-and-laughs-3.html
Comment 31 Nigel Metheringham 2008-08-26 07:09:09 EDT
Note that the blog article[1] that is causing the current surge of activity
says that:-

  1. The patch this bug report links to only mitigates the performance
     slow down due to bless/overload  (it appears to be 30x faster than
     current stock 5.2 perl, but still 2x slower than own built perl --
     - figures were taken from response by "Chad Johnson")

  2. There is still an exponential performance slow down even with
     the patches applied - it just kicks in in a different manner.
     Meaning that the basic tests we have been using to show this bug
     may be inadequate.


[1] http://blog.vipul.net/2008/08/24/redhat-perl-what-a-tragedy/

I would suggest a more thorough review of the patch prior to update or 5.3 release, especially in light of the hate you currently appear to be gathering.
Comment 32 Dave Cross 2008-08-26 09:59:15 EDT
There's a blog post by the Perl Pumpking about this issue. He includes details of the patchsets that Red Hat are missing.

http://use.perl.org/~nicholas/journal/37274

Dave...
Comment 33 Marcela Mašláňová 2008-08-26 10:09:14 EDT
Yeah I know about it. I'm still working on it.
Comment 37 Marcela Mašláňová 2008-08-27 10:59:26 EDT
*** Bug 460095 has been marked as a duplicate of this bug. ***
Comment 39 Marcela Mašláňová 2008-08-27 11:20:57 EDT
Pre and post hotfix runs of the test case:
--------------------------------------------------------------------------
PRE:
[root@panda perl_xml_benchmark]#  perl -d:NYTProf xml_test.pl 

==================================================
Test Case : 10 took 20.4251170158386 seconds
==================================================

==================================================
Test Case : 100 took 29.8565039634705 seconds
==================================================

==================================================
Test Case : 500 took 31.9550549983978 seconds
==================================================

==================================================
Test Case : 1000 took 31.2328250408173 seconds
==================================================

==================================================
Test Case : 10000 took 33.8995940685272 seconds
==================================================

==================================================
Test Case : 100000 took 57.2464160919189 seconds
==================================================

==================================================
Test Case : 1000000 took 254.903658151627 seconds
==================================================

--------------------------------------------------------------------------
POST:
[root@panda perl_xml_benchmark]#  perl -d:NYTProf xml_test.pl 

==================================================
Test Case : 10 took 14.1811721324921 seconds
==================================================

==================================================
Test Case : 100 took 19.1166498661041 seconds
==================================================

==================================================
Test Case : 500 took 19.3935630321503 seconds
==================================================

==================================================
Test Case : 1000 took 18.9018168449402 seconds
==================================================

==================================================
Test Case : 10000 took 20.9931077957153 seconds
==================================================

==================================================
Test Case : 100000 took 32.6134688854218 seconds
==================================================


==================================================
Test Case : 1000000 took 131.045488834381 seconds
==================================================
[root@panda perl_xml_benchmark]#
Comment 42 Peter 2008-08-28 06:47:16 EDT
I am sorry about that, but it seems that RH Support wakes up after i state
publicly it does not do it's job properly.
So I am doing it again:

--------------------------------------------------------
RedHat Support is not the best place to ask for support.
--------------------------------------------------------

And Yes, Mrs Quality Assurance Analyst i am very unhappy about the way
Redhat deals with this issue.

May i (i guess not only me) get perl 5.8.8 package for RHEL5 stripped of
all of these broken patches.

But If Steve Balmer's laugh did not wake you up i am afraid nothing else would.
Comment 44 Stepan Kasal 2008-08-28 12:09:04 EDT
(In reply to comment #28)
> What is/was RedHat trying to fix with whatever patch to Perl 5.8.8 that
> introduced this problem?

The patch is necessary to fix
http://rt.perl.org/rt3/Public/Bug/Display.html?id=34925

This is a bug in perl-5.8.8 which causes "overload and rebless" being interpreted
incorrectly; it's a real-life problem affecting modules on CPAN, e.g., Class::DBI.
Comment 48 Harald Klein 2008-08-29 12:20:02 EDT
tested perl-5.8.8-15.el5_2.1 (latest fix), looks much better:

[root@pinky perl_xml_benchmark]#  perl -d:NYTProf xml_test.pl 

==================================================
Test Case : 10 took 3.41057705879211 seconds
==================================================

==================================================
Test Case : 100 took 3.40202307701111 seconds
==================================================

==================================================
Test Case : 500 took 3.37363600730896 seconds
==================================================

==================================================
Test Case : 1000 took 3.33193588256836 seconds
==================================================

==================================================
Test Case : 10000 took 3.42702007293701 seconds
==================================================

==================================================
Test Case : 100000 took 3.35513401031494 seconds
==================================================

==================================================
Test Case : 1000000 took 3.37334394454956 seconds
==================================================

best regards,
Hari
Comment 52 Peter 2008-09-01 03:01:11 EDT
I can confirm that.
No more problems with Class::DBI with 5.8.8-15.el5_2.1 .
Comment 53 kelsey hudson 2008-09-03 13:36:00 EDT
Where can this updated package be found? I don't see it in the updates for RHEL5 nor the beta channel on RHN.  Am I missing something?
Comment 54 Peter 2008-09-03 14:29:05 EDT
As far as i know it is not the final RHEL5.3 package yet, just supported hostfix.
You need to request it via Redhat Support.
Comment 56 Radek Vokal 2008-09-08 03:15:28 EDT
We can't offer a solution to our customers that we don't support. Marking comment 55 as private.
Comment 57 R P Herrold 2008-09-08 10:31:01 EDT
It seems like Red Hat cannot figure out its mind these days, and is blowing hot and cold on what in the world Bugzilla is for.

Earlier in THIS VERY BUG:

https://bugzilla.redhat.com/show_bug.cgi?id=379791#c16

'Since Bugzilla is not a customer support tool, it's unlikely that you can
receive response in this way.' per lsmid@redhat.com 

... hmmm.  It also appears that comments #19 through #25 in this bug have been elided. It is hard to get an accurate read, when a non-insider viewing the bug sees only the happy talk

Either Bugzilla is something for support or it is not.  'Bugzilla is not a support tool' [citation below] is pretty unambiguous from cevich@redhat.com; 'We can't offer a solution to our customers that we don't support' by rvokal@redhat.com in #56 mischaracterizes the pointer to a solution CentOS have offered -- no support offer is made by Red Hat; no imprimatur from Red Hat was suggested by the post.

https://bugzilla.redhat.com/show_bug.cgi?id=448014#c56

  -------  Comment #56 From  Chris Evich (cevich@redhat.com )  2008-07-24 17:31:16 EDT   (-) [reply] -------

While defects can be reported using Red Hat's Bugzilla, please be advised that
Bugzilla is not a support tool and thus does not offer any Service Level
Agreement (SLA) for issues reported through that channel.  If you require 
technical support assistance, please direct those requests as per you support
contract with us.  Our technical support staff would be glad to assist you with
issues relating to this bug.

===================

Please restore visibility to #56

-- Russ herrold
Comment 59 Nigel Metheringham 2008-09-18 04:02:14 EDT
[For information, since this bug has a bigger readership than many newsgroups...]

Related bug #460308 has been closed off as an advisory and errata packages
have been issued.

See
   http://rhn.redhat.com/errata/RHBA-2008-0876.html

Bug #460308 should be reopened if the solution does not work for you.
Comment 71 errata-xmlrpc 2009-01-20 16:10:34 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0117.html

Note You need to log in before you can comment on or make changes to this bug.