Bug 574210 - mono stack completely broken with latest glibc update
Summary: mono stack completely broken with latest glibc update
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: 13
Hardware: All
OS: Linux
low
urgent
Target Milestone: ---
Assignee: H.J. Lu
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 554909
TreeView+ depends on / blocked
 
Reported: 2010-03-16 20:59 UTC by Christian Krause
Modified: 2010-03-24 00:50 UTC (History)
6 users (show)

Fixed In Version: glibc-2.11.90-16
Clone Of:
Environment:
Last Closed: 2010-03-24 00:50:28 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
A patch for memcmp-sse3.S (2.65 KB, patch)
2010-03-18 12:45 UTC, H.J. Lu
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Novell 588540 0 None None None Never

Description Christian Krause 2010-03-16 20:59:23 UTC
Description of problem:
The recent glibc update
https://admin.fedoraproject.org/updates/glibc-2.11.90-15
broke the whole mono stack. Most of the mono programs just crash with similar back traces:


Unhandled Exception: System.ArgumentNullException: Argument cannot be null.
Parameter name: CollectionBase.OnValidate: Invalid parameter value passed to method: null
  at System.Collections.CollectionBase.OnValidate (System.Object value) [0x00000] in <filename unknown>:0 
  at System.Collections.CollectionBase.System.Collections.IList.Add (System.Object value) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.ExpressionCollection.Add (System.Text.RegularExpressions.Syntax.Expression e) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.Assertion..ctor () [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.ExpressionAssertion..ctor () [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.Parser.ParseGroupingConstruct (System.Text.RegularExpressions.RegexOptions& options) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.Parser.ParseGroup (System.Text.RegularExpressions.Syntax.Group group, RegexOptions options, System.Text.RegularExpressions.Syntax.Assertion assertion) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Syntax.Parser.ParseRegularExpression (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Regex.CreateMachineFactory (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Regex.InitNewRegex () [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Regex.Init () [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Regex..ctor (System.String pattern, RegexOptions options) [0x00000] in <filename unknown>:0 
  at System.Text.RegularExpressions.Regex..ctor (System.String pattern) [0x00000] in <filename unknown>:0 
  at Mono.Options.OptionSet..ctor (System.Converter`2 localizer) [0x00000] in <filename unknown>:0 
  at Mono.Options.OptionSet..ctor () [0x00000] in <filename unknown>:0 
  at Monodoc.Driver.Main (System.String[] args) [0x00000] in <filename unknown>:0 


Version-Release number of selected component (if applicable):
glibc-2.11.90-15.i686


How reproducible:
100%

Steps to Reproduce:
1. install e.g. mono-tools
2. start "monodoc"

  
Actual results:
backtrace (see above)


Additional info:
mono itself can't be compiled anymore:
https://koji.fedoraproject.org/koji/taskinfo?taskID=2057255

no other mono packages can be compiled:
https://koji.fedoraproject.org/koji/taskinfo?taskID=2057232

This is basically the same problem I have reported in bodhi for the previous update:
https://admin.fedoraproject.org/updates/glibc-2.11.90-14

Comment 1 H.J. Lu 2010-03-17 15:00:05 UTC
Is that possible to find a testcase?

Comment 2 H.J. Lu 2010-03-17 17:44:50 UTC
I am trying to reproduce the problem. Since glibc has
processor-specific functions, please post

# cat /proc/cpuinfo

Comment 3 Christian Krause 2010-03-17 17:55:46 UTC
#:~$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     P8400  @ 2.26GHz
stepping        : 6
cpu MHz         : 2267.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips        : 4522.09
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     P8400  @ 2.26GHz
stepping        : 6
cpu MHz         : 800.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips        : 4521.75
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

It looks like that the problem does only happen on x86 systems. I could not reproduce this on x86_64.

The problem can also be reproduced on Fedora's build machines:
https://koji.fedoraproject.org/koji/taskinfo?taskID=2057255

I've observed also that the mono backtraces contains references to regular expression handling. Were there any regexp-related changes between the two glibc versions?

Comment 4 H.J. Lu 2010-03-17 18:01:24 UTC
There are some regexp failures in glibc on Sparc:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43385

They could be related. It may be a gcc bug.

Comment 5 Christian Krause 2010-03-17 18:15:41 UTC
(In reply to comment #4)
> There are some regexp failures in glibc on Sparc:
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43385
> They could be related. It may be a gcc bug. 

The description in the gcc bug report indicates that the bug only happens in gcc 4.5: "gcc-4.4.x and gcc-4.3.x do not have this failure.". Since F-13 uses gcc 4.4.3 the gcc bug seems to be unrelated to the mono issue.

Comment 6 H.J. Lu 2010-03-17 18:18:36 UTC
Gcc 4.4.3 in Fedora 13 contains some changes backported from trunk,
aka gcc 4.5.0, which aren't in FSF gcc 4.4.3.

Comment 7 H.J. Lu 2010-03-17 19:42:10 UTC
Something seems wrong with regexp on Fedora 13. When I build glibc,
I got

/usr/local/bin/ld: /export/build/gnu/glibc/build-i686-linux/nptl/tst-_res1.o: undefined reference to symbol '_res'
/usr/local/bin/ld: note: '_res' is defined in DSO /export/build/gnu/glibc/build-i686-linux/nptl/tst-_res1mod1.so so try adding it to the linker command line
/export/build/gnu/glibc/build-i686-linux/nptl/tst-_res1mod1.so: could not read symbols: Invalid operation
collect2: ld returned 1 exit status

Comment 8 Christian Krause 2010-03-17 20:49:59 UTC
(In reply to comment #7)
> Something seems wrong with regexp on Fedora 13. When I build glibc,

I've started the same test some hours ago, too. ;-)

On my F13 (fully updated) I could successfully rebuild glibc (make local in glibc/F-13).

I have also re-tested the failed glibc test cases from the mentioned gcc bug report: bug-regex20 and bug-regex11 work fine in F13 even with the new glibc.

Comment 9 H.J. Lu 2010-03-17 22:04:48 UTC
Glibc 2.11.90-12 is OK. 2.11.90-13 and above are bad.

Comment 10 H.J. Lu 2010-03-18 05:38:06 UTC
It is a memcmp bug. I have a patch.

Comment 11 H.J. Lu 2010-03-18 12:45:08 UTC
Created attachment 401004 [details]
A patch for memcmp-sse3.S

This patch updates %xmm3 when exit from loop.

Comment 12 Fedora Update System 2010-03-18 15:24:03 UTC
glibc-2.11.90-16 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/glibc-2.11.90-16

Comment 13 H.J. Lu 2010-03-18 17:24:15 UTC
After upgrading to glibc-2.11.90-16, monodoc works for me.

Comment 14 Christian Krause 2010-03-18 22:10:59 UTC
(In reply to comment #13)
> After upgrading to glibc-2.11.90-16, monodoc works for me.    

I've just tested the new glibc in F13 - the problem seems to be fully solved! 

- I've tested a bunch of mono applications which do all work now again without any problems.
- Compiling mono itself works again, too.
- No other regressions found so far in the new build: rebooting the system, desktop environment etc. work without any issues.

H.J., thank you very much for the quick investigation and the bug fix!

Comment 15 Fedora Update System 2010-03-23 02:25:24 UTC
glibc-2.11.90-16 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update glibc'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/glibc-2.11.90-16

Comment 16 Fedora Update System 2010-03-24 00:50:23 UTC
glibc-2.11.90-16 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.