Bug 1652872

Summary: glibc: Compose instability leads to upgrade problems related to glibc-headers.i686
Product: [Fedora] Fedora Reporter: Maciej Żenczykowski <zenczykowski>
Component: glibcAssignee: Carlos O'Donell <codonell>
Status: CLOSED CANTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: andrea.vai, aoliva, arjun.is, aros, codonell, danielsun3164, dj, fweimer, law, mfabian, nouveau, pfrankli, rth, siddhesh, zenczykowski
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-23 11:57:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Maciej Żenczykowski 2018-11-23 11:21:12 UTC
[root@eonwe ~]# rpm -qa | egrep glibc | sort
glibc-2.28-17.fc29.i686
glibc-2.28-17.fc29.x86_64
glibc-all-langpacks-2.28-17.fc29.x86_64
glibc-common-2.28-17.fc29.x86_64
glibc-devel-2.28-17.fc29.i686
glibc-devel-2.28-17.fc29.x86_64
glibc-headers-2.28-17.fc29.i686
glibc-headers-2.28-17.fc29.x86_64
glibc-static-2.28-17.fc29.x86_64

[root@eonwe ~]# dnf upgrade
Last metadata expiration check: 1:41:43 ago on Fri 23 Nov 2018 01:38:42 AM PST.
Dependencies resolved.

 Problem: cannot install both glibc-headers-2.28-20.fc29.x86_64 and glibc-headers-2.28-17.fc29.x86_64
  - glibc-headers-2.28-17.fc29.i686 has inferior architecture
  - cannot install the best update candidate for package glibc-headers-2.28-17.fc29.x86_64
  - problem with installed package glibc-headers-2.28-17.fc29.i686
=============================================================================================================================================
 Package                              Arch                          Version                             Repository                      Size
=============================================================================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 glibc-headers                        x86_64                        2.28-20.fc29                        updates                        463 k

Transaction Summary
=============================================================================================================================================
Skip  1 Package

Nothing to do.
Complete!





[root@eonwe ~]# dnf distro-sync
Last metadata expiration check: 1:42:02 ago on Fri 23 Nov 2018 01:38:42 AM PST.
Dependencies resolved.

 Problem 1: package glibc-headers-2.28-9.fc29.i686 requires glibc = 2.28-9.fc29, but none of the providers can be installed
  - cannot install both glibc-2.28-9.fc29.i686 and glibc-2.28-20.fc29.i686
  - cannot install both glibc-2.28-9.fc29.x86_64 and glibc-2.28-20.fc29.x86_64
  - cannot install the best update candidate for package glibc-headers-2.28-17.fc29.i686
  - cannot install the best update candidate for package glibc-2.28-17.fc29.i686
  - cannot install the best update candidate for package glibc-2.28-17.fc29.x86_64
 Problem 2: glibc-headers-2.28-9.fc29.i686 has inferior architecture
  - cannot install both glibc-headers-2.28-9.fc29.x86_64 and glibc-headers-2.28-20.fc29.x86_64
  - problem with installed package glibc-headers-2.28-17.fc29.i686
  - cannot install the best update candidate for package glibc-headers-2.28-17.fc29.x86_64
  - glibc-headers-2.28-17.fc29.i686 does not belong to a distupgrade repository
=============================================================================================================================================
 Package                                  Arch                        Version                             Repository                    Size
=============================================================================================================================================
Downgrading:
 glibc                                    i686                        2.28-9.fc29                         fedora                       3.5 M
 glibc                                    x86_64                      2.28-9.fc29                         fedora                       3.8 M
 glibc-all-langpacks                      x86_64                      2.28-9.fc29                         fedora                        25 M
 glibc-common                             x86_64                      2.28-9.fc29                         fedora                       795 k
 glibc-devel                              i686                        2.28-9.fc29                         fedora                       1.0 M
 glibc-devel                              x86_64                      2.28-9.fc29                         fedora                       1.0 M
 glibc-headers                            i686                        2.28-9.fc29                         fedora                       460 k
 glibc-headers                            x86_64                      2.28-9.fc29                         fedora                       460 k
 glibc-static                             x86_64                      2.28-9.fc29                         fedora                       1.7 M
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 glibc                                    i686                        2.28-9.fc29                         fedora                       3.5 M
 glibc                                    x86_64                      2.28-9.fc29                         fedora                       3.8 M
 glibc                                    i686                        2.28-20.fc29                        updates                      3.5 M
 glibc                                    x86_64                      2.28-20.fc29                        updates                      3.8 M
 glibc-headers                            x86_64                      2.28-9.fc29                         fedora                       460 k
 glibc-headers                            x86_64                      2.28-20.fc29                        updates                      463 k
Skipping packages with broken dependencies:
 glibc-headers                            i686                        2.28-9.fc29                         fedora                       460 k

Transaction Summary
=============================================================================================================================================
Downgrade  9 Packages
Skip       7 Packages

Total download size: 38 M
Is this ok [y/N]: n
Operation aborted.

Comment 1 Maciej Żenczykowski 2018-11-23 11:22:58 UTC
At a guess glibc-headers-2.28-20.fc29.i686 is missing from the repos...

Comment 2 Maciej Żenczykowski 2018-11-23 11:29:24 UTC
Fetched glibc from https://koji.fedoraproject.org/koji/buildinfo?buildID=1165441

[root@eonwe ~]# ls *rpm
glibc-2.28-20.fc29.i686.rpm                  glibc-common-2.28-20.fc29.i686.rpm    glibc-headers-2.28-20.fc29.i686.rpm
glibc-2.28-20.fc29.x86_64.rpm                glibc-common-2.28-20.fc29.x86_64.rpm  glibc-headers-2.28-20.fc29.x86_64.rpm
glibc-all-langpacks-2.28-20.fc29.i686.rpm    glibc-devel-2.28-20.fc29.i686.rpm     glibc-static-2.28-20.fc29.i686.rpm
glibc-all-langpacks-2.28-20.fc29.x86_64.rpm  glibc-devel-2.28-20.fc29.x86_64.rpm   glibc-static-2.28-20.fc29.x86_64.rpm

[root@eonwe ~]# rpm -hvU glibc-*.rpm
error: Failed dependencies:
        libxcrypt-static(x86-32) >= 4.0.0 is needed by glibc-static-2.28-20.fc29.i686

Comment 3 Florian Weimer 2018-11-23 11:36:40 UTC
Have you enabled the updates-testing repository?

Comment 4 Maciej Żenczykowski 2018-11-23 11:38:03 UTC
Looks like libxcrypt-static is a red herring, I didn't actually have glibc-static i686 installed.

[root@eonwe ~]# rpm -q libxcrypt-static
libxcrypt-static-4.3.4-1.fc29.x86_64

[root@eonwe ~]# rm glibc-static-2.28-20.fc29.i686.rpm 
rm: remove regular file 'glibc-static-2.28-20.fc29.i686.rpm'? y

[root@eonwe ~]# rpm -hvU glibc-*.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
        file /usr/bin/ldd conflicts between attempted installs of glibc-common-2.28-20.fc29.i686 and glibc-common-2.28-20.fc29.x86_64

... something still seems wrong ...
apparently only need one common

[root@eonwe ~]# rpm -q glibc-common
glibc-common-2.28-17.fc29.x86_64
[root@eonwe ~]# rm glibc-common-2.28-20.fc29.i686.rpm 
rm: remove regular file 'glibc-common-2.28-20.fc29.i686.rpm'? y
[root@eonwe ~]# rpm -hvU glibc-*.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:glibc-all-langpacks-2.28-20.fc29 ################################# [  5%]
   2:glibc-common-2.28-20.fc29        ################################# [ 11%]
   3:glibc-2.28-20.fc29               ################################# [ 16%]
   4:glibc-headers-2.28-20.fc29       ################################# [ 21%]
   5:glibc-devel-2.28-20.fc29         ################################# [ 26%]
   6:glibc-all-langpacks-2.28-20.fc29 ################################# [ 32%]
   7:glibc-headers-2.28-20.fc29       ################################# [ 37%]
   8:glibc-2.28-20.fc29               ################################# [ 42%]
   9:glibc-devel-2.28-20.fc29         ################################# [ 47%]
  10:glibc-static-2.28-20.fc29        ################################# [ 53%]
Cleaning up / removing...
  11:glibc-devel-2.28-17.fc29         ################################# [ 58%]
  12:glibc-2.28-17.fc29               ################################# [ 63%]
  13:glibc-static-2.28-17.fc29        ################################# [ 68%]
  14:glibc-devel-2.28-17.fc29         ################################# [ 74%]
  15:glibc-headers-2.28-17.fc29       ################################# [ 79%]
  16:glibc-headers-2.28-17.fc29       ################################# [ 84%]
  17:glibc-2.28-17.fc29               ################################# [ 89%]
  18:glibc-all-langpacks-2.28-17.fc29 ################################# [ 95%]
  19:glibc-common-2.28-17.fc29        ################################# [100%]

[root@eonwe ~]# dnf upgrade
Last metadata expiration check: 1:54:15 ago on Fri 23 Nov 2018 01:38:42 AM PST.
Dependencies resolved.
Nothing to do.
Complete!

[root@eonwe ~]# dnf distro-sync
Last metadata expiration check: 1:54:22 ago on Fri 23 Nov 2018 01:38:42 AM PST.
Error: 
 Problem: package glibc-all-langpacks-2.28-20.fc29.i686 requires glibc = 2.28-20.fc29, but none of the providers can be installed
  - glibc-2.28-20.fc29.i686 has inferior architecture
  - cannot install both glibc-2.28-9.fc29.x86_64 and glibc-2.28-20.fc29.x86_64
  - glibc-2.28-9.fc29.i686 has inferior architecture
  - package glibc-devel-2.28-9.fc29.x86_64 requires glibc = 2.28-9.fc29, but none of the providers can be installed
  - problem with installed package glibc-devel-2.28-20.fc29.x86_64
  - package glibc-devel-2.28-20.fc29.x86_64 requires glibc-headers = 2.28-20.fc29, but none of the providers can be installed
  - cannot install both glibc-headers-2.28-9.fc29.x86_64 and glibc-headers-2.28-20.fc29.x86_64
  - glibc-headers-2.28-9.fc29.i686 has inferior architecture
  - problem with installed package glibc-headers-2.28-20.fc29.i686
  - glibc-headers-2.28-20.fc29.i686 does not belong to a distupgrade repository
  - problem with installed package glibc-all-langpacks-2.28-20.fc29.i686
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

[root@eonwe ~]# dnf list obsoletes
Last metadata expiration check: 1:54:30 ago on Fri 23 Nov 2018 01:38:42 AM PST.

[root@eonwe ~]# dnf list extras
Last metadata expiration check: 1:54:34 ago on Fri 23 Nov 2018 01:38:42 AM PST.
Extra Packages
glibc-all-langpacks.i686                                                2.28-20.fc29                                                 @System 
glibc-headers.i686                                                      2.28-20.fc29                                                 @System 
kernel-core.x86_64                                                      4.19.2-300.fc29                                              @updates
kernel-modules.x86_64                                                   4.19.2-300.fc29                                              @updates
kernel-modules-extra.x86_64                                             4.19.2-300.fc29                                              @updates


(and apparently installing glibc-all-langpacks.i686 was a screw up, so had to uninstall it and reinstall the x86_64 version, cause uninstalling it screwed things up even more)

------======------

So my interpretation is that:

  glibc-headers.i686 2.28-20.fc29 

didn't get published.

Comment 5 Maciej Żenczykowski 2018-11-23 11:39:27 UTC
# cat /etc/yum.repos.d/fedora-updates-testing.repo  | egrep enabled
enabled=0
enabled=0
enabled=0

I think no?

Comment 6 Maciej Żenczykowski 2018-11-23 11:43:39 UTC
The packages are definitely not coming from testing...

[root@eonwe ~]# dnf reinstall glibc\*
Last metadata expiration check: 0:00:40 ago on Fri 23 Nov 2018 03:40:38 AM PST.
Dependencies resolved.
=============================================================================================================================================
 Package                                  Arch                        Version                             Repository                    Size
=============================================================================================================================================
Reinstalling:
 glibc                                    i686                        2.28-20.fc29                        updates                      3.5 M
 glibc                                    x86_64                      2.28-20.fc29                        updates                      3.8 M
 glibc-all-langpacks                      x86_64                      2.28-20.fc29                        updates                       25 M
 glibc-common                             x86_64                      2.28-20.fc29                        updates                      797 k
 glibc-devel                              i686                        2.28-20.fc29                        updates                      1.0 M
 glibc-devel                              x86_64                      2.28-20.fc29                        updates                      1.0 M
 glibc-headers                            x86_64                      2.28-20.fc29                        updates                      463 k
 glibc-static                             x86_64                      2.28-20.fc29                        updates                      1.7 M

Transaction Summary
=============================================================================================================================================

Total download size: 37 M
Installed size: 256 M
Is this ok [y/N]:

Comment 7 Maciej Żenczykowski 2018-11-23 11:47:06 UTC
This is confusing... the glibc-headers i686 does not appear to be required any more now that I've upgraded...

Before:

[root@eonwe ~]# rpm -qa | egrep glibc | sort
glibc-2.28-17.fc29.i686
glibc-2.28-17.fc29.x86_64
glibc-all-langpacks-2.28-17.fc29.x86_64
glibc-common-2.28-17.fc29.x86_64
glibc-devel-2.28-17.fc29.i686
glibc-devel-2.28-17.fc29.x86_64
glibc-headers-2.28-17.fc29.i686
glibc-headers-2.28-17.fc29.x86_64
glibc-static-2.28-17.fc29.x86_64

Now:

[root@eonwe ~]# rpm -qa | egrep ^glibc- | sort
glibc-2.28-20.fc29.i686
glibc-2.28-20.fc29.x86_64
glibc-all-langpacks-2.28-20.fc29.x86_64
glibc-common-2.28-20.fc29.x86_64
glibc-devel-2.28-20.fc29.i686
glibc-devel-2.28-20.fc29.x86_64
glibc-headers-2.28-20.fc29.x86_64
glibc-static-2.28-20.fc29.x86_64

Perhaps some sort of packaging change?

Comment 8 Maciej Żenczykowski 2018-11-23 11:53:35 UTC
The system isn't complaining any more...

[root@eonwe ~]# dnf upgrade
Last metadata expiration check: 0:06:44 ago on Fri 23 Nov 2018 03:40:38 AM PST.
Dependencies resolved.
Nothing to do.
Complete!

[root@eonwe ~]# dnf distro-sync
Last metadata expiration check: 0:06:49 ago on Fri 23 Nov 2018 03:40:38 AM PST.
Dependencies resolved.
Nothing to do.
Complete!

======

Maybe there's a missing obsoletes? or maybe if these files are not arch specific they should be noarch?

I decompressed both glibc-headers rpms (i686 and x86_64) and the diff is:

[root@eonwe gl]# diff -Naur i686 x86_64
diff -Naur i686/HEADER x86_64/HEADER
--- i686/HEADER 2018-11-19 00:00:00.000000000 -0800
+++ x86_64/HEADER       2018-11-19 00:00:00.000000000 -0800
@@ -1,15 +1,15 @@
 Name        : glibc-headers
 Version     : 2.28
 Release     : 20.fc29
-Architecture: i686
+Architecture: x86_64
 Install Date: (not installed)
 Group       : Unspecified
 Size        : 2020193
 License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+ and GPLv2+ with exceptions and BSD and Inner-Net and ISC and Public Domain and GFDL
 Signature   : (none)
 Source RPM  : glibc-2.28-20.fc29.src.rpm
-Build Date  : Mon Nov 19 05:35:01 2018
-Build Host  : buildvm-32.phx2.fedoraproject.org
+Build Date  : Mon Nov 19 05:33:09 2018
+Build Host  : buildvm-30.phx2.fedoraproject.org
 Relocations : (not relocatable)
 Packager    : Fedora Project
 Vendor      : Fedora Project
diff -Naur i686/INFO/BUILDHOST x86_64/INFO/BUILDHOST
--- i686/INFO/BUILDHOST 2018-11-19 00:00:00.000000000 -0800
+++ x86_64/INFO/BUILDHOST       2018-11-19 00:00:00.000000000 -0800
@@ -1 +1 @@
-buildvm-32.phx2.fedoraproject.org
+buildvm-30.phx2.fedoraproject.org
diff -Naur i686/INFO/BUILDTIME x86_64/INFO/BUILDTIME
--- i686/INFO/BUILDTIME 2018-11-19 00:00:00.000000000 -0800
+++ x86_64/INFO/BUILDTIME       2018-11-19 00:00:00.000000000 -0800
@@ -1 +1 @@
-Mon Nov 19 05:35:01 2018
+Mon Nov 19 05:33:09 2018
diff -Naur i686/INFO/PROVIDES x86_64/INFO/PROVIDES
--- i686/INFO/PROVIDES  2018-11-19 00:00:00.000000000 -0800
+++ x86_64/INFO/PROVIDES        2018-11-19 00:00:00.000000000 -0800
@@ -1,3 +1,3 @@
 glibc-headers = 2.28-20.fc29
-glibc-headers(i686)  
-glibc-headers(x86-32) = 2.28-20.fc29
+glibc-headers(x86-64) = 2.28-20.fc29
+glibc-headers(x86_64)  

Since the content of the packages appears identical... why do they have a different set of provides (and/or why aren't they noarch: but maybe they do differ on other archs).  It seems like at least glibc-headers x86_64 should have provided i686/x86-32 ?

Eh, anyway, my machine is fixed... and I don't know how it got in this state or what triggered the problem.

Comment 9 Florian Weimer 2018-11-23 11:57:56 UTC
The usual situation is this:

$ rpm -q glibc-devel glibc-headers
glibc-devel-2.26-30.fc27.x86_64
glibc-devel-2.26-30.fc27.i686
glibc-headers-2.26-30.fc27.x86_64

That is, only the glibc-devel package is multilib'ed.  glibc-headers is good for both architectures, similar to glibc-common.

There are no conflicts which prevent the installation of glibc-headers, though, so if you installed it manually at one point, you will experience problems during upgrades.

Comment 10 Maciej Żenczykowski 2018-11-23 12:01:15 UTC
Fair enough... but note there's some publishing inconsistency, since the:
 glibc-headers i686 2.28-9.fc29 fedora 460 k
version of the package does get found by dnf... (see comment #1 for more details)

Comment 11 Florian Weimer 2018-11-25 12:57:55 UTC
*** Bug 1653039 has been marked as a duplicate of this bug. ***

Comment 12 Florian Weimer 2018-11-26 07:59:09 UTC
*** Bug 1653183 has been marked as a duplicate of this bug. ***

Comment 13 Artem S. Tashkinov 2018-12-04 19:21:40 UTC
This has now slipped into Fedora 28:

 Problem 1: cannot install both glibc-headers-2.27-35.fc28.x86_64 and glibc-headers-2.27-32.fc28.x86_64
  - glibc-headers-2.27-32.fc28.i686 has inferior architecture
  - cannot install the best update candidate for package glibc-headers-2.27-32.fc28.x86_64
  - problem with installed package glibc-headers-2.27-32.fc28.i686
 Problem 2: package glibc-headers-2.27-32.fc28.i686 requires glibc = 2.27-32.fc28, but none of the providers can be installed
  - cannot install both glibc-2.27-35.fc28.i686 and glibc-2.27-32.fc28.i686
  - cannot install both glibc-2.27-35.fc28.x86_64 and glibc-2.27-32.fc28.x86_64
  - cannot install the best update candidate for package glibc-headers-2.27-32.fc28.i686
  - cannot install the best update candidate for package glibc-2.27-32.fc28.i686
  - cannot install the best update candidate for package glibc-2.27-32.fc28.x86_64
================================================================================
 Package              Arch          Version                Repository      Size
================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 glibc                i686          2.27-35.fc28           updates        3.4 M
 glibc                x86_64        2.27-35.fc28           updates        3.6 M
 glibc-headers        x86_64        2.27-35.fc28           updates        460 k

Transaction Summary
================================================================================
Skip  3 Packages

Nothing to do.
Complete!

Show I open another bug report? Why is it happening again?

Comment 14 Florian Weimer 2018-12-12 10:31:17 UTC
*** Bug 1658498 has been marked as a duplicate of this bug. ***

Comment 15 Florian Weimer 2018-12-12 10:35:42 UTC
We cannot fix this in the glibc package.  I raised this issue again on the development list:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/PI2MQIR77AN66GJ3BIS4MSEYXXRXO44E/

The workaround is to remove the glibc-headers.i686 package manually.

Comment 16 Florian Weimer 2018-12-14 12:09:41 UTC
*** Bug 1658497 has been marked as a duplicate of this bug. ***

Comment 17 Florian Weimer 2019-05-07 11:34:54 UTC
*** Bug 1707354 has been marked as a duplicate of this bug. ***

Comment 18 Roy 2019-05-07 11:45:13 UTC
Uninstalling glibc-headers.i686 is an unsatisfactory work-around, at least for me. As a nouveau OSS graphics driver developer (or on behalf of others in this community) I require glibc-headers.i686 for a 32-bit cross-compiled mesa build ([0]) in order to test the impact of changes on binary 32-bit applications e.g. those distributed through Steam. Without the ability to keep glibc-headers.i686 in sync with the rest of glibc(.686, .x86_64?) on my system, my workflow is severely disturbed.

[0] https://kojipkgs.fedoraproject.org//packages/mesa/19.0.3/1.fc30/data/logs/i686/root.log

Comment 19 Florian Weimer 2019-05-07 11:48:07 UTC
(In reply to Roy from comment #18)
> Uninstalling glibc-headers.i686 is an unsatisfactory work-around, at least
> for me. As a nouveau OSS graphics driver developer (or on behalf of others
> in this community) I require glibc-headers.i686 for a 32-bit cross-compiled
> mesa build ([0]) in order to test the impact of changes on binary 32-bit
> applications e.g. those distributed through Steam. Without the ability to
> keep glibc-headers.i686 in sync with the rest of glibc(.686, .x86_64?) on my
> system, my workflow is severely disturbed.
> 
> [0]
> https://kojipkgs.fedoraproject.org//packages/mesa/19.0.3/1.fc30/data/logs/
> i686/root.log

This is a 32-bit build root, which obviously has glibc-headers.i686.  It does not use a multilib compose, after all.

When you build 32-bit programs on a multilib x86_64 installation, the role of glibc-headers.i686 is assumed by glibc-headers.x86_64, in the same way that bash.x86_64 supersedes bash.i686.  There is no loss of functionality by removing this package and installing the x86_64 version.

Comment 20 Roy 2019-05-07 12:14:52 UTC
I see. This was not an assumption I dared to make; for such low-level components there's the off-chance that some arch-specific headers are generated rather than #ifdef'd to perfection. I'm glad the kernel is alone in this situation and that I therefore do not need to install both packages. In fact, having one overwrite the includes of the other sounds undesirable. That being said, I have not verified that rpmbuild will not complain about missing dependencies when targeting i686 on a x86_64 build platform. Even though the required include files might be the same, the rpm spec could still insist on the package. I will check this as soon as the opportunity arises.

I do wonder: If these headers are not architecture-specific, is there a good reason against composing this package for "noarch"... other than the potential legacy nightmare of having to alter thousands of spec files?

Comment 21 Florian Weimer 2019-05-08 11:12:56 UTC
(In reply to Roy from comment #20)
> I do wonder: If these headers are not architecture-specific, is there a good
> reason against composing this package for "noarch"... other than the
> potential legacy nightmare of having to alter thousands of spec files?

These header files are the same on i686 and x86_64, but they are still different from the other architectures.