Is this really a distribution thing?  Isn't the "primary" arch stuff set in rpm
itself, and what is really needed for ppc64 is the ability to have only some
apps prefer ppc64 and the rest not?

Doesn't seem to be a distribution problem. The primary versions of these
packages are available in the tree; it's just that the installer chooses to
install the secondary version instead.

Created attachment 150771 [details]
install.log from offending installation

(In reply to comment #1)
> Is this really a distribution thing?  

It's a distribution thing because things work the way the do based on the policy
we've set for the distro.

OK... what policy is that? We currently have ppc32 as 'primary' and ppc64 as
'secondary' on ppc64 hardware, don't we? What policy causes the installer to
install the secondary version of these packages and not the primary version?

(In reply to comment #5)
> OK... what policy is that? We currently have ppc32 as 'primary' and ppc64 as
> 'secondary' on ppc64 hardware, don't we? What policy causes the installer to
> install the secondary version of these packages and not the primary version?

I do believe there are two policies at work here.  During compose, there is a
policy about what is the primary arch and what is the multilib arch, and during
INSTALL there is a policy about what is the preferred arch and what is the
secondary arch.  For most arches, these are one in the same, but for PPC I think
they are different.  During compose, 'ppc64' is treated as the multilib arch and
thus a limited number of ppc64 packages are brought into the compose.  However
during install, I do believe that ppc64 is treated as the preferred arch and the
binaries from .ppc64 packages will be used instead of the binaries from the .ppc
arches.

Please correct me if I'm wrong here.

RPM has its own policy regarding file "colours", which means that if a given
package is simultaneously installed for _both_ 32-bit and 64-bit versions,
certain file "conflicts" are allowed and in the case of executables, the 64-bit
one will land in /usr/bin (or wherever).

That policy in itself is dubious on ppc -- I suspect it ought to be favouring
the 32-bit version since that's the primary arch. But that's a different question.

It doesn't explain why we get _only_ the 64-bit package installed in some cases.
As far as I can tell, rpmUtils.arch.getBestArch() still returns 'ppc' on ppc64
hardware, which is the correct thing to do.

Building evolution fails because /usr/bin/perl is 64-bit. After fixing that, it
then fails thus:

    Package cairo was not found in the pkg-config search path.
    Perhaps you should add the directory containing `cairo.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'cairo', required by 'Pango Cairo', not found

Of course, cairo-devel was one of the packages only installed for the secondary
architecture and not the primary arch. Trying to remedy that leads to other
strange behaviour...

[dwmw2@net2-100 devel]$ rpm -q cairo-devel
cairo-devel-1.4.0-1.fc7
[dwmw2@net2-100 devel]$ rpm -q --qf %{ARCH}\\n cairo-devel
ppc64

[root@net2-100 ~]# yum install cairo-devel.ppc
Loading "installonlyn" plugin
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
Checking deps for cairo-devel.ppc 0-1.4.2-1.fc7 - u
--> Processing Dependency: cairo = 1.4.2-1.fc7 for package: cairo-devel
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
Checking deps for cairo-devel.ppc 0-1.4.2-1.fc7 - u
Checking deps for cairo.ppc64 0-1.4.0-1.fc7 - None
Checking deps for cairo.ppc64 0-1.4.2-1.fc7 - u

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 cairo-devel             ppc        1.4.2-1.fc7      development       145 k
Updating for dependencies:
 cairo                   ppc64      1.4.2-1.fc7      development       452 k

Transaction Summary
=============================================================================
Install      1 Package(s)         
Update       1 Package(s)         
Remove       0 Package(s)         

Total download size: 597 k
Is this ok [y/N]: y
Downloading Packages:
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating  : cairo                        ######################### [1/4] 
  Installing: cairo-devel                  ######################### [2/4] 
  Cleanup   : cairo-devel                  ######################### [3/4]
  Cleanup   : cairo                        ######################### [4/4]

Installed: cairo-devel.ppc 0:1.4.2-1.fc7
Dependency Updated: cairo.ppc64 0:1.4.2-1.fc7
Complete!
[root@net2-100 ~]#  rpm -q --qf %{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\\n cairo
cairo-devel
cairo-1.4.0-1.fc7.ppc
cairo-1.4.2-1.fc7.ppc64
cairo-devel-1.4.2-1.fc7.ppc


I don't remember our multilib support ever being this broken. If we can't fix it
in time for FC7 then we need to go back to shipping far fewer ppc64 packages.
IBM's requirements for RHEL5 don't apply to Fedora. 

(In reply to comment #5)
> OK... what policy is that? We currently have ppc32 as 'primary' and ppc64 as
> 'secondary' on ppc64 hardware, don't we? What policy causes the installer to
> install the secondary version of these packages and not the primary version?

ping?

FC6 seems to have similar behaviour too. It just isn't so problematic because we
don't have so many ppc64 packages. Here's a list of PPC64-only packages after an
install of the Fedora Unity 20070315 spin...

gnome-keyring-devel
nss-devel
libstdc++-devel
pygobject2-devel
e2fsprogs-devel
dbus-glib-devel
pycairo-devel
sqlite-devel
startup-notification-devel
kernel-headers
redhat-artwork
cyrus-sasl-md5
audiofile-devel
libsepol-devel
libIDL-devel
libxslt-devel
esound-devel
libXv-devel
librsvg2-devel
pilot-link-devel
libgcrypt-devel
libfontenc-devel
libidn-devel
fontconfig-devel
alsa-lib-devel
xorg-x11-proto-devel
libcroco-devel
neon-devel
libXpm-devel
libgpg-error-devel
nspr-devel
cairo-devel
libXinerama-devel
libgnomeprint22-devel
paps
kernel
kernel-devel
libXi-devel
mesa-libGLU-devel

Ah, and here's a simple case where yum alone gets it wrong...

[root@net2-100 ~]# yum install cups
Loading "installonlyn" plugin
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
Checking deps for cups.ppc 1-1.2.10-1.fc7 - u
--> Processing Dependency: paps >= 0.6.6-9 for package: cups
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
Checking deps for paps.ppc64 0-0.6.6-18.fc7 - u
Checking deps for cups.ppc 1-1.2.10-1.fc7 - u

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 cups                    ppc        1:1.2.10-1.fc7   development       3.2 M
Installing for dependencies:
 paps                    ppc64      0.6.6-18.fc7     development        35 k

Transaction Summary
=============================================================================
Install      2 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 3.2 M
Is this ok [y/N]: 


No it bloody isn't! I want paps.ppc :)

Relevant part of 'yum -d 100 install cups' looks like this...

Resolving for requirement: paps >= 0.6.6-9
Searching pkgSack for dep: paps
Potential match for paps from paps - 0.6.6-18.fc7.ppc
Matched paps - 0.6.6-18.fc7.ppc to require for paps
Potential match for paps from paps - 0.6.6-18.fc7.ppc64
Matched paps - 0.6.6-18.fc7.ppc64 to require for paps
TSINFO: Marking paps - 0.6.6-18.fc7.ppc64 as install for cups

We chose the wrong one. Reassigning to yum.

How 'bout this...

--- arch.py~    2007-03-22 15:42:21.000000000 +0000
+++ arch.py     2007-03-27 22:48:32.000000000 +0100
@@ -56,7 +56,7 @@ arches = {
 
 # this computes the difference between myarch and targetarch
 def archDifference(myarch, targetarch):
-    if myarch == targetarch:
+    if getBestArch(myarch) == targetarch:
         return 1
     if arches.has_key(myarch):
         ret = archDifference(arches[myarch], targetarch)
@@ -261,8 +261,9 @@ def getMultiArchInfo(arch = canonArch):
 # get the best usual userspace arch for the arch we're on.  this is
 # our arch unless we're on an arch that uses the secondary as its
 # userspace (eg ppc64, sparc64)
-def getBestArch():
-    arch = canonArch
+def getBestArch(arch=None):
+    if arch is None:
+        arch = canonArch
 
     if arch.startswith("sparc64"):
         arch = "sparc"

Oh, that breaks for getBestArchFromList(["ppc64"]).

I'll go away now and leave it to someone who understands the code.

so I want to make sure I understand this properly:

your system returns what when you run this:

python -c "import rpmUtils.arch; print rpmUtils.arch.getCanonArch()"

you seem to be saying that:
 - if your system returns ppc64 and we have ppc and ppc64 pkgs available then we
should install the ppc package and not the ppc64 package, is that correct?

what about the other ppc64-like archs? do they act the same way?
This is the ordering for ppc arches:
ppc64pseries and ppc64iseries > ppc64
pp64 > ppc
ppc > noarch

you're saying that actually it is:
ppc64pseries and ppc64iseries > ppc64
ppc > ppc64
ppc64 > noarch

but for a ppc 32 system it is:
ppc > noarch
everything else is not compatible

is that correct?

(In reply to comment #15)
> python -c "import rpmUtils.arch; print rpmUtils.arch.getCanonArch()"

ppc64

> you seem to be saying that:
>  - if your system returns ppc64 and we have ppc and ppc64 pkgs available then we
> should install the ppc package and not the ppc64 package, is that correct?
>
> what about the other ppc64-like archs? do they act the same way?


Yes, absolutely. Not just other ppc64-like arches, in fact -- it covers SPARC
and probably other 64-bit architectures. 32-bit code is more efficient, in
general -- 64-bit addresses are mostly just bloat. The only reason we favour
64-bit on x86_64 is because i386 is so register-starved that the benefits of
x86_64 mode actually outweigh the bloat. It's x86_64 which is the special case here.

When you say 'other ppc64-like arches' ITYM ppc64pseries and ppc64iseries --
those architectures were only ever used for the kernel, and are not any more; we
have a generic PPC64 kernel image which runs on all ppc64 machines (hopefully
including, by the end of today, PlayStation 3). 


> you're saying that actually it is:
> ppc64pseries and ppc64iseries > ppc
> ppc > ppc64
> ppc64 > noarch

Yes, that's the ordering we expect -- and something similar on SPARC.
Note that I corrected what I _think_ was a typo on the first line of the above.
And note that the kernel is a special case because obviously you want the one
which matches the hardware.

(In fact, gdb is currently a special case too, since 32-bit gdb cannot debug
64-bit programs. But we can deal with that differently. It'll still DTRT at the
moment because we install _both_ versions of GDB, and rpm chooses the 64-bit
filecolours when it installs them. So ignore gdb; I mention it only for
completeness.)

> but for a ppc 32 system it is:
> ppc > noarch
> everything else is not compatible

Right.

IIRC the only reason both versions of gdb are installed is because you asked for
it to be installed by name, rather than having it pulled in via deps.  Things
asked for by name will install both arches if they are available, but the deps
those bring in will only be installed for the "preferred" arch.  If we switch
this, anytime something Requires gdb and you don't have it installed already,
you're only going to get the ppc32 version of it.

Kernel is special because anaconda takes care of making sure you have the right one.

(In reply to comment #17)
> IIRC the only reason both versions of gdb are installed is because you asked for
> it to be installed by name, rather than having it pulled in via deps.  Things
> asked for by name will install both arches if they are available, but the deps
> those bring in will only be installed for the "preferred" arch.  If we switch
> this, anytime something Requires gdb and you don't have it installed already,
> you're only going to get the ppc32 version of it.

That's fine. Needing 64-bit gdb is _rare_ because most of userspace is 32-bit.
You really shouldn't have all that 64-bit crap lying around. As long as it's
_available_, it's fine not to install it by default.
 
Let's not screw over the rest of the distro just to work around a gdb shortcoming :)

Arch preference ordering for SPARC:

sparc32:

sparcv9 > sparcv8 > sparc

sparc64:

sparcv9v > sparcv9 > sparcv8 > sparc

We really only ever want sparc64 for kernel on sparc64.

OK, much the same as ppc64 then.

I think the conclusion was that we were going to remove the unwanted 64-bit
packages from the 'ppc' repo -- we'll leave a core of 64-bit packages, but the
rest of the 64-bit packages will be available in a _separate_ repository. That
matches what you do for SPARC64 too, I believe (although we'll probably be
making _all_ ppc64 packages available).

The minimal set of packages required is something like:

 kernel (obviously)
 gdb (for debugging kernels and maybe userspace) 
 glibc, readline (pulled in by gdb anyway)
 kexec (32-bit kexec doesn't boot 64-bit kernels)

okay, I'm looking at this bug and I think this can be done by just implementing
a special arch ordering for ppc64 in much the same way we have a 'special' one
for sparc.

for example right now the code works like this:
In [2]: rpmUtils.arch.getBestArchFromList(['ppc', 'ppc64'], myarch='ppc64')
Out[2]:'ppc64'

You think what we should be doing is:
In [2]: rpmUtils.arch.getBestArchFromList(['ppc', 'ppc64'], myarch='ppc64')
Out[2]:'ppc'

is that right?
And keep in mind - getBestArchFromList can't know about the pkgname - it is only
an arch-comparison function.

Would we have to rely upon anaconda to select the ppc64 kernel instead of the
ppc one?  What about after the fact if somebody did a yum install kernel?  Would
they get the ppc one?  Will we have to finally split up the tree to have a pure
ppc32 tree and a ppc64 tree that doesn't have a 32bit kernel, or any other of
the 32bit packages that just won't work on a ppc64 install?

Why would we have a kernel in a tree that cannot sensibly be run by the system?
It would be like having an i686 kernel in the x86_64 tree, wouldn't it?

This is the way the ppc trees have been up to date.  The entire ppc32 tree plus
all the ppc64 "multilib" packages, which includes the 64bit kernel are all in
the same tree.  The bootloader is smart enough to load the right kernel at boot
time, and a ppc32 install won't see any ppc64 packages as available.  However on
the ppc64 side of things this gets messy, and maybe it's time to change this. 
Perhaps we can take a queue from RHEL5 and do multiple repos in the tree that
their use depends on the arch booted.  Say there is a ppc32 and a ppc64 repo,
the ppc32 is pure 32bit, the ppc64 repo is mostly hardlinked ppc32 packages
except the kernel and such, with a sprinkling of what ppc64 packages are really
needed. Not that pungi knows how to do this yet, nor won't for F7, but it is
something to think about.  If the x86 bootloaders had this ability we could
potentially do it for x86(_64) too.

(In reply to comment #21)
> You think what we should be doing is:
> In [2]: rpmUtils.arch.getBestArchFromList(['ppc', 'ppc64'], myarch='ppc64')
> Out[2]:'ppc'

Yes. I think the SPARC ordering needs to be fixed to choose 32-bit first, too.

Regarding the kernel -- yes, we rely on anaconda to pick the right kernel. But
anaconda's had the special case to pick the right kernel for as long as I can
remember. That's fine, surely?

'yum install kernel' implies that the user didn't have a kernel in the first
place. Not something we really expect to be a common case, I feel. Is is really
worth having another almost-identical repo just so there can be only one kernel?

It's more than just the kernel, and special anaconda hacks should be avoided
whenever necessary, because anaconda isn't always what generates your system,
like in the case of a LiveCD where yum is used to populate an install root
without anaconda interaction.  We want the logic to handle these things at the
common level which is yum (or rpm, but meh...)

I think the common level we want is yum, not rpm. Even the existing bits which
rpm does to choose between 32-bit and 64-bit (with file colours) want to die in
favour of just letting yum choose the right package in the first place.

We _could_ move the special case into something like a yum plugin, although I
agree that it's a bit icky. Probably better than having a completely separate
repository with all the same packages but the kernel, though.

Alternatively, I wonder if the 'proposed' "Multilib" RPM tag discussed in bug
#235758 can also express the information required here -- that this is a package
where the only 64-bit version should be installed on 64-bit hardware. Maybe it'd
be another new tag.

Or maybe a 'Conflicts: CPU.ppc32' in the 64-bit kernel package? Do we have
virtual packages representing the CPU like that? Is it possible?

I don't think we have anything that Provides CPU.ppc32' or whatever.

The second repo would just be hardlinks, except for the different packages. 
This I think would be a happier situation than having a complete ppc32 spin and
a complete ppc64 spin, each with their own iso sets, like we do for i386/x86_64.

Using a separate repo means entirely different sets of ISO images too, remember
-- and you were very much _not_ keen on adding new repositories a few days ago,
even when the files could be hard linked.

How hard would it be to do the virtual 'CPU.ppc32'?

No no, I'm talking a second repo ON the iso set.

Instead of iso/Fedora/<packages>  you'd have:

iso/Fedora/ppc32/<packages>
iso/Fedora/ppc64/<packages>

or something similar.  Anaconda has some code in it already for enabling
multiple repos on an iso set, it could detect which kernel you're running (since
the bootloader gets that right) and enable either the 32 or the 64 repo. 
Hardlinks save space it would be no bigger/smaller than the ppc discs were before.

The problem with virtual CPU.ppc32 is what package do you add that to?

Ah, OK. You can do hardlinks on iso9660? 
Doesn't even matter if not -- we could go to three repositories. One for ppc32
packages in general, one for ppc32 kernel, one for ppc64 kernel (since we don't
really want to ship any other ppc64 stuff by default, ideally).

I was thinking the virtual CPU.ppc32 would provided at runtime by rpm or yum
itself. Hence 'virtual' :)

Actually, if you're going to have anaconda select the right repository based on
the kernel that's currently running, why is that so much easier than just
selecting the right kernel and using only one repository as we've been doing for
years?

That discussion should probably be elsewhere though; this bug is filed against
yum because of the misordering of architecture preferences.

Any progress on that? It should be relatively simple but you _really_ don't want
me trying to hack it up for myself, do you? You've _seen_ my attempts at python...

Re: comment #31: FYI: Per-system virtual Provides: are trivially configured in rpm-4.4.3 and later:

   echo "CPU.ppc32" >> /etc/rpm/sysinfo/Providename

I tried an install with getBestArchFromList() hacked like this...

    if "ppc" in archlist:
        return "ppc"
    
It seems to work very nicely. The correct kernel is installed, and all the
correct 32-bit packages are installed too.

hm, except for metacity-devel, for which I only have the ppc64 version.
Strangely. But on the whole, a massive improvement :)

Created attachment 153486 [details]
patch

This patch picks 'ppc' from the list first, if we're on ppc64. There are
probably better ways of doing it for someone who can do python better than I
can, but this works.

Oh, that should be myarch.startswith("ppc64") rather than myarch=="ppc64" -- I
thought that the speshul 'ppc64pseries' arch had gone away now, but it does
still seem to exist in yum. No clue why -- it should just be ppc64, really. We
should remove that bit from getCanonPPCArch(), I think.

so, using some of the code we already had I sorta did something similar to what
you did, david.

http://linux.duke.edu/~skvidal/patches/ppc-arch-fix.patch

It seems to produce the right outputs for x86_64, ppc64 and sparc64.

okay, that patch wasn't completely correct. Let's see if this is more-better:

http://linux.duke.edu/~skvidal/patches/ppc-arch-fix-2.patch

once more, just for fun:
http://linux.duke.edu/~skvidal/patches/ppc-arch-fix-3.patch

I think this one gets all the edge cases of doom.

Created attachment 153523 [details]
patch

I found another edge case, admittedly contrived:
 $ python -c 'import rpmUtils.arch ; print
rpmUtils.arch.getBestArchFromList(("noarch", "sparc64"), "sparc64");'
noarch

The attached patch should get that one right too.

okay, I think that's got it.

Seth committed this and it should be in 3.1.7