Bug 190116

Summary: Yum fails to see virtual packages for obsolete sub-packages
Product: [Fedora] Fedora Reporter: Michael Schwendt <bugs.michael>
Component: yumAssignee: Jeremy Katz <katzj>
Status: CLOSED RAWHIDE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 5CC: j, katzj, nicolas.mailhot, scop, sundaram
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
URL: https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=532
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-09-18 20:29:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michael Schwendt 2006-04-27 18:27:40 UTC
This is the same as
https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=532
(mistakingly closed as NOTABUG)

Affected are at least FC5, FC4 and FC3.

I rate this as "high" because this defect shields update packages
and build requirements from being seen by Yum when installed with
"yum install ...".

[...]

In the repository you have a package called "foo" which

  Obsoletes: foo-devel < %{version}-%{release}
  Provides: foo-devel = %{version}-%{release}

and does this in order to eliminate/merge an old sub-package "foo-devel".

[Note that it could be explained with several different scenarios, e.g.
an ordinary package name change, or a pair of packages "foo" and "libfoo",
where the updated "foo" package now includes the library after the
developers eliminated its public API. In either case, real-world scenarios].

When you run:

  yum install foo-devel

Yum does not see the virtual package "Provides: foo-devel = ..." in
package "foo" _if_ old binaries of the foo-devel sub-package are still
present in the repository. It is said that Yum stops looking for virtual
forms of "foo-devel" when it finds a package with that name. Effectively,
a common mechanism for obsoleting packages and for creating virtual
packages does not work with Yum. On the contrary, if the old
stripesnoop-devel package is removed from the repository, a
"yum install stripesnoop-devel" works fine and finds the virtual
package inside the stripesnoop package.


Example with FC3:

$ sudo yum list available | grep stripe
stripesnoop.i386                         1.5-5.fc3       extras
stripesnoop-devel.i386                   1.5-2.fc3       extras

Note the different version-release. Note also that is fails to see
the newer virtual stripesnoop-devel-1.5-5.fc3 inside stripesnoop.

$ rpm -qp --provides stripesnoop-1.5-5.fc3.i386.rpm 
stripesnoop-devel = 1.5-5.fc3
stripesnoop = 1.5-5.fc3

Hence:

$ sudo yum install stripesnoop-devel
Setting up Install Process
Setting up Repos
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Package stripesnoop-devel.i386 0:1.5-2.fc3 set to be updated
--> Running transaction check
--> Processing Dependency: stripesnoop = 1.5-2.fc3 for package: stripesnoop-devel
--> Finished Dependency Resolution
Error: Missing Dependency: stripesnoop = 1.5-2.fc3 is needed by package
stripesnoop-devel


Note also that it seems it would install an _old_ stripesnoop-1.5-2.fc3
if that still existed in the repository.

Expected behaviour: It should have seen the newer virtual package
"stripesnoop-devel" inside package "stripesnoop".

Comment 1 Seth Vidal 2006-04-30 03:23:19 UTC
1. yum list only operates on package names-ver-rel.arch - it does not know or
describe anything about provides.

2. if a user runs yum list somepkg\* and sees somepkg-devel being the version
they want. Then they run yum install somepkg-devel and end up getting
someotherpkg (which provides somepkg-devel) do you think that is appropriate
behavior?

Comment 2 Michael Schwendt 2006-04-30 08:19:55 UTC
It's not "yum list" alone, but read one.

> 2.

Yes, that would be the appropriate behaviour as we do have _virtual
packages_ which Yum is aware of in other cases, e.g. FC5:

  yum install libGL-devel

installs two virtual packages when no older packages with the same
name are present,

  yum remove libGL-devel

removes virtual packages and its dependencies, whereas

  yum list libGL-devel

doesn't know those virtual packages at all. The physical packages
are mesa-libGL-devel and mesa-libGLU-devel in case you raise doubts.
Another example:

  yum list links

finds no match, whereas

  yum install links

correctly finds and installs the virtual package inside package
"elinks".

You have a huge inconsistency there. Further, Yum should fetch the
latest EVR unless a specific EVR is specified as to not turn this
into a security issue. Developers (and some ordinary users) install
BuildRequires with "yum install" and should gets the latest.

We can kick the entire concept of Obsoletes/Provides for virtual
packages and virtual sub-packages if Yum doesn't support this cleanly.

And in case this is a big PITA for you, Seth (you sound like you're
in NOTABUG mood once more, since question 2 in your reply sounds
quite rhetorical), as I've pointed out before, as an inconvenient
alternative we may need strict and tiresome policies for the packagers
to request removal of old {sub-}packages from the repository as to not
create conflicts with Yum's understanding of virtual packages and
Obsoletes/Provides. And then this would remain a confusing defect in
Yum, see examples above.

Comment 3 Seth Vidal 2006-04-30 12:33:06 UTC
Michael, will you get rid of the attitude. There was nothing rhetorical in the
second question at all. I hope you realize you're making me less willing to help
you the more you act like a jerk.

Comment 4 Jeremy Katz 2006-05-01 18:47:36 UTC
So, I see a few things at work here:
1) Some inconsistency in the handling of virtual provides around the install
command.  Seth -- I remember there being a specific reason that was added, but
I'm blanking on what it was right now
2) The repositories containing old versions of packages causing some weird cases
to be hit

While we could fix the second just by not keeping old packages around, it could
still occur with multiple repositories.  I lean towards saying that, at least by
default, virtual provides shouldn't be searched at all for installs.  Primarily
because there will always be oddities like this and the only way to get real
consistency is to force people to ask for what they really want.  eg, if you
want to match on provides, explicitly asking for it so that there _aren't_
questions about what's meant.

Comment 5 Ville Skyttä 2006-05-01 22:11:21 UTC
(In reply to comment #4)
> While we could fix the second just by not keeping old packages around

Not always possible/feasible; think for example FC5 base repo and FC5 updates.

> at least by default, virtual provides shouldn't be searched at all for 
> installs.

Surely you mean not searched if that something is matched by a package name?
IOW, for example "yum install webserver" would still work?

> eg, if you
> want to match on provides, explicitly asking for it so that there _aren't_
> questions about what's meant.

By "explicitly asking for it" you mean qualifying the argument to "yum install"
so that it produces only one hit, or something like a new install command
("install-provides") would take provides/obsoletes into account? 

FWIW, while it can be argued that the current implementation is ok, I still
think that not taking provides/obsoletes into account in "yum install" is
counterintuitive and the bigger evil.  It results in more (error prone human)
packager work as specfiles can't be shared between distros as much/reliably as
before.  It is also weird for end users who must know that even if their "yum
install foo" command succeeds, they should run "yum update" right after it to
ensure that they have the latest "foo" installed.

Does the depsolver take provides/obsoletes into account under the hood?  Like if
I install a package that depends on foo, does yum search for foo just as if I'd
do a direct "yum install foo", or would it consider provides/obsoletes in that case?

Comment 6 Michael Schwendt 2006-05-01 22:37:27 UTC
The latter. For dependencies, it takes a look at provides/obsoletes, and
highest NEVR wins as expected.


Comment 7 Seth Vidal 2006-05-01 23:41:17 UTC
(In reply to comment #4)
> So, I see a few things at work here:
> 1) Some inconsistency in the handling of virtual provides around the install
> command.  Seth -- I remember there being a specific reason that was added, but
> I'm blanking on what it was right now

for mock and friends. So people could do: yum install /usr/bin/foo or
'webserver' and have it work like they'd expect. It was to dispense with the two
step operation of:
yum resolvedep foo
yum install pkg-that-resolvedep-found

 
> While we could fix the second just by not keeping old packages around, it could
> still occur with multiple repositories.  I lean towards saying that, at least by
> default, virtual provides shouldn't be searched at all for installs.  Primarily
> because there will always be oddities like this and the only way to get real
> consistency is to force people to ask for what they really want.  eg, if you
> want to match on provides, explicitly asking for it so that there _aren't_
> questions about what's meant.


When I looked at the problem on saturday I thought that the following would work:

1. continue looking up pkgs for install as we have been
2. before we add a pkg to the transaction set we check to make sure it is not
obsoleted by something else. If it is obsoleted then we added the obsoleting pkg
instead.

It'd be pretty simple to add that into the install,update functions and/or into
the tsInfo methods.

thoughts?



Comment 8 Seth Vidal 2006-05-01 23:43:31 UTC
removing my other address since I really don't need to get this twice.

Comment 9 Jeremy Katz 2006-05-02 00:48:14 UTC
Ah, right.  That sounds like it should work without being too bad.

Comment 10 Michael Schwendt 2006-05-02 11:35:45 UTC
That would be different from what the current repsolver does
(cf. bottom of comment 5 and comment 6).

It looks at versioned "Provides" always, regardless of whether
there is an "Obsoletes". Highest NEVR wins, whether virtual or
not.

The packaging policies would need to change, so that for every
versioned "Provides" there must be a corresponding "Obsoletes". Even
in cases where a new small pkg "foo" just "Provides: foo-devel = ...",
because creating a separate sub-package is not justified.

Comment 11 Seth Vidal 2006-05-02 12:08:07 UTC
(In reply to comment #10)
> That would be different from what the current repsolver does
> (cf. bottom of comment 5 and comment 6).

I don't see how it is any different at all. The results would end up the same
after the next time a user ran an 'update' anyway - as the obsoletes grabber
would catch it.
 
> It looks at versioned "Provides" always, regardless of whether
> there is an "Obsoletes". Highest NEVR wins, whether virtual or
> not.

But it wouldn't be a nevr b/c it is explicitly NOT a name.

Look at this case
foo-1.1-1 provides sucko = 0:4.1-2
bar-1.1-1 provides sucko = 0:4.1-1

baz requires sucko >= 0:4.1


So we search provides and find that both foo and bar match. Now you're
suggesting we compare the versioned provide to see which one is newer but there
is NO definition to do that.

Yum would compare foo and bar to see which one is newer, finding neither of them
 it would compare the names of the packages and take the smaller and earlier
one. Yes it is preposterous but it gets us around -debuginfo pkgs which is where
it comes from - it is derived from anaconda circa rhl9 or fc1, iirc.


> The packaging policies would need to change, so that for every
> versioned "Provides" there must be a corresponding "Obsoletes". Even
> in cases where a new small pkg "foo" just "Provides: foo-devel = ...",
> because creating a separate sub-package is not justified.

if a foo-devel existed before, that's correct. You must obsolete it. If a
foo-devel never existed before then no it is not necessary.



Comment 12 Michael Schwendt 2006-05-02 14:20:21 UTC
> if a foo-devel existed before, that's correct. You must obsolete
> it. If a foo-devel never existed before then no it is not necessary.

Even with changed behaviour, a versioned "Provides" must always be
accompanied with a versioned "Obsoletes". You cannot predict that
a package, which might be added later [to any enabled repository],
would not share the same namespace and shield the "Provides" from
being seen. Because it remains the case that

  libfoo-1.0-3.i386.rpm

wins over

  Provides: libfoo = 1.2-1

because the latter is not even looked at at install/update time,
and a "yum update" won't fix that either. Unless it becomes

  Provides: libfoo = 1.2-1
  Obsoletes: libfoo < 1.2-1

so we need a strict packaging policy for this: No virtual packages
without a Provides/Obsoletes pair!


Okay, so far so good. Provided that yum gets fixed. So it does take
a look at Obsoletes and Provides at install-time.


> foo-1.1-1 provides sucko = 0:4.1-2
> bar-1.1-1 provides sucko = 0:4.1-1
>
> baz requires sucko >= 0:4.1

That's a completely different scenario. This dependency means
explicitly that ANY version of sucko >= 0:4.1 is sufficient.

On the contrary, "yum install sucko" should get the newest,
just like "yum install firefox" and "yum install glibc-devel" do.

By default, all packages even contain

  Provides: %name = %{?epoch}%{?epoch::}%version-%release

so it should be valid to compare physical packages and "Provides"
against eachother. And in case

  foo-4.1-2 provides foo-devel = 4.1-2

does not mean it is compared with

  foo-devel-4.0-1 provides foo = 4.0-1

that doesn't sound good and asks for the same strict policy on
Provides/Obsoletes as commented on above.


Comment 13 Seth Vidal 2006-05-24 07:23:10 UTC
okay I've implemented the code I mentioned in comment 7.
Patches are:
https://lists.dulug.duke.edu/pipermail/yum-cvs-commits/2006-May/000734.html
and
https://lists.dulug.duke.edu/pipermail/yum-cvs-commits/2006-May/000733.html

to yum 2.6.X

Could you give those a test and let's see if we can work out the rest of this
bug after that?


Comment 14 Michael Schwendt 2006-05-29 17:47:04 UTC
Applied against yum-2.6.1-0.fc5 and "works for me" so far, especially
with the "stripesnoop-devel -> stripesnoop" test-case. I'm going to
keep Yum patched for further testing.


Comment 15 Nicolas Mailhot 2006-06-19 17:11:44 UTC
*** Bug 195823 has been marked as a duplicate of this bug. ***

Comment 16 Rahul Sundaram 2006-07-13 04:15:25 UTC

I can still this see this on my FC5 system with a yum from Fedora devel tree. If
you need more information, let me know. 

# rpm -q yum
yum-2.9.2-3

#yum list libGL-devel

Setting up repositories
Reading repository metadata in from local files


# yum install libGL-devel

Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Package mesa-libGL-devel.i386 0:6.4.2-6.FC5.3 set to be updated
--> Running transaction check
--> Processing Dependency: libX11-devel for package: mesa-libGL-devel
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Package libX11-devel.i386 0:1.0.0-3.FC5.0 set to be updated
--> Running transaction check
--> Processing Dependency: xorg-x11-proto-devel for package: libX11-devel
--> Processing Dependency: libXdmcp-devel for package: libX11-devel
--> Processing Dependency: libXau-devel for package: libX11-devel
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Package xorg-x11-proto-devel.i386 0:7.0-6 set to be updated
---> Package libXdmcp-devel.i386 0:1.0.0-2.2 set to be updated
---> Package libXau-devel.i386 0:1.0.0-2.2 set to be updated
--> Running transaction check

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 mesa-libGL-devel        i386       6.4.2-6.FC5.3    updates           116 k
Installing for dependencies:
 libX11-devel            i386       1.0.0-3.FC5.0    updates           662 k
 libXau-devel            i386       1.0.0-2.2        core               11 k
 libXdmcp-devel          i386       1.0.0-2.2        core              7.1 k
 xorg-x11-proto-devel    i386       7.0-6            core              263 k

Transaction Summary
=============================================================================
Install      5 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 1.0 M
Is this ok [y/N]:


Comment 17 Seth Vidal 2006-07-13 04:24:10 UTC
(In reply to comment #16)
> 
> I can still this see this on my FC5 system with a yum from Fedora devel tree. If
> you need more information, let me know. 
> 
> # rpm -q yum
> yum-2.9.2-3
> 

I don't see how you're claiming you're seeing the bug.


What is wrong with how yum is resolving for your command?