Bug 439986 - XML => .sqlite conversion update breakage
XML => .sqlite conversion update breakage
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: yum-metadata-parser (Show other bugs)
8
All Linux
low Severity low
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-01 04:38 EDT by Ralf Corsepius
Modified: 2014-09-01 10:38 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-09 02:45:20 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
repodata/* as having been created by createrepo (335.14 KB, application/octet-stream)
2008-04-01 10:03 EDT, Ralf Corsepius
no flags Details
repodata/* as having been created by createrepo -d (708.19 KB, application/octet-stream)
2008-04-01 10:06 EDT, Ralf Corsepius
no flags Details

  None (edit)
Description Ralf Corsepius 2008-04-01 04:38:05 EDT
Description of problem:
yum doesn't seem to be able to process package deps correctly.

Scenario:
A local repo with 2 packages:
foo-1.25-0.20080401.3.fc8.i386.rpm
foo-devel-1.25-0.20080401.3.fc8.i386.rpm

foo-devel requires foo:
# rpm -q --requires -p foo-devel-1.25-0.20080401.3.fc8.i386.rpm | grep foo 
foo = 1.25-0.20080401.3.fc8

foo provides
# rpm -q --provides -p  foo-1.25-0.20080401.3.fc8.i386.rpm | grep foo
foo = 1.25-0.20080401.3.fc8

foo is already installed:
# rpm -q foo
foo-1.13.25-0.20080401.3.fc8

# yum install foo-devel
...    
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package foo-devel.i386 0:1.25-0.20080401.3.fc8 set to be updated
--> Processing Dependency: foo = 1.25-0.20080401.3.fc8 for package: foo-devel
--> Finished Dependency Resolution
Error: Missing Dependency: foo = 1.25-0.20080401.3.fc8 is needed by package
foo-devel

WTH ?!??

Installing from the same repos on the same machine with apt-get, installation
works flawlessly.

Version-Release number of selected component (if applicable):
yum-3.2.8-2.fc8
repos are using *.xml repodata having been created by createrepo-0.4.11-2.fc8
(no sqlite repos)

How reproducible:
Always. I am able to reproduce the effect with several different packages.

Expected results:
Function.

Additional info:
The example above is on Fedora i386, with no yum plugins installed. 
It's also reproducible on Fedora x86_64.

# rpm -qa 'yum*'
yum-3.2.8-2.fc8
yum-metadata-parser-1.1.2-1.fc8
yum-utils-1.1.11-1.fc8
Comment 1 Ralf Corsepius 2008-04-01 04:40:09 EDT
Urgh, cut'n'pasto:

The "foo is already installed" sections above should read:

> foo is already installed:
> # rpm -q foo
> foo-1.25-0.20080401.3.fc8
Comment 2 Ralf Corsepius 2008-04-01 05:05:35 EDT
Further insights:

Regenerating repodata with
createrepo -d
instead of
createrepo

lets "yum install foo-devel" succeed.

=> Something is broken with either yum's xml metadata proceeding or with
createrepo 's *.xml generation.

This renders Fedora 8 unsuitable as host to host repositories for distros not
supporting sqlite metadata :/

Comment 3 Seth Vidal 2008-04-01 07:30:44 EDT
I'm a little confused are you saying this is a createrepo bug or a yum bug? What
version of createrepo created the original repository? Can you post the original
xml files so I can see what it was or was-not seeing?

Was apt-get using the same xml files or the db files?

Was there a local yum cache already or had that been cleaned/expired?
Comment 4 Ralf Corsepius 2008-04-01 07:50:48 EDT
(In reply to comment #3)
> I'm a little confused are you saying this is a createrepo bug or a yum bug?
I don't know.

Symptoms are:
* If I use xml-metadata ("rm -f repodata; createrepo .") to create my local
repo's repodata/* on the server, "yum install foo-devel" fails on clients. 
* If I use sqlite-metadata ("rm -f repodata; createrepo -d") to create my local
repo's repodata/* on the server, "yum install foo-devel" succeeds on clients.

> What
> version of createrepo created the original repository?
I rebuilt them from scratch using
createrepo-0.4.11-2.fc8

> Can you post the original
> xml files so I can see what it was or was-not seeing?
I'll try to do so, some time later today ...

> Was apt-get using the same xml files or the db files?
apt-get succeeded using the same xml files, which failed in yum.
(The repo did not contain any db files at the time, I checked).

> Was there a local yum cache already or had that been cleaned/expired?
Good question. I have "metadata_expire=0" in my local repo's yum.repos.d/*.repo.
Comment 5 James Antill 2008-04-01 09:53:22 EDT
 Also is it possible for you to try the version in rawhide:

yum install pygpgme
yum update yum --enablerepo=development
Comment 6 Ralf Corsepius 2008-04-01 10:03:51 EDT
Created attachment 299895 [details]
repodata/* as having been created by createrepo

"yum install dpkg-devel" fails on an FC8/i386 when the server contains this
repodata.
Comment 7 Ralf Corsepius 2008-04-01 10:06:32 EDT
Created attachment 299897 [details]
repodata/* as having been created by createrepo -d

"yum install dpkg-devel" succeeds with this repodata.

Except of i386/repodata having been regenerated, the repo's contents is
identical to the one from previous attachment.
Comment 8 Ralf Corsepius 2008-04-01 10:25:14 EDT
(In reply to comment #5)
>  Also is it possible for you to try the version in rawhide:

I gave this (yum from fc9 on fc8/i386) a try, but it doesn't seem to help.

[root@mccallum ~]# rpm -q yum
yum-3.2.13-1.fc9

The symptoms are the same:

With xml-repodata: yum install dpkg-devel fails:

[root@mccallum ~]# yum install dpkg-devel
livna                     100% |=========================| 2.1 kB    00:00     
fedora                    100% |=========================| 2.1 kB    00:00     
adobe-linux-i386          100% |=========================|  951 B    00:00     
corsepiu-rtems            100% |=========================|  951 B    00:00     
rtems-4.9                 100% |=========================|  951 B    00:00     
updates                   100% |=========================| 2.3 kB    00:00     
corsepiu-extra            100% |=========================|  951 B    00:00     
primary.xml.gz            100% |=========================|  48 kB    00:00     
corsepiu-e: ################################################## 265/265
packman                   100% |=========================|  951 B    00:00     
rtems-4.8                 100% |=========================|  951 B    00:00     
rtems-4.7                 100% |=========================|  951 B    00:00     
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package dpkg-devel.i386 0:1.13.25-0.20080401.3.fc8 set to be updated
--> Processing Dependency: dpkg = 1.13.25-0.20080401.3.fc8 for package: dpkg-devel
--> Finished Dependency Resolution
dpkg-devel-1.13.25-0.20080401.3.fc8.i386 from corsepiu-extra has depsolving problems
  --> Missing Dependency: dpkg = 1.13.25-0.20080401.3.fc8 is needed by package
dpkg-devel-1.13.25-0.20080401.3.fc8.i386 (corsepiu-extra)
Error: Missing Dependency: dpkg = 1.13.25-0.20080401.3.fc8 is needed by package
dpkg-devel-1.13.25-0.20080401.3.fc8.i386 (corsepiu-extra)


With createrepo -d created repodata, it succeeds:

[root@mccallum ~]# yum install dpkg-devel
livna                     100% |=========================| 2.1 kB    00:00     
fedora                    100% |=========================| 2.1 kB    00:00     
adobe-linux-i386          100% |=========================|  951 B    00:00     
corsepiu-rtems            100% |=========================|  951 B    00:00     
rtems-4.9                 100% |=========================|  951 B    00:00     
updates                   100% |=========================| 2.3 kB    00:00     
corsepiu-extra            100% |=========================| 1.9 kB    00:00     
primary.sqlite.bz2        100% |=========================|  84 kB    00:00     
packman                   100% |=========================|  951 B    00:00     
rtems-4.8                 100% |=========================|  951 B    00:00     
rtems-4.7                 100% |=========================|  951 B    00:00     
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package dpkg-devel.i386 0:1.13.25-0.20080401.3.fc8 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 dpkg-devel              i386       1.13.25-0.20080401.3.fc8  corsepiu-extra   
168 k

Transaction Summary
=============================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total size: 168 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: dpkg-devel                   ######################### [1/1] 

Installed: dpkg-devel.i386 0:1.13.25-0.20080401.3.fc8
Complete!

Comment 9 Seth Vidal 2008-04-01 21:26:22 EDT
okay, here's the confusion I'm having:

the process which createrepo uses to generate the sqlite files is identical to
the one yum uses to generate them. It's the same library, it's the same call.

Just for grins:
 the machine you're running createrepo on and the machine you're running yum on,
are they carrying the same version of yum-metadata-parser?

Comment 10 Ralf Corsepius 2008-04-02 00:34:27 EDT
(In reply to comment #9)
>  the machine you're running createrepo on and the machine you're running yum on,
> are they carrying the same version of yum-metadata-parser?
Yes, except that the server is x86_64, while the client is i386:

[root@mccallum ~]# rpm -q --qf "%{name}-%{version}-%{arch}.%{release}\n"
yum-metadata-parser
yum-metadata-parser-1.1.2-i386.1.fc8

[root@beck ~]# rpm -q --qf "%{name}-%{version}-%{arch}.%{release}\n"
yum-metadata-parser
yum-metadata-parser-1.1.2-x86_64.1.fc8

but ... meanwhile, I think, this is a cache handling issue in yum. 
As it seems to me, yum doesn't check if /var/cache/yum/<repo>/*.{xml,sqlite}*
are still valid rsp. doesn't invalidate (remove/ignore) those files having
become invalid.

At least manually rm'ing the cache on the client after the repodata has been
regenerated on the server (and switching between "createrepo" and "createrepo
-d"), seems to remedy the issue on the client.

When not "rm'img" the cached files, the issue reappears, when regenerating the
repodata on the server with "createrepo", when it once had been generated with
"createrepo -d", before.

Wild guess: yum reuses some *.sqlite files from its cache, even when repoml.xml
doesn't reference it?
Comment 11 James Antill 2008-04-02 01:20:12 EDT
Ralf, in the repo/.xml files do you have a pkg listed more than once[1] -- say
the same pkg in multiple sub-dirs?


[1] Same: <checksum type="sha"
pkgid="YES">116560e64df5201ecd4c342db6ed0569a3e59751</checksum> entry?
Comment 12 Darrell R. Kresge 2008-06-17 16:22:05 EDT
I started to file this as a new issue, but given the similarity, this may be the
solution.  Try applying the little patch at the end -- if it doesn't help, I'll
re-open this as a different bug:


Subject: matchingPrcos call fails to identify installed dependencies


Abstract:

Given installed 'package-1.0.0' requiring '1.0.0 <= dependency < 2.0.0' (also
installed)

Attempting to install 'dependency-2.0.0' will not detect that the the installed
package 'package' requires disposition.


Reason:

depsolve.py::_checkRemove() -> transactioninfo.py::getRequires(*prov) to find
list of packages requiring *prov
                           -> transactioninfo.py::getOldRequires(*prov)
                           -> rpmsack.py::getRequires(*prov)
                           -> packages.py::matchingPrcos('requires', *prov) for
po in requiring packages
                           -> rpmUtils::miscUtils::rangeCompare(*prov, tup) for
tup in requirements(self)

Since rangeCompare expects arguments of (reqtup, provtup), and is logically
implemented in terms of these concepts, the above
usage is conceptually reversed and invalid.


Potential patch:


Index: packages.py
===================================================================
--- packages.py (revision 45261)
+++ packages.py (working copy)
@@ -323,8 +323,13 @@
                    r = self.rel
                #(e, v, r) = (self.epoch, self.ver, self.rel)

-            matched = rpmUtils.miscutils.rangeCompare(
-                reqtuple, (n, f, (e, v, r)))
+            if prcotype == 'requires':
+                matched = rpmUtils.miscutils.rangeCompare(
+                    (n, f, (e, v, r)), reqtuple)
+            else:
+                matched = rpmUtils.miscutils.rangeCompare(
+                    reqtuple, (n, f, (e, v, r)))
+
            if matched:
                result.append((n, f, (e, v, r)))


Comment 13 James Antill 2008-08-07 18:11:07 EDT
 Both the .xml files look identical, and the .sqlite looks like what should be generated form the .xml ... I think the problem is with the optimisation in yum-metadata-parser where instead of generating the .sqlite file from scratch when it gets a new .XML file it'll update it in place. I've seen what I thought was this before, but getting a reproducer is probably impossible until someone sees what the problem is *sigh*.

 Given the best practise of always using "createrepo -d", I think we should just rm this optimisation if we want to keep supporting the bad repos. that only provide .xml files.
Comment 14 Bug Zapper 2008-11-26 05:20:50 EST
This message is a reminder that Fedora 8 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 8.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '8'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 8's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 8 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 15 Bug Zapper 2009-01-09 02:45:20 EST
Fedora 8 changed to end-of-life (EOL) status on 2009-01-07. Fedora 8 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 16 Ralf Corsepius 2014-09-01 10:38:46 EDT
OK, the Redhat bureaucrats have sent me a needinfo reminder about this 4 year+ closed bug - Thank you for forcing me to wasting my and your time!

Note You need to log in before you can comment on or make changes to this bug.