Bug 444739

Created attachment 304223 [details]
/var/cache/yum directory

Created attachment 304224 [details]
rpm -qa

Created attachment 304225 [details]
yum list all

Created attachment 304226 [details]
yum -d 100 update

log is truncated and I have pressed Ctrl+C at the end

We are using RHN Satellite 5.0.1 on https://rlx-1-12.rhndev.redhat.com/ and 
system is registered to the rhel-i386-server-5 channel there.

 So if I use the latest upstream code then almost instantly I get:

--> Finished Dependency Resolution
glibc-common-2.5-18.el5_1.1.i386 from rhel-i386-server-5 has depsolving problems
  --> glibc-common conflicts with glibc
1:cups-1.2.4-11.14.el5_1.6.i386 from rhel-i386-server-5 has depsolving problems
  --> Missing Dependency: cups-libs = 1:1.2.4-0 is needed by package
1:cups-1.2.4-11.14.el5_1.6.i386 (rhel-i386-server-5)
Error: glibc-common conflicts with glibc
Error: Missing Dependency: cups-libs = 1:1.2.4-0 is needed by package
1:cups-1.2.4-11.14.el5_1.6.i386 (rhel-i386-server-5)

...and that problem isn't a yum problem ... it's an RHN bug:

% rpm -qp --conflicts ./glibc-common-2.5-18.el5_1.1.i386.rpm 
glibc < 2.5
glibc > 2.5
% sqlite3 /var/cache/yum/rhel-i386-server-5/primary.xml.gz.sqlite
sqlite> select packages.name, packages.version, packages.release,
conflicts.name, conflicts.flags, conflicts.version, conflicts.release from
conflicts,packages where conflicts.pkgKey=packages.pkgKey and
packages.name='glibc-common' and conflicts.name = 'glibc';
glibc-common|2.5|12|glibc|GT|2.5|0
glibc-common|2.5|12|glibc|LT|2.5|0
glibc-common|2.5|18|glibc|GT|2.5|0
glibc-common|2.5|18|glibc|LT|2.5|0
glibc-common|2.5|18.el5_1.1|glibc|GT|2.5|0
glibc-common|2.5|18.el5_1.1|glibc|LT|2.5|0

...this shows that the epoch is moving from NULL into '0', so yum is doing the
correct thing with the metadata it is given but that MD is _wrong_.
 Now there are hits for "select * from conflicts WHERE relase IS NULL", so it's
not happening _all_ the time ... which is probably even more bizarre.

 Ok, I think I've moved this to the correct product/version/etc. ... I'm not
sure if we should be updating the severity/priority of this, Jan said this isn't
happening on hosted but do we know how often it happens on Satellite?

 Ok, moving back to yum ... *sigh*.
 I forget that RHN doesn't give yum .sqlite files ... and checking the .xml
files they are fine.

 XML is:

 <package type="rpm">
    <name>glibc-common</name>
    <arch>i386</arch>
    <version epoch="0" ver="2.5" rel="18.el5_1.1" />
    <checksum type="md5" pkgid="YES">b6a8c810bf8fa286cebbc055b5870667</checksum>
[...]
      <rpm:conflicts>
        <rpm:entry name="glibc" flags="LT"                         epoch="0" ver
="2.5" />
        <rpm:entry name="glibc" flags="GT"                         epoch="0" ver
="2.5" />
      </rpm:conflicts>

 Ok, so if I do:

mv /var/cache/yum/rhel-i386-server-5/primary.xml.gz.sqlite
/var/cache/yum/rhel-i386-server-5/primary.xml.gz.old-sqlite

...and then run "yum update", everything works.

 Indeed the sqlite file is fine now:

sqlite> select packages.name, packages.version, packages.release,
conflicts.name, conflicts.flags, conflicts.version, conflicts.release from
conflicts,packages where conflicts.pkgKey=packages.pkgKey and
packages.name='glibc-common' and conflicts.name = 'glibc';
glibc-common|2.5|12|glibc|GT|2.5|
glibc-common|2.5|12|glibc|LT|2.5|
glibc-common|2.5|18|glibc|GT|2.5|
glibc-common|2.5|18|glibc|LT|2.5|
glibc-common|2.5|18.el5_1.1|glibc|GT|2.5|
glibc-common|2.5|18.el5_1.1|glibc|LT|2.5|

 That means we have a "fix" for anyone who experiences this problem, "yum clean
dbcache" will fix it ... also if RHN moves to generating the .sqlite data (which
they should do for 5.2, for speed improvements) ... that'll make sure it doesn't
happen to anyone on hosted.

 This also probably explains why it worked on RHN hosted, when you did the move
from the satellite yum probably got new MD and so regenerated new .sqlite data
... which is then non-broken.

 Ok, it's magic ... if I downgrade to:

# rpm -q yum yum-metadata-parser
yum-3.0.1-5.el5.noarch
yum-metadata-parser-1.0-8.fc6.i386

...and remove the MD, then everything is generated correctly.

 Ok, so we've found the bug, and it's all y-m-p and it's "pretty bad".
 If you do start with a working .sqlite file and do:

 update db_info set checksum = '1';
 update conflicts set release = '0';

...then yum/y-m-p will notice that the checksum is wrong, but it has an
"optimization" that instead of regenerating the .sqilte file from scratch it'll
just delete the old data and put the new in (with the new checksum).
 However after it does this there are _zero_ rows with "release is NULL", they
are all release is '0'. Also if you change the '0' to a '1' above, all the NULL
release fields will have '1' in them ... so it's just keeping the old data around.

 Ok, that was apparently a bad test ... so yum notices that the package checksum
is the same as in the XML, and so doesn't update the conflicts. And if I also do:

update packages set pkgId = pkgId || '-new';

...then everything works again.

 So that _implies_ a possible real cause:

1. download bad XML from satellite.
2. yum converts to .sqlite
3. download "good" XML from satellite.
4. yum notices that pkg checksum == old pkg checksum, doesn't update anything

...is this possible? Can you explain the steps for reproducing the problem?

 Ok, so an RHN person pointed us to: rhbz#240168 ... which fixes this exact
problem in sat. (generating bad XML).
 That fix has not been backported to 5.0.1 yet.

 Ok, RHN are going to fix this in 5.0.2 ... dup'ing against their bug.

*** This bug has been marked as a duplicate of 444878 ***