Bug 232037

Created attachment 149957 [details]
Screenshot from a customer site of an errata with missing package links.

Created attachment 149960 [details]
Database output from provided query

Customer output from executing the following queries, note that 10736 is the
errata ID for the errata the customer provided a screenshot for.

select * from rhnerratapackage
where errata_id = 10736;
select * from rhnerratafile
where errata_id = 10736;
select * from rhnerratafilepackage
where errata_file_id in (
	select id from rhnerratafile where errata_id = 10736
	);
select * from rhnerratafilechannel
where errata_file_id in (
	select id from rhnerratafile where errata_id = 10736
	);
select * from rhnchannel where id in (
	select channel_id from rhnerratafilechannel
	where errata_file_id in (
		select id from rhnerratafile where errata_id = 10736
		)
	);

Packages are appearing without links due to missing entries in
rhnErrataFilePackage that should correspond to entries in rhnErrataFile.

Examining the output from the above customer data and running the same queries
against the production database to verify the number of results being returned
for the same errata we see the following:

rhnErrataPackage: 7 rows for customer, 27 in production
rhnErrataFile: 78 rows for customer, 79 in production
rhnErrataFilePackage: 28 rows for customer, 74 in production

It seems likely all three tables are out of sync with production.

The problem sounds similar to 208465 where errata packages were being deleted
after syncing a new channel. We attempted to have the customer perform some of
the actions listed in that ticket to see if it would remedy the problem, namely
re-running satellite-sync with a -c option for all their existing channels, and
running the sql provided as a fix in that ticket. Neither approach seemed to
remedy their problem.

We suspect there is a similar issue with syncing new channels in the code that
deals with the errata file tables.

Jan added the following with regard to the customers sql output from the
previous comment:

"""""
So we are looking at the files (rhnerratafile records)

340289
9a273bb88feb2ba6de0a729f4303da77
/ftp/pub/redhat/linux/updates/enterprise/4WS/en/os/i386/kernel-2.6.9-42.0.8.EL.i686.rpm

which has got the link, and

340300
8c9145fdf63eef95fb496e66c38d4bc7
/ftp/pub/redhat/linux/updates/enterprise/4WS/en/os/x86_64/kernel-2.6.9-42.0.8.EL.x86_64.rpm

which does not.

There is a 9577, 340289 record in rhnerratafilepackage and 104, 340289
in rhnerratafilechannel, but only 116, 340300 record in
rhnerratafilechannel -- no record in rhnerratafilepackage.

Well, we already knew this since it was the only possible way that the
outer join in the errata_packages select could have returned a record
with a null rhnErrataFileChannel.package_id.

It seems to me that the cause might be similar to bug 208465 -- a sync
that adds new channel (and does not list the old ones) probably removes
the records. The problem is that unlike rhnerratapackage,
rhnerratafilepackage content cannot be deduced from other tables -- it
is the primary source.
"""""

Currently working on installing a 4.0.7 statellite and syncing channels
individually, then piecing together a query that will discover when the problem
has occurred. Aparently the individuals most likely to know the code have since
left us, but I'm also working on reading through it to try and piece together a
possible cause.

Any update?

Thanks!

Justin

No additional info yet, I've had to put this ticket aside for a couple days in
light of some other issues which are still ongoing. Currently just instructed to
work on this when I can find some free time. If there's any urgency to it's
resolution please feel free to let me know and I'll check with my lead about
reprioritizing.

Satellite sync code appears to check the last modified time of an errata to
determine if it needs to resync it, explaining why the customer saw no change
with their corrupted entries when they satellite-synced again.

For an update on progress, I've been digging through the code for the last
couple days and still am unable to locate where the bug is. While doing so I
noticed we have a --force-all-packages flag for satellite-sync, so I've been
working on a --force-all-errata flag so we can hopefully force the database to
fix itself in the event that we can't locate the bug soon. The code that syncs
these tables is very confusing and the original author is no longer with the
company. The force all errata is not yet working, but hopefully in the proccess
we'll discover where the actual problem is taking place.

Need to move on to other bugs before the release, setting this issue aside again
and will resume as soon as inspiration strikes or time allows.

--force-all-errata is now implemented and working. To verify I went and
intentionally deleted entries from rhnErrataFilePackage and verified that the
links in the UI disappear as per the customers issue. After this doing a
satellite-sync -cchan1 -cchan2 --force-all-errata, the entries in the table are
restored.

Given that we can't locate in the code (yet) where the corruption might have
originated, can't reproduce the issue (we think), and haven't seen anyone else
with a similar problem yet, this workaround might be our best bet for now.

New Revision: 115725

Modified:
   trunk/eng/backend/satellite_tools/satsync.py

Would it be possible to back port this to the Sat. 4.0 sat-sync code?  That is
what the customer is using and I don't believe they will be able to update
anytime soon.

Justin

Should be no problem, will track down someone who can fill me in on everything
needed to do that and I'll try to have it backported by the weekend.

I started asking around for how to backport and was informed that because the
4.0 satellite is no longer maintained by Cliff Perry's group, the request needs
to go through the SEG hotfix process and eventually trickle back down to his group.

We have no way to reproduce the exact issue this customer saw at this time. The
problem however was missing entries in rhnErrataFilePackage, so we can reproduce
the issue by manually deleting such entries.

In the satellite UI choose an errata and view it's package list. Note they
should all have links, except for the source rpms. In my case I used
RHSA-2007:0085-2, which has the errata ID 695 (this may vary from satellite to
satellite, not sure)

Login to the database on the cmd line and execute the following query,
substituing  the errata ID you selected:

select unique(package_id) from rhnerratafilepackage where errata_file_id in
(select id from rhnerratafile where errata_id = 695);

I think each of these corresponds to one of the links we see in the UI. Choose
one of the ID's returned and execute:

delete from rhnerratafilepackage where errata_file_id in (select id from
rhnerratafile where errata_id = 695) and package_id = YOURPACKAGEIDHERE;

Refresh your UI list of pakages for this errata in the browser, links should now
disappear as the customer was seeing in the original ticket.

Now just re-run satellite-sync for the channels on this satellite and include
the --force-all-errata option. Once complete the links should be restored in the
UI and you should see the same results from the above select as you did prior to
running the delete.

verified

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

stage-verified using
rhn-satellite-5.0.0-40-redhat-linux-as-i386-4-embedded-oracle.iso
 on shaggy, moving to RELEASE_PENDING.

Closed for Satellite 500 Release.