Bug 232037 - Errata Packages Appear Without Links Following satellite-sync
Errata Packages Appear Without Links Following satellite-sync
Product: Red Hat Network
Classification: Red Hat
Component: RHN/Backend (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Devan Goodwin
Preethi Thomas
Depends On:
Blocks: 173427 277811
  Show dependency treegraph
Reported: 2007-03-13 13:45 EDT by Devan Goodwin
Modified: 2010-10-22 09:40 EDT (History)
4 users (show)

See Also:
Fixed In Version: sat500
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-06-25 23:06:47 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Screenshot from a customer site of an errata with missing package links. (2.93 MB, image/png)
2007-03-13 13:45 EDT, Devan Goodwin
no flags Details
Database output from provided query (106.99 KB, text/plain)
2007-03-13 13:51 EDT, Devan Goodwin
no flags Details

  None (edit)
Description Devan Goodwin 2007-03-13 13:45:27 EDT
Description of problem:

After performing a satellite-sync, some packages can appear without links when
viewing the details on an errata. 

Version-Release number of selected component (if applicable):

Surfaced in a customer deployment of 4.0.7 satellite.

How reproducible:


Steps to Reproduce:

Unknown as of yet.
Actual results:

Packages appear without links for certain channels when viewing an erratas details.

Expected results:

Packages should have proper links as they do when viewing the same errata in

Additional info:

Problem originally reported by Vishal Gaikwad in this email with specifics on
the results encountered by an actual customer:

Comment 1 Devan Goodwin 2007-03-13 13:45:29 EDT
Created attachment 149957 [details]
Screenshot from a customer site of an errata with missing package links.
Comment 2 Devan Goodwin 2007-03-13 13:51:24 EDT
Created attachment 149960 [details]
Database output from provided query

Customer output from executing the following queries, note that 10736 is the
errata ID for the errata the customer provided a screenshot for.

select * from rhnerratapackage
where errata_id = 10736;
select * from rhnerratafile
where errata_id = 10736;
select * from rhnerratafilepackage
where errata_file_id in (
	select id from rhnerratafile where errata_id = 10736
select * from rhnerratafilechannel
where errata_file_id in (
	select id from rhnerratafile where errata_id = 10736
select * from rhnchannel where id in (
	select channel_id from rhnerratafilechannel
	where errata_file_id in (
		select id from rhnerratafile where errata_id = 10736
Comment 3 Devan Goodwin 2007-03-13 14:19:12 EDT
Packages are appearing without links due to missing entries in
rhnErrataFilePackage that should correspond to entries in rhnErrataFile.

Examining the output from the above customer data and running the same queries
against the production database to verify the number of results being returned
for the same errata we see the following:

rhnErrataPackage: 7 rows for customer, 27 in production
rhnErrataFile: 78 rows for customer, 79 in production
rhnErrataFilePackage: 28 rows for customer, 74 in production

It seems likely all three tables are out of sync with production.

The problem sounds similar to 208465 where errata packages were being deleted
after syncing a new channel. We attempted to have the customer perform some of
the actions listed in that ticket to see if it would remedy the problem, namely
re-running satellite-sync with a -c option for all their existing channels, and
running the sql provided as a fix in that ticket. Neither approach seemed to
remedy their problem.

We suspect there is a similar issue with syncing new channels in the code that
deals with the errata file tables.

Jan added the following with regard to the customers sql output from the
previous comment:

So we are looking at the files (rhnerratafile records)


which has got the link, and


which does not.

There is a 9577, 340289 record in rhnerratafilepackage and 104, 340289
in rhnerratafilechannel, but only 116, 340300 record in
rhnerratafilechannel -- no record in rhnerratafilepackage.

Well, we already knew this since it was the only possible way that the
outer join in the errata_packages select could have returned a record
with a null rhnErrataFileChannel.package_id.

It seems to me that the cause might be similar to bug 208465 -- a sync
that adds new channel (and does not list the old ones) probably removes
the records. The problem is that unlike rhnerratapackage,
rhnerratafilepackage content cannot be deduced from other tables -- it
is the primary source.

Currently working on installing a 4.0.7 statellite and syncing channels
individually, then piecing together a query that will discover when the problem
has occurred. Aparently the individuals most likely to know the code have since
left us, but I'm also working on reading through it to try and piece together a
possible cause.
Comment 4 Justin Sherrill 2007-03-19 14:52:16 EDT
Any update?


Comment 5 Devan Goodwin 2007-03-19 15:31:09 EDT
No additional info yet, I've had to put this ticket aside for a couple days in
light of some other issues which are still ongoing. Currently just instructed to
work on this when I can find some free time. If there's any urgency to it's
resolution please feel free to let me know and I'll check with my lead about
Comment 6 Devan Goodwin 2007-04-19 08:14:51 EDT
Satellite sync code appears to check the last modified time of an errata to
determine if it needs to resync it, explaining why the customer saw no change
with their corrupted entries when they satellite-synced again.
Comment 7 Devan Goodwin 2007-04-20 11:26:36 EDT
For an update on progress, I've been digging through the code for the last
couple days and still am unable to locate where the bug is. While doing so I
noticed we have a --force-all-packages flag for satellite-sync, so I've been
working on a --force-all-errata flag so we can hopefully force the database to
fix itself in the event that we can't locate the bug soon. The code that syncs
these tables is very confusing and the original author is no longer with the
company. The force all errata is not yet working, but hopefully in the proccess
we'll discover where the actual problem is taking place.

Need to move on to other bugs before the release, setting this issue aside again
and will resume as soon as inspiration strikes or time allows.

Comment 8 Devan Goodwin 2007-05-02 11:58:18 EDT
--force-all-errata is now implemented and working. To verify I went and
intentionally deleted entries from rhnErrataFilePackage and verified that the
links in the UI disappear as per the customers issue. After this doing a
satellite-sync -cchan1 -cchan2 --force-all-errata, the entries in the table are

Given that we can't locate in the code (yet) where the corruption might have
originated, can't reproduce the issue (we think), and haven't seen anyone else
with a similar problem yet, this workaround might be our best bet for now.

New Revision: 115725

Comment 9 Justin Sherrill 2007-05-02 15:59:25 EDT
Would it be possible to back port this to the Sat. 4.0 sat-sync code?  That is
what the customer is using and I don't believe they will be able to update
anytime soon.

Comment 10 Devan Goodwin 2007-05-03 10:07:16 EDT
Should be no problem, will track down someone who can fill me in on everything
needed to do that and I'll try to have it backported by the weekend.
Comment 11 Devan Goodwin 2007-05-04 09:50:56 EDT
I started asking around for how to backport and was informed that because the
4.0 satellite is no longer maintained by Cliff Perry's group, the request needs
to go through the SEG hotfix process and eventually trickle back down to his group.
Comment 12 Devan Goodwin 2007-05-08 10:15:53 EDT
We have no way to reproduce the exact issue this customer saw at this time. The
problem however was missing entries in rhnErrataFilePackage, so we can reproduce
the issue by manually deleting such entries.

In the satellite UI choose an errata and view it's package list. Note they
should all have links, except for the source rpms. In my case I used
RHSA-2007:0085-2, which has the errata ID 695 (this may vary from satellite to
satellite, not sure)

Login to the database on the cmd line and execute the following query,
substituing  the errata ID you selected:

select unique(package_id) from rhnerratafilepackage where errata_file_id in
(select id from rhnerratafile where errata_id = 695);

I think each of these corresponds to one of the links we see in the UI. Choose
one of the ID's returned and execute:

delete from rhnerratafilepackage where errata_file_id in (select id from
rhnerratafile where errata_id = 695) and package_id = YOURPACKAGEIDHERE;

Refresh your UI list of pakages for this errata in the browser, links should now
disappear as the customer was seeing in the original ticket.

Now just re-run satellite-sync for the channels on this satellite and include
the --force-all-errata option. Once complete the links should be restored in the
UI and you should see the same results from the above select as you did prior to
running the delete.
Comment 13 Preethi Thomas 2007-05-08 13:08:36 EDT
Comment 15 RHEL Product and Program Management 2007-05-08 15:44:22 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
Comment 18 Máirín Duffy 2007-06-18 12:49:48 EDT
stage-verified using
 on shaggy, moving to RELEASE_PENDING.
Comment 19 Brandon Perkins 2007-06-25 23:06:47 EDT
Closed for Satellite 500 Release.

Note You need to log in before you can comment on or make changes to this bug.