Bug 650129 - rhnPackage.last_modified will lose its original values after an upgrade to Satellite 5.4
rhnPackage.last_modified will lose its original values after an upgrade to Sa...
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Upgrades (Show other bugs)
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Milan Zázrivec
Red Hat Satellite QA List
: Regression
Depends On:
Blocks: sat54-errata
  Show dependency treegraph
Reported: 2010-11-05 06:31 EDT by Milan Zázrivec
Modified: 2011-01-26 11:35 EST (History)
7 users (show)

See Also:
Fixed In Version: rhn-upgrade-
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2010-12-13 09:31:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
List of packages which exist on connected and do not exist on disconnected (840 bytes, text/plain)
2010-12-14 07:42 EST, Milan Zázrivec
no flags Details

  None (edit)
Description Milan Zázrivec 2010-11-05 06:31:19 EDT
Description of problem:
In Satellite 5.3 schema (and earlier) we have following trigger defined:

create or replace trigger
before insert or update on rhnPackage
for each row
    -- when we do a sat sync, we use last_modified to keep track
    -- of the upstream modification date.  So if we're setting
    -- it explicitly, don't override with sysdate.  But if we're
    -- not changing it, then this is a genuine update that needs
    -- tracking.
    -- we're not using is_satellite() here instead, because we
    -- might want to use this to keep webdev in sync.
    if :new.last_modified = :old.last_modified then
        :new.last_modified := sysdate;
    end if;
    :new.modified := sysdate;

During the schema upgrade to Satellite 5.4, we perform following
update (SHA-256 related change):

update rhnPackage p
       set checksum_id = (select id
                            from rhnChecksum c
                           where checksum_type_id = md5_id
                             and p.md5sum =  c.checksum);

The update above with the trigger defined before will effectively
set all rhnPackage.last_modified values to sysdate of the upgrade.

The rhnPackage.last_modified values are normally filed during
a satellite-sync with values retrieved from RHN (hosted) and
play an important role during incremental dump creation (decision
which packages will land in the time interval specified).

The rhnPackage.last_modified values set to sysdate of the upgrade
process render the creation of incremental dumps impossible.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Satellite 5.3, sync some content from RHN
2. Connect to the database with sqlplus
3. SQL> select last_modified from rhnPackage where rownum < 10;
4. Upgrade to Satellite 5.4
5. Connect to the database with sqlplus, perform the same query and
   compare with original values.
Actual results:
rhnPackage.last_modified values after the upgrade are set to sysdate.

Expected results:
rhnPackage.last_modified values stay preserved.

Additional info:
Subsequent satellite-sync done on the upgraded Satellite 5.4 will not
fix the last_modified values.
Comment 3 Milan Zázrivec 2010-11-22 12:23:28 EST
Another tables / columns showing the same problem after upgrade:
* rhnKickstartableTree.last_modified
* rhnKSTreeFile.last_modified
Comment 4 Milan Zázrivec 2010-11-23 15:36:20 EST
satellite.git SATELLITE-5.4:
Comment 5 Kermit Short 2010-11-23 16:02:19 EST
I recently upgraded from 5.3, where incremental export/import issues had been fixed to 5.4, where it is now broken again.  We're migrating from 4.2.3 to the 5.4 satellites, and 4.2.3 goes end of life late December.  When can we get this bug fixed??
Comment 6 Clifford Perry 2010-11-23 16:46:01 EST
This bug should be going to the QA team to test for an Errata within the next 1 ->2 days. As to when it is released, as soon as it and all associated bugs to the errata have passed testing. Ideally, as soon as we can. 

Comment 8 Milan Zázrivec 2010-12-01 06:24:29 EST
fix-imported-data script

satellite.git SATELLITE-5.4: b00ec5e77a50148b1978dff609b35beaaa0c9848
Comment 9 Milan Zázrivec 2010-12-01 07:21:51 EST
Fix for the problem described in this bug report has two parts:

1. Fixed schema upgrade scripts, so that new upgrades won't modify
the 'LAST_MODIFIED' values

2. Script which will fix the values on Satellites which already upgraded
and currently have incorrect data.

The script will be contained in updated rhn-upgrade package and can be
found at /etc/sysconfig/rhn/satellite-upgrade/fix-imported-data

The 'fix-imported-data' script does the following:
1. It retrieves channel, package and kickstart information
   from RHN (live sync), parent satellite (ISS) or a channel dump
2. It replaces following database tables / columns with the values
   retrieved in previous step (table.column):
   * rhnPackage.last_modified
   * rhnKickstartableTree.last_modified
   * rhnKSTreeFile.last_modified

There are two important things which need to be considered before
executing the script:
1. It needs to be run on the Satellite system which needs its data corrected.
2. Data source for the script. Can be: RHN (default source), ISS (parent
   satellite), or a disk export (channel dump). 

Typical use of the script (RHN is the source):

    cd /etc/sysconfig/rhn/satellite-upgrade
    ./fix-imported-data --verbose --log /tmp/fix-imported-data.log

When using parent satellite as the source (ISS):

    cd /etc/sysconfig/rhn/satellite-upgrade
    ./fix-imported-data --verbose --log /tmp/fix-imported-data.log \

When using channel dump as the source:

    cd /etc/sysconfig/rhn/satellite-upgrade
    ./fix-imported-data --verbose --log /tmp/fix-imported-data.log \
        --dump /path/to/the/dump
Comment 10 Milan Zázrivec 2010-12-01 08:26:03 EST
Schema change for upstream sources:

spacewalk.git master: de6ba4d02405aee3b047f7a7a4d59d44111c6882
Comment 16 Milan Zázrivec 2010-12-02 17:01:01 EST
The traceback described in comment #13 is caused by the simple fact
that not all channels we process have kickstartable tree(s)
associated -> imported_channel_kstrees() function returns 'None'.

I apologize for the inconvenience -- these situations certainly need
to be properly handled by the script.

Fix for the problem is quite simple:

--- a/satellite/upgrade/fix-imported-data
+++ b/satellite/upgrade/fix-imported-data
@@ -66,7 +66,7 @@ def imported_channel_kstrees(channel):
     h = rhnSQL.prepare(sql)
-    return h.fetchall_dict()
+    return h.fetchall_dict() or []
 def enable_trigger(trigger):
     h = rhnSQL.prepare("alter trigger %s enable" % trigger)

satellite.git SATELLITE-5.4: f3685fba39998101634b40a586d1d15da238f760

Updated rhn-upgrade packages:
* https://brewweb.devel.redhat.com/buildinfo?buildID=150652
* https://brewweb.devel.redhat.com/buildinfo?buildID=150648
Comment 26 Milan Zázrivec 2010-12-08 11:35:51 EST
proper time zone handling in fix-imported-data implemented in:
satellite.git SATELLITE-5.4: 3ef4f0bec06e175bcfe764063293d89462a4bd28

fix-imported-content now has functionality (--list-content option) for
listing content (packages, ks trees, ks files) available in the import source
(RHN, ISS, channel dump):
satellite.git SATELLITE-5.4: 4577ebd7a307bbd45d463a2ba02a012f9b399791
Comment 30 Milan Zázrivec 2010-12-08 17:36:41 EST
fix-imported-data, fix for the following traceback:

Traceback (most recent call last):
  File "/etc/sysconfig/rhn/satellite-upgrade/fix-imported-data", line 350, in ?
    message("  Package %s does not seem to be present in the import source,
  skipping" % p, 1, 0)
  NameError: name 'p' is not defined

satellite.git SATELLITE-5.4: 6310332a8a7575cd7dd985faa70ff0ef04389ce8

rhn-upgrade- builds:

Comment 37 Milan Zázrivec 2010-12-10 05:47:00 EST
fix-imported-data: request channels from RHN according to valid entitlements:

satellite.git SATELLITE-5.4: 2aae79f43af78ad8559f648b0b2b5350fcb0c31f
Comment 40 errata-xmlrpc 2010-12-13 09:31:11 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.