Bug 650165 - Incremental dump creation in Satellite 5.4 is flawed
Incremental dump creation in Satellite 5.4 is flawed
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server (Show other bugs)
Unspecified Unspecified
high Severity medium
: ---
: ---
Assigned To: Michael Mráka
Šimon Lukašík
Depends On:
Blocks: sat54-errata
  Show dependency treegraph
Reported: 2010-11-05 09:04 EDT by Milan Zazrivec
Modified: 2011-02-01 11:16 EST (History)
4 users (show)

See Also:
Fixed In Version: spacewalk-backend-1.2.13-25
Doc Type: Bug Fix
Doc Text:
Cause: Consequence: Fix: Result:
Story Points: ---
Clone Of:
Last Closed: 2011-02-01 11:16:24 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Milan Zazrivec 2010-11-05 09:04:02 EDT
Description of problem:
From satellite_tools/disk_dumper/dumper.py:

_query_get_package_ids_by_date_limits = rhnSQL.Statement("""
        select package_id
          from rhnPackage rp, rhnChannelPackage rcp
         where rcp.channel_id = :channel_id
         and rcp.package_id = rp.id
         and (rcp.modified >= TO_Date(:lower_limit, 'YYYYMMDDHH24MISS')
              or rp.last_modified >= TO_Date(:lower_limit, 'YYYYMMDDHH24MISS')
         and (rcp.modified <= TO_Date(:upper_limit, 'YYYYMMDDHH24MISS')
              or rp.last_modified <= TO_Date(:upper_limit, 'YYYYMMDDHH24MISS')

rhnPackage.last_modified values are (ordinarilly) being populated during
satellite-sync w/ RHN and represent the time the package was modified
on RHN side (i.e. it is a value we take from RHN and store it so as it is).

rhnChannelPackage.modified values are being filed during satellite-sync
and are set to sysdate of the sync (they represent the moment when the
particular package became part of that particular channel, which is
the moment of the sync).

For a particular package to become part of an incremental dump (defined by
:lower_limit and :upper_limit placeholders), either of rhnPackage.last_modified
rhnChannelPackage.modified has to fit into the interval specified.

Imagine you've just synced a RHEL-5-Server channel from RHN (i.e.
rhnChannelPackage.modified values are very fresh) and want to export its
content from recent past (for example starting week ago, finishing 2 days ago).
Your situation would look like:

lower_limit < upper_limit < all rhnChannelPackage.modified values

Since most of the rhnPackage.last_modified values will be older than
your specified lower_limit (the channel content is simply much older),


(rcp.modified >= TO_Date(:lower_limit, 'YYYYMMDDHH24MISS')
 or rp.last_modified >= TO_Date(:lower_limit, 'YYYYMMDDHH24MISS')
) and
(rcp.modified <= TO_Date(:upper_limit, 'YYYYMMDDHH24MISS')
 or rp.last_modified <= TO_Date(:upper_limit, 'YYYYMMDDHH24MISS')

condition will be true for most of the packages contained in the channel
and your incremental dump will contain pretty much the whole channel.

I believe rhn-satellite-exporter needs to be able to export the incremental
channel content based on rhnChannelPackage.modified values, and based on
rhnPackage.last_modified values but definitely must not 'OR' the two values.

In another words, when creating incremental content, you need to be able
to specify, whether you wish to select the packages based on the date
they were born in RHN or they were actually synced to your Satellite.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. New Satellite 5.4 installation, sync some content from RHN
2. Create incremental export starting 7 days ago, finishing 2 days ago
3. Check the created export
Actual results:
The export contains whole channel.

Expected results:
You need to be able to specify what date you want to select the packages
for the export.

Additional info:
Comment 2 Milan Zazrivec 2010-11-18 09:53:25 EST
Needless to say, the same logic needs to apply to erratas, ks trees and
ks files (satellite_tools/disk_dumper/iss.py)
Comment 3 Michael Mráka 2010-11-22 05:04:08 EST
We agreed to create new options --use-sync-date and --use-rhn-date and let user choose which one use (default is --use-rhn-date) rather than trying how to put them together.

Spacewalk git:
commit 44e62bc2296f5a36b9e736ef094d5533ea6f4f2a
    650165 - let user specify which date for incremental export use
commit 2e8568e823c0df4fc6cf6e8b4faff6a2a9183bca
    650165 - propagate use_rhn_date down to Dumper
Comment 4 Miroslav Suchý 2010-12-16 12:25:01 EST
cherry-picked to satellite.git as commits:
Comment 6 Šimon Lukašík 2011-01-11 07:39:33 EST
Flipping back to Assigned - for review

I have done very incremental export of channel, which should not contain
any packages. Directories packages_short and packages does not exists,
although channel.xml does contain all packages.

This breaks the import of the dump.
Comment 8 Michael Mráka 2011-01-17 08:22:01 EST
Fixed in spacewalk git
commit ab481c31b699933cd186f9b68f5a0e4781a2b100
    650165 - _ChannelDumper should also understand use_rhn_date
Comment 9 Michael Mráka 2011-01-17 08:23:35 EST
Backported to satellite git as
commit 88fa61bc4aa9b2a541a656ed73d9af1bc06963b1
    650165 - _ChannelDumper should also understand use_rhn_date
    (cherry picked from commit ab481c31b699933cd186f9b68f5a0e4781a2b100)
Comment 11 Šimon Lukašík 2011-01-24 12:17:31 EST
Changing to Verified:

Testing procedure:
 - Verified scenario from comment 0
 - Created various dumps with using new options 
   (--use-rhn-date, --use-sync-date)
 - Manual check of dump/channels/*/channel.xml.gz for number of pkgs.
 - Verified that all the dumps created with new packages were importable
 - At least some of dumps were imported on different satellite

Example of dumps created, for both of the options (x represents of sync):
<x-2,x-2>, <x-7,x-2>, <x-7,x+2>, <x-365,x-2>, <x-2,x-2>, <0, x-2>

Verified against:

Most of the testing was done on x86_64 only.
Comment 12 Florian Nadge 2011-01-27 06:26:04 EST
Please be so kind and add a few key words to the technical note of this
bugzilla entry using the following structure:





For details, see:

Comment 13 Florian Nadge 2011-01-27 06:26:04 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    New Contents:



Comment 14 Miroslav Suchý 2011-01-27 07:11:11 EST
There is already nice text in errata.
Comment 15 errata-xmlrpc 2011-02-01 11:16:24 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.