Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1419867 - it takes too long to re-sync already synced channel
Summary: it takes too long to re-sync already synced channel
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Satellite Synchronization
Version: 580
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Jan Dobes
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On:
Blocks: sat580-low
TreeView+ depends on / blocked
 
Reported: 2017-02-07 09:45 UTC by Jan Hutař
Modified: 2017-06-21 12:09 UTC (History)
1 user (show)

Fixed In Version: spacewalk-backend-2.5.3-79
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-21 12:09:01 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Jan Hutař 2017-02-07 09:45:20 UTC
Description of problem:
On 5.6 or 5.7, resync of already synced channel takes 3 to 6 minutes (e.g. to get errata). On 5.8Beta it takes almost 20 minutes.


Version-Release number of selected component (if applicable):
Satellite-5.8-RHEL-6-20170118.0-Satellite-x86_64


How reproducible:
1 of 1 attempt


Steps to Reproduce:
1. Sync channel on 5.8Beta and then measure how long resync takes


Actual results:
Re-sync of already fully synced channel takes about 3x more time than on 5.7.


Expected results:
Duration should be similar to 5.7

Comment 1 Jan Dobes 2017-02-07 09:51:30 UTC
Can you tell which part of sync takes most of the time? My suspect is the first part where are evaluated packages which are needed to sync.

Comment 3 Jan Hutař 2017-02-09 23:01:12 UTC
(In reply to Jan Dobes from comment #1)
> Can you tell which part of sync takes most of the time? My suspect is the
> first part where are evaluated packages which are needed to sync.

IMO you are right. Please see attached output.

Comment 4 Jan Dobes 2017-02-10 12:33:57 UTC
I think the main reason why this part is slow when channel is synced is because there is performed sha sum on each package RPM on file system which is referenced from repodata and decided if it matches file in repository. satellite-sync was not doing it most likely. I'm not sure how this could be optimized other way than by simplifying these checks.

Comment 5 Jan Dobes 2017-02-17 16:40:50 UTC
I don't like the fact each cdn-sync/reposync run needs to go compute checksum of every RPM in channel. It can't just compare the checksum value from Satellite database because it's sha256, in repodata are often sha1 checksums. Maybe would be useful to do some caching of these computed checksums or compute them only with some special option.

Comment 6 Jan Dobes 2017-02-27 17:34:55 UTC
I implemented some kind of cache for this in file /var/cache/rhn/reposync/checksum_cache. It caches computed checksums for given path in Satellite package storage. It's global for all packages in Satellite, for RHEL 6 base channel it created cache file with size like 4 MB. Let's see how it performs and split it into multiple files or compress it if necessary.

spacewalk master:
40f794c894a8cc4d5b664415ff0153fd93b2ff40

I'm going to see for other ways how to optimize the run and maybe add more fixes here.

Comment 7 Jan Dobes 2017-03-01 15:42:57 UTC
fixing previous commit in spacewalk master:

49f910e6b539336229ce4407f180df4eb35f57e9

Comment 9 Jan Dobes 2017-03-02 15:05:01 UTC
fixing pylint in spacewalk master:

68a25b6efc6ede6e630a4d14e8bb6099fb578011

Comment 10 Jan Dobes 2017-03-02 15:38:59 UTC
skip package to channel subscription when there was no package synced:

spacewalk master:

b4456e8a87df3787561c6d6dca8efb7f40c53f05

Comment 12 Jan Dobes 2017-03-02 15:41:54 UTC
correct commit from comment #10 is:

b716ceb5fd7262958a72b1449551968232d9ce80

Comment 13 Jan Dobes 2017-03-03 15:22:50 UTC
one small improvement in kickstart sync, spacewalk master:

8cc501a68d8ce491cc26aeeecbd91f993496ad59

Comment 15 Jan Dobes 2017-03-03 16:46:33 UTC
improving errata import to not import always all errata, adding --force-all-errata same as in satellite-sync, spacewalk master:

0614dc0a022e9a70bb68091e3d804fc7a656e020
786c7802a35988f8760e00fe60821d225fa0ec82

Comment 17 Jan Hutař 2017-03-03 21:00:56 UTC
Another possible scenario:

1. I copy /var/satellite with synced rhel-x86_64-server-6 from another Satellite
   to my fresh Satellite
2. Run cdn-sync -c rhel-x86_64-server-6

I have feeling that all packages are being downloaded (sync seems to be very slow). Does this match with the way it is supposed to work (checksum caching and so on)?

If so, should this be fixed or at least reported as a different bugzilla?

Comment 19 Jan Dobes 2017-03-06 14:13:31 UTC
one more fix in spacewalk master:

5833e299e4955c257170beab187e5798ad315c95

Comment 21 Jan Dobes 2017-03-06 14:43:26 UTC
(In reply to Jan Hutař from comment #17)
> Another possible scenario:
> 
> 1. I copy /var/satellite with synced rhel-x86_64-server-6 from another
> Satellite
>    to my fresh Satellite
> 2. Run cdn-sync -c rhel-x86_64-server-6
> 
> I have feeling that all packages are being downloaded (sync seems to be very
> slow). Does this match with the way it is supposed to work (checksum caching
> and so on)?
> 
> If so, should this be fixed or at least reported as a different bugzilla?

I'm afraid this scenario is not supported. In this case RPMs are on filesystem but they are not in database. cdn-sync needs to download it again because it doesn't know where in the filesystem the RPMs could be - the path depends on checksum of each RPM -> cdn-sync can only know about packages if they are in the database (where is path already) or if they are downloaded in stagging directory - /var/satellite/redhat/NULL/stage. It's not possible to get the checksummed-path from nothing and detect the RPM somewhere in /var/satellite/redhat/NULL/*.

In any case, after download, RPMs metadata need to be imported to the database and this is the slowest part by far.

Comment 23 Jan Hutař 2017-03-06 21:40:39 UTC
(In reply to Jan Dobes from comment #21)
> (In reply to Jan Hutař from comment #17)
> > [...]
> I'm afraid this scenario is not supported. In this case RPMs are on
> filesystem but they are not in database. cdn-sync needs to download it again
> because it doesn't know where in the filesystem the RPMs could be - the path
> depends on checksum of each RPM -> cdn-sync can only know about packages if
> they are in the database (where is path already) or if they are downloaded
> in stagging directory - /var/satellite/redhat/NULL/stage. It's not possible
> to get the checksummed-path from nothing and detect the RPM somewhere in
> /var/satellite/redhat/NULL/*.

Hmm, so I can just move these packages into "stage/" and cdn-sync should not download these right? Can I safely pre-cache any packages I'm about to download by this approach? I assume if I place corrupted data into "stage/", cdn-sync will handle it and re-download?

Comment 25 Jan Dobes 2017-03-07 09:40:18 UTC
(In reply to Jan Hutař from comment #23)
> (In reply to Jan Dobes from comment #21)
> > (In reply to Jan Hutař from comment #17)
> > > [...]
> > [...]
> 
> Hmm, so I can just move these packages into "stage/" and cdn-sync should not
> download these right? Can I safely pre-cache any packages I'm about to
> download by this approach? I assume if I place corrupted data into "stage/",
> cdn-sync will handle it and re-download?

Correct, you can pick RPMs from directories and put them into stage directory, cdn-sync will check if they match packages from repository.


Note You need to log in before you can comment on or make changes to this bug.