Bug 1540981

Hi Team,

As a workaround, we can move the checksum_cache file on the satellite server to a temporary location.

------
# mv /var/cache/reposync/checksum_cache /tmp/
# cdn-sync
------

Deepannagaraj N.

There is probably not enough space in /var/cache location. Was disk space increased on partition containing /var/cache? The checksum_cache file is used for speed up subsequent cdn-sync runs without need to recompute checksum of all RPMs in channel. If this file gets too big, it can be safely deleted.

Hello Jan,

Thank you for the quick response.

Yes, the disk space is increased and after that the cdn-sync fails to run and the error is also not more clear so it will be difficult for the customer to understand.

Deepannagaraj N.

What is following command output?
# df -h

Because Satellite is using multiple locations to store files, depending on customer's disk partition layout some partitions may require more space than other.

I agree the error message could be clearer.

Hello Jan,

In my test Satellite server, I have everything mounted under "/".

~~~~~~
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_vm-lv_root
                      181G  135G   38G  79% /
tmpfs                 2.4G  4.0K  2.4G   1% /dev/shm
/dev/sda1             477M   75M  377M  17% /boot
/dev/mapper/vg_vm-lv_home
                      9.8G   36M  9.2G   1% /home
~~~~~~

Above size is after increasing the disk space.

Deepannagaraj N.

Not sure, if we can reproduce this, as I remember at least two other BZs dealing with limited space not hitting this issue, but let's try.
If there was an internal reproducer, that would be great.

I suppose we could update the error message when the file gets corrupted or there isn't enough disk space to advise user to delete this file or check if there is enough space.

Thanks Deepannagaraj!
Ack, Jan.

An corrupted /var/opt/satellite/var/cach/reposync/checksum_cache file caused the problem.
The cdn-sync command gave a strange python error, sugestion to give a more clean error when this problem rises again.
After i deleted the /var/opt/satellite/var/cach/reposync/checksum_cache, the file was rebuild and i was able to sync again.

Sugestion:
better error handling of the cdn-sync command or in case of error automaticly rebuild (or delete) the file.

Reproduce:
replace the /var/opt/satellite/var/cach/reposync/checksum_cache file with a broken one.

(In reply to Sander Eerkes from comment #10)
> An corrupted /var/opt/satellite/var/cach/reposync/checksum_cache file caused
> the problem.
> The cdn-sync command gave a strange python error, sugestion to give a more
> clean error when this problem rises again.
> After i deleted the /var/opt/satellite/var/cach/reposync/checksum_cache, the
> file was rebuild and i was able to sync again.
> 
> Sugestion:
> better error handling of the cdn-sync command or in case of error
> automaticly rebuild (or delete) the file.
> 
> Reproduce:
> replace the /var/opt/satellite/var/cach/reposync/checksum_cache file with a
> broken one.

Concur. One note - checksum_cache path is actually

  /var/cache/rhn/reposync/checksum_cache

Proposed wording change:

===
[root@host ~]# cdn-sync -c rhn-tools-rhel-x86_64-server-6
11:50:19 ======================================
11:50:19 | Channel: rhn-tools-rhel-x86_64-server-6
11:50:19 ======================================
11:50:19 Sync of channel started.
SYNC ERROR: Perhaps checksum_cache is corrupted? Remove /var/cache/rhn/reposync/checksum_cache and retry.
<type 'exceptions.EOFError'>
[root@host ~]# rm /var/cache/rhn/reposync/checksum_cache
[root@host ~]# cdn-sync -c rhn-tools-rhel-x86_64-server-6
11:50:31 ======================================
11:50:31 | Channel: rhn-tools-rhel-x86_64-server-6
11:50:31 ======================================
11:50:31 Sync of channel started.
11:50:31 Repo URL: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhn-tools/os
11:50:32 Packages in repo:               226
11:50:32 No new packages to sync.
11:50:33 Repo https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhn-tools/os has comps file fbecca9e03f6e988b833143039e9583454cc84f7-comps.xml.
11:50:33 Repo https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhn-tools/os has 61 errata.
11:50:33 No new errata to sync.
11:50:34 Kickstartable tree not detected (no valid treeinfo file)
11:50:34 Sync of channel completed in 0:00:02.
11:50:34 Total time: 0:00:02
[root@host-8-243-229 ~]#

spacewalk.github:
daac1eb0e4f8c9b301d9793e0be93d20776c1a70

Let's not make comments private unless we have to - discussion that isn't exposing customer data or internal machine names is fine to have in public comments.

The error i got with the checksum_cache file was diffrent since i think the file was partly corrupted.
After i removed the file the problem was solved.
See my error message:

https://drive.google.com/file/d/1Qcxfdk8AEQ3iN8Vh6JbjjulONjP29Ren/view?usp=sharing

ps. i made a screenshot as i was not able to copy and paste it here.

The file was corrupted due to a new full sync of a new channel.
After the file system went full to 100% we aborted the cdn-sync.
Then we increased the file system, after the resize it was 65%. (75G free)
Then we ran cdn-sync again and at that moment we saw the error.
The error was realy strange, see the above link.
When we removed the checksum_cache file everything was back to normal.
So in my case the error message was not clear at all, thats why this ticket was created.

Created attachment 1395828 [details]
error message when syncing

Error due to (partly?) corruped checksum_cache file.

(In reply to Sander Eerkes from comment #18)
> The file was corrupted due to a new full sync of a new channel.
> After the file system went full to 100% we aborted the cdn-sync.
> Then we increased the file system, after the resize it was 65%. (75G free)
> Then we ran cdn-sync again and at that moment we saw the error.
> The error was realy strange, see the above link.
> When we removed the checksum_cache file everything was back to normal.
> So in my case the error message was not clear at all, thats why this ticket
> was created.

Sanders - yup, exactly so. The proposed fix makes the problem and possible fix  more clear:

===
SYNC ERROR: Perhaps checksum_cache is corrupted? Remove /var/cache/rhn/reposync/checksum_cache and retry.
===

Reproduced using the reproducer from the comment #7 on spacewalk-backend-2.5.3-160. Creating an empty /var/cache/rhn/reposync/checksum_cache file was enough to reproduce the error during the cdn-sync.

After updating to spacewalk-backend-2.5.3-162, "corrupting" the checksum_cache  file again, and running the cdn-sync, there was a following output and error message:

# cdn-sync -c rhel-x86_64-server-7 --no-packages
05:25:20 ======================================
05:25:20 | Channel: rhel-x86_64-server-7
05:25:20 ======================================
05:25:20 Sync of channel started.
SYNC ERROR: Perhaps checksum_cache is corrupted? Remove /var/cache/rhn/reposync/checksum_cache and retry.
<type 'exceptions.EOFError'>

After the checksum_cache file the cdn-sync was successful again.

VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0393

(In reply to Grant Gainey from comment #20)
> (In reply to Sander Eerkes from comment #18)
> > The file was corrupted due to a new full sync of a new channel.
> > After the file system went full to 100% we aborted the cdn-sync.
> > Then we increased the file system, after the resize it was 65%. (75G free)
> > Then we ran cdn-sync again and at that moment we saw the error.
> > The error was realy strange, see the above link.
> > When we removed the checksum_cache file everything was back to normal.
> > So in my case the error message was not clear at all, thats why this ticket
> > was created.
> 
> Sanders - yup, exactly so. The proposed fix makes the problem and possible
> fix  more clear:
> 
> ===
> SYNC ERROR: Perhaps checksum_cache is corrupted? Remove
> /var/cache/rhn/reposync/checksum_cache and retry.
> ===


Hi Grant,

Is this also fixed for Satellite 6.X or is there no checksum_cache in 6.X?

Gr, Sander.

(In reply to Sander Eerkes from comment #25)
> (In reply to Grant Gainey from comment #20)
> > (In reply to Sander Eerkes from comment #18)
> > > The file was corrupted due to a new full sync of a new channel.
> > > After the file system went full to 100% we aborted the cdn-sync.
> > > Then we increased the file system, after the resize it was 65%. (75G free)
> > > Then we ran cdn-sync again and at that moment we saw the error.
> > > The error was realy strange, see the above link.
> > > When we removed the checksum_cache file everything was back to normal.
> > > So in my case the error message was not clear at all, thats why this ticket
> > > was created.
> > 
> > Sanders - yup, exactly so. The proposed fix makes the problem and possible
> > fix  more clear:
> > 
> > ===
> > SYNC ERROR: Perhaps checksum_cache is corrupted? Remove
> > /var/cache/rhn/reposync/checksum_cache and retry.
> > ===
> 
> 
> Hi Grant,
> 
> Is this also fixed for Satellite 6.X or is there no checksum_cache in 6.X?
> 
> Gr, Sander.

Hi Sander,

Sat6 is a completely different product - there's no overlap between this BZ and anything in Sat6 or its subprojects (pulp would be the thing that fills this particular niche, I think) cdn-sync is a Sat5-only thing.