Bug 1473784

+++ This bug was initially created as a clone of Bug #1420288 +++

Description of problem:
I compare inter satellite sync with oracle and postgresql database. And on both systems are problem with inconsistent synced cache.

Oracle:

SYNC ERROR: unhandled exception occurred:

(Check logs/email for potentially more detail)

<rhnFault class (code = 54, text = 'ORA-01400: cannot insert NULL into ("RHNSAT"."RHNKSTREEFILE"."CHECKSUM_ID")
')>
(54, 'ORA-01400: cannot insert NULL into ("RHNSAT"."RHNKSTREEFILE"."CHECKSUM_ID")\n', '\n     Package Upload Failed due to uniqueness constraint violation.\n     Make sure the package does not have any duplicate dependencies or\n     does not already exists on the server\n     ')

Postgresql:

ERROR: Encountered IntegrityError: 
null value in column "checksum_id" violates not-null constraint
DETAIL:  Failing row contains (19, EULA, null, 8445, 2008-09-02 19:30:57+02, 2017-02-08 11:59:46.996481+01, 2017-02-08 11:59:46.996481+01).

consider removing satellite-sync cache at /var/cache/rhn/satsync/* and re-run satellite-sync with same options.
If this error persits after removing cache, please contact Red Hat support.

Version-Release number of selected component (if applicable):
spacewalk-backend-tools-2.5.3-49.1.el6sat.noarch

How reproducible:
always

Steps to Reproduce:
1. probably switch rhn/cdn parent (stage/prod)

Actual results:
satellite oracle and postgresql behave different

Expected results:
Behavior is same.

--- Additional comment from Pavel Studeník on 2017-02-08 09:05:28 EST ---

It looks that it is not possible to sync kickstarts (full base channel) from other satellite 5.8 by ISS. After I remove cache and try to sync again then I receive error again

rm -rf /var/cache/rhn/*

>>  satellite-sync --iss-parent=foo.redhat.com --ca-cert=/root/RHN-ORG-TRUSTED-SSL-CERT  -c rhel-x86_64-server-5 --no-rpms --no-package

I tried to sync rhel 5/6/7 and the error was same for all combination.

--- Additional comment from Tomas Lestach on 2017-02-08 11:19:59 EST ---

I hope the reproducer steps are sufficient for us.

--- Additional comment from Pavel Studeník on 2017-02-10 11:11:06 EST ---

Reproducer:

1) Install Satellite 5.8 and sync rhn channel from CDN.
2) Enable on the satellite ISS for new slave satellite.
3) Install slave Satellite 5.8 and try to sync synced rhn channel from master .

>> satellite-sync --iss-parent sat58.master.example.com --ca-cert /root/RHN-ORG-TRUSTED-SSL-CERT  -c rhel-x86_64-server-6 --no-packages --no-rpms
....
Downloading:######################################## - complete
17:07:24 
17:07:24 Downloading kickstartable trees files
17:07:24    Retrieving / parsing kickstart tree files: rhel-x86_64-server-6 (NONE RELEVANT)
17:07:24 
17:07:24 Importing channel errata
17:07:24    Importing *relevant* errata: rhel-x86_64-server-6 (NONE RELEVANT)
17:07:24 Importing kickstartable trees (8)
ERROR: Encountered IntegrityError: 
null value in column "checksum_id" violates not-null constraint
DETAIL:  Failing row contains (9, EULA, null, 8794, 2010-09-02 22:59:26+02, 2017-02-10 17:07:22.566023+01, 2017-02-10 17:07:22.566023+01).

--- Additional comment from Jan Dobes on 2017-02-10 12:37:11 EST ---

I reproduced it with two Spacewalk nightlies.

Also found a one-line patch for it. But it's weird to me because it seems like it never worked.

Pavel, do you know if it works with 2 Satellite 5.7 machines?

--- Additional comment from Pavel Studeník on 2017-02-13 10:51:05 EST ---

I tested different environments:

Satellite 5.7 synced from Satellite 5.7 - PASS
Satellite 5.8 synced from Satellite 5.8 - FAIL
Satellite 5.7 synced from Satellite 5.8 - FAIL 
Satellite 5.8 synced from Satellite 5.7 - PASS

Problem is caused when Satellite 5.8 is master.

--- Additional comment from Jan Dobes on 2017-02-13 12:29:21 EST ---

(In reply to Pavel Studeník from comment #5)
> I tested different environments:
> 
> Satellite 5.7 synced from Satellite 5.7 - PASS
> Satellite 5.8 synced from Satellite 5.8 - FAIL
> Satellite 5.7 synced from Satellite 5.8 - FAIL 
> Satellite 5.8 synced from Satellite 5.7 - PASS
> 
> Problem is caused when Satellite 5.8 is master.

Great news, thanks!

--- Additional comment from Jan Dobes on 2017-02-14 12:23:53 EST ---

All kickstart files synced from RHN are in database with MD5 checksum. cdn-sync inserts these files with SHA256 checksum. During ISS, slave Satellite can parse only MD5 checksum from XML export, any other checksum is ignored and Satellite tries to insert null value.

This bug is basically present also in Satellite 5.6 and 5.7 and occurs on ISS of kickstarts synced with spacewalk-repo-sync.

--- Additional comment from Jan Dobes on 2017-02-14 12:32:15 EST ---

fixed in spacewalk master:

54c4bab31eec49a26f7fefebbdd4551eb89db1ab

--- Additional comment from Jan Dobes on 2017-02-14 12:36:37 EST ---

cherry-picked to satellite.git(SATELLITE-5.8):

72d5d32ad1050967f8e0353e375ca615923d034a


--- Additional comment from Radovan Drazny on 2017-05-03 06:40:50 EDT ---

Doesn't seem to be fixed. 

# yum list spacewalk-backend
Loaded plugins: product-id, search-disabled-repos, security, subscription-manager
Installed Packages
spacewalk-backend.noarch             2.5.3-111.el6sat                            @red-hat-satellite

Synced "rhel-x86_64-server-6" channel from CDN on Sat 5.8. 
Set-up ISS environment and attempted to do an ISS using a Sat 5.7 server as a slave:

# satellite-sync --iss-parent=sat58server.example.com -c rhel-x86_64-server-6 --no-packages --no-rpms                                                                                            
06:26:08 Red Hat Satellite - live synchronization                                                                                                                                                                                           
06:26:08    url: https://sat58server.example.com                                                                                                                                                           
06:26:08    debug/output level: 1                                                                                                                                                                                                           
06:26:08    db:  rhnuser/<password>@rhnschema                                                                                                                                                                                               
06:26:08                                                                                                                                                                                                                                    
06:26:08 Retrieving / parsing orgs data                                                                                                                                                                                                     
06:26:08 orgs data complete                                                                                                                                                                                                                 
06:26:08                                                                                                                                                                                                                                    
06:26:08 Retrieving / parsing channel-families data                                                                                                                                                                                         
06:26:09 channel-families data complete                                                                                                                                                                                                     
06:26:09                                                                                                                                                                                                                                    
06:26:09 Retrieving / parsing product names data                                                                                                                                                                                            
06:26:09 product names data complete                                                                                                                                                                                                        
06:26:09                                                                                                                                                                                                                                    
06:26:09 Retrieving / parsing arches data                                                                                                                                                                                                   
06:26:09 arches data complete                                                                                                                                                                                                               
06:26:09                                                                                                                                                                                                                                    
06:26:09 Retrieving / parsing additional arches data                                                                                                                                                                                        
06:26:09 additional arches data complete                                                                                                                                                                                                    
06:26:09                                                                                                                                                                                                                                    
06:26:09 Retrieving / parsing channel data                                                                                                                                                                                                  
06:26:10    p = previously imported/synced channel
06:26:10    . = channel not yet imported/synced
06:26:10    base-channels:
06:26:10       p rhel-x86_64-server-6                     19484       full import from Wed May  3 06:15:48 2017
06:26:10 
06:26:10 Channel data complete
06:26:10 
06:26:10 Downloading errata data
06:26:10    Retrieving / parsing errata data: rhel-x86_64-server-6 (NONE RELEVANT)
06:26:10 Downloading errata data complete
06:26:10 
06:26:10 Downloading kickstartable trees metadata
06:26:10    Retrieving / parsing kickstart data: rhel-x86_64-server-6 (9)
            ________________________________________
Downloading:######################################## - complete
06:26:11 
06:26:11 Downloading kickstartable trees files
06:26:11    Retrieving / parsing kickstart tree files: rhel-x86_64-server-6 (NONE RELEVANT)
06:26:11 
06:26:11 Importing channel errata
06:26:11    Importing *relevant* errata: rhel-x86_64-server-6 (NONE RELEVANT)
06:26:11 Importing kickstartable trees (9)
ERROR: Encountered IntegrityError: 
null value in column "checksum_id" violates not-null constraint
DETAIL:  Failing row contains (10, Server/repodata/productid.gz, null, 1719, 2011-05-10 18:51:37-04, 2017-05-03 06:26:10.303182-04, 2017-05-03 06:26:10.303182-04).

consider removing satellite-sync cache at /var/cache/rhn/satsync/* and re-run satellite-sync with same options.
If this error persits after removing cache, please contact Red Hat support.

Doing "rm -rf /var/cache/rhn/satsync/*" and re-running the ISS ended with the same error.

--- Additional comment from Jan Dobes on 2017-05-03 07:39:17 EDT ---

If I remember correctly, this is totally expected because the patch is not backported to Satellite 5.7.

This bug occurs when you are consuming kickstart data obtained using cdn-sync or spacewalk-repo-sync because import code can't handle anything other than MD5.

This bug is present on all Satellite versions, this bugzilla fixes it on Satellite 5.8.

Comment #5 is misleading because neither of:

Satellite 5.7 synced from Satellite 5.7 - PASS
Satellite 5.8 synced from Satellite 5.7 - PASS

will pass if channels with kickstarts on 5.7 master are populated using spacewalk-repo-sync and not satellite-sync.

--- Additional comment from Jan Dobes on 2017-05-09 18:59:01 EDT ---

Based on previous communication, switching this back to ON_QA.

--- Additional comment from Radovan Drazny on 2017-05-11 09:28:27 EDT ---

Ok, tested ISS between two Sat 5.8 servers (and both PostgreSQL and Oracle) using the reproducer from comment #3. Sync was successful in both cases. 
Problem with Satellite 5.7 described in comments #11 and #12 remains. According to devels, it will require an erratum to fix the older version of Satellite.

VERIFIED