Description of problem:


* A host is connected to a pulp-node. 
* A pulp server is syncing repositories to the pulp-node.
* The network between the pulp-node and the pulp server is shaky and not always reliable.

[root@pulp-01 pulp]# pulp-admin node sync run --node-id pulp-node-01



when the host tries to download a certain package from the synced repository on the pulp-node with yum, it fails with: 

https://pulp-node-01/pulp/repos/custom-repository/custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm: [Errno 14] PYCURL ERROR 22 - "NSS: client certificate not found (nickname not specified)"
Trying other mirror.
Could not download/verify pkg custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch: failure: custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm from custom-repository: [Errno 256] No more mirrors to try.



Comparing the rpm:s on the pulp-server (pulp-01) with the corresponding rpm on the pulp-node (pulp-node-01) it turns out that the sha1sum on the pulp-node mismatches.
It can be that there was a network glitch or other error during the sync operation.


[root@pulp-01 pulp]# sha1sum /var/lib/pulp/content/rpm/custom-package/7.2.0/8.Final_redhat_8.ep6.el6/noarch/198f23330b188b9997efdd7b4e1e955097ebaba6/custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm
198f23330b188b9997efdd7b4e1e955097ebaba6  /var/lib/pulp/content/rpm/custom-package/7.2.0/8.Final_redhat_8.ep6.el6/noarch/198f23330b188b9997efdd7b4e1e955097ebaba6/custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm


[root@pulp-node-01 ~]# sha1sum /var/lib/pulp/content/rpm/custom-package/7.2.0/8.Final_redhat_8.ep6.el6/noarch/198f23330b188b9997efdd7b4e1e955097ebaba6/custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm
ff620e1a6bc5636e3ef71dbd5d1808e3f7dd36d5  /var/lib/pulp/content/rpm/custom-package/7.2.0/8.Final_redhat_8.ep6.el6/noarch/198f23330b188b9997efdd7b4e1e955097ebaba6/custom-package-7.2.0-8.Final_redhat_8.ep6.el6.noarch.rpm



* Subsequent publishes and syncs of the pulp-node does not remove/replace the mismatching rpm. 
* it is not possible on the pulp-node to remove the rpm with pulp-admin: it says task completed, but the RPM is still there.
* removing the rpm manually fails: the "pulp-admin rpm repo publish" job on the repo containing that rpm fails.
* manually copying the rpm from pulp-server to the pulp-node, and then running a publish of the repo containing that rpm on the pulp-node "fixes" the issue. 


Version-Release number of selected component (if applicable):
2.2.0

How reproducible:
sometimes

Steps to Reproduce:
1. create a pulp-node
2. sync a big repository from a pulp server to the pulp-node until one rpm turns up with a sha1sum which is different on the pulp-node (probably adding some latency inbetween and/or restarting httpd on the pulp-node d can help reproduce this)
3. install that rpm from a server connected to that yum repository on the pulp-node

Actual results:

* rpm does not install. 
* pulp-admin on pulp-node cannot remove the rpm.
* sync jobs to the pulp-node does not remove the corrupt rpm.

Expected results:

all rpm:s which appears in the repo after being published should be installable.




Additional info: