Bug 1807151

Summary: capsule sync failed on Docker content DKR1008: Could not find registry API at https://satellite.example:5000 (Katello::Errors::PulpError)
Product: Red Hat Satellite Reporter: Ganesh Payelkar <gpayelka>
Component: InstallationAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: dkliban, ekohlvan, jsherril, zhunting
Target Milestone: 6.7.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: foreman-installer-1.24.1.20-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-14 13:38:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ganesh Payelkar 2020-02-25 17:42:25 UTC
Description of problem:

capsule sync failed on Docker content  DKR1008: Could not find registry API at https://satellite.example:5000 (Katello::Errors::PulpError) 


Version-Release number of selected component (if applicable):

satellite-capsule-6.7.0-5.beta.el7sat.noarch
pulp-docker-plugins-3.2.5-1.el7sat.noarch
python-pulp-docker-common-3.2.5-1.el7sat.noarch

How reproducible:
New installation of 6.7 Beta

Steps to Reproduce:
1. Create a product and then create a docker repo for https://registry.access.redhat.com with Docker upstream rhel 

OR 

Create a product and then create a docker repo for https://registry.redhat.io with Docker upstream rhel7.7

2. Sync the repo

3. On satellite it will sync properly, it errored when capsule sync start  


Content Counts
Content Type
Container Image Manifests 	21
Container Image Manifest Lists 	7
Container Image Tags 	        9 

Actual results: 


Feb 25 22:42:44 vm pulp: urllib3.connectionpool:INFO: [65d03eb8] Starting new HTTPS connection (1): satellite.example.com
Feb 25 22:42:44 vm pulp: nectar.downloaders.threaded:ERROR: [65d03eb8] Skipping requests to satellite.example.com:5000 due to repeated connection failures: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618)
Feb 25 22:42:44 vm pulp: pulp.server.async.tasks:INFO: [65d03eb8] Task failed : [65d03eb8-5234-4bb5-b434-4c5efda9690c] : Could not find registry API at https://satellite.example.com:5000
Feb 25 22:42:44 vm pulp: celery.app.trace:INFO: [65d03eb8] Task pulp.server.managers.repo.sync.sync[65d03eb8-5234-4bb5-b434-4c5efda9690c] raised expected: PulpCodedException()
Feb 25 22:42:44 vm pulp: celery.app.trace:INFO: [2eb39e9f] Task pulp.server.async.tasks._release_resource[2eb39e9f-7bb5-4ff2-b1cc-1bd93c07f800] succeeded in 0.00347492890432s: None



Expected results:


Additional info:

I have custom SSL configured on satellite and capsules.

Comment 8 Justin Sherrill 2020-03-03 16:26:55 UTC
It looks like this change broke this:  https://projects.theforeman.org/issues/28043

We are configuring crane's CA cert to be the server ca, while we're using an uber cert to authenticate against it generated from the default CA.  

Ewoud, thoughts on this?

Comment 9 Ewoud Kohl van Wijngaarden 2020-03-03 16:33:33 UTC
That sounds like a misconfiguration and should be configuring Katello to expect the right certificate. The question is what the right certificate is. As a user it makes a lot more sense to me be presented the server certificate because your docker/podman is more likely to have the server certificate present than the default CA. If I'm wrong, I'll gladly hear it.

Comment 10 Justin Sherrill 2020-03-03 18:18:14 UTC
The main apache server is configured with: 

  SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"

I'd expect crane to be configured similarly ?  Unless you mean that that is also wrong?

Comment 11 Ewoud Kohl van Wijngaarden 2020-03-04 10:17:04 UTC
(In reply to Justin Sherrill from comment #10)
> The main apache server is configured with: 
> 
>   SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
>   SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
> 
> I'd expect crane to be configured similarly ?  Unless you mean that that is
> also wrong?

This is what I would expect to see for crane as well.

Comment 12 Eric Helms 2020-03-06 13:18:13 UTC
Created redmine issue https://projects.theforeman.org/issues/29278 from this bug

Comment 13 Bryan Kearney 2020-03-06 15:05:56 UTC
Upstream bug assigned to ehelms

Comment 14 Bryan Kearney 2020-03-06 15:05:58 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29278 has been resolved.

Comment 16 Devendra Singh 2020-03-24 07:53:04 UTC
Verification step:
Satellite Version: 6.7 Snap16

1. Create a product and then create a docker repo for https://registry.access.redhat.com with Docker upstream rhel 
2. Sync the repo, Repository synced completed successfully.
3- Trigger capsule sync and it also completed successfully

# hammer capsule content synchronize --id 2
[..........................................................................................................................................................................................................] [100%]


Mar 24 03:23:22 capsule-upgrade pulp: urllib3.connectionpool:INFO: [e3a341c5] Starting new HTTPS connection (34): satellite.com
Mar 24 03:23:22 capsule-upgrade pulp: nectar.downloaders.threaded:INFO: [e3a341c5] Download succeeded: https://satellite.com:5000/v2/default_organization-dev-test_container_1112-docker_test-docker_repo_test/manifests/7.2-84.
Mar 24 03:23:22 capsule-upgrade pulp: nectar.downloaders.threaded:INFO: [e3a341c5] Download succeeded: https://satellite.com:5000/v2/default_organization-dev-test_container_1112-docker_test-docker_repo_test/manifests/7.7.
Mar 24 03:23:22 capsule-upgrade pulp: urllib3.connectionpool:INFO: [e3a341c5] Resetting dropped connection: satellite.com
Mar 24 03:23:22 capsule-upgrade pulp: nectar.downloaders.threaded:INFO: [e3a341c5] Download succeeded: https://satellite.com:5000/v2/default_organization-dev-test_container_1112-docker_test-docker_repo_test/manifests/**********************************.
Mar 24 03:23:22 capsule-upgrade pulp: nectar.downloaders.threaded:INFO: [e3a341c5] Download succeeded: https://satellite.com:5000/v2/default_organization-dev-test_container_1112-docker_test-docker_repo_test/manifests/**********************************.
Mar 24 03:23:22 capsule-upgrade pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[d5a6487c-67b5-4417-acff-f381bedcdb74]


Didn't get any error message in satellite's crane logs.

Comment 17 Bryan Kearney 2020-04-14 13:38:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454