1219961 – Unable to synchronize content

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1219961 - Unable to synchronize content

Summary: Unable to synchronize content

Keywords:
Status:	CLOSED DUPLICATE of bug 1222912
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Content Management
Sub Component:
Version:	6.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	Unspecified
Assignee:	Katello Bug Bin
QA Contact:	Katello QA List
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-05-08 20:19 UTC by Andrew Schofield
Modified:	2017-02-23 20:04 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-06-11 18:23:59 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Foreman debug from usl10149336 (2.65 MB, application/x-xz) 2015-05-12 21:00 UTC, Andrew Schofield	no flags	Details
View All

Description Andrew Schofield 2015-05-08 20:19:26 UTC

Description of problem:

(related to https://bugzilla.redhat.com/show_bug.cgi?id=1207167)

Unable to synchronize content to Capsule server used our own CA. Capsule is registered in Satellite.

Errors in /var/log/messsages on Capsule:

May  8 16:05:41 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May  8 16:05:41 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
May  8 16:06:51 usl10149338 ntpd[2036]: 0.0.0.0 0613 03 spike_detect -0.005171 s
May  8 16:07:28 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://usl10149336.am.hedani.net:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
May  8 16:07:28 usl10149338 goferd: [INFO][worker-0] root:473 - connecting to usl10149336.am.hedani.net:5647...
May  8 16:07:28 usl10149338 goferd: [INFO][worker-0] root:513 - Disconnected
May  8 16:07:28 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May  8 16:07:28 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
May  8 16:09:15 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://usl10149336.am.hedani.net:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
May  8 16:09:15 usl10149338 goferd: [INFO][worker-0] root:473 - connecting to usl10149336.am.hedani.net:5647...
May  8 16:09:15 usl10149338 goferd: [INFO][worker-0] root:513 - Disconnected
May  8 16:09:15 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May  8 16:09:15 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
May  8 16:11:02 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://usl10149336.am.hedani.net:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
May  8 16:11:02 usl10149338 goferd: [INFO][worker-0] root:473 - connecting to usl10149336.am.hedani.net:5647...
May  8 16:11:02 usl10149338 goferd: [INFO][worker-0] root:513 - Disconnected
May  8 16:11:02 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May  8 16:11:02 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
May  8 16:12:49 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://usl10149336.am.hedani.net:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
May  8 16:12:49 usl10149338 goferd: [INFO][worker-0] root:473 - connecting to usl10149336.am.hedani.net:5647...
May  8 16:12:49 usl10149338 goferd: [INFO][worker-0] root:513 - Disconnected
May  8 16:12:49 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May  8 16:12:49 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Version-Release number of selected component (if applicable):

Satellite - 6.1
RHEL - 6.6

How reproducible:


Steps to Reproduce:
1. Satellite server setup using out internal CA
2. Registered a new capsule to this (capsule shows as ok in Satellite. Feature refresh etc seems to work). 
3. Promoting content to an environment which the capsule should get hangs.

Actual results:

Errors as detailed above, sync hangs.

Expected results:

No errors in /var/log/messages and synchronization should be successfull (or timeout with a useful errror!).

Additional info:

Capsule certs generated as per:

capsule-certs-generate --capsule-fqdn usl10149338.am.hedani.net --certs-tar /root/usl10149338.am.hedani.net-certs.tar --server-cert /root/usl10149338.crt --server-cert-req /root/usl10149338.csr --server-ca-cert /root/server.ca --server-key /root/usl10149338.key --certs-update-all


Capsule installed as per:

capsule-installer --parent-fqdn "usl10149336.am.hedani.net" --register-in-foreman "true" --foreman-oauth-key "vhLAz3SsiVVqCnhT55y7h4HPFrhzXA3C" --foreman-oauth-secret "cxhfnEdZvwhVJGCB7GLH9pw2DFbZ6rBu" --pulp-oauth-secret "4xmv3aX5CRvXsZaBfvTyaxfx7jjm9sDn" --certs-tar "/root/usl10149338.am.hedani.net-certs.tar" --puppet "true" --puppetca "true" --pulp "true" --certs-update-all

Capsule installer completed ok with no errors, registered in Satellite ok.

Comment 1 RHEL Program Management 2015-05-08 20:41:47 UTC

Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Mike McCune 2015-05-12 02:13:33 UTC

The error:

gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected

indicates a failure to connect to QPID on the Satellite.

I'm assuming you checked that the capsule host can connect to port 5647 on usl10149336.am.hedani.net ?

can we get a katello-debug archive from the Satellite attached to this BZ?

Comment 4 Mike McCune 2015-05-12 02:14:17 UTC

oops, I meant 'foreman-debug', apologies.

Comment 5 Andrew Schofield 2015-05-12 20:56:44 UTC

Apologies for the delay:

(usl10149336 is master, usl10149338 is capsule)

I can connect to 5647 (not sure I should be getting data back but...)

[root@usl10149338 dev]# cat < /dev/tcp/usl10149336/5647
^C
[root@usl10149338 dev]#

And


[root@usl10149336 x86_64]# tcpdump 'host usl10149338 && port 5647'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:55:13.198147 IP usl10149338.am.hedani.net.43150 > usl10149336.am.hedani.net.5647: Flags [S], seq 3047965331, win 14600, options [mss 1460,sackOK,TS val 353252131 ecr 0,nop,wscale 7], length 0
16:55:13.198165 IP usl10149336.am.hedani.net.5647 > usl10149338.am.hedani.net.43150: Flags [S.], seq 1415294448, ack 3047965332, win 14480, options [mss 1460,sackOK,TS val 353638726 ecr 353252131,nop,wscale 7], length 0
16:55:13.198264 IP usl10149338.am.hedani.net.43150 > usl10149336.am.hedani.net.5647: Flags [.], ack 1, win 115, options [nop,nop,TS val 353252131 ecr 353638726], length 0
16:55:13.804777 IP usl10149338.am.hedani.net.43150 > usl10149336.am.hedani.net.5647: Flags [F.], seq 1, ack 1, win 115, options [nop,nop,TS val 353252738 ecr 353638726], length 0
16:55:13.805198 IP usl10149336.am.hedani.net.5647 > usl10149338.am.hedani.net.43150: Flags [F.], seq 1, ack 2, win 114, options [nop,nop,TS val 353639332 ecr 353252738], length 0
16:55:13.805332 IP usl10149338.am.hedani.net.43150 > usl10149336.am.hedani.net.5647: Flags [.], ack 2, win 115, options [nop,nop,TS val 353252738 ecr 353639332], length 0
^C
6 packets captured
10 packets received by filter
0 packets dropped by kernel

[root@usl10149336 x86_64]# lsof -i:5647
COMMAND    PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
qdrouterd 1558 qpidd    6u  IPv4   13266      0t0  TCP *:5647 (LISTEN)
qdrouterd 1558 qpidd    9u  IPv4   79343      0t0  TCP usl10149336.am.hedani.net:5647->usl10149338.am.hedani.net:40881 (ESTABLISHED)
qdrouterd 1558 qpidd   10u  IPv4   79473      0t0  TCP usl10149336.am.hedani.net:5647->usl10149338.am.hedani.net:40882 (ESTABLISHED)
qdrouterd 1558 qpidd   11u  IPv4   79555      0t0  TCP usl10149336.am.hedani.net:5647->usl10149338.am.hedani.net:40883 (ESTABLISHED)
qdrouterd 1558 qpidd   12u  IPv4   79676      0t0  TCP usl10149336.am.hedani.net:5647->usl10149338.am.hedani.net:40884 (ESTABLISHED)
qdrouterd 1558 qpidd   13u  IPv4   79797      0t0  TCP usl10149336.am.hedani.net:5647->usl10149338.am.hedani.net:40885 (ESTABLISHED)
qdrouterd 1558 qpidd   14u  IPv4   79956      0t0  TCP usl1014933

Comment 6 Andrew Schofield 2015-05-12 21:00:35 UTC

Created attachment 1024776 [details]
Foreman debug from usl10149336

Comment 7 Mike McCune 2015-05-27 22:27:03 UTC

From the capsule, can you try the following openssl command to check if the capsule can connect to the Satellite, just insert your Satellite's hostname on that second -connect param:


# openssl s_client -connect satellite.example.com:5647 -cert /etc/pki/consumer/bundle.pem -key /etc/pki/consumer/bundle.pem -state -debug

if it works correctly it should look like:

# openssl s_client -connect satellite.example.com:5647 -cert /etc/pki/consumer/bundle.pem -key /etc/pki/consumer/bundle.pem -state -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
....
---
Acceptable client certificate CA names
/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=satellite.example.com
---
SSL handshake has read 3776 bytes and written 1915 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: D5A36BD2450C1F15B1E53580908AD8A9E7ADCF36F55CB9399FF124B6498FCDEE
    Session-ID-ctx: 
    Master-Key: 48B8F446D2940B0967ED344D046BABBD5366EA35FC0F1ECA92610312AA7A937405B55B050399D720E37D6494BB5AAE6E
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1432764739
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

Comment 8 Andrew Schofield 2015-05-28 15:57:44 UTC

Thanks Mike:

# openssl s_client -connect usl10149336.am.hedani.net:5647 -cert /etc/pki/consumer/bundle.pem -key /etc/pki/consumer/bundle.pem -state -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x288e6e0 [0x288e760] (249 bytes => 249 (0xF9))
0000 - 16 03 01 00 f4 01 00 00-f0 03 03 55 67 35 6f 0f   ...........Ug5o.
0010 - c9 18 14 b5 5e c3 23 ae-84 fa 6b 12 7f 06 51 f6   ....^.#...k...Q.
0020 - ba 22 2f 15 32 99 65 4d-3f bb 65 00 00 84 c0 30   ."/.2.eM?.e....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b   .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a   .j.9.8.....2...*
0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0060 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67   .+.'.#.........g
0070 - 00 40 00 33 00 32 c0 12-c0 08 00 9a 00 99 00 45   .@.3.2.........E
0080 - 00 44 00 16 00 13 c0 31-c0 2d c0 29 c0 25 c0 0e   .D.....1.-.).%..
0090 - c0 04 c0 0d c0 03 00 9c-00 3c 00 2f 00 96 00 41   .........<./...A
00a0 - 00 0a 00 07 c0 11 c0 07-c0 0c c0 02 00 05 00 04   ................
00b0 - 00 ff 01 00 00 43 00 0b-00 04 03 00 01 02 00 0a   .....C..........
00c0 - 00 08 00 06 00 19 00 18-00 17 00 23 00 00 00 0d   ...........#....
00d0 - 00 22 00 20 06 01 06 02-06 03 05 01 05 02 05 03   .". ............
00e0 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02   ................
00f0 - 02 03 01 01 00 0f 00 01-01                        .........
SSL_connect:SSLv2/v3 write client hello A
read from 0x288e6e0 [0x2893cc0] (7 bytes => 0 (0x0))
139991291250504:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 249 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---


So the next question is how to fix up my SSL errors? Certs were generated via the following command:

Satellite:
----------

# capsule-certs-generate --capsule-fqdn usl10149338.am.hedani.net --certs-tar /root/usl10149338.am.hedani.net-certs.tar --server-cert /root/usl10149338.crt --server-cert-req /root/usl10149338.csr --server-ca-cert /root/server.ca --server-key /root/usl10149338.key --certs-update-all

# capsule-certs-generate --capsule-fqdn usl10149338.am.hedani.net --certs-tar /root/usl10149338.am.hedani.net-certs.tar --server-cert /root/usl10149338.crt --server-cert-req /root/usl10149338.csr --server-ca-cert /root/server.ca --server-key /root/usl10149338.key --certs-update-all
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-proxy-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-apache for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-broker for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-client-cert for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-puppet-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-router-server for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-proxy for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-router-client for update
Installing             Done                                               [100%] [.......................................]
  Success!

  To finish the installation, follow these steps:
<--snip-->
  The full log is at /var/log/katello-installer/capsule-certs-generate.log
[root@usl10149336 ~]#

Capsule:
--------

(copied tar file accross)

# capsule-installer --parent-fqdn "usl10149336.am.hedani.net" --register-in-foreman "true" --foreman-oauth-key <authkey> --foreman-oauth-secret <authsecret> --pulp-oauth-secret <authsecret> --certs-tar "/root/usl10149338.am.hedani.net-certs.tar" --puppet "true" --puppetca "true" --pulp "true" --certs-update-all
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-apache for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-router-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-client-cert for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-broker for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-proxy for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-puppet-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-qpid-router-server for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-client for update
Marking certificate /root/ssl-build/usl10149338.am.hedani.net/usl10149338.am.hedani.net-foreman-proxy-client for update
Installing             Done                                               [100%] [.......................................]
  Success!
  * Capsule is running at https://usl10149338.am.hedani.net:9090
  The full log is at /var/log/capsule-installer/capsule-installer.log

Comment 9 Andrew Schofield 2015-05-28 16:48:36 UTC

Ok, 1 step forward (I think). For some reason my bundle.pem file wasn't being renewed. I ran subscription-manager unregister and noticed that bundle.pem still existed even though cert.pem and key.pem had gone. A subscription-manager register renewed the cert.epm and key.epm files but didn't update bundle.pem. So I did a unregister and removed bundle.pem and then a register and viola bundle.pem was created using the contents of cert.pem and key.pem. After the register I reran the capsule-installer just to try and be safe.

However. All is still not well. 

# openssl s_client -connect usl10149336.am.hedani.net:5647 -cert /etc/pki/consumer/bundle.pem -key /etc/pki/consumer/bundle.pem -state -debug
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
CONNECTED(00000003)
<--snip-->
---
Acceptable client certificate CA names
/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=usl10149336.am.hedani.net
Server Temp Key: DH, 2048 bits
---
SSL handshake has read 5003 bytes and written 1996 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-GCM-SHA384
    Session-ID: 1E1852C09450FCC78175F811F1A041F7A139D081D91EAD8033574A98EE1592CC
    Session-ID-ctx:
    Master-Key: 8FBD046299ECD16270FA4491C2F73EB89150811342BC68B02E7A8435CF9FDA17D73EF9F63207B406C608689CD1530EB7
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
<--snip-->

    Start Time: 1432831204
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

And there are still gofer errors:

May 28 12:44:14 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://usl10149336.am.hedani.net:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
May 28 12:44:14 usl10149338 goferd: [INFO][worker-0] root:473 - connecting to usl10149336.am.hedani.net:5647...
May 28 12:44:14 usl10149338 goferd: [INFO][worker-0] root:513 - Disconnected
May 28 12:44:14 usl10149338 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://usl10149336.am.hedani.net:5647, failed: Connection amqps://usl10149336.am.hedani.net:5647 disconnected
May 28 12:44:14 usl10149338 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

And I still can't sync content from my master.

Comment 10 Andrew Schofield 2015-06-11 18:23:59 UTC


*** This bug has been marked as a duplicate of bug 1222912 ***

Note You need to log in before you can comment on or make changes to this bug.