Bug 847074 - client package install hang with continuous logging of Errno 111 Connection refused error with port 5671 connect
Summary: client package install hang with continuous logging of Errno 111 Connection r...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Katello
Classification: Retired
Component: Installer
Version: 1.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Lukas Zapletal
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: 852788
TreeView+ depends on / blocked
 
Reported: 2012-08-09 15:30 UTC by sosan.ng
Modified: 2012-08-30 13:46 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 852788 (view as bug list)
Environment:
Last Closed: 2012-08-30 13:46:54 UTC
Embargoed:


Attachments (Terms of Use)
katello WebUI Add Package hang screen capture (213.74 KB, image/png)
2012-08-09 15:33 UTC, sosan.ng
no flags Details
Katello Server katello-debug-certificates output (13.98 KB, application/octet-stream)
2012-08-17 17:29 UTC, sosan.ng
no flags Details
Katello Client /etc/rhsm/ca/candlepin-local.pem content (16.35 KB, application/octet-stream)
2012-08-17 17:30 UTC, sosan.ng
no flags Details
katello-debug tarball (872.07 KB, application/x-gzip)
2012-08-21 04:30 UTC, sosan.ng
no flags Details
katello-debug command run screen capture (13.85 KB, application/octet-stream)
2012-08-21 04:31 UTC, sosan.ng
no flags Details
/etc/httpd/conf.d/katello.conf (2.49 KB, text/plain)
2012-08-21 04:32 UTC, sosan.ng
no flags Details
/etc/httpd/conf.d/pulp.conf (2.46 KB, text/plain)
2012-08-21 04:33 UTC, sosan.ng
no flags Details
katello-configure command run screen capture (14.33 KB, application/octet-stream)
2012-08-21 15:18 UTC, sosan.ng
no flags Details
package install error (18.58 KB, application/octet-stream)
2012-08-21 15:19 UTC, sosan.ng
no flags Details
redhat.repo (14.12 KB, application/octet-stream)
2012-08-23 03:45 UTC, sosan.ng
no flags Details

Description sosan.ng 2012-08-09 15:30:05 UTC
Description of problem:

While trying to install package onto katello client system, the "Adding Package" task hangs with continuous logging of "[Errno 111] Connection refused" with port 5671 connect attempt on both the server and client.  

Following are the errors being logged on both the katello server and client:

=== katello server iptables port 5671 setting ===
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5671 

=== katello server pulp.log error ===
2012-08-09 10:09:57,360 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:11:57,456 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:11:57,457 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 34]: [Errno 111] Connection refused2012-08-09 10:11:57,457 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:13:57,546 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:13:57,547 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 35]: [Errno 111] Connection refused2012-08-09 10:13:57,547 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:15:57,647 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:15:57,647 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 36]: [Errno 111] Connection refused2012-08-09 10:15:57,648 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds

=== katello client /var/log/gofer/agent.log ===
2012-08-09 10:06:17,267 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:08:17,350 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:08:17,420 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 32]: [Errno 111] Connection refused2012-08-09 10:08:17,420 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:10:17,496 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:10:17,534 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 33]: [Errno 111] Connection refused2012-08-09 10:10:17,534 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:12:17,594 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:12:17,629 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 34]: [Errno 111] Connection refused2012-08-09 10:12:17,629 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:14:17,700 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:14:17,732 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 35]: [Errno 111] Connection refused2012-08-09 10:14:17,733 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds

Version-Release number of selected component (if applicable):
# rpm -qa|grep katello
katello-certs-tools-1.1.7-1.el6.noarch
katello-glue-candlepin-1.0.2-1.el6.noarch
katello-cli-1.0.1-1.el6.noarch
katello-glue-foreman-1.0.2-1.el6.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-glue-pulp-1.0.2-1.el6.noarch
katello-all-1.0.2-1.el6.noarch
katello-selinux-1.0.1-1.el6.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-cli-common-1.0.1-1.el6.noarch
katello-common-1.0.2-1.el6.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-repos-1.0.3-1.el6.noarch
katello-configure-1.0.1-1.el6.noarch
katello-1.0.2-1.el6.noarch

How reproducible:
Tried to install package onto client system two times and both tasks hung the same way

Steps to Reproduce:
1.  under WebUI "Content" page "Manage Packages" section, type in package name (eg. xterm) and click "Add"
2.  
3.
  
Actual results:
Under "Content" page "Name" section, "Adding Package" circular progress icon keeps running


Expected results:
Package should be installed onto the client system

Additional info:

Comment 1 sosan.ng 2012-08-09 15:33:50 UTC
Created attachment 603290 [details]
katello WebUI Add Package hang screen capture

Comment 3 Lukas Zapletal 2012-08-15 10:10:48 UTC
Hello,

can you please do the following?

1) Which version of katello-agent do you use on the *CLIENT*.

2) Can you check qpidd is running on the *SERVER*?

3) Restart qpidd and try again.

4) Also check and apply updates on the client and try again if the above does not help.

Thanks for report.

Comment 4 Lukas Zapletal 2012-08-15 10:13:40 UTC
Oh I noticed you are using nightly build. We have released Katello 1.0 last week. You may like to install that version (sorry upgrade from nightly to 1.0 is not supported - but you can try).

This version of agent works with 1.0:

katello-agent-1.0.6-1.el6.noarch.rpm

http://www.katello.org/katello-1-0-released/

Comment 5 sosan.ng 2012-08-16 18:41:55 UTC
Hello,

I tried out the directions in this updates and the install package onto the client system is still hanging.  

Following are the directions I tried:

1) Which version of katello-agent do you use on the *CLIENT*.

katello-agent-1.0.4-1.fc16.noarch.  

I ran "yum update katello-agent" and updated it to the katello-agent-1.0.6-1.fc16.noarch version.  I unregister and re-register the client system to the same katello server with the nightly version but it didn't correct the package install hang issue. 

I then installed another server using Fedora 16, ran yum update and installed the official V1.0 Katello software.  Unregistered the client system to the original server and registered it to the V1.0 Katello server.  However, the package install from the server to the client still hangs. The /var/log/gofer/agent.log file logs the same set of connection attempt to port 5671 but refused warnings continuously, there's an exception that occurred during the init phase prior to the continuous warnings logging.  Is this the cause of the package install hang?  Is there a way to correct or workaround this issue?

Following is the V1.0 katello software installed on the katello server with Fedora 16:

katello-cli-1.0.1-1.fc16.noarch
katello-glue-foreman-1.0.4-1.fc16.noarch
katello-common-1.0.4-1.fc16.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-repos-1.0.3-1.fc16.noarch
katello-1.0.4-1.fc16.noarch
katello-configure-1.0.1-1.fc16.noarch
katello-certs-tools-1.1.7-1.fc16.noarch
katello-glue-candlepin-1.0.4-1.fc16.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-selinux-1.0.1-1.fc16.noarch
katello-glue-pulp-1.0.4-1.fc16.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-all-1.0.4-1.fc16.noarch
katello-cli-common-1.0.1-1.fc16.noarch

=======  from client /var/log/gofer/agent.log ======

2012-08-16 10:27:04,027 [WARNING][e7a361ab-ec70-4aae-aadd-6695174bb3ab] close_engine() @ driver.py:444 - recoverable error[attempt 1]: [Errno 111] Connection refused
2012-08-16 10:27:04,027 [WARNING][e7a361ab-ec70-4aae-aadd-6695174bb3ab] close_engine() @ driver.py:446 - sleeping 1 seconds
2012-08-16 10:27:04,067 [INFO][PathMonitor1] __init__() @ connection.py:486 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = True
2012-08-16 10:27:04,067 [INFO][PathMonitor1] __init__() @ connection.py:497 - Connection Built: host: mccvm182.fcux.usa.hp.com, port: 443, handler: /katello/api
2012-08-16 10:27:04,067 [INFO][PathMonitor1] report_enabled() @ katelloplugin.py:382 - reporting: {'enabled_repos': {'repos': []}}
2012-08-16 10:27:04,140 [ERROR][PathMonitor1] __notify() @ pmon.py:150 - /etc/yum.repos.d/redhat.repo
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gofer/pmon.py", line 148, in __notify
    cb(path)
  File "/usr/lib/gofer/plugins/katelloplugin.py", line 143, in changed
    uep.report_enabled(uuid, report.content)
  File "/usr/lib/gofer/plugins/katelloplugin.py", line 384, in report_enabled
    return self.conn.request_put(method, report)
  File "/usr/lib/python2.7/site-packages/rhsm/connection.py", line 394, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib/python2.7/site-packages/rhsm/connection.py", line 337, in _request
    response = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1027, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.7/socket.py", line 430, in readline
    data = recv(1)
  File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 228, in read
    return self._read_bio(size)
  File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 213, in _read_bio
    return m2.ssl_read(self.ssl, size, self._timeout)
SSLError: tlsv1 alert unknown ca
2012-08-16 10:27:05,027 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671
2012-08-16 10:27:06,515 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 2]: [Errno 111] Connection refused
2012-08-16 10:27:06,515 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 2 seconds
2012-08-16 10:27:08,518 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671
2012-08-16 10:27:08,550 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 3]: [Errno 111] Connection refused
2012-08-16 10:27:08,551 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 4 seconds
...
...
2012-08-16 13:35:23,667 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671
2012-08-16 13:35:23,700 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 101]: [Errno 111] Connection refused
2012-08-16 13:35:23,701 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds
2012-08-16 13:37:23,793 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671
2012-08-16 13:37:23,860 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 102]: [Errno 111] Connection refused
2012-08-16 13:37:23,861 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds
............

On the server, it logs a similar set of warnings and error with connection attempts to port 5671 but refused:

===== from server /var/log/pulp/pulp.log =====

2012-08-16 13:53:51,846 7872:140510450673408: qpid.messaging:WARNING: driver:446 sleeping 120 seconds
2012-08-16 13:55:51,919 7872:140510450673408: qpid.messaging:WARNING: driver:523 trying: localhost:5671
2012-08-16 13:55:51,921 7872:140510450673408: qpid.messaging:WARNING: driver:444 recoverable error[attempt 120]: [Errno 111] Connection refused
2012-08-16 13:55:51,921 7872:140510450673408: qpid.messaging:WARNING: driver:446 sleeping 120 seconds
2012-08-16 13:57:52,011 7872:140510450673408: qpid.messaging:WARNING: driver:523 trying: localhost:5671
2012-08-16 13:57:52,012 7872:140510450673408: qpid.messaging:WARNING: driver:444 recoverable error[attempt 121]: [Errno 111] Connection refused


2) Can you check qpidd is running on the *SERVER*?
# service qpidd status
qpidd.service - LSB: start or stop qpidd
	  Loaded: loaded (/etc/rc.d/init.d/qpidd)
	  Active: active (running) since Thu, 16 Aug 2012 01:33:31 -0400; 12h ago
	  CGroup: name=systemd:/system/qpidd.service
		  â”” 20073 /usr/sbin/qpidd --data-dir /var/lib/qpidd --daemon

3) Restart qpidd and try again.
I restarted qpidd using "sevice qpidd restart" but the install package tasks still hangs.  These hung tasks don't timeout and persists across the client reboot.  Is there a way to cancel these tasks up and reinitiate them?

4) Also check and apply updates on the client and try again if the above does not help.
I ran "yum update" on the client system and tried the package install from the katello server again with the same result - hung install package task.

Thank you very much for your help with this issue!

Comment 6 Lukas Zapletal 2012-08-17 14:27:18 UTC
I just tested agent with nightly and it is working. Can you please show me /etc/gofer/plugins/katelloplugin.conf?

Then use katello-debug-certificates tool (in the PATH) both on the server and the client. Check this:

1) CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of /usr/share/katello/candlepin-cert.crt (SERVER).

2) NSS DB - CA matches SERIAL of the candlepin-cert.crt (both SERVER)

3) CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of the NSS DB - Broker (SERVER).

4) If you don't mind I would like to see outputs of both (you can attach private, no worries there are no sensitive data in it just serials - feel free to scramble it).

Also I'm interested in what subscription-manager identity shows. Thats for client side. Try to unregister, reinstall candlepin-consumer-xyz package and register again and make sure consumer certs (/etc/pki/consumer/ and /etc/rhsm/ca/candlepin-local.crt) changed. Also send output of the:

rpm -ql candlepin-cert-consumer-your_server_fqdn

It should list the certificate that was deployed. Check also fqdn - it MUST match with the server you are working with. Also, make sure you have full FQDN and also DNS is set properly.

Now since you have issues with qpidd also from the pulp side, it looks its something wrong with it. Can you check updates on both server and client? And when I say updates, I mean whole system :-) We are interested in goferd, pulp, qpidd server and client packages. If you don't mind, just update both and restart to see if it helps.

Comment 7 sosan.ng 2012-08-17 17:27:21 UTC
(In reply to comment #6)
> I just tested agent with nightly and it is working. Can you please show me
> /etc/gofer/plugins/katelloplugin.conf?

# cat /etc/gofer/plugins/katelloplugin.conf
@import:/etc/rhsm/rhsm.conf:server:hostname(host)

[main]
enabled=1
requires=package

[messaging]
uuid=
url=ssl://$(host):5671
cacert=/etc/rhsm/ca/candlepin-local.pem
clientcert=/etc/pki/consumer/bundle.pem

[reboot]
allow=1
delay=+1
-----------
In the /etc/rhsm/rhsm.conf, the hostname is FQDN, with the additional proxy_hostname and poxy_port setting.

> Then use katello-debug-certificates
> tool (in the PATH) both on the server and the client. Check this:
The katello-debug-certificates tool is only available on the server and not the client. I do see a serial parameter in the client /etc/rhsm/ca/candlepin-local.pem  file and it does match the server CA_SERIAL.  Please see the attached files katello-server-cert.docx and katello-client-cert.docx.

1)
> CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL
> entry of /usr/share/katello/candlepin-cert.crt (SERVER).

CA_SERIAL from the client /etc/rhsm/ca/candlepin-local.pem:
# fgrep serial /etc/rhsm/ca/candlepin-local.pem
                serial:BA:30:7B:69:B3:1F:D7:61

CA SERIAL from the server /usr/share/katello/candlepin-cert.crt:
/usr/share/katello/candlepin-cert.crt
  CA_SERIAL:BA:30:7B:69:B3:1F:D7:61

2) NSS DB - CA
> matches SERIAL of the candlepin-cert.crt (both SERVER)

The NSS DB - CA output that got displayed from the katello-debug-certificates
tool reported a "Could not find cert: broker: File not found" message.  The CA_SERIAL does match candlepin-cert.crt one:

/etc/rhsm/ca/candlepin-local.pem
  N/A
NSS DB - Broker Key

NSS DB - CA
  DN: "CN=mccvm182.fcux.usa.hp.com,OU=Cloud BU,O=Red Hat,L=Raleigh
  SERIAL: 00:ba:30:7b:69:b3:1f:d7:61
  CA: Is a CA with no maximum path length.
  CA_SERIAL: 00:ba:30:7b:69:b3:1f:d7:61
certutil: Could not find cert: broker
: File not found
NSS DB - Broker
  N/A

/usr/share/katello/candlepin-cert.crt
  CA_SERIAL:BA:30:7B:69:B3:1F:D7:61

3) CA_SERIAL of the
> /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of the NSS DB
> - Broker (SERVER).

There isn't a CA_SERIAL being shown under the NSS DB - Broker output, it shows "N/A" with a "Could not find cert: broker: File not found" message right beforehand:

certutil: Could not find cert: broker
: File not found
NSS DB - Broker
  N/A

The serial in the client /etc/rhsm/ca/candlepin-local.pem:
# fgrep serial /etc/rhsm/ca/candlepin-local.pem
                serial:BA:30:7B:69:B3:1F:D7:61

4) If you don't mind I would like to see outputs of both
> (you can attach private, no worries there are no sensitive data in it just
> serials - feel free to scramble it).

Please see the attached files katello-server-cert.docx and katello-client-cert.docx.

Also I'm interested in what
> subscription-manager identity shows. Thats for client side. Try to
> unregister, reinstall candlepin-consumer-xyz package and register again and
> make sure consumer certs (/etc/pki/consumer/ and
> /etc/rhsm/ca/candlepin-local.crt) changed. Also send output of the:

Before:
# subscription-manager identity 
Current identity is: 8627f4f6-d953-4a67-aca8-17fe240ee270
name: mccvm179.fcux.usa.hp.com
org name: HP_LM_Org
org id: 8f720136392dec0c01392fb4f3fd0003
# rpm -qa|grep candlepin
candlepin-cert-consumer-mccvm182.fcux.usa.hp.com-1.0-1.noarch
[root@mccvm179 ca]# rpm -ql candlepin-cert-consumer-mccvm182.fcux.usa.hp.com-1.0-1.noarch
/etc/rhsm/ca/candlepin-local.pem

unregister and reinstall candlepin-consumer-xyz package but can't register again due to the unregister errors:

After the client system is unregistered via the katello WebUI (System > Remove System), a couple sets of the following 410 Gone errors are logged and the client system fail to re-register with the "Validation failed: Name has already been taken".  What is the correct procedure to clean up the client system from katello so the client system can re-register?  The System tab page from the katello WebUI keeps displaying the folowing error as well:

[ERROR: 2012-08-17 13:06:17 #1145] Rendering 500:Resources::Candlepin::CandlepinResource: 410 Gone {"displayMessage":"Consumer 8627f4f6-d953-4a67-aca8-17fe240ee270 has been deleted","deletedId":"8627f4f6-d953-4a67-aca8-17fe240ee270"} (GET /candlepin/consumers/8627f4f6-d953-4a67-aca8-17fe240ee270/events)

# yum -y --nogpgcheck reinstall http://$KATELLO_HOSTNAME/pub/candlepin-cert-consumer-$KATELLO_HOSTNAME-1.0-1.noarch.rpm

# subscription-manager clean
All local data removed
[root@mccvm179 ca]# subscription-manager register --force --username=admin --password=admin
Validation failed: Name has already been taken

rpm -ql
> candlepin-cert-consumer-your_server_fqdn

It should list the certificate
> that was deployed. Check also fqdn - it MUST match with the server you are
> working with. Also, make sure you have full FQDN and also DNS is set
> properly.

Now since you have issues with qpidd also from the pulp side, it
> looks its something wrong with it. Can you check updates on both server and
> client? And when I say updates, I mean whole system :-) We are interested in
> goferd, pulp, qpidd server and client packages. If you don't mind, just
> update both and restart to see if it helps.

I ran "yum check-update" on both the server and client, the only update needed was gdb and I applied it anyway so both are up to date:
=== server ===
[root@mccvm182 log]# yum check-update
Loaded plugins: langpacks, presto, refresh-packagekit
[root@mccvm182 log]# 

=== client ===
[root@mccvm179 ca]# yum check-update
Loaded plugins: langpacks, presto, product-id, refresh-packagekit, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[root@mccvm179 ca]# 

Thank you very much for your help!

Comment 8 sosan.ng 2012-08-17 17:29:16 UTC
Created attachment 605231 [details]
Katello Server katello-debug-certificates output

Comment 9 sosan.ng 2012-08-17 17:30:12 UTC
Created attachment 605232 [details]
Katello Client /etc/rhsm/ca/candlepin-local.pem content

Comment 10 Lukas Zapletal 2012-08-20 15:40:18 UTC
Okay, the installation is obviously not correct - katello-debug-certificates script is unable to print NSS DB certificates. Did you see any errors during the installation? Can you attach me katello-debug output? Note it is a different script that creates a tarball of all log files (it strips out all passwords from there). I really need to see all main.log logfiles that our intaller created. I think this could be installer-related issue.

Also - did you change the hostname after installation by chance?

To correctly unregister a system run subscriptioin-manager unregister. You can also delete it from katello. Please note "clean" command just deletes local data and does not send it to the katello server. Use this for debugging purposes. You can also use --name to change the name of the system you are registering to avoid that naming issue.

I am also interested in:

$ ls /etc/pki/katello/nssdb -la
$ certutil -d /etc/pki/katello/nssdb -L
$ certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10
$ certutil -d /etc/pki/katello/nssdb -L -n broker | head -n10

Comment 11 sosan.ng 2012-08-21 04:29:03 UTC
(In reply to comment #10)
> Okay, the installation is obviously not correct - katello-debug-certificates
> script is unable to print NSS DB certificates. Did you see any errors during
> the installation? Can you attach me katello-debug output? Note it is a
> different script that creates a tarball of all log files (it strips out all
> passwords from there). I really need to see all main.log logfiles that our
> intaller created. I think this could be installer-related issue.

I didn't see any errors during the installation.  Following is the screen capture of the katello-configure run.  I am attaching the katello-debug run tarball, since it can't find the katello.conf and pulp.conf file under the /etc/httpd.d/ directory, these files are being attached here too (from /etc/httpd/conf.d/).  It also can't find thumbslug etc and log files which didn't get installed as part of the katello install, not sure if it's needed.

-----------------------------
[root@mccvm182 ~]# katello-configure
Starting Katello configuration
The top-level log file is [/var/log/katello/katello-configure-20120816-013225/main.log]
Creating Katello database user
############################################################ ... OK
Creating Katello database
############################################################ ... OK
Creating Candlepin database user
############################################################ ... OK
Populating Katello database schema
############################################################ ... OK
Initializing Katello data
############################################################ ... OK
-----------------------------

Also - did
> you change the hostname after installation by chance?

I didn't change the hostname after installation.  However, due to this issue I've been experimenting with the /etc/pulp/pulp.conf file and modified the following lines in the /etc/pulp/pulp.conf file, it didn't help correct this issue though:-

37c37
< url:        ssl://localhost:5671
---
> url:        ssl://mccvm182.fcux.usa.hp.com:5671

Since we need to use a proxy, the following settings were added before running the katello-configure command.  I've been wondering if the proxy setting needs to be set in other katello related configuration files as well or is pulp.conf the only place this setting is needed.

93a104,105
> proxy_url:http://16.85.88.10
> 
94a107,109
> proxy_port:8080


To correctly
> unregister a system run subscriptioin-manager unregister. You can also
> delete it from katello. Please note "clean" command just deletes local data
> and does not send it to the katello server. Use this for debugging purposes.
> You can also use --name to change the name of the system you are registering
> to avoid that naming issue.

I unregistered the system from the katello GUI only.  I'll try out using the subscription-manager unregister command.


I am also interested in:

$ ls
> /etc/pki/katello/nssdb -la
$ certutil -d /etc/pki/katello/nssdb -L
$
> certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10
$ certutil -d
> /etc/pki/katello/nssdb -L -n broker | head -n10

[root@mccvm182 ~]# ls /etc/pki/katello/nssdb -la
total 104
drwxr-xr-x. 2 root katello  4096 Aug 16 01:32 .
drwxr-x---. 3 root katello  4096 Aug 16 01:33 ..
-rw-r-----. 1 root katello 65536 Aug 16 01:33 cert8.db
-rw-r-----. 1 root katello 16384 Aug 16 01:33 key3.db
-rw-r-----. 1 root katello 16384 Aug 16 01:32 secmod.db

[root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

ca                                                           CT,C,c
mccvm182.fcux.usa.hp.com - Red Hat                           u,u,u

[root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            00:ba:30:7b:69:b3:1f:d7:61
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "CN=mccvm182.fcux.usa.hp.com,OU=Cloud BU,O=Red Hat,L=Raleigh,
            ST=North Carolina,C=US"
        Validity:
            Not Before: Thu Aug 16 05:32:37 2012

[root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L -n broker | head -n10
certutil: Could not find cert: broker
: File not found
[root@mccvm182 ~]# 

Thank you very much for your help!

Comment 12 sosan.ng 2012-08-21 04:30:37 UTC
Created attachment 605828 [details]
katello-debug tarball

Comment 13 sosan.ng 2012-08-21 04:31:58 UTC
Created attachment 605829 [details]
katello-debug command run screen capture

Comment 14 sosan.ng 2012-08-21 04:32:35 UTC
Created attachment 605830 [details]
/etc/httpd/conf.d/katello.conf

Comment 15 sosan.ng 2012-08-21 04:33:04 UTC
Created attachment 605831 [details]
/etc/httpd/conf.d/pulp.conf

Comment 16 Lukas Zapletal 2012-08-21 08:24:04 UTC
Ok this is a bug we already fixed in master:

https://github.com/Katello/katello/pull/447

To fix this, do the following steps:

# rm -f /etc/pki/katello/nssdb/*db
# katello-configure --answer-file=/etc/katello/katello-configure.conf -b

Attach the output of the last command here, it should regenerate NSS database. Then you should be able to connect. If not, restart Katello services:

# katello-service restart

And then show the output of:

# certutil -d /etc/pki/katello/nssdb -L

You should see "broker" there.

Once you confirm me the fix, I will most likely do update for Katello 1.0 and update Known Problems page.

Comment 17 sosan.ng 2012-08-21 15:17:50 UTC
(In reply to comment #16)
> Ok this is a bug we already fixed in master:
> https://github.com/Katello/katello/pull/447

To fix this, do the following
> steps:

# rm -f /etc/pki/katello/nssdb/*db
# katello-configure
> --answer-file=/etc/katello/katello-configure.conf -b

Attach the output of
> the last command here, it should regenerate NSS database. Then you should be
> able to connect. If not, restart Katello services:

# katello-service
> restart

After removal of the *db files and regenerated them using the katello-configure command you provided, the install package tasks no longer hang.  However, they are failing the package install with the "No package available to install" error.  I ran katello-service restart and rebooted the client system but the same failure persists.  The client system is subscirbed to the repo that contains the package.  Please see the attachment "package install error" for the screen captures of the error as well as the katello CLI output that shows the client system is subscribed to the repo.  I also don't see the subscribed repo from the "yum repolist" command run and can't yum install the package from the subscribed repo (please see the screen capture in the attachement "package install error" as well).  Do you know what I need to do to fix this?

You should see "broker" there.

# certutil -d /etc/pki/katello/nssdb
> -L

[root@mccvm182 katello]# certutil -d /etc/pki/katello/nssdb -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

ca                                                           CT,C,c
broker                                                       u,u,u


Once you confirm me the fix, I will most
> likely do update for Katello 1.0 and update Known Problems page.

Do I need to reinstall Katello 1.0 or can I run yum update install on katello-all to pick up this fix?

Thank you very much for your help!

Comment 18 sosan.ng 2012-08-21 15:18:53 UTC
Created attachment 605966 [details]
katello-configure command run screen capture

Comment 19 sosan.ng 2012-08-21 15:19:45 UTC
Created attachment 605967 [details]
package install error

Comment 20 Lukas Zapletal 2012-08-22 11:59:59 UTC
Ok now it looks good.

Did you promote your repo?

What environment is the consumer (client) registered to?

You need to have the content in the same environment as the client, otherwise you wont see the content.

Create environment, let's say "test", promote packages there (or whole product), register the client against environment "test" and then you will be able to consume it.

Comment 21 sosan.ng 2012-08-23 03:45:14 UTC
I didn't promote the repos.  Thank you so much for your helpful tip!

Initially the client was registered to the "common" environment created off of the top level "Library" environment.  After creating the "test_env" off of the "common" environment, I created a changeset "CS_c179" with the "test_repo" and "local_repo" products, then promoted it to the "test_env".  I used the Katello GUI to subscribe to the "test_repo" and "local_repo" products under the "test_env" environment, then re-registered the client system to the new environment "test_env" using the subscription-manager command, ran "subscription-manager refresh" and then "yum repolist".  The /etc/yum.repos.d/redhat.repo file is now updated with the repo configuration information (please see attachment redhat.repo).  However, the yum repolist command run returned the "HTTP Error 403 - Forbidden :" errors on trying to access these repos repomd.xml files.  I checked the CloudForms User's Guide on promote changset section, and the Infrastructure and Application Deployment Fundamentals on Connecting Instances to System Engine section, I don't see additional steps needed.  Do you know how I can correct this repo access error?  

[root@mccvm179 yum.repos.d]# subscription-manager list --consumed 
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+

Subscription Name:    	test_repo
Provides:             	test_repo
SKU:                  	1345125668164
Contract:             	None
Account:              	None
Serial Number:        	8125559394500063678
Active:               	True
Quantity Used:        	1
Service Level:        	
Service Type:         	
Starts:               	08/15/2012
Ends:                 	08/08/2042

Subscription Name:    	local_repo
Provides:             	local_repo
SKU:                  	1345128551424
Contract:             	None
Account:              	None
Serial Number:        	3467786245544411526
Active:               	True
Quantity Used:        	1
Service Level:        	
Service Type:         	
Starts:               	08/15/2012
Ends:                 	08/08/2042


[root@mccvm179 yum.repos.d]# yum repolist
Loaded plugins: langpacks, presto, product-id, refresh-packagekit, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml
Trying other mirror.
https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml
Trying other mirror.
repo id                            repo name                                                       status
HP_LM_Org_local_repo_c_repo        c_repo                                                               0
HP_LM_Org_local_repo_w_repos       w_repos                                                              0
HP_LM_Org_local_repo_zebra_repo    zebra_repo                                                           0
HP_LM_Org_test_repo_brew_test_repo brew_test_repo                                                       0
HP_LM_Org_test_repo_zoo_repo       zoo_repo                                                             0
fedora                             Fedora 16 - x86_64                                              25,098
fedora-katello                     integrates together a series of open source systems management       1
fedora-subscription-manager        Tools and libraries for Red Hat subscription management.            13
pulp-v1-stable                     Pulp v1                                                             20
updates                            Fedora 16 - x86_64 - Updates                                    10,007
repolist: 35,139

Thank you very much for your patience and help!

Comment 22 sosan.ng 2012-08-23 03:45:45 UTC
Created attachment 606438 [details]
redhat.repo

Comment 23 Lukas Zapletal 2012-08-27 08:25:57 UTC
The 403 is thrown when your consumer certificate AND/OR entitlement certificate are not correct.

Katello project is used mainly for Red Hat repositories from Content Delivery Network, therefore for each repository there is a product and entitlements imported from a manifest file.

If you work with custom yum repositories, it works the same way. For each (custom) product you create, unlimited subscription is automatically created by Katello. You need to register and subscribe to a content you want to consume at the moment. Both organization and environment must match. If one of these things is not correct, yum is not able to access the content (with 403 error).

Therefore you need to check two things:

1) Client was registered to the correct organization and environment during subscription-manager register command (see the cli options).

2) Subscription was consumed for each yum custom product you want consume repositories from. Retrieve pool id and use it in the subscription-manager subscribe command to properly subscribe to the content.

You can do both from subscription-manager or also via Katello UI or CLI. If you do the latter, please note it takes a while (up to 4 hours) until new information is propagated to consumers. You can speed this up using subscription-manager refresh command.

For example in the Katello UI case, you need to navigate to your systems list, open a system, list the available subscriptions and apply for it. Then wait or use the "refresh" command on the client to receive subscriptions, then you can consume content.

Comment 24 Lukas Zapletal 2012-08-30 13:46:54 UTC
Hello, since I don't read any other issues, I am marking the bug as done. Please feel free to reopen or file a new one.


Note You need to log in before you can comment on or make changes to this bug.