Description of problem: While trying to install package onto katello client system, the "Adding Package" task hangs with continuous logging of "[Errno 111] Connection refused" with port 5671 connect attempt on both the server and client. Following are the errors being logged on both the katello server and client: === katello server iptables port 5671 setting === ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5671 === katello server pulp.log error === 2012-08-09 10:09:57,360 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:11:57,456 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:11:57,457 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 34]: [Errno 111] Connection refused2012-08-09 10:11:57,457 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:13:57,546 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:13:57,547 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 35]: [Errno 111] Connection refused2012-08-09 10:13:57,547 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds2012-08-09 10:15:57,647 1831:140366013228800: qpid.messaging:WARNING: driver:523 trying: localhost:56712012-08-09 10:15:57,647 1831:140366013228800: qpid.messaging:WARNING: driver:444 recoverable error[attempt 36]: [Errno 111] Connection refused2012-08-09 10:15:57,648 1831:140366013228800: qpid.messaging:WARNING: driver:446 sleeping 120 seconds === katello client /var/log/gofer/agent.log === 2012-08-09 10:06:17,267 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:08:17,350 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:08:17,420 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 32]: [Errno 111] Connection refused2012-08-09 10:08:17,420 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:10:17,496 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:10:17,534 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 33]: [Errno 111] Connection refused2012-08-09 10:10:17,534 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:12:17,594 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:12:17,629 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 34]: [Errno 111] Connection refused2012-08-09 10:12:17,629 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds2012-08-09 10:14:17,700 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm172.fcux.usa.hp.com:56712012-08-09 10:14:17,732 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 35]: [Errno 111] Connection refused2012-08-09 10:14:17,733 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds Version-Release number of selected component (if applicable): # rpm -qa|grep katello katello-certs-tools-1.1.7-1.el6.noarch katello-glue-candlepin-1.0.2-1.el6.noarch katello-cli-1.0.1-1.el6.noarch katello-glue-foreman-1.0.2-1.el6.noarch katello-qpid-client-key-pair-1.0-1.noarch katello-glue-pulp-1.0.2-1.el6.noarch katello-all-1.0.2-1.el6.noarch katello-selinux-1.0.1-1.el6.noarch katello-candlepin-cert-key-pair-1.0-1.noarch katello-cli-common-1.0.1-1.el6.noarch katello-common-1.0.2-1.el6.noarch katello-qpid-broker-key-pair-1.0-1.noarch katello-repos-1.0.3-1.el6.noarch katello-configure-1.0.1-1.el6.noarch katello-1.0.2-1.el6.noarch How reproducible: Tried to install package onto client system two times and both tasks hung the same way Steps to Reproduce: 1. under WebUI "Content" page "Manage Packages" section, type in package name (eg. xterm) and click "Add" 2. 3. Actual results: Under "Content" page "Name" section, "Adding Package" circular progress icon keeps running Expected results: Package should be installed onto the client system Additional info:
Created attachment 603290 [details] katello WebUI Add Package hang screen capture
Hello, can you please do the following? 1) Which version of katello-agent do you use on the *CLIENT*. 2) Can you check qpidd is running on the *SERVER*? 3) Restart qpidd and try again. 4) Also check and apply updates on the client and try again if the above does not help. Thanks for report.
Oh I noticed you are using nightly build. We have released Katello 1.0 last week. You may like to install that version (sorry upgrade from nightly to 1.0 is not supported - but you can try). This version of agent works with 1.0: katello-agent-1.0.6-1.el6.noarch.rpm http://www.katello.org/katello-1-0-released/
Hello, I tried out the directions in this updates and the install package onto the client system is still hanging. Following are the directions I tried: 1) Which version of katello-agent do you use on the *CLIENT*. katello-agent-1.0.4-1.fc16.noarch. I ran "yum update katello-agent" and updated it to the katello-agent-1.0.6-1.fc16.noarch version. I unregister and re-register the client system to the same katello server with the nightly version but it didn't correct the package install hang issue. I then installed another server using Fedora 16, ran yum update and installed the official V1.0 Katello software. Unregistered the client system to the original server and registered it to the V1.0 Katello server. However, the package install from the server to the client still hangs. The /var/log/gofer/agent.log file logs the same set of connection attempt to port 5671 but refused warnings continuously, there's an exception that occurred during the init phase prior to the continuous warnings logging. Is this the cause of the package install hang? Is there a way to correct or workaround this issue? Following is the V1.0 katello software installed on the katello server with Fedora 16: katello-cli-1.0.1-1.fc16.noarch katello-glue-foreman-1.0.4-1.fc16.noarch katello-common-1.0.4-1.fc16.noarch katello-qpid-broker-key-pair-1.0-1.noarch katello-repos-1.0.3-1.fc16.noarch katello-1.0.4-1.fc16.noarch katello-configure-1.0.1-1.fc16.noarch katello-certs-tools-1.1.7-1.fc16.noarch katello-glue-candlepin-1.0.4-1.fc16.noarch katello-candlepin-cert-key-pair-1.0-1.noarch katello-selinux-1.0.1-1.fc16.noarch katello-glue-pulp-1.0.4-1.fc16.noarch katello-qpid-client-key-pair-1.0-1.noarch katello-all-1.0.4-1.fc16.noarch katello-cli-common-1.0.1-1.fc16.noarch ======= from client /var/log/gofer/agent.log ====== 2012-08-16 10:27:04,027 [WARNING][e7a361ab-ec70-4aae-aadd-6695174bb3ab] close_engine() @ driver.py:444 - recoverable error[attempt 1]: [Errno 111] Connection refused 2012-08-16 10:27:04,027 [WARNING][e7a361ab-ec70-4aae-aadd-6695174bb3ab] close_engine() @ driver.py:446 - sleeping 1 seconds 2012-08-16 10:27:04,067 [INFO][PathMonitor1] __init__() @ connection.py:486 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = True 2012-08-16 10:27:04,067 [INFO][PathMonitor1] __init__() @ connection.py:497 - Connection Built: host: mccvm182.fcux.usa.hp.com, port: 443, handler: /katello/api 2012-08-16 10:27:04,067 [INFO][PathMonitor1] report_enabled() @ katelloplugin.py:382 - reporting: {'enabled_repos': {'repos': []}} 2012-08-16 10:27:04,140 [ERROR][PathMonitor1] __notify() @ pmon.py:150 - /etc/yum.repos.d/redhat.repo Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gofer/pmon.py", line 148, in __notify cb(path) File "/usr/lib/gofer/plugins/katelloplugin.py", line 143, in changed uep.report_enabled(uuid, report.content) File "/usr/lib/gofer/plugins/katelloplugin.py", line 384, in report_enabled return self.conn.request_put(method, report) File "/usr/lib/python2.7/site-packages/rhsm/connection.py", line 394, in request_put return self._request("PUT", method, params) File "/usr/lib/python2.7/site-packages/rhsm/connection.py", line 337, in _request response = conn.getresponse() File "/usr/lib64/python2.7/httplib.py", line 1027, in getresponse response.begin() File "/usr/lib64/python2.7/httplib.py", line 407, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status line = self.fp.readline() File "/usr/lib64/python2.7/socket.py", line 430, in readline data = recv(1) File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 228, in read return self._read_bio(size) File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 213, in _read_bio return m2.ssl_read(self.ssl, size, self._timeout) SSLError: tlsv1 alert unknown ca 2012-08-16 10:27:05,027 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671 2012-08-16 10:27:06,515 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 2]: [Errno 111] Connection refused 2012-08-16 10:27:06,515 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 2 seconds 2012-08-16 10:27:08,518 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671 2012-08-16 10:27:08,550 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 3]: [Errno 111] Connection refused 2012-08-16 10:27:08,551 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 4 seconds ... ... 2012-08-16 13:35:23,667 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671 2012-08-16 13:35:23,700 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 101]: [Errno 111] Connection refused 2012-08-16 13:35:23,701 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds 2012-08-16 13:37:23,793 [WARNING][Thread-1] connect() @ driver.py:523 - trying: mccvm182.fcux.usa.hp.com:5671 2012-08-16 13:37:23,860 [WARNING][Thread-1] close_engine() @ driver.py:444 - recoverable error[attempt 102]: [Errno 111] Connection refused 2012-08-16 13:37:23,861 [WARNING][Thread-1] close_engine() @ driver.py:446 - sleeping 120 seconds ............ On the server, it logs a similar set of warnings and error with connection attempts to port 5671 but refused: ===== from server /var/log/pulp/pulp.log ===== 2012-08-16 13:53:51,846 7872:140510450673408: qpid.messaging:WARNING: driver:446 sleeping 120 seconds 2012-08-16 13:55:51,919 7872:140510450673408: qpid.messaging:WARNING: driver:523 trying: localhost:5671 2012-08-16 13:55:51,921 7872:140510450673408: qpid.messaging:WARNING: driver:444 recoverable error[attempt 120]: [Errno 111] Connection refused 2012-08-16 13:55:51,921 7872:140510450673408: qpid.messaging:WARNING: driver:446 sleeping 120 seconds 2012-08-16 13:57:52,011 7872:140510450673408: qpid.messaging:WARNING: driver:523 trying: localhost:5671 2012-08-16 13:57:52,012 7872:140510450673408: qpid.messaging:WARNING: driver:444 recoverable error[attempt 121]: [Errno 111] Connection refused 2) Can you check qpidd is running on the *SERVER*? # service qpidd status qpidd.service - LSB: start or stop qpidd Loaded: loaded (/etc/rc.d/init.d/qpidd) Active: active (running) since Thu, 16 Aug 2012 01:33:31 -0400; 12h ago CGroup: name=systemd:/system/qpidd.service â”” 20073 /usr/sbin/qpidd --data-dir /var/lib/qpidd --daemon 3) Restart qpidd and try again. I restarted qpidd using "sevice qpidd restart" but the install package tasks still hangs. These hung tasks don't timeout and persists across the client reboot. Is there a way to cancel these tasks up and reinitiate them? 4) Also check and apply updates on the client and try again if the above does not help. I ran "yum update" on the client system and tried the package install from the katello server again with the same result - hung install package task. Thank you very much for your help with this issue!
I just tested agent with nightly and it is working. Can you please show me /etc/gofer/plugins/katelloplugin.conf? Then use katello-debug-certificates tool (in the PATH) both on the server and the client. Check this: 1) CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of /usr/share/katello/candlepin-cert.crt (SERVER). 2) NSS DB - CA matches SERIAL of the candlepin-cert.crt (both SERVER) 3) CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of the NSS DB - Broker (SERVER). 4) If you don't mind I would like to see outputs of both (you can attach private, no worries there are no sensitive data in it just serials - feel free to scramble it). Also I'm interested in what subscription-manager identity shows. Thats for client side. Try to unregister, reinstall candlepin-consumer-xyz package and register again and make sure consumer certs (/etc/pki/consumer/ and /etc/rhsm/ca/candlepin-local.crt) changed. Also send output of the: rpm -ql candlepin-cert-consumer-your_server_fqdn It should list the certificate that was deployed. Check also fqdn - it MUST match with the server you are working with. Also, make sure you have full FQDN and also DNS is set properly. Now since you have issues with qpidd also from the pulp side, it looks its something wrong with it. Can you check updates on both server and client? And when I say updates, I mean whole system :-) We are interested in goferd, pulp, qpidd server and client packages. If you don't mind, just update both and restart to see if it helps.
(In reply to comment #6) > I just tested agent with nightly and it is working. Can you please show me > /etc/gofer/plugins/katelloplugin.conf? # cat /etc/gofer/plugins/katelloplugin.conf @import:/etc/rhsm/rhsm.conf:server:hostname(host) [main] enabled=1 requires=package [messaging] uuid= url=ssl://$(host):5671 cacert=/etc/rhsm/ca/candlepin-local.pem clientcert=/etc/pki/consumer/bundle.pem [reboot] allow=1 delay=+1 ----------- In the /etc/rhsm/rhsm.conf, the hostname is FQDN, with the additional proxy_hostname and poxy_port setting. > Then use katello-debug-certificates > tool (in the PATH) both on the server and the client. Check this: The katello-debug-certificates tool is only available on the server and not the client. I do see a serial parameter in the client /etc/rhsm/ca/candlepin-local.pem file and it does match the server CA_SERIAL. Please see the attached files katello-server-cert.docx and katello-client-cert.docx. 1) > CA_SERIAL of the /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL > entry of /usr/share/katello/candlepin-cert.crt (SERVER). CA_SERIAL from the client /etc/rhsm/ca/candlepin-local.pem: # fgrep serial /etc/rhsm/ca/candlepin-local.pem serial:BA:30:7B:69:B3:1F:D7:61 CA SERIAL from the server /usr/share/katello/candlepin-cert.crt: /usr/share/katello/candlepin-cert.crt CA_SERIAL:BA:30:7B:69:B3:1F:D7:61 2) NSS DB - CA > matches SERIAL of the candlepin-cert.crt (both SERVER) The NSS DB - CA output that got displayed from the katello-debug-certificates tool reported a "Could not find cert: broker: File not found" message. The CA_SERIAL does match candlepin-cert.crt one: /etc/rhsm/ca/candlepin-local.pem N/A NSS DB - Broker Key NSS DB - CA DN: "CN=mccvm182.fcux.usa.hp.com,OU=Cloud BU,O=Red Hat,L=Raleigh SERIAL: 00:ba:30:7b:69:b3:1f:d7:61 CA: Is a CA with no maximum path length. CA_SERIAL: 00:ba:30:7b:69:b3:1f:d7:61 certutil: Could not find cert: broker : File not found NSS DB - Broker N/A /usr/share/katello/candlepin-cert.crt CA_SERIAL:BA:30:7B:69:B3:1F:D7:61 3) CA_SERIAL of the > /etc/rhsm/ca/candlepin-local.pem (CLIENT) matches SERIAL entry of the NSS DB > - Broker (SERVER). There isn't a CA_SERIAL being shown under the NSS DB - Broker output, it shows "N/A" with a "Could not find cert: broker: File not found" message right beforehand: certutil: Could not find cert: broker : File not found NSS DB - Broker N/A The serial in the client /etc/rhsm/ca/candlepin-local.pem: # fgrep serial /etc/rhsm/ca/candlepin-local.pem serial:BA:30:7B:69:B3:1F:D7:61 4) If you don't mind I would like to see outputs of both > (you can attach private, no worries there are no sensitive data in it just > serials - feel free to scramble it). Please see the attached files katello-server-cert.docx and katello-client-cert.docx. Also I'm interested in what > subscription-manager identity shows. Thats for client side. Try to > unregister, reinstall candlepin-consumer-xyz package and register again and > make sure consumer certs (/etc/pki/consumer/ and > /etc/rhsm/ca/candlepin-local.crt) changed. Also send output of the: Before: # subscription-manager identity Current identity is: 8627f4f6-d953-4a67-aca8-17fe240ee270 name: mccvm179.fcux.usa.hp.com org name: HP_LM_Org org id: 8f720136392dec0c01392fb4f3fd0003 # rpm -qa|grep candlepin candlepin-cert-consumer-mccvm182.fcux.usa.hp.com-1.0-1.noarch [root@mccvm179 ca]# rpm -ql candlepin-cert-consumer-mccvm182.fcux.usa.hp.com-1.0-1.noarch /etc/rhsm/ca/candlepin-local.pem unregister and reinstall candlepin-consumer-xyz package but can't register again due to the unregister errors: After the client system is unregistered via the katello WebUI (System > Remove System), a couple sets of the following 410 Gone errors are logged and the client system fail to re-register with the "Validation failed: Name has already been taken". What is the correct procedure to clean up the client system from katello so the client system can re-register? The System tab page from the katello WebUI keeps displaying the folowing error as well: [ERROR: 2012-08-17 13:06:17 #1145] Rendering 500:Resources::Candlepin::CandlepinResource: 410 Gone {"displayMessage":"Consumer 8627f4f6-d953-4a67-aca8-17fe240ee270 has been deleted","deletedId":"8627f4f6-d953-4a67-aca8-17fe240ee270"} (GET /candlepin/consumers/8627f4f6-d953-4a67-aca8-17fe240ee270/events) # yum -y --nogpgcheck reinstall http://$KATELLO_HOSTNAME/pub/candlepin-cert-consumer-$KATELLO_HOSTNAME-1.0-1.noarch.rpm # subscription-manager clean All local data removed [root@mccvm179 ca]# subscription-manager register --force --username=admin --password=admin Validation failed: Name has already been taken rpm -ql > candlepin-cert-consumer-your_server_fqdn It should list the certificate > that was deployed. Check also fqdn - it MUST match with the server you are > working with. Also, make sure you have full FQDN and also DNS is set > properly. Now since you have issues with qpidd also from the pulp side, it > looks its something wrong with it. Can you check updates on both server and > client? And when I say updates, I mean whole system :-) We are interested in > goferd, pulp, qpidd server and client packages. If you don't mind, just > update both and restart to see if it helps. I ran "yum check-update" on both the server and client, the only update needed was gdb and I applied it anyway so both are up to date: === server === [root@mccvm182 log]# yum check-update Loaded plugins: langpacks, presto, refresh-packagekit [root@mccvm182 log]# === client === [root@mccvm179 ca]# yum check-update Loaded plugins: langpacks, presto, product-id, refresh-packagekit, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. [root@mccvm179 ca]# Thank you very much for your help!
Created attachment 605231 [details] Katello Server katello-debug-certificates output
Created attachment 605232 [details] Katello Client /etc/rhsm/ca/candlepin-local.pem content
Okay, the installation is obviously not correct - katello-debug-certificates script is unable to print NSS DB certificates. Did you see any errors during the installation? Can you attach me katello-debug output? Note it is a different script that creates a tarball of all log files (it strips out all passwords from there). I really need to see all main.log logfiles that our intaller created. I think this could be installer-related issue. Also - did you change the hostname after installation by chance? To correctly unregister a system run subscriptioin-manager unregister. You can also delete it from katello. Please note "clean" command just deletes local data and does not send it to the katello server. Use this for debugging purposes. You can also use --name to change the name of the system you are registering to avoid that naming issue. I am also interested in: $ ls /etc/pki/katello/nssdb -la $ certutil -d /etc/pki/katello/nssdb -L $ certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10 $ certutil -d /etc/pki/katello/nssdb -L -n broker | head -n10
(In reply to comment #10) > Okay, the installation is obviously not correct - katello-debug-certificates > script is unable to print NSS DB certificates. Did you see any errors during > the installation? Can you attach me katello-debug output? Note it is a > different script that creates a tarball of all log files (it strips out all > passwords from there). I really need to see all main.log logfiles that our > intaller created. I think this could be installer-related issue. I didn't see any errors during the installation. Following is the screen capture of the katello-configure run. I am attaching the katello-debug run tarball, since it can't find the katello.conf and pulp.conf file under the /etc/httpd.d/ directory, these files are being attached here too (from /etc/httpd/conf.d/). It also can't find thumbslug etc and log files which didn't get installed as part of the katello install, not sure if it's needed. ----------------------------- [root@mccvm182 ~]# katello-configure Starting Katello configuration The top-level log file is [/var/log/katello/katello-configure-20120816-013225/main.log] Creating Katello database user ############################################################ ... OK Creating Katello database ############################################################ ... OK Creating Candlepin database user ############################################################ ... OK Populating Katello database schema ############################################################ ... OK Initializing Katello data ############################################################ ... OK ----------------------------- Also - did > you change the hostname after installation by chance? I didn't change the hostname after installation. However, due to this issue I've been experimenting with the /etc/pulp/pulp.conf file and modified the following lines in the /etc/pulp/pulp.conf file, it didn't help correct this issue though:- 37c37 < url: ssl://localhost:5671 --- > url: ssl://mccvm182.fcux.usa.hp.com:5671 Since we need to use a proxy, the following settings were added before running the katello-configure command. I've been wondering if the proxy setting needs to be set in other katello related configuration files as well or is pulp.conf the only place this setting is needed. 93a104,105 > proxy_url:http://16.85.88.10 > 94a107,109 > proxy_port:8080 To correctly > unregister a system run subscriptioin-manager unregister. You can also > delete it from katello. Please note "clean" command just deletes local data > and does not send it to the katello server. Use this for debugging purposes. > You can also use --name to change the name of the system you are registering > to avoid that naming issue. I unregistered the system from the katello GUI only. I'll try out using the subscription-manager unregister command. I am also interested in: $ ls > /etc/pki/katello/nssdb -la $ certutil -d /etc/pki/katello/nssdb -L $ > certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10 $ certutil -d > /etc/pki/katello/nssdb -L -n broker | head -n10 [root@mccvm182 ~]# ls /etc/pki/katello/nssdb -la total 104 drwxr-xr-x. 2 root katello 4096 Aug 16 01:32 . drwxr-x---. 3 root katello 4096 Aug 16 01:33 .. -rw-r-----. 1 root katello 65536 Aug 16 01:33 cert8.db -rw-r-----. 1 root katello 16384 Aug 16 01:33 key3.db -rw-r-----. 1 root katello 16384 Aug 16 01:32 secmod.db [root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ca CT,C,c mccvm182.fcux.usa.hp.com - Red Hat u,u,u [root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L -n ca | head -n10 Certificate: Data: Version: 3 (0x2) Serial Number: 00:ba:30:7b:69:b3:1f:d7:61 Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=mccvm182.fcux.usa.hp.com,OU=Cloud BU,O=Red Hat,L=Raleigh, ST=North Carolina,C=US" Validity: Not Before: Thu Aug 16 05:32:37 2012 [root@mccvm182 ~]# certutil -d /etc/pki/katello/nssdb -L -n broker | head -n10 certutil: Could not find cert: broker : File not found [root@mccvm182 ~]# Thank you very much for your help!
Created attachment 605828 [details] katello-debug tarball
Created attachment 605829 [details] katello-debug command run screen capture
Created attachment 605830 [details] /etc/httpd/conf.d/katello.conf
Created attachment 605831 [details] /etc/httpd/conf.d/pulp.conf
Ok this is a bug we already fixed in master: https://github.com/Katello/katello/pull/447 To fix this, do the following steps: # rm -f /etc/pki/katello/nssdb/*db # katello-configure --answer-file=/etc/katello/katello-configure.conf -b Attach the output of the last command here, it should regenerate NSS database. Then you should be able to connect. If not, restart Katello services: # katello-service restart And then show the output of: # certutil -d /etc/pki/katello/nssdb -L You should see "broker" there. Once you confirm me the fix, I will most likely do update for Katello 1.0 and update Known Problems page.
(In reply to comment #16) > Ok this is a bug we already fixed in master: > https://github.com/Katello/katello/pull/447 To fix this, do the following > steps: # rm -f /etc/pki/katello/nssdb/*db # katello-configure > --answer-file=/etc/katello/katello-configure.conf -b Attach the output of > the last command here, it should regenerate NSS database. Then you should be > able to connect. If not, restart Katello services: # katello-service > restart After removal of the *db files and regenerated them using the katello-configure command you provided, the install package tasks no longer hang. However, they are failing the package install with the "No package available to install" error. I ran katello-service restart and rebooted the client system but the same failure persists. The client system is subscirbed to the repo that contains the package. Please see the attachment "package install error" for the screen captures of the error as well as the katello CLI output that shows the client system is subscribed to the repo. I also don't see the subscribed repo from the "yum repolist" command run and can't yum install the package from the subscribed repo (please see the screen capture in the attachement "package install error" as well). Do you know what I need to do to fix this? You should see "broker" there. # certutil -d /etc/pki/katello/nssdb > -L [root@mccvm182 katello]# certutil -d /etc/pki/katello/nssdb -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ca CT,C,c broker u,u,u Once you confirm me the fix, I will most > likely do update for Katello 1.0 and update Known Problems page. Do I need to reinstall Katello 1.0 or can I run yum update install on katello-all to pick up this fix? Thank you very much for your help!
Created attachment 605966 [details] katello-configure command run screen capture
Created attachment 605967 [details] package install error
Ok now it looks good. Did you promote your repo? What environment is the consumer (client) registered to? You need to have the content in the same environment as the client, otherwise you wont see the content. Create environment, let's say "test", promote packages there (or whole product), register the client against environment "test" and then you will be able to consume it.
I didn't promote the repos. Thank you so much for your helpful tip! Initially the client was registered to the "common" environment created off of the top level "Library" environment. After creating the "test_env" off of the "common" environment, I created a changeset "CS_c179" with the "test_repo" and "local_repo" products, then promoted it to the "test_env". I used the Katello GUI to subscribe to the "test_repo" and "local_repo" products under the "test_env" environment, then re-registered the client system to the new environment "test_env" using the subscription-manager command, ran "subscription-manager refresh" and then "yum repolist". The /etc/yum.repos.d/redhat.repo file is now updated with the repo configuration information (please see attachment redhat.repo). However, the yum repolist command run returned the "HTTP Error 403 - Forbidden :" errors on trying to access these repos repomd.xml files. I checked the CloudForms User's Guide on promote changset section, and the Infrastructure and Application Deployment Fundamentals on Connecting Instances to System Engine section, I don't see additional steps needed. Do you know how I can correct this repo access error? [root@mccvm179 yum.repos.d]# subscription-manager list --consumed +-------------------------------------------+ Consumed Subscriptions +-------------------------------------------+ Subscription Name: test_repo Provides: test_repo SKU: 1345125668164 Contract: None Account: None Serial Number: 8125559394500063678 Active: True Quantity Used: 1 Service Level: Service Type: Starts: 08/15/2012 Ends: 08/08/2042 Subscription Name: local_repo Provides: local_repo SKU: 1345128551424 Contract: None Account: None Serial Number: 3467786245544411526 Active: True Quantity Used: 1 Service Level: Service Type: Starts: 08/15/2012 Ends: 08/08/2042 [root@mccvm179 yum.repos.d]# yum repolist Loaded plugins: langpacks, presto, product-id, refresh-packagekit, subscription-manager This system is receiving updates from Red Hat Subscription Management. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/c_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/w_repos/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/local_repo/zebra_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/brew_test_repo/repodata/repomd.xml Trying other mirror. https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden : https://mccvm182.fcux.usa.hp.com/pulp/repos/HP_LM_Org/test_env//custom/test_repo/zoo_repo/repodata/repomd.xml Trying other mirror. repo id repo name status HP_LM_Org_local_repo_c_repo c_repo 0 HP_LM_Org_local_repo_w_repos w_repos 0 HP_LM_Org_local_repo_zebra_repo zebra_repo 0 HP_LM_Org_test_repo_brew_test_repo brew_test_repo 0 HP_LM_Org_test_repo_zoo_repo zoo_repo 0 fedora Fedora 16 - x86_64 25,098 fedora-katello integrates together a series of open source systems management 1 fedora-subscription-manager Tools and libraries for Red Hat subscription management. 13 pulp-v1-stable Pulp v1 20 updates Fedora 16 - x86_64 - Updates 10,007 repolist: 35,139 Thank you very much for your patience and help!
Created attachment 606438 [details] redhat.repo
The 403 is thrown when your consumer certificate AND/OR entitlement certificate are not correct. Katello project is used mainly for Red Hat repositories from Content Delivery Network, therefore for each repository there is a product and entitlements imported from a manifest file. If you work with custom yum repositories, it works the same way. For each (custom) product you create, unlimited subscription is automatically created by Katello. You need to register and subscribe to a content you want to consume at the moment. Both organization and environment must match. If one of these things is not correct, yum is not able to access the content (with 403 error). Therefore you need to check two things: 1) Client was registered to the correct organization and environment during subscription-manager register command (see the cli options). 2) Subscription was consumed for each yum custom product you want consume repositories from. Retrieve pool id and use it in the subscription-manager subscribe command to properly subscribe to the content. You can do both from subscription-manager or also via Katello UI or CLI. If you do the latter, please note it takes a while (up to 4 hours) until new information is propagated to consumers. You can speed this up using subscription-manager refresh command. For example in the Katello UI case, you need to navigate to your systems list, open a system, list the available subscriptions and apply for it. Then wait or use the "refresh" command on the client to receive subscriptions, then you can consume content.
Hello, since I don't read any other issues, I am marking the bug as done. Please feel free to reopen or file a new one.