+++ This bug was initially created as a clone of Bug #1457938 +++ Description of problem: when trying to do hammer host-collection erratum install, installation sub task error returned Version-Release number of selected component (if applicable): * candlepin-0.9.54.21-1.el7.noarch * candlepin-selinux-0.9.54.21-1.el7.noarch * foreman-1.11.0.76-1.el7sat.noarch * foreman-compute-1.11.0.76-1.el7sat.noarch * foreman-debug-1.11.0.76-1.el7sat.noarch * foreman-discovery-image-3.1.1-23.el7sat.noarch * foreman-ec2-1.11.0.76-1.el7sat.noarch * foreman-gce-1.11.0.76-1.el7sat.noarch * foreman-installer-1.11.0.15-2.el7sat.noarch * foreman-installer-katello-3.0.0.84-1.el7sat.noarch * foreman-libvirt-1.11.0.76-1.el7sat.noarch * foreman-openstack-1.11.0.76-1.el7sat.noarch * foreman-ovirt-1.11.0.76-1.el7sat.noarch * foreman-postgresql-1.11.0.76-1.el7sat.noarch * foreman-proxy-1.11.0.7-1.el7sat.noarch * foreman-rackspace-1.11.0.76-1.el7sat.noarch * foreman-selinux-1.11.0.4-1.el7sat.noarch * foreman-vmware-1.11.0.76-1.el7sat.noarch * katello-3.0.0-20.el7sat.noarch * katello-certs-tools-2.4.0-1.el7sat.noarch * katello-client-bootstrap-1.3.0-1.el7sat.noarch * katello-common-3.0.0-20.el7sat.noarch * katello-debug-3.0.0-20.el7sat.noarch * katello-default-ca-1.0-1.noarch * katello-installer-base-3.0.0.84-1.el7sat.noarch * katello-selinux-3.0.1.2-1.el7sat.noarch * katello-server-ca-1.0-1.noarch * katello-service-3.0.0-20.el7sat.noarch * openldap-2.4.40-13.el7.x86_64 * pulp-client-1.0-1.noarch * pulp-docker-plugins-2.0.3.1-1.el7sat.noarch * pulp-katello-1.0.2-1.el7sat.noarch * pulp-ostree-plugins-1.1.3.3-1.el7sat.noarch * pulp-puppet-plugins-2.8.7.1-1.el7sat.noarch * pulp-puppet-tools-2.8.7.1-1.el7sat.noarch * pulp-rpm-plugins-2.8.7.13-1.el7sat.noarch * pulp-selinux-2.8.7.11-1.el7sat.noarch * pulp-server-2.8.7.11-1.el7sat.noarch * python-ldap-2.4.15-2.el7.x86_64 * tfm-rubygem-ldap_fluff-0.4.5-1.el7sat.noarch * tfm-rubygem-net-ldap-0.13.0-1.el7sat.noarch How reproducible: Always Steps to Reproduce: 1. Setup 2 hosts and a host-collection subscribed to some products containing erratum 2. run: hammer host-collection erratum install --id="col-id" --organization-id="org-id" --errata="errata-id" <<< Message,Id Successfully scheduled installation of errata,263eea3b-3ac1-4576-9b66-85ce46ec93e9 3. to wait the task to finish run: hammer -v -u admin -p changeme task progress --id="263eea3b-3ac1-4576-9b66-85ce46ec93e9" Actual results: 2 task(s), 0 success, 2 fail Expected results: 2 task(s), 2 success, 0 fail and errata successfully installed Additional info: 1- in the UI Minitor > tasks the tasks error is: Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host? 2- at one of the virtual machine /var/log/messages contain error Jun 1 10:44:19 139890008902672 goferd: [ERROR][worker-0] gofer.messaging.adapter.connect:33 - connect: proton+amqps://dell-pe-fm120-2a.rhts.eng.bos.redhat.com:5647, failed: Connection amqps://dell-pe-fm120-2a.rhts.eng.bos.redhat.com:5647 disconnected Jun 1 10:44:19 139890008902672 goferd: [INFO][worker-0] gofer.messaging.adapter.connect:35 - retry in 51 seconds --- Additional comment from Djebran Lezzoum on 2017-06-01 11:18 EDT --- --- Additional comment from Djebran Lezzoum on 2017-06-01 11:20 EDT --- --- Additional comment from Djebran Lezzoum on 2017-06-01 11:22:13 EDT --- related bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1405428 https://bugzilla.redhat.com/show_bug.cgi?id=1418026 --- Additional comment from Djebran Lezzoum on 2017-06-01 11:25:19 EDT --- full automation test log: http://pastebin.test.redhat.com/490019 --- Additional comment from RHEL Product and Program Management on 2017-06-01 11:25:51 EDT --- Since this issue was entered in Red Hat Bugzilla, the pm_ack has been set to + automatically for the next planned release
6.3 Version-Release number of selected component (if applicable): * candlepin-2.0.35-1.el7.noarch * candlepin-selinux-2.0.35-1.el7.noarch * foreman-1.15.0-1.el7sat.noarch * foreman-compute-1.15.0-1.el7sat.noarch * foreman-debug-1.15.0-1.el7sat.noarch * foreman-discovery-image-3.1.1-22.el7sat.noarch * foreman-ec2-1.15.0-1.el7sat.noarch * foreman-gce-1.15.0-1.el7sat.noarch * foreman-installer-1.15.0-1.el7sat.noarch * foreman-installer-katello-3.4.0-2.el7sat.noarch * foreman-libvirt-1.15.0-1.el7sat.noarch * foreman-openstack-1.15.0-1.el7sat.noarch * foreman-ovirt-1.15.0-1.el7sat.noarch * foreman-postgresql-1.15.0-1.el7sat.noarch * foreman-proxy-1.15.0-2.el7sat.noarch * foreman-rackspace-1.15.0-1.el7sat.noarch * foreman-selinux-1.15.0-1.el7sat.noarch * foreman-vmware-1.15.0-1.el7sat.noarch * katello-3.4.0-3.el7sat.noarch * katello-ca-consumer-sat-r220-02.lab.eng.rdu2.redhat.com-1.0-1.noarch * katello-certs-tools-2.4.0-1.el7sat.noarch * katello-client-bootstrap-1.3.0-1.el7sat.noarch * katello-common-3.4.0-3.el7sat.noarch * katello-debug-3.4.0-3.el7sat.noarch * katello-default-ca-1.0-1.noarch * katello-installer-base-3.4.0-2.el7sat.noarch * katello-selinux-3.0.2-1.el7sat.noarch * katello-server-ca-1.0-1.noarch * katello-service-3.4.0-3.el7sat.noarch * openldap-2.4.40-13.el7.x86_64 * pulp-client-1.0-1.noarch * pulp-docker-plugins-2.3.0-1.el7sat.noarch * pulp-katello-1.0.2-1.el7sat.noarch * pulp-ostree-plugins-1.2.1-1.el7sat.noarch * pulp-puppet-plugins-2.12.2-1.el7sat.noarch * pulp-puppet-tools-2.12.2-1.el7sat.noarch * pulp-rpm-plugins-2.12.2-1.el7sat.noarch * pulp-selinux-2.12.2-1.el7sat.noarch * pulp-server-2.12.2-1.el7sat.noarch * python-ldap-2.4.15-2.el7.x86_64 * tfm-rubygem-ldap_fluff-0.4.6-1.el7sat.noarch * tfm-rubygem-net-ldap-0.15.0-1.el7sat.noarch
I see similar behaviour in my tests: the connection from goferd to satellite is fine, but the messages are not picked up from the queue: pulp.agent.91ab5c1b-9cc9-49e6-9022-495761a518d2 Y 6 6 0 3.52k 3.52k 0 0 1 I also tried installing the whole client stack (katello-agent, gofer, proton) on my client VM, but this did not help.
moving to ON_DEV as qpid 1.36 was delivered in latest snap (15) and fixes the issues for me
VERIFIED on sat6.3.0-16 systemd[1]: Started Gofer Agent. systemd[1]: Starting Gofer Agent... goferd[4777]: [INFO][Thread-1] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/demo goferd[4777]: [WARNING][MainThread] gofer.agent.plugin:639 - plugin:demo, DISABLED goferd[4777]: [INFO][Thread-2] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin goferd[4777]: [INFO][Thread-3] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin goferd[4777]: [INFO][MainThread] gofer.agent.plugin:682 - plugin:katelloplugin loaded using: /usr/lib/gofer/plugins/katelloplugin.py goferd[4777]: [INFO][MainThread] rhsm.connection:830 - Connection built: host=sat-server port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False goferd[4777]: [INFO][MainThread] katelloplugin:164 - Using /etc/rhsm/ca/katello-default-ca.pem as the ca cert for qpid connection goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647 goferd[4777]: [INFO][MainThread] gofer.agent.main:87 - agent started. goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647 goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647 goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://sat-server:5647 goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647 goferd[4777]: [INFO][worker-0] gofer.agent.plugin:368 - plugin:katelloplugin, attached => pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647 goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647 # updating a package sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647 sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647 sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647 sat-servergoferd[4777]: [INFO][worker-0] gofer.rmi.dispatcher:603 - call: Content.update() sn=206e036f-6f18-4b4b-985f-e7004b7e576a data={u'task_id': u'10c459a8-9774-4b9b-87e5-626d11969b02', u'consumer_id': u'c90cc938-16e7-4bc4-b9f8-39c73bc77cab'} sat-servergoferd[4777]: Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id, sat-servergoferd[4777]: : search-disabled-repos, subscription-manager, tracer_upload sat-servergoferd[4777]: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. sat-servergoferd[4777]: Running Transaction Check Sep 20 08:04:00 sat-servergoferd[4777]: Uploading Package Profile Sep 20 08:04:00 sat-servergoferd[4777]: Uploading Tracer Profile Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] pulp_rpm.handlers.rpmtools:232 - Updated: python-urllib3-1.10.2-3.el7sat.noarch Sep 20 08:04:03 sat-serveryum[4777]: Updated: python-urllib3-1.10.2-3.el7sat.noarch Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.agent.rmi:193 - Request: 206e036f-6f18-4b4b-985f-e7004b7e576a, committed Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.agent.rmi:147 - Request: 206e036f-6f18-4b4b-985f-e7004b7e576a processed in: 8.371 (seconds) Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://sat-server:5647
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0338