Bug 1457977

Summary:	goferd failed to connect to satellite amqps service
Product:	Red Hat Satellite	Reporter:	Djebran Lezzoum <dlezzoum>
Component:	Qpid	Assignee:	Mike Cressman <mcressma>
Status:	CLOSED ERRATA	QA Contact:	Katello QA List <katello-qa-list>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.3.0	CC:	afox, bbuckingham, bkearney, egolov, ehelms, jcallaha, katello-qa-list, mcressma, mmccune, mmithaiw, rplevka, xiaoxwan
Target Milestone:	Unspecified	Keywords:	AutomationBlocker, Regression, Triaged
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	The 'hammer host-collection erratum install` installation failed with a sub-task error. With the latest update to qpid, this is now fixed.	Story Points:	---
Clone Of:	1457938	Environment:
Last Closed:	2018-02-21 12:57:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1457938
Bug Blocks:

Description Djebran Lezzoum 2017-06-01 16:25:56 UTC

+++ This bug was initially created as a clone of Bug #1457938 +++

Description of problem:
when trying to do hammer host-collection erratum install, installation sub task error returned

Version-Release number of selected component (if applicable):
* candlepin-0.9.54.21-1.el7.noarch
* candlepin-selinux-0.9.54.21-1.el7.noarch
* foreman-1.11.0.76-1.el7sat.noarch
* foreman-compute-1.11.0.76-1.el7sat.noarch
* foreman-debug-1.11.0.76-1.el7sat.noarch
* foreman-discovery-image-3.1.1-23.el7sat.noarch
* foreman-ec2-1.11.0.76-1.el7sat.noarch
* foreman-gce-1.11.0.76-1.el7sat.noarch
* foreman-installer-1.11.0.15-2.el7sat.noarch
* foreman-installer-katello-3.0.0.84-1.el7sat.noarch
* foreman-libvirt-1.11.0.76-1.el7sat.noarch
* foreman-openstack-1.11.0.76-1.el7sat.noarch
* foreman-ovirt-1.11.0.76-1.el7sat.noarch
* foreman-postgresql-1.11.0.76-1.el7sat.noarch
* foreman-proxy-1.11.0.7-1.el7sat.noarch
* foreman-rackspace-1.11.0.76-1.el7sat.noarch
* foreman-selinux-1.11.0.4-1.el7sat.noarch
* foreman-vmware-1.11.0.76-1.el7sat.noarch
* katello-3.0.0-20.el7sat.noarch
* katello-certs-tools-2.4.0-1.el7sat.noarch
* katello-client-bootstrap-1.3.0-1.el7sat.noarch
* katello-common-3.0.0-20.el7sat.noarch
* katello-debug-3.0.0-20.el7sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-base-3.0.0.84-1.el7sat.noarch
* katello-selinux-3.0.1.2-1.el7sat.noarch
* katello-server-ca-1.0-1.noarch
* katello-service-3.0.0-20.el7sat.noarch
* openldap-2.4.40-13.el7.x86_64
* pulp-client-1.0-1.noarch
* pulp-docker-plugins-2.0.3.1-1.el7sat.noarch
* pulp-katello-1.0.2-1.el7sat.noarch
* pulp-ostree-plugins-1.1.3.3-1.el7sat.noarch
* pulp-puppet-plugins-2.8.7.1-1.el7sat.noarch
* pulp-puppet-tools-2.8.7.1-1.el7sat.noarch
* pulp-rpm-plugins-2.8.7.13-1.el7sat.noarch
* pulp-selinux-2.8.7.11-1.el7sat.noarch
* pulp-server-2.8.7.11-1.el7sat.noarch
* python-ldap-2.4.15-2.el7.x86_64
* tfm-rubygem-ldap_fluff-0.4.5-1.el7sat.noarch
* tfm-rubygem-net-ldap-0.13.0-1.el7sat.noarch


How reproducible:
Always

Steps to Reproduce:
1. Setup 2 hosts and a host-collection subscribed to some products containing erratum
2. run:
   hammer host-collection erratum install --id="col-id" --organization-id="org-id" --errata="errata-id"
<<< Message,Id
Successfully scheduled installation of errata,263eea3b-3ac1-4576-9b66-85ce46ec93e9

3. to wait the task to finish run:
    hammer -v -u admin -p changeme  task progress --id="263eea3b-3ac1-4576-9b66-85ce46ec93e9"
     

Actual results:
2 task(s), 0 success, 2 fail

Expected results:
2 task(s), 2 success, 0 fail
and errata successfully installed

Additional info:
1- in the UI Minitor > tasks 
    the tasks error is:
     Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host?

2- at one of the virtual machine  /var/log/messages contain error
   Jun  1 10:44:19 139890008902672 goferd: [ERROR][worker-0] gofer.messaging.adapter.connect:33 - connect: proton+amqps://dell-pe-fm120-2a.rhts.eng.bos.redhat.com:5647, failed: Connection amqps://dell-pe-fm120-2a.rhts.eng.bos.redhat.com:5647 disconnected
Jun  1 10:44:19 139890008902672 goferd: [INFO][worker-0] gofer.messaging.adapter.connect:35 - retry in 51 seconds

--- Additional comment from Djebran Lezzoum on 2017-06-01 11:18 EDT ---



--- Additional comment from Djebran Lezzoum on 2017-06-01 11:20 EDT ---



--- Additional comment from Djebran Lezzoum on 2017-06-01 11:22:13 EDT ---

related bugs: 
https://bugzilla.redhat.com/show_bug.cgi?id=1405428
https://bugzilla.redhat.com/show_bug.cgi?id=1418026

--- Additional comment from Djebran Lezzoum on 2017-06-01 11:25:19 EDT ---

full automation test log:
http://pastebin.test.redhat.com/490019

--- Additional comment from RHEL Product and Program Management on 2017-06-01 11:25:51 EDT ---

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been
set to + automatically for the next planned release

Comment 1 Djebran Lezzoum 2017-06-01 16:27:58 UTC

6.3 
Version-Release number of selected component (if applicable):
* candlepin-2.0.35-1.el7.noarch
* candlepin-selinux-2.0.35-1.el7.noarch
* foreman-1.15.0-1.el7sat.noarch
* foreman-compute-1.15.0-1.el7sat.noarch
* foreman-debug-1.15.0-1.el7sat.noarch
* foreman-discovery-image-3.1.1-22.el7sat.noarch
* foreman-ec2-1.15.0-1.el7sat.noarch
* foreman-gce-1.15.0-1.el7sat.noarch
* foreman-installer-1.15.0-1.el7sat.noarch
* foreman-installer-katello-3.4.0-2.el7sat.noarch
* foreman-libvirt-1.15.0-1.el7sat.noarch
* foreman-openstack-1.15.0-1.el7sat.noarch
* foreman-ovirt-1.15.0-1.el7sat.noarch
* foreman-postgresql-1.15.0-1.el7sat.noarch
* foreman-proxy-1.15.0-2.el7sat.noarch
* foreman-rackspace-1.15.0-1.el7sat.noarch
* foreman-selinux-1.15.0-1.el7sat.noarch
* foreman-vmware-1.15.0-1.el7sat.noarch
* katello-3.4.0-3.el7sat.noarch
* katello-ca-consumer-sat-r220-02.lab.eng.rdu2.redhat.com-1.0-1.noarch
* katello-certs-tools-2.4.0-1.el7sat.noarch
* katello-client-bootstrap-1.3.0-1.el7sat.noarch
* katello-common-3.4.0-3.el7sat.noarch
* katello-debug-3.4.0-3.el7sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-base-3.4.0-2.el7sat.noarch
* katello-selinux-3.0.2-1.el7sat.noarch
* katello-server-ca-1.0-1.noarch
* katello-service-3.4.0-3.el7sat.noarch
* openldap-2.4.40-13.el7.x86_64
* pulp-client-1.0-1.noarch
* pulp-docker-plugins-2.3.0-1.el7sat.noarch
* pulp-katello-1.0.2-1.el7sat.noarch
* pulp-ostree-plugins-1.2.1-1.el7sat.noarch
* pulp-puppet-plugins-2.12.2-1.el7sat.noarch
* pulp-puppet-tools-2.12.2-1.el7sat.noarch
* pulp-rpm-plugins-2.12.2-1.el7sat.noarch
* pulp-selinux-2.12.2-1.el7sat.noarch
* pulp-server-2.12.2-1.el7sat.noarch
* python-ldap-2.4.15-2.el7.x86_64
* tfm-rubygem-ldap_fluff-0.4.6-1.el7sat.noarch
* tfm-rubygem-net-ldap-0.15.0-1.el7sat.noarch

Comment 5 Evgeni Golov 2017-06-22 11:17:04 UTC

I see similar behaviour in my tests: the connection from goferd to satellite is fine, but the messages are not picked up from the queue:

  pulp.agent.91ab5c1b-9cc9-49e6-9022-495761a518d2                           Y                      6     6      0    3.52k  3.52k       0         0     1

I also tried installing the whole client stack (katello-agent, gofer, proton) on my client VM, but this did not help.

Comment 11 Evgeni Golov 2017-09-14 14:24:01 UTC

moving to ON_DEV as qpid 1.36 was delivered in latest snap (15) and fixes the issues for me

Comment 12 Roman Plevka 2017-09-20 12:17:15 UTC

VERIFIED
on sat6.3.0-16


systemd[1]: Started Gofer Agent.
systemd[1]: Starting Gofer Agent...
goferd[4777]: [INFO][Thread-1] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/demo
goferd[4777]: [WARNING][MainThread] gofer.agent.plugin:639 - plugin:demo, DISABLED
goferd[4777]: [INFO][Thread-2] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin
goferd[4777]: [INFO][Thread-3] gofer.rmi.store:114 - Using: /var/lib/gofer/messaging/pending/katelloplugin
goferd[4777]: [INFO][MainThread] gofer.agent.plugin:682 - plugin:katelloplugin loaded using: /usr/lib/gofer/plugins/katelloplugin.py
goferd[4777]: [INFO][MainThread] rhsm.connection:830 - Connection built: host=sat-server port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ verify=False
goferd[4777]: [INFO][MainThread] katelloplugin:164 - Using /etc/rhsm/ca/katello-default-ca.pem as the ca cert for qpid connection
goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647
goferd[4777]: [INFO][MainThread] gofer.agent.main:87 - agent started.
goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647
goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647
goferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://sat-server:5647
goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647
goferd[4777]: [INFO][worker-0] gofer.agent.plugin:368 - plugin:katelloplugin, attached => pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab
goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647
goferd[4777]: [INFO][pulp.agent.c90cc938-16e7-4bc4-b9f8-39c73bc77cab] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647


# updating a package

sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://sat-server:5647
sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://sat-server:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://sat-server:5647
sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://sat-server:5647
sat-servergoferd[4777]: [INFO][worker-0] gofer.rmi.dispatcher:603 - call: Content.update() sn=206e036f-6f18-4b4b-985f-e7004b7e576a data={u'task_id': u'10c459a8-9774-4b9b-87e5-626d11969b02', u'consumer_id': u'c90cc938-16e7-4bc4-b9f8-39c73bc77cab'}
sat-servergoferd[4777]: Loaded plugins: enabled_repos_upload, langpacks, package_upload, product-id,
sat-servergoferd[4777]: : search-disabled-repos, subscription-manager, tracer_upload
sat-servergoferd[4777]: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
sat-servergoferd[4777]: Running Transaction Check
Sep 20 08:04:00 sat-servergoferd[4777]: Uploading Package Profile
Sep 20 08:04:00 sat-servergoferd[4777]: Uploading Tracer Profile
Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] pulp_rpm.handlers.rpmtools:232 - Updated: python-urllib3-1.10.2-3.el7sat.noarch
Sep 20 08:04:03 sat-serveryum[4777]: Updated: python-urllib3-1.10.2-3.el7sat.noarch
Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.agent.rmi:193 - Request: 206e036f-6f18-4b4b-985f-e7004b7e576a, committed
Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.agent.rmi:147 - Request: 206e036f-6f18-4b4b-985f-e7004b7e576a processed in: 8.371 (seconds)
Sep 20 08:04:03 sat-servergoferd[4777]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://sat-server:5647

Comment 15 errata-xmlrpc 2018-02-21 12:57:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0338