Bug 1344215

Summary: [Protocol Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77 while upgrading sat6.1.9 -> sat6.2 snap15
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: UpgradesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, bkearney, dzhukous, mshimura, oshtaier, sghai, swadeley, syangsao, tpapaioa
Target Milestone: UnspecifiedKeywords: Reopened, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
During a Satellite 6.2 upgrade, as part of the host unification feature, if a host and content host are found with same name but within different organizations, the content host will be unregistered. This will remove the content host from the Satellite. Consequently, once the content host has been unregistered by the upgrade process, it is possible that errors will be observed in /var/log/messages as the content host may still attempt to interact with the Satellite via the agent. The errors can include ones similar to the following: [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77 Workaround Prior to performing the Satellite 6.2 upgrade, the user should execute the pre-upgrade script on the Satellite 6.1 Server to obtain a summary of the content hosts that will be unregistered. The script may be executed as: foreman-rake katello:upgrade_check Prior to the upgrade, the user should then update the hosts and content hosts to ensure that they are in the correct organizations. If this is not done prior to the upgrade, the user will need to re-register each of those content hosts using the Red Hat Subscription Manager command-line tool after the upgrade has completed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-14 15:51:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    
Attachments:
Description Flags
foreman-debug none

Description Sachin Ghai 2016-06-09 08:19:37 UTC
Description of problem:
I was trying upgrade from  sat6.1.9 -> sat6.2 snap15 on rhel7, and installer was throwing following error message in /var/log/messages:

 
==> /var/log/messages <==
Jun  9 03:51:04 qe-sat6-upgrade-rhel7 qpidd: 2016-06-09 03:51:04 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:51:04 qe-sat6-upgrade-rhel7 qpidd[9999]: 2016-06-09 03:51:04 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:51:46 qe-sat6-upgrade-rhel7 qpidd: 2016-06-09 03:51:46 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:51:46 qe-sat6-upgrade-rhel7 qpidd[9999]: 2016-06-09 03:51:46 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:52:27 qe-sat6-upgrade-rhel7 qpidd: 2016-06-09 03:52:27 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:52:27 qe-sat6-upgrade-rhel7 qpidd[9999]: 2016-06-09 03:52:27 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:53:09 qe-sat6-upgrade-rhel7 qpidd[9999]: 2016-06-09 03:53:09 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:53:09 qe-sat6-upgrade-rhel7 qpidd: 2016-06-09 03:53:09 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77
Jun  9 03:53:50 qe-sat6-upgrade-rhel7 qpidd[9999]: 2016-06-09 03:53:50 [Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77

Version-Release number of selected component (if applicable):
upgrade from  sat6.1.9 -> sat6.2 snap15 on rhel7,

How reproducible:


Steps to Reproduce:
0 set 6.2 repo
1. yum update -y
2. satellite-installer --scenario satellite --upgrade


Actual results:
[Protocol] error Error on attach: Node not found: pulp.agent.737d5dde-2ec8-4b21-9933-3e7fa45cbf77

Expected results:

no error message should be thrown in logs while installer is running
Additional info:

Comment 1 Sachin Ghai 2016-06-09 08:43:38 UTC
Created attachment 1166208 [details]
foreman-debug

Comment 3 Brad Buckingham 2016-06-09 19:51:51 UTC
Sachin, is this error from SNAP 15 or 15.1?

Neither myself or Chris are seeing the error; however, we do have different data and we are running on 15.1.

Comment 4 Sachin Ghai 2016-06-10 06:48:49 UTC
@Brad: this issue is appearing on upgrade with snap15 and snap15.1 on rhel7. And I see this error consistently while upgrade and a bunch of times.

Comment 5 Brad Buckingham 2016-06-10 18:59:42 UTC
In discussing this issue with Justin and Jeff, it appears that the issue is due to the fact that the during the upgrade some of the content hosts (pulp consumers) can be deleted from the server.  Unfortunately, the agent running on the client (content host) does not know that it has been deleted and it may still reach out to Satellite triggering the error to be generated in the Satellite logs.  Once rhsmcertd runs on the client, it will detect that it is no longer registered and stop sending the requests to the server.

This behavior exists today in Satellite 6.1 as well.  Example scenario would be, client registers, goferd is running on the client and the content host is deleted from the UI.

Comment 7 Brad Buckingham 2016-06-10 19:12:34 UTC
Aside from rhsmcertd running on the client, re-registering it should also cause the error messages to stop.

Comment 8 Sachin Ghai 2016-06-13 10:30:17 UTC
@Brad: To fix this if changes are required at client end with updates to katello-agent then let's document this in release notes.

Comment 9 Brad Buckingham 2016-06-13 12:40:19 UTC
Thanks Sachin, flagging the bug for release notes.

Comment 11 Bryan Kearney 2016-07-27 11:34:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Comment 27 Bryan Kearney 2017-09-14 15:51:44 UTC
Confirmed with CEE and ENG that this case is resolved. Closing this out.