Bug 1267543 - goferd fails to start after upgrading to 2.6.6-2 when having pending tasks
goferd fails to start after upgrading to 2.6.6-2 when having pending tasks
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: katello-agent (Show other bugs)
6.1.1
All Linux
medium Severity high (vote)
: GA
: --
Assigned To: satellite6-bugs
Sachin Ghai
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-30 07:17 EDT by Pavel Moravec
Modified: 2016-07-27 07:02 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 07:02:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Pavel Moravec 2015-09-30 07:17:43 EDT
Description of problem:
Upgrading gofer to 2.6.6-2, if it has a pending task under /var/lib/gofer/messaging/pending/katelloplugin/ directory, an attempt to restart goferd service fails (details below).

Removing the task helps, but the task wont be executed.


Version-Release number of selected component (if applicable):
gofer-2.6.6-2.el7sat.noarch


How reproducible:
100% (I guess)


Steps to Reproduce:
1. Stop goferd on capsule when it has a pending task under /var/lib/gofer/messaging/pending/katelloplugin/ directory.
2. Upgrade gofer to 2.6.6-2
3. service goferd restart


Actual results:
3. fails with:
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - cannot concatenate 'str' and 'NoneType' objects
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - Traceback (most recent call last):
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 39, in _fn
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     return fn(*args, **keywords)
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 805, in open
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     self._impl.open()
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 44, in _fn
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     return fn(messenger, *args, **kwargs)
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/producer.py", line 72, in open
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     self.connection.open()
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 96, in open
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     connector = Connector.find(self.url)
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 994, in find
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     domain_id = URL(url).canonical
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/url.py", line 112, in canonical
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 -     url += self.host
Sep 30 13:05:08 toledo-capsule goferd: [ERROR][worker-0] gofer.messaging.adapter.model:43 - TypeError: cannot concatenate 'str' and 'NoneType' objects


Expected results:
goferd service successfully restarted.


Additional info:
- Debugging a little:
/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py:96 has:

connector = Connector.find(self.url)

it is called for None instead of a real URL.

- content of json file under /var/lib/gofer/messaging/pending/katelloplugin/ attached
Comment 6 Brad Buckingham 2016-04-25 09:41:10 EDT
gofer 2.7 (e.g. 2.7.6) is currently in the Satellite 6.2 repositories; therefore, based on comment 4, moving this bug to ON_QA for verification.
Comment 7 Sachin Ghai 2016-07-21 07:24:19 EDT
On clients I can see goferd 2.7.6 package is available:

[root@rhel7 ~]# rpm -qa | grep gofer
gofer-2.7.6-1.el7sat.noarch
python-gofer-proton-2.7.6-1.el7sat.noarch
python-gofer-2.7.6-1.el7sat.noarch


I can install packages on client from sat6 webUI without any Issue. I restarted it couple of times and didn't see any issue.

[root@rhel7 ~]# systemctl status goferd.service
● goferd.service - Gofer Agent
   Loaded: loaded (/usr/lib/systemd/system/goferd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-07-21 10:19:40 UTC; 1h 4min ago
 Main PID: 1194 (python)
   CGroup: /system.slice/goferd.service
           └─1194 python /usr/bin/goferd --foreground

Jul 21 10:19:46 rhel7.idmqe.lab.eng.bos.redhat.com goferd[1194]: [INFO][pulp.agent.21389180-23a9-442d-8851-af37f5020d86] gofer.messaging.adap...m:5647
Jul 21 10:19:46 rhel7.idmqe.lab.eng.bos.redhat.com goferd[1194]: [INFO][pulp.agent.21389180-23a9-442d-8851-af37f5020d86] gofer.messaging.adap...m:5647
Comment 8 Sachin Ghai 2016-07-21 08:11:41 EDT
Hey Pavel,

I don't see /var/lib/gofer/messaging/pending/katelloplugin on capsule as you mentioned in steps to reproduce. However its available at client. Could you please confirm ?
Comment 9 Pavel Moravec 2016-07-23 04:22:47 EDT
(In reply to Sachin Ghai from comment #7)
> On clients I can see goferd 2.7.6 package is available:
> 
> [root@rhel7 ~]# rpm -qa | grep gofer
> gofer-2.7.6-1.el7sat.noarch
> python-gofer-proton-2.7.6-1.el7sat.noarch
> python-gofer-2.7.6-1.el7sat.noarch
> 
> 
> I can install packages on client from sat6 webUI without any Issue. I
> restarted it couple of times and didn't see any issue.

Did you try:
1) Having older goferd version (some pre-2.6.6-2 version) on client
2) Issue package/errata install
3) Monitor /var/lib/gofer/messaging/pending/katelloplugin directory on the client.
4) Once the directory gets a json file, stop goferd and upgrade it.
5) Try to start it (having the json file in the dir).

(In reply to Sachin Ghai from comment #8)
> Hey Pavel,
> 
> I don't see /var/lib/gofer/messaging/pending/katelloplugin on capsule as you
> mentioned in steps to reproduce. However its available at client. Could you
> please confirm ?

The directory needs to be also on capsule since goferd stores pending tasks there in either way (Capsule is also a Content Host). Try to invoke capsule sync for a new capsule / newly added to some environment, and check that dir after a while.

Anyway, trying the reproducer on current goferd version from 6.1.9, I cant reproduce - seems the bug has been fixed meantime. Let try the scenario above on a content host during an upgrade to 6.2, only.
Comment 10 Bryan Kearney 2016-07-27 07:02:54 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Note You need to log in before you can comment on or make changes to this bug.