Bug 1027500 - Race condition between tasking and agent servers on pulp startup.
Race condition between tasking and agent servers on pulp startup.
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: z_other (Show other bugs)
Master
Unspecified Unspecified
high Severity high
: ---
: 2.3.0
Assigned To: Jeff Ortel
Preethi Thomas
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-06 18:53 EST by Jeff Ortel
Modified: 2013-12-09 09:31 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-09 09:31:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeff Ortel 2013-11-06 18:53:46 EST
In application.py, the agent/direct/services.statup() configures the python-gofer lib AFTER tasking is initialized.  Restarted tasks that message the agent attempt to do so before gofer is configured with SSL certificates.

The result is:

Qpid connection fails and goes into retry loop.

The fix:

Configure the python-gofer lib before starting either the tasking or messaging services.
Comment 1 Jeff Ortel 2013-11-06 19:50:24 EST
https://github.com/pulp/pulp/pull/707
Comment 3 Jeff Ortel 2013-11-07 10:21:25 EST
build: 2.3.0-0.29.beta
Comment 4 Jeff Ortel 2013-11-15 10:41:43 EST
Steps to reproduce:

1. Have pulp installed with qpid configured for SSL.
2. Have a consumer registered.
3. Shutdown qpidd
4. Using pulp-admin, install a package on the consumer.  The task will not complete because the broker is down.
5. Stop httpd.
6. Start qpidd.
7. Start httpd.
8. Note in pulp.log that when the package install task is restarted, you dont see gofer/qpid connection errors in the log and the install task completes.
9. Go have a beer ... you've earned it.
Comment 5 Preethi Thomas 2013-11-15 15:11:25 EST
verified

followed the steps above

[root@pulp-v2-server ~]# service qpidd stop
Stopping Qpid AMQP daemon:                                 [  OK  ]

[root@pulp-v2-server ~]# pulp-admin rpm consumer package install run -n cat --consumer-id consumer1
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

^C[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  agent_bind
Resources:   consumer1 (consumer), zoo (repository), yum_distributor
             (repository_distributor)
State:       Successful
Start Time:  2013-11-15T20:01:57Z
Finish Time: 2013-11-15T20:01:57Z
Result:      
  Details:     
    Yum Distributor: 
      Details:   
      Repo Id:   zoo
      Succeeded: True
  Num Changes: 0
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     c5537e28-eed6-40c4-a7d9-e182122f207a

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:04:01Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# service qpidd start
Starting Qpid AMQP daemon:                                 [  OK  ]
[root@pulp-v2-server ~]# service httpd start
Starting httpd:                                            [  OK  ]



[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  profile_create
Resources:   consumer1 (consumer), rpm (content_unit)
State:       Successful
Start Time:  2013-11-15T20:08:35Z
Finish Time: 2013-11-15T20:08:36Z
Result:      ****
Task Id:     b6e8bb12-16b9-4251-9eb0-38854e010a8e

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:06:29Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# 


[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Successful
Start Time:  2013-11-15T20:09:12Z
Finish Time: 2013-11-15T20:09:16Z
Result:      
  Details:     
    Rpm: 
      Details:   
        Deps:     
        Errors:   
        Resolved: 
          Arch:    noarch
          Epoch:   0
          Name:    cat
          Qname:   cat-1.0-1.noarch
          Release: 1
          Repoid:  zoo
          Version: 1.0
      Succeeded: True
  Num Changes: 1
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]#
Comment 6 Preethi Thomas 2013-12-09 09:31:01 EST
Pulp 2.3 released.

Note You need to log in before you can comment on or make changes to this bug.