Bug 1027500

Summary: Race condition between tasking and agent servers on pulp startup.
Product: [Retired] Pulp Reporter: Jeff Ortel <jortel>
Component: z_otherAssignee: Jeff Ortel <jortel>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: high Docs Contact:
Priority: high    
Version: MasterCC: jmontleo, pthomas
Target Milestone: ---Keywords: Triaged
Target Release: 2.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-09 14:31:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Ortel 2013-11-06 23:53:46 UTC
In application.py, the agent/direct/services.statup() configures the python-gofer lib AFTER tasking is initialized.  Restarted tasks that message the agent attempt to do so before gofer is configured with SSL certificates.

The result is:

Qpid connection fails and goes into retry loop.

The fix:

Configure the python-gofer lib before starting either the tasking or messaging services.

Comment 1 Jeff Ortel 2013-11-07 00:50:24 UTC
https://github.com/pulp/pulp/pull/707

Comment 3 Jeff Ortel 2013-11-07 15:21:25 UTC
build: 2.3.0-0.29.beta

Comment 4 Jeff Ortel 2013-11-15 15:41:43 UTC
Steps to reproduce:

1. Have pulp installed with qpid configured for SSL.
2. Have a consumer registered.
3. Shutdown qpidd
4. Using pulp-admin, install a package on the consumer.  The task will not complete because the broker is down.
5. Stop httpd.
6. Start qpidd.
7. Start httpd.
8. Note in pulp.log that when the package install task is restarted, you dont see gofer/qpid connection errors in the log and the install task completes.
9. Go have a beer ... you've earned it.

Comment 5 Preethi Thomas 2013-11-15 20:11:25 UTC
verified

followed the steps above

[root@pulp-v2-server ~]# service qpidd stop
Stopping Qpid AMQP daemon:                                 [  OK  ]

[root@pulp-v2-server ~]# pulp-admin rpm consumer package install run -n cat --consumer-id consumer1
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

^C[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  agent_bind
Resources:   consumer1 (consumer), zoo (repository), yum_distributor
             (repository_distributor)
State:       Successful
Start Time:  2013-11-15T20:01:57Z
Finish Time: 2013-11-15T20:01:57Z
Result:      
  Details:     
    Yum Distributor: 
      Details:   
      Repo Id:   zoo
      Succeeded: True
  Num Changes: 0
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     c5537e28-eed6-40c4-a7d9-e182122f207a

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:04:01Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# service qpidd start
Starting Qpid AMQP daemon:                                 [  OK  ]
[root@pulp-v2-server ~]# service httpd start
Starting httpd:                                            [  OK  ]



[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  profile_create
Resources:   consumer1 (consumer), rpm (content_unit)
State:       Successful
Start Time:  2013-11-15T20:08:35Z
Finish Time: 2013-11-15T20:08:36Z
Result:      ****
Task Id:     b6e8bb12-16b9-4251-9eb0-38854e010a8e

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:06:29Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# 


[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Successful
Start Time:  2013-11-15T20:09:12Z
Finish Time: 2013-11-15T20:09:16Z
Result:      
  Details:     
    Rpm: 
      Details:   
        Deps:     
        Errors:   
        Resolved: 
          Arch:    noarch
          Epoch:   0
          Name:    cat
          Qname:   cat-1.0-1.noarch
          Release: 1
          Repoid:  zoo
          Version: 1.0
      Succeeded: True
  Num Changes: 1
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]#

Comment 6 Preethi Thomas 2013-12-09 14:31:01 UTC
Pulp 2.3 released.