Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours

Bug 1027500

Summary: Race condition between tasking and agent servers on pulp startup.
Product: [Retired] Pulp Reporter: Jeff Ortel <jortel>
Component: z_otherAssignee: Jeff Ortel <jortel>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: high Docs Contact:
Priority: high    
Version: MasterCC: jmontleo, pthomas
Target Milestone: ---Keywords: Triaged
Target Release: 2.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-09 09:31:01 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Jeff Ortel 2013-11-06 18:53:46 EST
In application.py, the agent/direct/services.statup() configures the python-gofer lib AFTER tasking is initialized.  Restarted tasks that message the agent attempt to do so before gofer is configured with SSL certificates.

The result is:

Qpid connection fails and goes into retry loop.

The fix:

Configure the python-gofer lib before starting either the tasking or messaging services.
Comment 1 Jeff Ortel 2013-11-06 19:50:24 EST
https://github.com/pulp/pulp/pull/707
Comment 3 Jeff Ortel 2013-11-07 10:21:25 EST
build: 2.3.0-0.29.beta
Comment 4 Jeff Ortel 2013-11-15 10:41:43 EST
Steps to reproduce:

1. Have pulp installed with qpid configured for SSL.
2. Have a consumer registered.
3. Shutdown qpidd
4. Using pulp-admin, install a package on the consumer.  The task will not complete because the broker is down.
5. Stop httpd.
6. Start qpidd.
7. Start httpd.
8. Note in pulp.log that when the package install task is restarted, you dont see gofer/qpid connection errors in the log and the install task completes.
9. Go have a beer ... you've earned it.
Comment 5 Preethi Thomas 2013-11-15 15:11:25 EST
verified

followed the steps above

[root@pulp-v2-server ~]# service qpidd stop
Stopping Qpid AMQP daemon:                                 [  OK  ]

[root@pulp-v2-server ~]# pulp-admin rpm consumer package install run -n cat --consumer-id consumer1
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...

^C[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  agent_bind
Resources:   consumer1 (consumer), zoo (repository), yum_distributor
             (repository_distributor)
State:       Successful
Start Time:  2013-11-15T20:01:57Z
Finish Time: 2013-11-15T20:01:57Z
Result:      
  Details:     
    Yum Distributor: 
      Details:   
      Repo Id:   zoo
      Succeeded: True
  Num Changes: 0
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     c5537e28-eed6-40c4-a7d9-e182122f207a

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:04:01Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# service qpidd start
Starting Qpid AMQP daemon:                                 [  OK  ]
[root@pulp-v2-server ~]# service httpd start
Starting httpd:                                            [  OK  ]



[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  profile_create
Resources:   consumer1 (consumer), rpm (content_unit)
State:       Successful
Start Time:  2013-11-15T20:08:35Z
Finish Time: 2013-11-15T20:08:36Z
Result:      ****
Task Id:     b6e8bb12-16b9-4251-9eb0-38854e010a8e

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Running
Start Time:  2013-11-15T20:06:29Z
Finish Time: Incomplete
Result:      Incomplete
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@pulp-v2-server ~]# 


[root@pulp-v2-server ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  unit_install
Resources:   consumer1 (consumer)
State:       Successful
Start Time:  2013-11-15T20:09:12Z
Finish Time: 2013-11-15T20:09:16Z
Result:      
  Details:     
    Rpm: 
      Details:   
        Deps:     
        Errors:   
        Resolved: 
          Arch:    noarch
          Epoch:   0
          Name:    cat
          Qname:   cat-1.0-1.noarch
          Release: 1
          Repoid:  zoo
          Version: 1.0
      Succeeded: True
  Num Changes: 1
  Reboot:      
    Details:   
    Scheduled: False
  Succeeded:   True
Task Id:     88834467-ff50-4933-9390-5419faba6605


[root@pulp-v2-server ~]#
Comment 6 Preethi Thomas 2013-12-09 09:31:01 EST
Pulp 2.3 released.