Bug 1104320

Created attachment 901871 [details]
Screen log of immediate repo sync call

Description of problem:
Currently, pulp allows requests even though the pulp_worker-*.services are not 
fully operational.
A related scenario is a repo sync request right after a pulp_worker restart, where pulp_database.available queues entries get lost from mongo and stale reservation appears in the pulp_database.reserved_resources collection.

Version-Release number of selected component (if applicable):
- Fedora 20 (systemd)
- pulp-server-2.4.0-0.19.beta.fc20.noarch

How reproducible:
~100% (on my system)

Steps to Reproduce:
1. create a rpm repo (via pulp-admin)
   pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed \
   http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
2. systemctl restart pulp_worker-{0..8}.service ; \
   pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo


Actual results:
- a task queue lost even though the both the pulp_worker processes are up and running
- stale resource reservation item

  [root@ec2-79-125-97-56 ~]# fg
  mongo
  > db.available_queues.find()
  { "_id" : "reserved_resource_worker-0.compute.amazonaws.com", "last_heartbeat" : ISODate("2014-06-03T17:07:26.060Z"), "num_reservations" : 0 }
  > db.reserved_resources.find()
  { "_id" : "repository:zoo", "assigned_queue" : null, "num_reservations" : 1 }
  > db.task_status.find()
  > db.available_queues.find()
  { "_id" : "reserved_resource_worker-0.compute.amazonaws.com", "last_heartbeat" : ISODate("2014-06-03T17:08:08.281Z"), "num_reservations" : 0 }
  >

Expected results:
503 Service Unavailable Response until the pulp_worker-*.service prepare all necessary resources
i.e. Pulp server (wsgi) doesn't allow requests while pulp_worker-*.service aren't fully operational 

Additional info: