Bug 1021011

+++ This bug was initially created as a clone of Bug #1019909 +++

Description of problem:
If pulp seeds from a mongodb replica set, and the mongo PRIMARY is re-elected, pulp fails to reconnect to the new PRIMARY.

Version-Release number of selected component (if applicable):
pulp-server-2.2.0-0.20.beta.git.0.d54a854.el6eng.cdn.1.noarch
mongo server buildinfo - 2.4.6 (EPEL)
pymongo-2.1.1-1.el6.x86_64

How reproducible:
very

Steps to Reproduce:
1. have a mongo replica set, where mongodb01.web.stage.our.domain.com is PRIMARY
2. setup pulp to seed from a host in the mongodb replica set
 /etc/pulp/server.conf [database] seeds: mongodb01.web.stage.our.domain.com:27017
3. Start the pulp sever (and pulp-manage-db)(ensure it's functioning correctly)
3. on the mongodb rs, have mongodb01 step down from PRIMARY ( rs.stepDown() )
4. make calls to the pulp server ( `pulp-admin login -u admin -p S3krit` )

Actual results:
   An internal error occurred on the Pulp server. More information can be found in the client log file ~/.pulp/admin.log.
=== START ~/.pulp/admin.log =====
2013-10-16 10:22:45,466 - ERROR - Client-side exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/core.py", line 478, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.6/site-packages/okaara/cli.py", line 974, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/extensions.py", line 224, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/usr/lib/pulp/admin/extensions/pulp_server_info/pulp_cli.py", line 35, in types
    all_types = self.context.server.server_info.get_types()
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server_info.py", line 33, in get_types
    return self.server.GET(path)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 84, in GET
    return self._request('GET', path, queries)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 142, in _request
    self._handle_exceptions(response_code, response_body)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 183, in _handle_exceptions
    raise code_class_mappings[response_code](response_body)
PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/types/ failed with 401 - Pulp exception occurred: AuthenticationFailed
2013-10-16 10:23:05,446 - ERROR - Exception occurred:
        href:      /pulp/api/v2/actions/login/
        method:    POST
        status:    500
        error:     create_index operation failed on pulp2_database.users: database connection still down after 3 tries
        traceback: None
        data:      {u'args': [u'create_index operation failed on pulp2_database.users: database connection still down after 3 tries']}
=== END ~/.pulp/admin.log =====

Expected results:
   Successfully logged in. Session certificate will expire at Oct 23 14:24:33 2013 GMT.

Additional info:
If I can get the original PRIMARY node elected back as PRIMARY, then everything on pulp begins working again.

This is enough of an issue to block us from promoting pulp v2 to production.

--- Additional comment from Michael Hrivnak on 2013-10-18 14:45:48 EDT ---

This is likely a regression. Unless it's particularly inconvenient, I think it makes sense to fix this in 2.2 and get it into our 2.2.1 release.