Bug 986070

Summary:	Pulp causes MongoDB to segfault
Product:	[Retired] Pulp	Reporter:	Jim <jgreenleaf>
Component:	z_other	Assignee:	Sayli Karmarkar <skarmark>
Status:	CLOSED NOTABUG	QA Contact:	Preethi Thomas <pthomas>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	2.1.1	CC:	cperry, jgreenleaf
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-24 15:52:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jim 2013-07-18 21:58:12 UTC

Description of problem:

Pulp causes mongodb to segfault


Version-Release number of selected component (if applicable):

2.1.3-1


mongodb-2.2.3-4
pymongo-2.1.1-1


How reproducible:


Steps to Reproduce:

# service mongod status
mongod dead but pid file exists
# service mongod restart
Stopping mongod:                                           [FAILED]
Starting mongod:                                           [  OK  ]
# service mongod status
mongod (pid  30930) is running...
# pulp-manage-db 
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 4
Migration package pulp_rpm.migrations is up to date at version 7
Database migrations complete.
Loading content types.
Content types loaded.
# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
# service mongod status
mongod dead but pid file exists

This has only happened since we upgraded to 2.1.3-1 --- it started happening intermittently on our servers, and one of them has gotten into the state wherein it does this every time

Actual results:

from /var/log/mongodb/mongodb.log

Thu Jul 18 16:46:07 [conn1] build index pulp_database.units_yum_repo_metadata_file { data_type: 1 }
Thu Jul 18 16:46:07 [conn1] build index done.  scanned 4 total records. 0 secs
Thu Jul 18 16:46:12 Invalid access at address: 0 from thread: conn3

Thu Jul 18 16:46:12 Got signal: 11 (Segmentation fault).

Thu Jul 18 16:46:12 Backtrace:
0x97d6b6 0x548f8d 0x5492f7 0x3a4140f4a0 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x97d6b6]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x39d) [0x548f8d]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x247) [0x5492f7]
 /lib64/libpthread.so.0() [0x3a4140f4a0]



Expected results:

No SEGFAULT

Additional info:

Notice the address that it is trying to access.... 

It looks like a threading problem.

Comment 1 Jim 2013-07-23 18:10:36 UTC

Here are the pulp.log logs from a single instance of this bug:

2013-07-23 13:06:05,070 pulp.server.db.connection:INFO: Attempting Database connection with seeds = localhost:27017
2013-07-23 13:06:05,071 pulp.server.db.connection:INFO: Database connection established with: seeds = localhost:27017, name = pulp_database
2013-07-23 13:06:05,385 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/yum_distributor/yum_distributor.conf
2013-07-23 13:06:05,386 pulp.plugins.loader.manager:INFO: Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group,yum_repo_metadata_file
2013-07-23 13:06:05,388 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/iso_distributor/iso_distributor.conf
2013-07-23 13:06:05,388 pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2013-07-23 13:06:05,389 pulp.plugins.loader.loading:INFO: Cannot load plugin: Groupdistributor has no module: yum_distributor.groupdistributor
2013-07-23 13:06:05,389 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/distributors/iso_distributor/iso_distributor.conf
2013-07-23 13:06:05,390 pulp.plugins.loader.manager:INFO: Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
2013-07-23 13:06:05,391 pulp.plugins.loader.loading:INFO: Cannot load plugin: Groupimporter has no module: yum_importer.groupimporter
2013-07-23 13:06:05,407 pulp.plugins.loader.loading:INFO: Loading config file: /usr/lib/pulp/plugins/importers/yum_importer/yum_importer.conf
2013-07-23 13:06:05,408 pulp.plugins.loader.manager:INFO: Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file
2013-07-23 13:06:05,421 pulp.plugins.loader.manager:INFO: Loaded plugin rpm_pkg_profiler for types: rpm
2013-07-23 13:06:05,423 pulp.plugins.loader.manager:INFO: Loaded plugin rpm_errata_profiler for types: erratum
2013-07-23 13:06:05,428 pulp.plugins.loader.manager:INFO: Loaded plugin nodes_http_distributor for types: node
2013-07-23 13:06:05,431 pulp.plugins.loader.manager:INFO: Loaded plugin iso_distributor for types: iso
2013-07-23 13:06:05,434 pulp.plugins.loader.manager:INFO: Loaded plugin iso_importer for types: iso
2013-07-23 13:06:05,860 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 2
2013-07-23 13:06:05,860 pulp.server.dispatch.task:ERROR: connection closed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association.py", line 351, in unassociate_by_criteria
    unassociate_units = association_query_manager.get_units(repo_id, criteria=criteria)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association_query.py", line 116, in get_units
    return self.get_units_by_type(repo_id, type_id, criteria=criteria)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/unit_association_query.py", line 331, in get_units_by_type
    units = list(cursor)
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 703, in next
    if len(self.__data) or self._refresh():
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 666, in _refresh
    self.__uuid_subtype))
  File "/usr/lib64/python2.6/site-packages/pymongo/cursor.py", line 616, in __send_message
    **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 889, in _send_message_with_response
    raise AutoReconnect(str(e))
AutoReconnect: connection closed
2013-07-23 13:06:05,920 pulp.server.dispatch.task:INFO: FAILURE: Task 69458161-aec6-42bc-94af-60ed3f2ee85a: CallRequest: RepoUnitAssociationManager.unassociate_by_criteria(u'bt_laser', {'unit_sort': None, 
'remove_duplicates': False, 'skip': None, 'association_fields': None, 'association_filters': {}, '_id': ObjectId('51e85dc1ded29455e1000020'), 'id': '51e85dc1ded29455e1000020', 'unit_filters': {'$and': [{'n
ame': {'$regex': 'bt_laser'}}, {'version': {'$regex': '1931'}}, {'release': {'$regex': '2'}}], '_id': {'$in': [u'6f6de4a7-5fd8-49bd-810b-4e8edbc9456a', u'8fccbca1-1f18-4d24-b3f2-1c6786100c24', u'c8233bcf-7c4f-4e71-add0-5bd2d2c9ef99', u'60bcd6a1-3b78-44dc-8b89-d3ac415bdaf8', u'b0b68ffd-c3a2-4ecb-b0d3-71870b24d408', u'2832aa64-64e5-4efd-b0d8-efcba28b025f', u'4030f0ad-449c-4e42-970c-1ead4ea7b6d8', u'442b3953-c
5ae-4721-a1d6-f299dfb49d01', u'bbd21514-8623-430b-b563-6209d14b45d8', u'd350f79f-30cc-4151-8a35-c18b898e7834', u'ca5f87b4-e7fc-4e85-adbb-f26b18e7ade8', u'162f59dd-043c-4127-b041-105bd1735747', u'9532160e-2
c06-41a0-a0e1-fd55fc19f3fd', u'acc7eccc-646e-448d-9fa2-53fca60e1fc7', u'c63ea245-6e9b-4822-99a6-b73c8d537557', u'f03993cd-ad8d-4766-afbf-2e26ba99b400', u'3c6c0481-e43b-482c-b363-ace75d3a58ea', u'8f25b45f-f
5c7-491c-8463-a3e3e4e53872', u'5027dd46-1a94-49be-b14e-f166a6e7e6f7', u'4c262336-45f0-4080-a07e-8154f012a479', u'96872a9b-9a49-48c5-9d5c-1fea6c46fb26', u'edc77973-fff2-4118-bc4f-220d4b6966b2', u'eccb2ac2-1
0bd-4904-95e5-f780777916bb', u'da0de168-089c-4451-99aa-e6108a88428f', u'fb3750d2-a493-4279-9cf7-efb8e763eda8', u'cd9c57c0-9c38-416a-853b-603ffcbb368b', u'd6935cec-a973-48dd-8d68-ac913c2cfcf1', u'2defd4c7-8
5d4-4763-9365-1bf2a857b9c4', u'ace7146e-d003-444b-9ecb-7a71c4577ca9', u'a82d7256-7ff2-4080-9ecf-2360354a9d81', u'65fd97f9-e057-4597-a18f-71ce0abf2bca', u'944672bb-5364-42da-830e-b00a68bdc3c5', u'4b96e374-b003-45df-bd0e-51c07bce2cb5', u'26bcaffc-a911-4df5-bf0f-41f411b06f73', u'651caf6e-c7fb-488f-ac1e-007be35b41bc', u'49a12744-6116-45a2-acff-20d299e9169e', u'5e438169-d913-4b1f-8eb2-e4bd6cf8e17d', u'dc9ce60d-3dbd-413f-9adc-95113767b7bf', u'd0dacb56-776e-4b07-a6c2-f99a32ca874c', u'95a5327d-354d-449d-9342-c1afa35ea5bb', u'c31904e9-6873-4204-8973-a0e7db85de29', u'fb582850-5df5-4f06-bd7c-a61164d8ddae', u'35c4d80c-b1e7-438e-80b2-0f0a0495d733', u'90e3fa09-1744-41b4-9434-fa5d8d63859b', u'51242e5d-12cf-4942-833f-9190e05239c6', u'1aa4135d-0ec2-42c3-8ee0-8f8f9bef723d', u'56119dad-ccd3-4be1-adeb-876ce885530b', u'f92f4feb-da7c-4c8e-b2c7-d32f8dcd5e31', u'679064d5-5ba2-497e-9d1e-31da35d46efe', u'9eb211bf-7a0d-49a7-b34b-28f7e46b3b6f', u'1a579aa7-258c-4073-9b4a-325560f6f7ab', u'0b6c6730-089f-4095-9a4f-4b04ae5530a4', u'c67fc903-d44a-41a4-8b88-c8ddc0ff6f7b', u'ae34f77b-9b37-4422-9b68-7781a3ed59c2', u'56947e90-0ba0-4b87-a28c-f0fec66a1807', u'bd308ade-9226-4e2d-98ba-01610d2f9f20', u'6fb017de-03e3-446b-af43-20fdf507e775', u'18c0ccd1-1c75-4f80-8e11-8efcbed5e580', u'5bb2493d-58e1-4f4d-b5fd-4437309f73d2', u'91dc2350-a5f2-4a94-af40-fef4f8d030d7', u'1beb1214-c640-43d9-8137-879fc011f5f4', u'11512444-f46b-4077-a618-296fe4ed8ab5', u'0e3c8d6f-f6b4-4f6e-8e08-a23aabdbadc8', u'2e282097-9481-4d81-9405-045d1e9bcb78', u'dfb818bf-2181-40b8-ab62-77446f8c7cfc', u'7324b02e-a7cd-446f-8695-2e5fe0f50a38', u'7e067ca5-f528-4ffb-bd40-3ab114f12cff', u'a4b7f358-44b6-4ba3-858c-6a4017783095', u'fd414505-b815-452c-b2fd-b066fc43da90', u'9aa3791e-8be2-4981-b9d9-61d70e3ae7a4', u'76bc8e7a-49ec-4d81-b2ff-82343beb157c', u'831c6956-8af3-4087-b568-352ef8919f72', u'cf662c4a-2eff-423d-acc7-408c2d520878', u'6d61c1f0-f360-4ece-bde7-24aaaa0eabbe', u'092adddf-ac3d-44b7-9751-eb92c9bd06ae', u'9adb998d-d68e-4ad6-8e8a-0c415714c4ca', u'1cc6a494-34cb-4674-ba29-64fe213f0352', u'5e160bc3-852b-4484-afe1-53d89672d11c', u'e7e6294d-2104-4e3b-82bf-0a9625673aba', u'5ca2ae25-db0d-40b5-af71-54e51cf29b12', u'1d95899e-7150-4efe-b5ff-6ff1c17247a1', u'a08b43be-8555-4e4c-8af3-9593b0332142', u'476d1c1c-d568-4a1b-972c-65500237fda6', u'dae10292-e6d4-45fb-8197-5dfbc5a83ddc', u'9f642b97-4586-4a44-95e1-2d949f947bf5', u'20304f3d-a4d5-4149-aeb7-0f9e96a8da92', u'9559fe3b-6de9-4118-ae26-cf3a43181f4a', u'259efef1-68f0-4147-9ce8-88f001863e91', u'4906473e-1920-4709-8c12-72f6166c2ff0', u'27a617de-91ae-4557-880a-26a99e8db401', u'06b6a8fe-adb2-46bd-b638-daa2da1bc294', u'7b99135b-2983-4a57-a2a4-617dc7e2fccc', u'56ca64ba-ae86-4940-905b-c7c716539b1b', u'685f0995-6984-495f-8592-195dfef5c2e7', u'3a31d599-9905-434d-bd08-dda983bd8958', u'5481666c-40ab-40b4-b76b-5e98d3f0f7b6', u'cf08a540-b66a-4956-b499-1b5746687722', u'd43d372a-39b6-4221-9410-a8ba1a45170f', u'7aca9e21-8f4c-425c-9df3-afb9174f13f2', u'9b29aa7b-5368-44bb-a4c1-02a7a006f0f0', u'e303ebe5-38f1-4256-a38d-8bf36e9109e9', u'55c7826f-0c63-4221-aa92-a1985eb6271d', u'df4255b2-9a51-4257-b87a-27b18f288a4f', u'ce7e9a34-067e-4998-9246-4908c859b9e5', u'26739561-12f5-4fdc-83c6-f04730486615', u'99d2fd90-405d-4365-a5be-f592388a4f8b', u'4f189005-94cc-477c-8bdc-0547e0100dfd', u'2054ae7c-b6c8-4865-b3a5-dfff3ac0cc5e', u'eca11179-a9e4-4f4d-afcf-468fbf5839a3', u'fda3b488-eb52-473c-96f3-67b3bf09a2b6', u'87b08f41-1fa0-4e40-9b47-1ec3595e1c4f', u'c1ff516d-3751-4be6-a217-8272e00ab751', u'd98d06c4-420d-4446-9f2a-fb8be1ff4abb', u'02947069-fe93-4715-85a1-052ead880cd7', u'14dfe251-9ee3-4273-953f-c9f1902d618c', u'b946cdfa-a8fd-44b9-9ca0-05149f640b3d', u'cc43fd84-4da4-43eb-a91e-c4c5eb353956', u'ab885465-b745-4197-827b-010c9d2368b9', u'20e30159-d571-428b-af02-50ecb940b9e6', u'ed50c420-058a-4d96-8e5f-3e18c85d389d', u'a0f459bc-08cc-49bf-af86-12333d788714', u'ad1908a2-b3e7-411a-a6f7-e7da3985c196', u'92f2280b-707f-4948-9f02-e80f327e4cf4', u'9bc0bd2f-76e4-4151-ab72-17630fd355aa', u'cfd0ecd4-4fbc-4c4d-9047-7e0c846679be', u'7936f61f-3c80-4cc4-b939-8ff8d8a15508', u'7853cd6d-2374-421e-8589-f5abddece470', u'2a074ee4-8e56-436c-9134-ddf22f966b89', u'6366960d-8925-417e-8956-ff075263d546', u'a800db20-a90d-4c8e-83e7-565a6de7769b', u'768a42db-bfde-4976-85c0-31c2d0f34622', u'7fab452c-b8b7-4324-97ca-a0560b9263cc', u'9da7e836-b000-405f-80fd-f56fa47c994c', u'5e207563-2126-4197-a961-a13aa15230ab', u'5e0c3a1f-de1d-4a92-ad69-cb362f2c8b3d', u'b0476e52-91e4-46e7-b9a2-900a32d616f5', u'fdea5723-cfcd-4224-a1e0-6bd0190b5b5d', u'8bdcea2e-7abf-46b7-bdd6-35495d5682a8', u'e128aeca-fdec-49b2-95a5-b6e0d3973930', u'44152ff6-0849-46d9-b6df-e4542c385b8e', u'2fe04e45-27cb-4357-bda3-b6c38b1c670c', u'89f41bbe-1045-4319-b429-486561f4e490', u'2002eda5-88b9-44b2-8436-54f639b1f03d', u'75c07d98-f9b5-40fb-82d9-98c81740b10a', u'27351522-598b-4055-88f9-af4b05148180', u'd1404bc4-5f46-4c31-bf99-dcd755044bcb', u'5ea4a9bd-fc22-4a93-8a90-597bc07f3183', u'4ef5a4db-380c-42b9-b9f7-8ed2d690caa7', u'ae8a1f3a-b88d-4679-9cd2-4f4a10990494', u'6e5d2480-80b8-4cdd-8b28-5c76b4526ede', u'ed871c4e-b679-4ac2-834f-081a4b29384d', u'a5ab2f5d-68aa-4d9c-b80e-94d49f081933', u'e3355abc-6bad-4add-af0a-2748f4bdcb6b', u'56fab946-5241-41de-a4b2-7277cf031da1', u'dd21e395-37d1-45d7-9ed4-e0ef40b9dd55', u'797afa0a-6334-4299-a96e-d99a567aa8ca', u'38bce226-6ac4-4e7a-b0cc-2faf2d0e1917', u'6d2fc565-c3d7-4e25-bbd6-03249c334404', u'396b507a-7143-4ada-983d-4f165e5bc34b', u'8c4a004f-3162-4fb0-a3ca-2f981d438304', u'0ae93500-b3ba-4466-b774-c29e0074f2fa', u'18f3def9-aee9-403e-8524-0087a6e78bcd', u'ba56facb-4b24-4487-9123-62249bb3f080', u'610bd2e0-6265-40e5-8de3-d0ea301a9d75', u'8721c5e1-1946-46fa-93d1-7004d612ad43', u'94d43b75-24de-41ee-97d2-e5118d996ff7', u'274cd031-4713-4f41-ae70-f65772db800d', u'2eac516a-1b22-4dae-9561-28bde94c5ed0', u'e563ba57-a6e2-4ab2-8a22-7641c1e26219', u'9bb37188-9f86-4f43-8d52-2f9e71f55004', u'70c6d66b-7d9e-457d-aeb9-42fc82fd52f1', u'f2160315-df71-4fe7-93d4-b0ee20b9223c', u'd3d04681-de98-4953-b7f3-86fe15b519b1', u'a42a3713-4911-4e4c-a3ef-7a335a876459', u'dea33e81-511b-4830-9bdc-7f84a2d5130c', u'952bcb1a-66dd-44c1-be8f-b7744d1f06d4', u'a95861c7-3488-48f2-ae86-eac01f56dd6c', u'977709fb-cced-47a9-844e-4750bc63aea9', u'6a3912fb-4e44-408c-a366-ca32c14d669b', u'f1e6cfab-2a13-4956-8db6-07475b36907d', u'ebefd2aa-924a-4feb-ba97-e1ac19d3f0c4', u'4a5ad562-5410-4fd6-942c-e45ff8a72e61', u'4e113d0f-aa9a-4535-acfb-52080565bf54', u'd69f14d0-224c-4d15-a6dc-711a12f15a0e', u'175c5fec-5f5a-4c29-a7eb-ba8af145756c', u'81be605e-8f7d-4a32-bf6f-5b9479418ca9', u'bb8f589b-1c5d-4e3c-94e1-9daa4d2db11f', u'5990a3cb-4002-4743-a0af-4b1dff80852f', u'46bfc2eb-f581-4a0a-beb5-f01ed90deab0', u'5d157bb3-5104-4119-9128-6b85431c02d0', u'b0e9fcf1-f395-4100-91f7-03596c1fd2f0', u'eea31269-50f8-47fa-ad27-2bd09f84ef95', u'6a25475e-d004-4e6c-94f8-d976926a12f6', u'4ab5a0a2-342e-4b2a-879a-5f7419d636d9', u'559ce8cb-bc6d-45c4-8ab2-48798f46f662', u'bff65156-0202-49e0-9e8e-1debb476c06c', u'814f98bd-028c-403f-baba-ada253d647c6', u'6c145392-2a7a-4787-89d2-5362593ee0fc', u'6998116d-9607-4a14-b2ed-0c67f6327a79', u'82c6f3c2-0fac-4a9c-9391-5432ce6c0ff6', u'fb659adf-386a-4124-8825-0f93e0a27ded', u'34c3de92-adb8-4f1f-8429-c5f6a9a78d25', u'a13402be-b05b-4688-8499-a6551c6e75c6', u'a21ec383-1f35-4393-9132-c4024462952e', u'b84a80b1-72c1-4a37-8155-91dd6bfda3f8', u'a8820e07-7579-4acb-8451-c478cd393302', u'48e06204-482e-444d-b74d-101ad83af1f6', u'f965d577-01d9-421c-95ae-ad0d7194aaa4', u'113d7961-af88-4069-a8b4-779af7753d9b', u'1b305b37-fcab-4583-8bac-80b6707b6f0d', u'4e615577-4b78-4cb9-be20-61cb52f558ba', u'2f1da237-44db-4f6c-b854-90f296d94de9', u'2c7e6c0e-a422-4707-9f48-64b5f376d93e', u'1c7f26b2-1347-458a-a6f5-fb861bf27e6a', u'322836e3-3f74-436e-bf3b-c3869760c8de', u'3dd2bf18-c71e-426c-b4a9-1d6562d4e39e', u'b40146d6-41f0-4484-b66a-4a98086da96e', u'066e1fda-b601-4556-b131-2155d841b09f', u'310e1e32-0b23-46b0-99d7-9f05983304d4', u'5f27a8f4-12ed-460d-8b36-763ac9d5955e', u'a0f88873-e7ec-4e55-8c97-943e7619b772', u'38209a1d-6a00-40ec-809c-613d866343cb', u'3e4fa0a1-8dd8-4b6e-bf40-78e4cb0caea5', u'3434a0d0-767c-428d-b9a1-9f62365d04f4', u'8fff0c17-0d2a-4262-aa8c-f829029c2739', u'7cdcb317-5d76-401f-8a96-3d14b2a1a0ee', u'75bb558f-eb92-460d-93c9-90b187774727', u'e1c927f7-32e0-4c3e-9701-06ea7777ebe6', u'82bd04b4-7357-46bd-b50c-76b7c030060b', u'211ffec3-4834-471e-a274-02b964cb8c6d', u'a17d5393-33ca-44d0-a3c0-eb5da33ec05c', u'f85bb8d7-df88-4eb9-af48-76d6e38ac880', u'999fc680-3a5f-4c28-a0b1-8cf768543b2c', u'e0154386-1301-4e80-a865-b6430b13f690', u'db52b6c7-d7ed-48fa-a05c-29a4f50b4397', u'193f5acb-1975-46b1-9d0a-6a9bff3a8fb3', u'628d13b8-f03e-4bcc-926e-36d61a4a36da', u'8cb6ed10-fe29-49a4-9982-6050bbec2733', u'aad0acb4-900f-450b-a16d-6bfa73081920', u'43cf5cce-c2d2-4ef1-883b-b2999c36dad5', u'bf9eaec8-0c02-4134-baa8-279b25d6f03b', u'c746fd59-2df3-4c41-81fc-b477c2db1069', u'ae48f3a0-7bf5-461a-a79c-cfd2fa9c67a6', u'15e859c4-58ba-45b1-a6cb-70841b4e2b20', u'8cea2d89-3ab7-4148-be81-413da53ab677', u'f6a5baa0-aef2-4594-b033-2fb7645db7ce', u'0b8b44e1-541d-49e2-8171-296671e14287', u'dbd16178-1fe7-437e-8ba4-b21caf400fef', u'ceba9fa2-6572-4c60-b676-2111c0579ece', u'e763a476-126d-4393-becb-a6d5d4db769a', u'57b29f57-c362-4f09-ba2f-29d290c04fda', u'5da077e1-4379-4288-bb3e-c9ceafbec853', u'355b7af6-0471-472d-b001-6d9fe0be16f6', u'50d1b1fa-0b1c-4ee9-ae4b-0903f13ea01d', u'6610a878-04dc-44ea-b8a0-4abfa166d177', u'3cb490c5-230c-4b15-88f5-da72318c2209', u'a0905a9e-a6e7-4ecd-9859-db511e10591e', u'3a7068bd-ca9f-4cb3-b16f-cf97e205e829', u'ffd23479-7ddd-47e9-9a42-0700c20df20e', u'513bea1b-2bd0-4b2c-8a08-acfe38d7afd3', u'4737f1db-f95c-4032-89f8-c308564d42e3', u'377c2400-4b49-4631-acd2-793856b54c21', u'36207071-9132-42a0-bcb7-967737c000e8', u'41f5f48e-27a2-4f0d-93e8-4937384ca537', u'476ae611-8bab-45b5-8206-1897b3aa19ae', u'bddb9c46-2d15-484b-a8bc-9c420b6f3910', u'd1092dd9-48ab-4767-aff4-27f24c2a1177', u'942c1642-1941-43c2-b4f4-eca17a9eec29', u'c65cdf6a-f04f-4f07-a717-d1349325a6dc', u'c62fd484-558f-492b-a18d-35025d55c7c4', u'486a0844-4a95-4e77-b720-d6754aadf8c0', u'3195b352-8a38-4abb-b34b-13a710ba17b1', u'963aaa8a-13a2-4d12-93ce-db677b8a0406', u'46a16e74-4155-4d93-a1ec-e61b891e9380', u'12da86da-2082-4048-bb3e-50f619af34db', u'b4371597-8fef-4384-8931-626ddb1c808a', u'f471732f-73b3-48f7-96f2-32f338a60389', u'e257a292-2dc2-475f-96c1-fdea241f1734', u'5365b100-a54f-4b84-bb8e-4a7f81685dd7', u'3cbfedfb-74ea-4e3f-a76b-1cce81346b9c', u'0b57cc35-cfc9-49c3-8fe1-7ce086bfb915', u'6f54491e-f284-407f-9f7e-fb990970b8a5', u'cb8ed5ed-1510-4280-9163-e4208edcdd04', u'485b9ea4-c785-4aa5-874e-75d1712bf8d6', u'b530b320-a25e-465a-b1b0-a6f9a1b097b5', u'fc0a208f-54e2-4d87-b851-2f9aeaca7cb6', u'438cd708-d177-4685-9177-be1e05bb2a8f', u'ba71e5a6-bd6d-4298-8acb-2e590bca3364', u'575239d5-dccf-4e00-bb1a-a6d73c17d5f3', u'eb740544-f06d-4f69-823f-622c7f4438b9', u'f18bcd51-bf83-4f9f-a34f-f06c439fb2b5', u'7a7e8a65-ae58-4f5f-8f1a-09090373a9fb', u'133caaf8-7a04-4d5d-beb2-5241221d5ec6', u'9e3e44a7-45af-4153-b766-dc327bf7e134', u'8dd46a53-3e57-481a-89c9-b87724280f00', u'1701c3ec-adfe-4dc6-af9c-65996225eb60', u'9f497889-cebb-4946-91e0-1405a011ad90']}}, 'association_sort': None, 'unit_fields': None, 'limit': None, 'type_ids': ['rpm']}, 'user', u'admin')
2013-07-23 13:06:06,161 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 1
2013-07-23 13:06:06,463 pulp.server.db.connection:WARNING: update operation failed on pulp_database.permissions: tries remaining: 0
2013-07-23 13:06:06,463 pulp.server.webservices.application:CRITICAL: *************************************************************
2013-07-23 13:06:06,464 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 2
2013-07-23 13:06:06,464 pulp.server.webservices.application:ERROR: The Pulp server encountered an unexpected failure during initialization
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 177, in wsgi_application
    _initialize_pulp()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 132, in _initialize_pulp
    dispatch_factory.initialize()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/factory.py", line 58, in initialize
    _initialize_coordinator()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/factory.py", line 31, in _initialize_coordinator
    _COORDINATOR.start()
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 80, in start
    self.execute_call_asynchronously(call_request)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 156, in execute_call_asynchronously
    self._process_tasks([task])
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/coordinator.py", line 251, in _process_tasks
    task_queue.enqueue(task)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 209, in enqueue
    task.call_life_cycle_callbacks(dispatch_constants.CALL_ENQUEUE_LIFE_CYCLE_CALLBACK)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 229, in call_life_cycle_callbacks
    hook(self.call_request, self.call_report)
  File "/usr/lib/python2.6/site-packages/pulp/server/auth/authorization.py", line 163, in __call__
    factory.permission_manager().grant(resource, self.user_name, operations)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/auth/permission/cud.py", line 158, in grant
    Permission.get_collection().save(permission, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 237, in save
    manipulate, safe, _check_keys=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 89, in retry
    (method.__name__, self.full_name, (self.retries + 1)))
PulpCollectionFailure: update operation failed on pulp_database.permissions: database connection still down after 3 tries
2013-07-23 13:06:06,757 pulp.server.webservices.application:CRITICAL: *************************************************************
2013-07-23 13:06:06,765 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 1
2013-07-23 13:06:07,067 pulp.server.db.connection:WARNING: remove operation failed on pulp_database.queued_calls: tries remaining: 0
2013-07-23 13:06:07,067 pulp.server.dispatch.task:ERROR: remove operation failed on pulp_database.queued_calls: database connection still down after 3 tries
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 214, in _call_complete_callback
    self.complete_callback(self)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 292, in _complete
    self.dequeue(task)
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/taskqueue.py", line 244, in dequeue
    self.queued_call_collection.remove({'_id': task.queued_call_id}, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 89, in retry
    (method.__name__, self.full_name, (self.retries + 1)))
PulpCollectionFailure: remove operation failed on pulp_database.queued_calls: database connection still down after 3 tries
2013-07-23 13:06:07,070 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 2
2013-07-23 13:06:07,372 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 1
2013-07-23 13:06:07,673 pulp.server.db.connection:WARNING: create_index operation failed on pulp_database.archived_calls: tries remaining: 0

Comment 2 Sayli Karmarkar 2013-07-24 15:52:36 UTC

Not a pulp bug.

Comment 3 Jim 2013-07-25 15:11:04 UTC

For posterity, it wound up being this bug: https://jira.mongodb.org/browse/SERVER-9014