Clone for Master +++ This bug was initially created as a clone of Bug #876234 +++ Description of problem: If you remove the consumer rpms and consumer related files, Missing resource error appears in the pulp.log Version-Release number of selected component (if applicable): [root@preethi ~]# rpm -qa pulp-rpm-server pulp-rpm-server-2.0.6-0.2.beta.noarch How reproducible: Steps to Reproduce: 1. Install the consumer-client packages and register a client to a pulp server 2. Remove the consumer-client rpms 3. Notice the error in pulp.log Actual results: Expected results: Additional info: From pulp.log 2012-11-13 10:32:08,447 16913:139677436737280: pulp.server.webservices.middleware.exception:ERROR: exception:46 Missing resource(s): consumer=preethi Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__ return self.app(environ, start_response) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__ return self.app(environ, start_response) File "/usr/lib/python2.7/site-packages/web/application.py", line 277, in wsgi result = self.handle_with_processors() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors return process(self.processors) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/web/application.py", line 555, in processor return handler() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/web/application.py", line 570, in processor result = handler() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process return self.handle() File "/usr/lib/python2.7/site-packages/web/application.py", line 228, in handle return self._delegate(fn, self.fvars, args) File "/usr/lib/python2.7/site-packages/web/application.py", line 411, in _delegate return f() File "/usr/lib/python2.7/site-packages/web/application.py", line 419, in <lambda> f = lambda: self._delegate_sub_application(pat, what) File "/usr/lib/python2.7/site-packages/web/application.py", line 444, in _delegate_sub_application return app.handle_with_processors() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors return process(self.processors) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/web/application.py", line 555, in processor return handler() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/web/application.py", line 570, in processor result = handler() File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process return self.handle() File "/usr/lib/python2.7/site-packages/web/application.py", line 228, in handle return self._delegate(fn, self.fvars, args) File "/usr/lib/python2.7/site-packages/web/application.py", line 394, in _delegate return handle_class(f) File "/usr/lib/python2.7/site-packages/web/application.py", line 385, in handle_class return tocall(*args) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 231, in _auth_decorator value = method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/consumers.py", line 484, in POST result = execution.execute_sync_created(self, call_request, link) File "/usr/lib/python2.7/site-packages/pulp/server/dispatch/task.py", line 123, in _run result = call(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/profile.py", line 43, in create return self.update(consumer_id, content_type, profile) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/profile.py", line 57, in update manager.get_consumer(consumer_id) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/cud.py", line 202, in get_consumer raise MissingResource(consumer=id) MissingResource: Missing resource(s): consumer=preethi
Jeff - I'd like to know what's happening here. This is weird. It's a pretty annoying problem if a consumer disappearing (which she kinda simulated here, but I suspect the agent is still running so it's not exactly like the consumer box up and died) and the server continues to log errors like this. I'm aligning to 2.0 Beta to investigate. Ping me when you know what's happening.
*** Bug 876234 has been marked as a duplicate of this bug. ***
This bug describes a condition whereby the consumer is unregistered from pulp but a valid consumer certifiate still exists on the consumer. The agent is still installed and running and because valid consumer certificates still exist on the box, it continues to send profile updates as scheduled. The agent detects registration/unregistration based on the existance of a valid consumer certificate. Under normal circumstances, the certificate is deleted locally when the consumer is unregistered using pulp-consumer. If the consumer is unregistered using pulp-admin, notification of the unregistration is sent to the agent which deletes the certificate. There are a few abnormal scenerios that I can think of that can produce this condition. The condition being that the consumer not longer exists in the server but the agent thinks it's still registered. 1. The server is reinstalled or DB is manually cleared. 2. Consumer is unregistered using pulp-admin and connectivity between the server and qpid broker cannot be established. Short term, this causes a condition where the agent continues to report package profiles but the consumer no longer exits. Once the connection to the broken is established, the condition is resolved. However, if httpd is restarted before the connection is established, the message to the agent will never be sent and the condition will never be resolved. 3. Consumer is unregistered using pulp-admin and connectivity between the agent and qpid broker cannot be established. Short term, this causes a condition where the agent continues to report package profiles but the consumer no longer exits. Once the connection to the broken is established, the condition is resolved. 4. A deleted consumer certificate reappears as the result of a restored backup on the consumer. 5. User manually installs consumer certificate in some other way. Since interruptions in the communication path between the server and consumer are unavoidable, we need to assume that this condition can exist for at least short durations. Here are some possible solutions: 1. Don't raise ResourceMissing when reporting profiles. Bad idea. 2. Have agent query to see if consumer exists before sending the profile. This approach subject to race conditions so not very solid. 3. Implement CRLs on the server. When consumer is unregistered, revoke the consumer certificate and REST calls to report package profile are rejected at the front door and don't show up as ResourceMissing in the logs. Thinking of this solution raises the issue of unregistered consumers continuing to consume content from protected repos when this condition exists. 4. TBD
Given this is an exceptional case she ran into, I'm punting this off of 2.0. There's not an easy solution so I'd rather take some time to step back and handle these sorts of consumer incongruities as its own story.
Moved to https://pulp.plan.io/issues/340