Bug 876677 - Missing resource error in pulp.log if you deletes the consumer without re-registering it
Summary: Missing resource error in pulp.log if you deletes the consumer without re-reg...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Pulp
Classification: Retired
Component: consumers
Version: Master
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: pulp-bugs
QA Contact: Preethi Thomas
URL:
Whiteboard:
: 876234 (view as bug list)
Depends On: 876234
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-14 17:20 UTC by Preethi Thomas
Modified: 2015-02-28 21:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 876234
Environment:
Last Closed: 2015-02-28 21:51:34 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 340 0 None None None Never

Description Preethi Thomas 2012-11-14 17:20:38 UTC
Clone for Master
+++ This bug was initially created as a clone of Bug #876234 +++

Description of problem:

If you remove the consumer rpms and consumer related files, Missing resource error appears in the pulp.log

Version-Release number of selected component (if applicable):
[root@preethi ~]# rpm -qa pulp-rpm-server
pulp-rpm-server-2.0.6-0.2.beta.noarch


How reproducible:


Steps to Reproduce:
1. Install the consumer-client packages and register a client to a pulp server
2. Remove the consumer-client rpms 
3. Notice the error in pulp.log
  
Actual results:


Expected results:


Additional info:

From pulp.log

2012-11-13 10:32:08,447 16913:139677436737280: pulp.server.webservices.middleware.exception:ERROR: exception:46 Missing resource(s): consumer=preethi
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 277, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 555, in processor
    return handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 570, in processor
    result = handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 228, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 411, in _delegate
    return f()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 419, in <lambda>
    f = lambda: self._delegate_sub_application(pat, what)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 444, in _delegate_sub_application
    return app.handle_with_processors()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 555, in processor
    return handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 570, in processor
    result = handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 228, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 394, in _delegate
    return handle_class(f)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 385, in handle_class
    return tocall(*args)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 231, in _auth_decorator
    value = method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/consumers.py", line 484, in POST
    result = execution.execute_sync_created(self, call_request, link)
  File "/usr/lib/python2.7/site-packages/pulp/server/dispatch/task.py", line 123, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/profile.py", line 43, in create
    return self.update(consumer_id, content_type, profile)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/profile.py", line 57, in update
    manager.get_consumer(consumer_id)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/cud.py", line 202, in get_consumer
    raise MissingResource(consumer=id)
MissingResource: Missing resource(s): consumer=preethi

Comment 1 Jay Dobies 2012-11-27 15:28:41 UTC
Jeff - I'd like to know what's happening here. This is weird. It's a pretty annoying problem if a consumer disappearing (which she kinda simulated here, but I suspect the agent is still running so it's not exactly like the consumer box up and died) and the server continues to log errors like this. I'm aligning to 2.0 Beta to investigate. Ping me when you know what's happening.

Comment 2 Jeff Ortel 2012-11-28 19:33:50 UTC
*** Bug 876234 has been marked as a duplicate of this bug. ***

Comment 3 Jeff Ortel 2012-11-28 21:13:13 UTC
This bug describes a condition whereby the consumer is unregistered from pulp but a valid consumer certifiate still exists on the consumer.  The agent is still installed and running and because valid consumer certificates still exist on the box, it continues to send profile updates as scheduled.  The agent detects registration/unregistration based on the existance of a valid consumer certificate.

Under normal circumstances, the certificate is deleted locally when the consumer is unregistered using pulp-consumer.  If the consumer is unregistered using pulp-admin, notification of the unregistration is sent to the agent which deletes the certificate.

There are a few abnormal scenerios that I can think of that can produce this condition.  The condition being that the consumer not longer exists in the server but the agent thinks it's still registered.

1. The server is reinstalled or DB is manually cleared.

2. Consumer is unregistered using pulp-admin and connectivity between the
   server and qpid broker cannot be established.  Short term, this causes
   a condition where the agent continues to report package profiles but
   the consumer no longer exits.  Once the connection to the broken
   is established, the condition is resolved.  However, if httpd is restarted
   before the connection is established, the message to the agent will never
   be sent and the condition will never be resolved.

3. Consumer is unregistered using pulp-admin and connectivity between the
   agent and qpid broker cannot be established.  Short term, this causes
   a condition where the agent continues to report package profiles but
   the consumer no longer exits.  Once the connection to the broken
   is established, the condition is resolved.

4. A deleted consumer certificate reappears as the result of a restored backup
   on the consumer.

5. User manually installs consumer certificate in some other way.

Since interruptions in the communication path between the server and consumer are unavoidable, we need to assume that this condition can exist for at least short durations.  Here are some possible solutions:

1. Don't raise ResourceMissing when reporting profiles.  Bad idea.

2. Have agent query to see if consumer exists before sending the profile.
   This approach subject to race conditions so not very solid.

3. Implement CRLs on the server.  When consumer is unregistered, revoke the
   consumer certificate and REST calls to report package profile are rejected
   at the front door and don't show up as ResourceMissing in the logs.  Thinking
   of this solution raises the issue of unregistered consumers continuing to
   consume content from protected repos when this condition exists.

4. TBD

Comment 4 Jay Dobies 2012-12-04 19:05:01 UTC
Given this is an exceptional case she ran into, I'm punting this off of 2.0. There's not an easy solution so I'd rather take some time to step back and handle these sorts of consumer incongruities as its own story.

Comment 5 Brian Bouterse 2015-02-28 21:51:34 UTC
Moved to https://pulp.plan.io/issues/340


Note You need to log in before you can comment on or make changes to this bug.