Bug 799032 - Having removed a CDS unable to register it again
Having removed a CDS unable to register it again
Status: CLOSED NOTABUG
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: RHUA (Show other bugs)
2.0.2
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jeff Ortel
wes hayutin
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-01 11:01 EST by mkovacik
Modified: 2012-03-12 15:44 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-01 12:54:56 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs from the CDS (740.90 KB, application/zip)
2012-03-01 11:08 EST, mkovacik
no flags Details

  None (edit)
Description mkovacik 2012-03-01 11:01:29 EST
Description of problem:
Having removed a CDS I'm unable to register it back. The call-stack below... 


Version-Release number of selected component (if applicable):
RHEL-6.2-RHUI-2.0.2-20120229.1-Server-x86_64-DVD1.iso, RHEL 6.2

How reproducible:
N/A

Steps to Reproduce:
1. register two CDSes and sync
2. delete a CDS
3. register the same CDS again
  
Actual results:
Having removed a CDS one can't register it back

Expected results:
Having removed a CDS one should be able to register it back

Additional info:
# rhua .rhui/rhui.log
Exception attempting to register CDS [ip-10-49-23-129.eu-west-1.compute.internal]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/rhui/tools/pulp_api.py", line 648, in register_cds
    self.cds_api.register(hostname, name=display_name, description='RHUI CDS', sync_schedule=schedule, cluster_id=CDS_CLUSTER_NAME)
  File "/usr/lib/python2.6/site-packages/pulp/client/api/cds.py", line 33, in register
    return self.server.POST(path, data)[1]
  File "/usr/lib/python2.6/site-packages/pulp/client/api/server.py", line 323, in POST
    return self._request('POST', path, body=body)
  File "/usr/lib/python2.6/site-packages/pulp/client/api/server.py", line 287, in _request
    raise ServerRequestError(response.status, message, traceback)
ServerRequestError: (500, u'PulpException: Communications error while attempting to initialize CDS [ip-10-49-23-129.eu-west-1.compute.internal]; check the server log for more information', u'Traceback (most recent call last):\n  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 48, in report_error\n    return method(self, *args, **kwargs)\n  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 127, in _auth_decorator\n    value = method(self, *args, **kwargs)\n  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/cds.py", line 76, in POST\n    cds = cds_api.register(hostname, name, description, sync_schedule=sync_schedule, cluster_id=cluster_id)\n  File "/usr/lib/python2.6/site-packages/pulp/server/api/cds.py", line 120, in register\n    secret = self.dispatcher.init_cds(cds)\n  File "/usr/lib/python2.6/site-packages/pulp/server/cds/dispatcher.py", line 92, in init_cds\n    secret = self._send(self._cds_stub(cds).initialize)\n  File "/usr/lib/python2.6/site-packages/pulp/server/cds/dispatcher.py", line 169, in _send\n    result = func(*args)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 72, in __call__\n    return self.stub._send(request, opts)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 133, in _send\n    return self.__send(request, options)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 164, in __send\n    any=opts.any)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 144, in send\n    return self.__getreply(sn, reader)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 181, in __getreply\n    return self.__onreply(envelope)\n  File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 197, in __onreply\n    raise RemoteException.instance(reply)')

# Screenlog

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Home =-

   r   manage repositories
   c   manage content delivery servers (CDS)
   s   synchronization status and scheduling
   e   create entitlement certificates and client configuration RPMs
   n   manage Red Hat entitlement certificates
   u   manage users
   i   manage identity certificate

                        Connected: ip-10-53-101-202.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (home) => c

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Content Delivery Server (CDS) Management =-

   l   list all CDS instances registered to the RHUI
   a   register (add) a new CDS instance
   d   unregister (delete) a CDS instance from the RHUI

                        Connected: ip-10-53-101-202.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (cds) => l

-= RHUI Content Delivery Servers =-

  ip-10-49-23-129.eu-west-1.compute.internal
  ip-10-49-107-204.eu-west-1.compute.internal

------------------------------------------------------------------------------
rhui (cds) => d

Select the CDS instances to be removed from the RHUI:
  -  1 : ip-10-49-23-129.eu-west-1.compute.internal
  -  2 : ip-10-49-107-204.eu-west-1.compute.internal
Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: 1

Select the CDS instances to be removed from the RHUI:
  x  1 : ip-10-49-23-129.eu-west-1.compute.internal
  -  2 : ip-10-49-107-204.eu-west-1.compute.internal
Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: c


All previously deployed repositories will be deleted when a CDS is unregistered.

The following CDS instances will be unregistered:
  ip-10-49-23-129.eu-west-1.compute.internal
Proceed? (y/n) y


Unregistering ip-10-49-23-129.eu-west-1.compute.internal...

The following CDS instances could not be contacted to be unregistered.
They may still be removed from the RHUI database, however the RHUI is unable
confirm that they have stopped hosting repositories.
  ip-10-49-23-129.eu-west-1.compute.internal

Forcibly remove these CDS instances? (y/n) y

Forcibly unregistering ip-10-49-23-129.eu-west-1.compute.internal...

------------------------------------------------------------------------------
rhui (cds) => l

-= RHUI Content Delivery Servers =-

  ip-10-49-107-204.eu-west-1.compute.internal

------------------------------------------------------------------------------
rhui (cds) => a

Prior to registering a CDS, the CDS must be configured and running.

Hostname of the CDS to register:
ip-10-49-23-129.eu-west-1.compute.internal

Display name for the CDS [ip-10-49-23-129.eu-west-1.compute.internal]:



The following CDS instance will be registered:
  Hostname: ip-10-49-23-129.eu-west-1.compute.internal
  Name:     ip-10-49-23-129.eu-west-1.compute.internal
Proceed? (y/n) y


Error registering ip-10-49-23-129.eu-west-1.compute.internal, check the log file for more information

------------------------------------------------------------------------------
rhui (cds) => l

-= RHUI Content Delivery Servers =-

  ip-10-49-107-204.eu-west-1.compute.internal

------------------------------------------------------------------------------
rhui (cds) => a

Prior to registering a CDS, the CDS must be configured and running.

Hostname of the CDS to register:
ip-10-49-23-129.eu-west-1.compute.internal

Display name for the CDS [ip-10-49-23-129.eu-west-1.compute.internal]:



The following CDS instance will be registered:
  Hostname: ip-10-49-23-129.eu-west-1.compute.internal
  Name:     ip-10-49-23-129.eu-west-1.compute.internal
Proceed? (y/n) y


Error registering ip-10-49-23-129.eu-west-1.compute.internal, check the log file for more information

------------------------------------------------------------------------------
rhui (cds) => a

Prior to registering a CDS, the CDS must be configured and running.

Hostname of the CDS to register:
ip-10-49-23-129.eu-west-1.compute.internal

Display name for the CDS [ip-10-49-23-129.eu-west-1.compute.internal]:



The following CDS instance will be registered:
  Hostname: ip-10-49-23-129.eu-west-1.compute.internal
  Name:     ip-10-49-23-129.eu-west-1.compute.internal
Proceed? (y/n) y


Error registering ip-10-49-23-129.eu-west-1.compute.internal, check the log file for more information
Comment 1 mkovacik 2012-03-01 11:08:28 EST
Created attachment 566875 [details]
logs from the CDS
Comment 2 James Slagle 2012-03-01 11:26:13 EST
From /var/log/gofer/agent.log on the CDS that failed to re-register:
2012-03-01 10:28:55,534 [ERROR][worker-0] __call__() @ dispatcher.py:488 - cdsplugin.initialize(), secret required
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 484, in __call__
    inst, method = self.resolve()
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 293, in resolve
    method = self.getmethod(inst)
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 325, in getmethod
    return self.permitted(method)
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 342, in permitted
    self.__authorized(fninfo, auth)
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 392, in __authorized
    raise SecretRequired(self.__cnfn())
SecretRequired: cdsplugin.initialize(), secret required
2012-03-01 10:28:55,535 [INFO][worker-0] sendreply() @ rmi.py:158 - 45de7b83-5041-43df-8796-0159b954fd0b processed in: 48 (ms)
Comment 3 James Slagle 2012-03-01 12:00:32 EST
The CDS was never unregistered, b/c it could not be contacted by the rhua.  Notice the following from the rhui-manager capture you pasted:

The following CDS instances could not be contacted to be unregistered.
They may still be removed from the RHUI database, however the RHUI is unable
confirm that they have stopped hosting repositories.
  ip-10-49-23-129.eu-west-1.compute.internal

Forcibly remove these CDS instances? (y/n) y

Forcibly unregistering ip-10-49-23-129.eu-west-1.compute.internal...


Is there any reason why this might be?  Perhaps there was a network hiccup b/c it received the next call to initialize again.  However, that call fails b/c the CDS still thinks it's registered.
Comment 4 mkovacik 2012-03-01 12:40:09 EST
No hickups---it is rather tester fault (mine); the actual steps to reproduce were:

Test 1
 1. Run rhui-manager
 2. Registered two CDS nodes ( CDS1 and CDS2).
 3. Create entitlement certs for client
 4. Create client config rpm and define one of the CDS node as Primary Load balancer e.g. CDS1
 5. Stop httpd service ot pulp-cds service on primary cds node.
 6. Gracefully removal of CDS1 ( unregister through rhui-manager).

Test 2 
 1. Run rhui-manager
 2. Add new repos on rhua
 3. Register new CDS node in same cluster

Please, note steps 5 and 6; the consequence is just what is described in Comment 3. I didn't notice I were removing forcibly...
Comment 5 mkovacik 2012-03-01 12:54:56 EST
Removing the secret file, all works as a charm; closing---not a bug...
Comment 6 James Slagle 2012-03-12 15:39:55 EDT
Released in RHUI 2.0.2

Note You need to log in before you can comment on or make changes to this bug.