Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 498903

Summary: Cobbler + Taskomatic == lots of tracebacks after running without error for a week
Product: Red Hat Satellite 5 Reporter: Clifford Perry <cperry>
Component: ProvisioningAssignee: Justin Sherrill <jsherril>
Status: CLOSED NOTABUG QA Contact: Steve Salevan <ssalevan>
Severity: medium Docs Contact:
Priority: low    
Version: 530CC: jsherril
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-05-26 15:40:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Clifford Perry 2009-05-04 11:22:19 UTC
Description of problem:
Installed rlx-1-18 with 4/24 ISO on 4/25. Everything seemed good, until :

05/03/2009 04:03 AM

At that point Taskomatic threw the following traceback email (note the time - CRON job time)

-------- Original Message --------
Subject: WEB TRACEBACK from rlx-1-18.rhndev.redhat.com
Date: Sun, 3 May 2009 04:03:02 -0400
From: RHN Satellite <dev-null>
To: cperry

com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException: We had an error trying to login.
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:57)
	at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:113)
	at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:73)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:72)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:49)
	at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:83)
	at com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(SingleThreadedTestableTask.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: redstone.xmlrpc.XmlRpcFault: cobbler.cexceptions.CX:'login failed: taskomatic_user'
	at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:443)
	at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
	at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
	... 8 more


After this event - Taskomatic every 10 minutes had generated the following traceback email:

-------- Original Message --------
Subject: WEB TRACEBACK from rlx-1-18.rhndev.redhat.com
Date: Sun, 3 May 2009 04:10:01 -0400
From: RHN Satellite <dev-null>
To: cperry

com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException: We had an error trying to login.
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:57)
	at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:113)
	at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:73)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.getConnection(CobblerXMLRPCHelper.java:92)
	at com.redhat.rhn.taskomatic.task.KickstartFileSyncTask.execute(KickstartFileSyncTask.java:66)
	at com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(SingleThreadedTestableTask.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: redstone.xmlrpc.XmlRpcFault: cobbler.cexceptions.CX:'login failed: taskomatic_user'
	at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:443)
	at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
	at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
	at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52)
	... 7 more


Version-Release number of selected component (if applicable):
Satellite-5.3.0-RHEL5-re20090424.1-i386-embedded-oracle.iso

How reproducible:

I bet - VERY :/ 

Steps to Reproduce:
1. Install, let it run, cron kicks in, does something, taskomatic errors and unable to re-authenticate. 
2.
3.
  
Actual results:

traceback emails every 10 minutes. 

Expected results:

No traceback. 

Additional info:
Gut feelings
1) CRON was doing something, causing an error. Taskomatic for some reason never was able to gracefully recover. I have *NOT* restarted anything on rlx-1-18 and leaving it alone with devel review. 

or
2) some authenticated session token expired after a week (maybe a taskomatic or cron job cleared the session token) and then taskomatic was not able to detect this and re-negotiate correctly with cobblerd. 

or
3) something else - maybe cobblerd just has a habit of dying on a weekly basis and needing restarts.

Comment 1 Clifford Perry 2009-05-04 11:30:38 UTC
From /var/log/cobblerd/cobblerd.log.1

2009-04-26 04:03:01,866 - api - authenticate; ['taskomatic_user', True]
2009-04-26 04:03:01,869 - api - login succeeded; user(taskomatic_user)

From /var/log/cobblerd/cobblerd.log


2009-05-03 04:03:00,064 - api - invalid token; user(???)
2009-05-03 04:03:00,065 - api - Exception occured: cobbler.cexceptions.CX
2009-05-03 04:03:00,065 - api - Exception value: 'invalid token: 9QJ7RWy+z6jyJ26OUBlMOibg8G1HatOTqA=='
2009-05-03 04:03:00,087 - api - Exception Info:
  File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 1567, in _dispatch
    return method_handle(*params)
   File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 1060, in token_check
    self.__validate_token(token)
   File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 968, in __validate_token
    raise CX(_("invalid token: %s" % token))

2009-05-03 04:03:00,094 - api - login attempt; user(taskomatic_user)
2009-05-03 04:03:00,491 - api - authenticate; ['taskomatic_user', False]
2009-05-03 04:03:00,492 - api - login failed; user(taskomatic_user)
2009-05-03 04:03:00,493 - api - Exception occured: cobbler.cexceptions.CX
2009-05-03 04:03:00,494 - api - Exception value: 'login failed: taskomatic_user'
2009-05-03 04:03:00,494 - api - Exception Info:
  File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 1567, in _dispatch
    return method_handle(*params)
   File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 1033, in login
    raise CX(_("login failed: %s") % login_user)


So taskomatic seems to login with old session token, fails, so tries to do a user authentication and fails also.

Comment 2 Justin Sherrill 2009-05-12 15:30:30 UTC
Cliff,

So after looking at this, it seems tomcat is simply dead.  This stops cobbler from being able to authenticate and causes taskomatic to spew errors.  All of the tracebacks above look like they are from taskomatic (none from tomcat).

When i run /etc/init.d/tomcat5 status I get :

[root@rlx-1-18 ~]# /etc/init.d/tomcat5  status
lock file found but no process running for pid 16615


"ps aux" shows no tomcat process either.

So tomcat died a hard one for sure and I don't see any indication as to why.  The only thing i know to do is restart tomcat and see if it happens again.  

Your thoughts cliff?

Comment 3 Justin Sherrill 2009-05-13 14:12:29 UTC
after talking with cliff, we're gonna restart taskomatic and see if it happens again.

Comment 4 Clifford Perry 2009-05-19 18:43:32 UTC
So far, did not happen again weekend of May 16/17

Comment 5 Clifford Perry 2009-05-26 15:40:35 UTC
Still no replication. Going to close this though.