Bug 847869

Summary: [as7] reload operation fails for standalone server - InvalidPluginConfigurationException
Product: [Other] RHQ Project Reporter: Stefan Negrea <snegrea>
Component: PluginsAssignee: Armine Hovsepyan <ahovsepy>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.4CC: ahovsepy, hrupp, jsanda, lzoubek, mfoley, myarboro, spinder
Target Milestone: CR02   
Target Release: JON 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 847674 Environment:
Last Closed: 2013-09-03 15:16:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 847674    
Bug Blocks: 707223    
Attachments:
Description Flags
reloadExceptionGUI
none
cridentials
none
credentials none

Description Stefan Negrea 2012-08-13 21:04:42 UTC
+++ This bug was initially created as a clone of Bug #847674 +++

Description of problem: When EAP server starts sending restart-required headers I am not able to reload it using UI.


Version-Release number of selected component (if applicable):
JON 3.1.1.ER1 + EAP6 GA

How reproducible:always


Steps to Reproduce:
1. Have EAP6 protected by user/pass running in standalone mode, imported with children => credentials are correct
2. make server require restart (for example by disabling ExampleDS datasource)
3. start 'reload' operation on server
  
Actual results: Reload operation fails with this.

org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
	at org.rhq.modules.plugins.jbossas7.ASConnection.handleAuthorizationFailureResponse(ASConnection.java:341)
	at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:266)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:433)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:374)
	at org.rhq.modules.plugins.jbossas7.StandaloneASComponent.invokeOperation(StandaloneASComponent.java:100)
	at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)


Expected results:


Additional info: Disabling a datasource is not the only way how to reproduce this BZ. It is enough to repeatedly run 'reload' operation on server, it fails for me on the 2nd attempt with above error message.

--- Additional comment from snegrea on 2012-08-13 17:04:05 EDT ---

The bug did not affect the restart operation itself. The plugin was not properly detecting when the server was up and operational after a successful restart. 

The code that was probing the server by sending an operation and looking for success or failure on the operation execution. If the server is not fully started, the operation execution could fail in surprising ways, thus throwing exceptions that bubbled up the stack. Updated the probing code to catch exceptions and just consider them failed attempts to connect to the server.

Comment 2 John Sanda 2012-08-14 02:16:38 UTC
Moving to ON_QA since JON 3.1.1 ER2 build is availble - https://brewweb.devel.redhat.com/buildinfo?buildID=228250

Comment 3 John Sanda 2012-08-20 16:34:44 UTC
This should not have been moved to ON_QA. It was changed to MODIFIED after the
ER2 build. This change will be available in the ER3 build.

Comment 4 John Sanda 2012-08-22 05:48:40 UTC
Moving to ON_QA. The JON 3.1.1 ER3 build is available at https://brewweb.devel.redhat.com/buildinfo?buildID=230321.

Comment 5 Armine Hovsepyan 2012-08-27 21:20:16 UTC
verified.

the bug is only possible to reproduce if rhq management user is not correctly installed <-- which is not a valid case.

Comment 6 Armine Hovsepyan 2012-08-27 22:03:37 UTC
After a discussion with Stefan - reopening bug.

Reproduction scenario is the following:
Start EAP standalone - install rhq user (rhqadmin rhqadmin) - disable ExampleDS -> EAP restart operation -> EAP reload operation ---- > reload returns exception (screen-shot attached below). Restart operation works correctly, while reload fails.

Scenario for having the original exception is:
Start eap standalone -> install rhq user (rhqadmin rhqadmin) -> change rhq user password with some incorrect one -> EAP reload operation ------> original exception (the one mentioned in bug handleAuthorizationFailureResponse....) is being visible. <--- not valid scenario imho.

Comment 7 Armine Hovsepyan 2012-08-27 22:04:13 UTC
Created attachment 607354 [details]
reloadExceptionGUI

Comment 8 Stefan Negrea 2012-08-29 08:50:08 UTC
Please retest. Updated all the wait methods for reload, shutdown, and restart to have the same design. All the methods now take into account possible exceptions for the test operation sent to the application server.

Also, please make sure that two commands do not get scheduled simultaneously. For example, do not schedule restart and then without waiting for the restart to complete immediately schedule a reload. This is not a supported scenario.




release/jon3.1.x branch commit:

http://git.fedorahosted.org/cgit/rhq/rhq.git/commit/?h=release/jon3.1.x&id=91db63a06a0612349faf89678884b6b9150b411f

Comment 9 John Sanda 2012-08-30 01:45:23 UTC
The CR1 build is available at
https://brewweb.devel.redhat.com/buildinfo?buildID=231258. Moving to ON_QA.

Comment 10 Libor Zoubek 2012-08-30 17:02:08 UTC
verified on JON 3.1.1.CR1

Comment 11 Armine Hovsepyan 2012-08-31 09:07:34 UTC
reopen.

Reproduction case:
1. Start eap standalone in full-ha mode.
2. Update cluster connection resource
3. Reload eap

Screnshot attached
No errors in all 3 (server, agent, eap) logs.

Comment 12 Armine Hovsepyan 2012-08-31 09:08:12 UTC
Created attachment 608439 [details]
cridentials

Comment 13 Stefan Negrea 2012-08-31 09:58:22 UTC
Please use the reload and restart operations with valid server configurations. This bug captures errors with the reload and restart operation in normal working conditions. 

The invalid configuration problem is captured by bug 852891. For the purpose and testing of this bug please do not update resources that result in invalid configuration. If there are other resource the result in invalid configuration and thus the reload method and restart method do not work please attach the comments and logs to bug 852891.

Comment 14 Armine Hovsepyan 2012-08-31 11:48:27 UTC
Created attachment 608494 [details]
credentials

Changed cluster connection resource - retry Interval Multiplier from 1 to 1.5 - reloaded eap, exception is visible on gui. The value o retry interval multiplier is changed.

Comment 15 Simeon Pinder 2012-08-31 20:49:15 UTC
I was not able to reproduce this issue.

On further analysis it looks like Comments 11, 12 and 14 were added to this BZ in error. This looks like the reproduce steps for 835701 and additionally this BZ was already set to verified by another QE member in comment 10.

Moving this back to ON_QA and will ping ccrouch and mfoley to confirm or take another look at this.

Comment 16 Armine Hovsepyan 2012-09-04 13:57:43 UTC
it seems it's reproducible only in RHEL6 environment.

My envs are:

1. RHEL6 - java version "jre-1.6.0-openjdk"
2. RHEL6 - java version "jre1.7.0"

Libor was able to reproduce the bug on RHEl6  - see comment  https://bugzilla.redhat.com/show_bug.cgi?id=835701#c16

Filips environement: Fedora 16 - jre-1.6.0-openjdk - bug is not reproducible at all.

Comment 17 Armine Hovsepyan 2012-09-04 14:02:30 UTC
Correct steps to reproduce:

1. Remove EAP from env.
2. Install new EAP
3. Start EAP (6) in standalone full-ha mode
4. Inventory EAP
5. Install RHQ user for EAP
6. Change the configuration of cluster connection/my cluster resource retry-interval-multiplier from 1 to 1.5 OR create a security domain resource and change it's cache type.
7. Reload EAP server

Actual result: reload fails for only first time with exception.


Expected result: no exception is visible in ui.

Additional information:
please get attached screen-shots - two different exceptions noticed jsut today.

Comment 18 Armine Hovsepyan 2012-09-04 15:26:59 UTC
since ordinary reload is fine, marking this bug as verified

new bug #854310 for the first reload case is created.

Comment 19 Heiko W. Rupp 2013-09-03 15:16:21 UTC
Bulk closing of old issues in VERIFIED state.