Bug 847869 - [as7] reload operation fails for standalone server - InvalidPluginConfigurationException
[as7] reload operation fails for standalone server - InvalidPluginConfigurati...
Product: RHQ Project
Classification: Other
Component: Plugins (Show other bugs)
Unspecified Unspecified
high Severity urgent (vote)
: CR02
: JON 3.1.1
Assigned To: Armine Hovsepyan
Mike Foley
Depends On: 847674
Blocks: as7-plugin
  Show dependency treegraph
Reported: 2012-08-13 17:04 EDT by Stefan Negrea
Modified: 2015-09-02 20:01 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 847674
Last Closed: 2013-09-03 11:16:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
reloadExceptionGUI (217.07 KB, image/png)
2012-08-27 18:04 EDT, Armine Hovsepyan
no flags Details
cridentials (170.06 KB, image/png)
2012-08-31 05:08 EDT, Armine Hovsepyan
no flags Details
credentials (164.86 KB, image/png)
2012-08-31 07:48 EDT, Armine Hovsepyan
no flags Details

  None (edit)
Description Stefan Negrea 2012-08-13 17:04:42 EDT
+++ This bug was initially created as a clone of Bug #847674 +++

Description of problem: When EAP server starts sending restart-required headers I am not able to reload it using UI.

Version-Release number of selected component (if applicable):
JON 3.1.1.ER1 + EAP6 GA

How reproducible:always

Steps to Reproduce:
1. Have EAP6 protected by user/pass running in standalone mode, imported with children => credentials are correct
2. make server require restart (for example by disabling ExampleDS datasource)
3. start 'reload' operation on server
Actual results: Reload operation fails with this.

org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
	at org.rhq.modules.plugins.jbossas7.ASConnection.handleAuthorizationFailureResponse(ASConnection.java:341)
	at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:266)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:433)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:374)
	at org.rhq.modules.plugins.jbossas7.StandaloneASComponent.invokeOperation(StandaloneASComponent.java:100)
	at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)

Expected results:

Additional info: Disabling a datasource is not the only way how to reproduce this BZ. It is enough to repeatedly run 'reload' operation on server, it fails for me on the 2nd attempt with above error message.

--- Additional comment from snegrea@redhat.com on 2012-08-13 17:04:05 EDT ---

The bug did not affect the restart operation itself. The plugin was not properly detecting when the server was up and operational after a successful restart. 

The code that was probing the server by sending an operation and looking for success or failure on the operation execution. If the server is not fully started, the operation execution could fail in surprising ways, thus throwing exceptions that bubbled up the stack. Updated the probing code to catch exceptions and just consider them failed attempts to connect to the server.
Comment 2 John Sanda 2012-08-13 22:16:38 EDT
Moving to ON_QA since JON 3.1.1 ER2 build is availble - https://brewweb.devel.redhat.com/buildinfo?buildID=228250
Comment 3 John Sanda 2012-08-20 12:34:44 EDT
This should not have been moved to ON_QA. It was changed to MODIFIED after the
ER2 build. This change will be available in the ER3 build.
Comment 4 John Sanda 2012-08-22 01:48:40 EDT
Moving to ON_QA. The JON 3.1.1 ER3 build is available at https://brewweb.devel.redhat.com/buildinfo?buildID=230321.
Comment 5 Armine Hovsepyan 2012-08-27 17:20:16 EDT

the bug is only possible to reproduce if rhq management user is not correctly installed <-- which is not a valid case.
Comment 6 Armine Hovsepyan 2012-08-27 18:03:37 EDT
After a discussion with Stefan - reopening bug.

Reproduction scenario is the following:
Start EAP standalone - install rhq user (rhqadmin rhqadmin) - disable ExampleDS -> EAP restart operation -> EAP reload operation ---- > reload returns exception (screen-shot attached below). Restart operation works correctly, while reload fails.

Scenario for having the original exception is:
Start eap standalone -> install rhq user (rhqadmin rhqadmin) -> change rhq user password with some incorrect one -> EAP reload operation ------> original exception (the one mentioned in bug handleAuthorizationFailureResponse....) is being visible. <--- not valid scenario imho.
Comment 7 Armine Hovsepyan 2012-08-27 18:04:13 EDT
Created attachment 607354 [details]
Comment 8 Stefan Negrea 2012-08-29 04:50:08 EDT
Please retest. Updated all the wait methods for reload, shutdown, and restart to have the same design. All the methods now take into account possible exceptions for the test operation sent to the application server.

Also, please make sure that two commands do not get scheduled simultaneously. For example, do not schedule restart and then without waiting for the restart to complete immediately schedule a reload. This is not a supported scenario.

release/jon3.1.x branch commit:

Comment 9 John Sanda 2012-08-29 21:45:23 EDT
The CR1 build is available at
https://brewweb.devel.redhat.com/buildinfo?buildID=231258. Moving to ON_QA.
Comment 10 Libor Zoubek 2012-08-30 13:02:08 EDT
verified on JON 3.1.1.CR1
Comment 11 Armine Hovsepyan 2012-08-31 05:07:34 EDT

Reproduction case:
1. Start eap standalone in full-ha mode.
2. Update cluster connection resource
3. Reload eap

Screnshot attached
No errors in all 3 (server, agent, eap) logs.
Comment 12 Armine Hovsepyan 2012-08-31 05:08:12 EDT
Created attachment 608439 [details]
Comment 13 Stefan Negrea 2012-08-31 05:58:22 EDT
Please use the reload and restart operations with valid server configurations. This bug captures errors with the reload and restart operation in normal working conditions. 

The invalid configuration problem is captured by bug 852891. For the purpose and testing of this bug please do not update resources that result in invalid configuration. If there are other resource the result in invalid configuration and thus the reload method and restart method do not work please attach the comments and logs to bug 852891.
Comment 14 Armine Hovsepyan 2012-08-31 07:48:27 EDT
Created attachment 608494 [details]

Changed cluster connection resource - retry Interval Multiplier from 1 to 1.5 - reloaded eap, exception is visible on gui. The value o retry interval multiplier is changed.
Comment 15 Simeon Pinder 2012-08-31 16:49:15 EDT
I was not able to reproduce this issue.

On further analysis it looks like Comments 11, 12 and 14 were added to this BZ in error. This looks like the reproduce steps for 835701 and additionally this BZ was already set to verified by another QE member in comment 10.

Moving this back to ON_QA and will ping ccrouch and mfoley to confirm or take another look at this.
Comment 16 Armine Hovsepyan 2012-09-04 09:57:43 EDT
it seems it's reproducible only in RHEL6 environment.

My envs are:

1. RHEL6 - java version "jre-1.6.0-openjdk"
2. RHEL6 - java version "jre1.7.0"

Libor was able to reproduce the bug on RHEl6  - see comment  https://bugzilla.redhat.com/show_bug.cgi?id=835701#c16

Filips environement: Fedora 16 - jre-1.6.0-openjdk - bug is not reproducible at all.
Comment 17 Armine Hovsepyan 2012-09-04 10:02:30 EDT
Correct steps to reproduce:

1. Remove EAP from env.
2. Install new EAP
3. Start EAP (6) in standalone full-ha mode
4. Inventory EAP
5. Install RHQ user for EAP
6. Change the configuration of cluster connection/my cluster resource retry-interval-multiplier from 1 to 1.5 OR create a security domain resource and change it's cache type.
7. Reload EAP server

Actual result: reload fails for only first time with exception.

Expected result: no exception is visible in ui.

Additional information:
please get attached screen-shots - two different exceptions noticed jsut today.
Comment 18 Armine Hovsepyan 2012-09-04 11:26:59 EDT
since ordinary reload is fine, marking this bug as verified

new bug #854310 for the first reload case is created.
Comment 19 Heiko W. Rupp 2013-09-03 11:16:21 EDT
Bulk closing of old issues in VERIFIED state.

Note You need to log in before you can comment on or make changes to this bug.