Hayk, does this happen for all driver types or only a specific driver?

It happens for all driver types and driver class types.

This looks odd - as if the class name in the driver config was supplied with actual quotes surrounding them.


---
WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "datasources"),
    ("jdbc-driver" => "mysql")
]) - failure description: "WFLYJCA0034: Unable to instantiate driver class \"com.mysql.jdbc.jdbc2.optional.MysqlXADataSource\". See log (WARN) for more details"
---

Would like to ask Mike T. about this as he recently was dealing with the JSON getting sent into the h-server.

Can someone attach what the hawkular websocket command (the JSON) that the MiQ UI console is sending looks like?

Also, someone attach the standalone.xml that results after the command was executed. I want to see what the jdbc driver/datasource section looks like.

Created attachment 1276366 [details]
standalone.xml

As it is mentioned in description, standalone.xml is not changed.
Attaching standalone.xml file.

OK, I ran a test over here. In our itests, we test the adding JDBC driver stuff. In the test, I gave it a bogus class name and it failed with the same error:

---
14:07:02,927 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 4) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "datasources"),
    ("jdbc-driver" => "mysql")
]) - failure description: "WFLYJCA0034: Unable to instantiate driver class \"com.mysql.jdbc.DriverXXX\". See log (WARN) for more details"
---

So the quotes are not a problem. But you can see I gave it a bad classname.

Therefore, I would like to ask if you are sure the driver class you provided is correct.

BTW: the WARN log message before this says the following - which tells me the class doesn't exist (which is true):

14:07:02,927 WARN  [org.jboss.as.connector.subsystems.datasources] (management-handler-thread - 4) WFLYJCA0003: Unable to instantiate driver class "com.mysql.jdbc.DriverXXX": java.lang.ClassNotFoundException: com.mysql.jdbc.DriverXXX from [Module "com.mysql:main" from local module loader @36b4cef0 (finder: local module finder @fad74ee (roots: /home/mazz/source/hawkular-agent/hawkular-javaagent-itest-parent/hawkular-javaagent-all-itests/hawkular-javaagent-cmdgw-itest/target/hawkular-javaagent-cmdgw-itest-1.0.0.Final-SNAPSHOT/modules,/home/mazz/source/hawkular-agent/hawkular-javaagent-itest-parent/hawkular-javaagent-all-itests/hawkular-javaagent-cmdgw-itest/target/hawkular-javaagent-cmdgw-itest-1.0.0.Final-SNAPSHOT/modules/system/layers/hawkular,/home/mazz/source/hawkular-agent/hawkular-javaagent-itest-parent/hawkular-javaagent-all-itests/hawkular-javaagent-cmdgw-itest/target/hawkular-javaagent-cmdgw-itest-1.0.0.Final-SNAPSHOT/modules/system/layers/base))]

I think I see the problem. We need to support setting the XA class name - looks like we don't support that yet.

I see this in EAP7 jboss cli output for the H2 driver:

/subsystem=datasources/jdbc-driver=h2/:read-resource(recursive=false)
{
    "outcome" => "success",
    "result" => {
        "deployment-name" => undefined,
        "driver-class-name" => undefined,
        "driver-datasource-class-name" => undefined,
        "driver-major-version" => undefined,
        "driver-minor-version" => undefined,
        "driver-module-name" => "com.h2database.h2",
        "driver-name" => "h2",
        "driver-xa-datasource-class-name" => "org.h2.jdbcx.JdbcDataSource",
        "jdbc-compliant" => undefined,
        "module-slot" => undefined,
        "profile" => undefined,
        "xa-datasource-class" => undefined
    }
}

We do not support setting driver-xa-datasource-class-name or xa-datasource-class (I have no idea why there are two of those). But I don't see any XA related settings in the Hawkular stuff to support either of them.

Looks like EAP 7.x and EAP 6.4 both "xa-datasource-class" and "driver-xa-datasource-class-name"

  * "driver-xa-datasource-class-name The fully qualified class name of the javax.sql.XADataSource implementation"
  * "xa-datasource-class XA datasource class"

See:

https://wildscribe.github.io/JBoss%20EAP/6.4.0/subsystem/datasources/jdbc-driver/index.html

and

https://wildscribe.github.io/Wildfly/10.0.0.Final/subsystem/datasources/jdbc-driver/index.html

I think we also should support

  * "jdbc-compliant Whether or not the driver is JDBC compliant"

See: https://issues.jboss.org/browse/HWKAGENT-174

(In reply to John Mazzitelli from comment #9)
> I think I see the problem. We need to support setting the XA class name -
> looks like we don't support that yet.
> 
> I see this in EAP7 jboss cli output for the H2 driver:
> 
> /subsystem=datasources/jdbc-driver=h2/:read-resource(recursive=false)
> {
>     "outcome" => "success",
>     "result" => {
>         "deployment-name" => undefined,
>         "driver-class-name" => undefined,
>         "driver-datasource-class-name" => undefined,
>         "driver-major-version" => undefined,
>         "driver-minor-version" => undefined,
>         "driver-module-name" => "com.h2database.h2",
>         "driver-name" => "h2",
>         "driver-xa-datasource-class-name" => "org.h2.jdbcx.JdbcDataSource",
>         "jdbc-compliant" => undefined,
>         "module-slot" => undefined,
>         "profile" => undefined,
>         "xa-datasource-class" => undefined
>     }
> }
> 
> We do not support setting driver-xa-datasource-class-name or
> xa-datasource-class (I have no idea why there are two of those). But I don't
> see any XA related settings in the Hawkular stuff to support either of them.

We actually have supported setting the XA stuff for some time now. I know I have added XA Datasources as I'm sure QE has also. And we do switch between using 'driver-class-name' for non-xa and 'xa-datasource-class' for XA.

I'm sure its a regression. Looking further into it...

(In reply to Mike Thompson from comment #12)
> > We do not support setting driver-xa-datasource-class-name or
> > xa-datasource-class (I have no idea why there are two of those). But I don't
> > see any XA related settings in the Hawkular stuff to support either of them.
> 
> We actually have supported setting the XA stuff for some time now. I know I
> have added XA Datasources as I'm sure QE has also. And we do switch between
> using 'driver-class-name' for non-xa and 'xa-datasource-class' for XA.
> 
> I'm sure its a regression. Looking further into it...

You are probably thinking of the XA stuff in the Datasources. This is related to JDBC Drivers, which has these two XA related properties on them. So when adding *JDBC Drivers* is where the problem is, I believe.

(In reply to John Mazzitelli from comment #13)
> (In reply to Mike Thompson from comment #12)
> > > We do not support setting driver-xa-datasource-class-name or
> > > xa-datasource-class (I have no idea why there are two of those). But I don't
> > > see any XA related settings in the Hawkular stuff to support either of them.
> > 
> > We actually have supported setting the XA stuff for some time now. I know I
> > have added XA Datasources as I'm sure QE has also. And we do switch between
> > using 'driver-class-name' for non-xa and 'xa-datasource-class' for XA.
> > 
> > I'm sure its a regression. Looking further into it...
> 
> You are probably thinking of the XA stuff in the Datasources. This is
> related to JDBC Drivers, which has these two XA related properties on them.
> So when adding *JDBC Drivers* is where the problem is, I believe.

Ah, XA in *JDBC Drivers* not Datasources. Ok, that is very likely true then.

I already had tested and verified the XA stuff in scope of BZ https://bugzilla.redhat.com/show_bug.cgi?id=1383611

So, right, this is a regression.

(In reply to Hayk Hovsepyan from comment #16)
> So, right, this is a regression.

I don't see how this could be a regression or how it ever worked - the cmdgw JSON command never had these xa datasource class name properties - which follows that the agent never set them.

See for MiQ UI changes: https://github.com/ManageIQ/manageiq-ui-classic/pull/1300
still need Hawular UI services build... and maybe more changes.

"add jdbc driver" is a feature that was added and tested for euwe release in October 2016. So it sounds like it's a regression, unless here some other driver types were tested that were not tested before.
Are those 2 additional properties mandatory? I didn't find it in the docs.
Also, the exception above didn't mention them, are there more descriptive exceptions in the log?

Actually, those 2 properties are not required (Required = false) for addDriver operation - it's in the docs  - select under Operations the "add" :

https://wildscribe.github.io/JBoss%20EAP/6.4.0/subsystem/datasources/jdbc-driver/index.html

https://wildscribe.github.io/Wildfly/10.0.0.Final/subsystem/datasources/jdbc-driver/index.html

So it looks like it's not the cause of the problem.

@Hayk - is that with the Javaagent?

@Mazz Could it be possible that the driver jar/module is somehow not put into the right place by the javaagent so that the add-driver operation will fail for that reason?
Also I don't recall if this would need any reload of the server in between (I guess not)

Me and Mike T know what the problem is :) We worked on it last week and the cmdgw/agent code has been merged and released. He has some MiQ work that he handed off - I believe most if not all of it was done.

FWIW: It has to do with the fact that when creating the jdbc driver, the XA datasource class name was being passed as the "non-XA" datasource class name, WildFly noticed that the given classname did not implement the proper API interface and threw an error. MiQ now supports setting the proper datasource class names (or at least Mike was working on that if he didn't finish it).

Can we then find out, why this used to work in the past (October last year) and fails now? Are newer WildFlys more strict?

My suspicion is the earlier tests weren't testing "XA JDBC Drivers" and thus weren't setting XA datasource classnames. I don't know if that is true or not - were we testing "XA JDBC Drivers" back in October?

(BTW, there is no such thing as "XA JDBC Drivers" - in WildFly there are just "JDBC Drivers" who happen to optionally have an additional "XA datasource class name" - I discussed this with Mike and he changed the related MiQ UI screen to match the way JDBC drivers are modeled in WildFly.)

Hayk, if this is a regression, please set the regression flag.

Mike T. is on PTO this week. Here's a PR he did last week that helps address this issue:

https://github.com/ManageIQ/manageiq-ui-classic/pull/1300

Mike and I worked on this end of last week. The hawkular cmdgw and hawkular agent have been updated and released that should work with this new code.

Make sure you are running with Mike's latest code in this PR. I do not know, however, if he fully finished this work. It is possible he was going to pass it off to another MiQ developer to finish it this week.

Note that the XA datasource class name is optional - if you only use this JDBC driver for non-xa datasources, it is not needed. However, if you are to use this JDBC driver with an XA datasource, I believe this is required to be specified (not 100% sure on that).

John, Hayk, I might be repeating myself, but if this is a regression, I don't understand how adding new fields is the solution. Please sync on the scenario that was tested now versus in the past to reach a final conclusion.

(In reply to Alissa from comment #31)
> John, Hayk, I might be repeating myself, but if this is a regression, I
> don't understand how adding new fields is the solution. Please sync on the
> scenario that was tested now versus in the past to reach a final conclusion.

Mike shows me some test file that showed the jdbc drivers and their configs that the tests were using. I don't know where they are. But what they showed me was the failure was to be expected until we added these new props.

Whether this truly is a regression or not, I don't know. Again, I can't see any way this would have passed with the old code because we simply did not support XA datasource class names in JDBC driver configs. If you were passing in XA datasource class names when creating JDBC drivers, it would have failed 100% of the time (because XA datasource class names placed in the normal datasource-class field that we had before would cause an error). This is why I think this isn't necessarily a regression - I think the tests are testing something new now (like XA datasource class names).

Here's Mike's PR - see my latest comment: https://github.com/ManageIQ/manageiq-ui-classic/pull/1300#issuecomment-300190220

Can someone show me what the previous test (the one that passed) actually tested? Because I have a feeling no one ever tested setting the XA datasource class name ON THE JDBC DRIVER (remember, we are talking about JDBC drivers here - NOT datasources).

If you DID try to set an XA datasource class name on a JDBC Driver, it would have failed. This is because up until late last week, the agent never even supported such a field - the agent was never able to set an XA datasource class name on a jdbc driver - so if you did try to set it, it would have failed with the error this bugzilla originally reported.

Unless I am completely misunderstanding what the problem is. But in that case, I need to see what you actually tested (code, config, etc) before that showed the test passing.

The previous testing scenario was:
1. Create JDBC Driver. Type JDBC Driver Class (not XA class) in "Driver Class" field.
2. Create XA Datasource. Type XA JDBC Driver Class in "Driver Class" field.
And it was working.

Right John, there was NOT an input field for XA JDBC Driver Class, and it does NOT exist even now.
What was added in Mike's PR: https://github.com/ManageIQ/manageiq-ui-classic/pull/876 was to automatically types XA JDBC Driver Class in the field "Driver Class". And here is the issue. 

And the PR https://github.com/ManageIQ/manageiq-ui-classic/pull/1300 which is sent now could be a solution. If there will be a back-end support of passing "XA Driver Class" in Agent.
Because EAP CLI itself supports creating JDBC Driver with specifying both "Driver Class" and "XA Datasource Driver Class".

And here the regression is, that when first creating XA JDBC Driver, it fails, and it causes later on failure of creating non-XA JDBC Driver as well. And this causes impossibility to create any Datasource.
So there is a regression from general functionality point of view, of creating XA Datasources, even though "XA JDBC Driver Class" field did not exist.

Karel Hala is taking over Mike's UI part since Mike is not available this week

(In reply to Hayk Hovsepyan from comment #34)
> Right John, there was NOT an input field for XA JDBC Driver Class, and it
> does NOT exist even now.
> What was added in Mike's PR:
> https://github.com/ManageIQ/manageiq-ui-classic/pull/876 was to
> automatically types XA JDBC Driver Class in the field "Driver Class". And
> here is the issue. 

Right. And that PR was wrong which I talked to Mike about last week. You can't put an XA datasource class as "Driver class" - that will fail when you try to deploy it in WildFly. So this PR is probably obsolete because it did things incorrectly.

> And the PR https://github.com/ManageIQ/manageiq-ui-classic/pull/1300 which
> is sent now could be a solution. If there will be a back-end support of
> passing "XA Driver Class" in Agent.
> Because EAP CLI itself supports creating JDBC Driver with specifying both
> "Driver Class" and "XA Datasource Driver Class".

Right. And this is what me and Mike worked on late last week. The agent now supports receiving this extra "XA Datasource Driver Class" in addition to the non-XA "Driver class" - this is the commons/agent code I mentioned earlier as being released and ready. We just need MIQ's PR 1300 to be merged so you can test this all out and see it work end-to-end. Without PR 1300, the MiQ console will still be broken.

> And here the regression is, that when first creating XA JDBC Driver, it
> fails, and it causes later on failure of creating non-XA JDBC Driver as
> well. And this causes impossibility to create any Datasource.
> So there is a regression from general functionality point of view, of
> creating XA Datasources, even though "XA JDBC Driver Class" field did not
> exist.

For the record, there is NO such thing as a "XA JDBC Driver" (or "Non-XA JDBC Driver"). To WildFly, its just a JDBC Driver. Every JDBC Driver can be associated with an optional "XA Datasource Class" - thus allowing the JDBC Driver to support both non-XA and XA. But to be sure, there is no such thing as a "XA JDBC Driver" or a "Non-XA JDBC Driver." There is just one notion of "JDBC Driver". (there *is* such a thing as "XA Datasource" and "Non-XA Datasource" - but those are datasources, not JDBC Drivers). So the MiQ Console should not be referencing "XA JDBC Driver" or "Non-XA JDBC Driver" because they are not a concept in WildFly. The Console should only be displaying and working with "JDBC Driver".

Now, it sounds like the regression mentioned above is testing something I wasn't aware of. This Bugzilla entry in the description showed the bug results in the error message "WFLYJCA0034: Unable to instantiate driver class \"com.mysql.jdbc.jdbc2.optional.MysqlXADataSource\"." This has to do with the missing XA datasource class name entry in the cmdgw/agent code.

For the "regression" replication procedures described above, it sounds like you attempt to add a JDBC Driver, it fails (due to mis-configuration of the JDBC Driver), you then attempt to add another JDBC Driver (with correct configuration this time) and it still fails (I hope I explained that correctly). If that is the case, then yes that would be considered a bug - but that is NOT dealing with the issues me and Mike worked on last week which this bugzilla originally reported (which was simply it was impossible to add a JDBC Driver with a XA datasource class name using the MiQ Console because the cmdgw and agent didn't even support such a thing. That specific issue should now be fixed in cmdgw, agent, and MiQ Console with PR #1300.)

From what I think I understand now, we need to do the following:

1) MiQ merges PR #1300 and using the latest hawkular code, someone should test that this BZ issue goes away. You should no longer see the error "WFLYJCA0034: Unable to instantiate driver class \"com.mysql.jdbc.jdbc2.optional.MysqlXADataSource\"" as reported in this BZ.

2) Try to then replicate that regression with the latest code - see if failing to add a jdbc driver the first time causes all subsequent attempts to also fail even though those subsequent attempts should succeed. If this is replicated, a new BZ should be entered since this seems to me to be a separate issue from this BZ.

Verified on 5.9.0.1.20171004215954_13427ce