Created attachment 1276625 [details] full agent log Description of problem: Two way ssl between agent and EAP 7.1 host controller using elytron is not working, host controller resource is down and following error is thrown to agent.log: 2017-05-05 09:51:38,626 WARN [ResourceContainer.invoker.daemon-4] (rhq.modules.plugins.wildfly10.HostControllerComponent)- Unable to detect HostController's process-type org.rhq.modules.plugins.wildfly10.json.ResultFailedException: Failed to read attribute [process-type] of address [] - response: Result{outcome='failed', failureDescription=WFLYDMHTTP0006: The security realm is not ready to process requests, see https://fbr-eap71-sec.bc.jonqe.lab.eng.bos.redhat.com:8443/error, rolledBack=true} Version-Release number of selected component (if applicable): JON3.3.8 + eap plugin pack 05.ER02, EAP 7.1.DR17 How reproducible: 3/3 Steps to Reproduce: 1. create trustore and keystore according to https://docs.jboss.org/author/display/WFLY/WildFly+Elytron+Security#WildFlyElytronSecurity-EnableTwowaySSL%2FTLSfortheManagementInterfacesusingtheElytronSubsystem 2. start domain controller 3. configure domain controller to use 2-way ssl: run jboss-cli.sh --connect --controller=${hostname}:9990 --file=enableTwoWaySSLForEAP7Domain.txt 4. import domain controller to inventory 5. navigate to domain controller resource->Inventory-Connection Setting 6. file proper values to Secure connection settings section Actual results: Resource is down and previous WARN is thrown to agent.log Expected results: Resource should be up and no warnings in agent.log Additional info: content of enableTwoWaySSLForEAP7Domain.txt: batch /host=master/subsystem=elytron/key-store=twoWayKS:add(path=/tmp/server.keystore.jks,credential-reference={clear-text=secret},type=JKS) /host=master/subsystem=elytron/key-store=twoWayTS:add(path=/tmp/server.truststore.jks,credential-reference={clear-text=secret},type=JKS) /host=master/subsystem=elytron/key-managers=twoWayKM:add(key-store=twoWayKS,algorithm="SunX509",credential-reference={clear-text=secret}) /host=master/subsystem=elytron/trust-managers=twoWayTM:add(key-store=twoWayTS,algorithm="SunX509") /host=master/subsystem=elytron/server-ssl-context=twoWaySSC:add(key-managers=twoWayKM,protocols=["TLSv1.2"],trust-managers=twoWayTM,want-client-auth=true,need-client-auth=true) run-batch # Enable HTTPS batch /host=master/core-service=management/management-interface=http-interface:write-attribute(name=ssl-context, value=twoWaySSC) run-batch reload --host=master Full agent.log attached, "peer not authenticated" are expected. Those were thrown when incorrect password was used in previous attempts. jboss-cli.sh connected to host controller successfully so configuration should be done correctly. One way ssl is working. Two way ssl is working in standalone mode.
Closing as not a bug because the issue is caused by bug on EAP side - https://issues.jboss.org/browse/JBEAP-10226 When rhqadmin user is added, it's working.
Reopening again. The bug on EAP side is fixed but JON agent is still throwing the error and the resource is marked as down. This affects both standalone and domain mode. http-interface by default uses security realm and configuration described above is just adding ssl-context so it looks like this: <http-interface ssl-context="twoWaySSC" security-realm="ManagementRealm"> The issue described by this bz disappears when security-realm is removed from http-interface or when rhqadmin user is added to mgmt-users.properties Not sure if it's bug on JON agent side or if it's EAP issue similar to JBEAP-10226. When using just legacy security realm (no ssl-context) two way SSL is working fine.
Filip, could you please attach the EAP server configuration?
Created attachment 1281838 [details] standalone xml
Added standalone xml. Note that mgmt-users.properties is empty. The issue can be workarounded by adding rhqadmin user to that file.
In case the security-realm is removed, :whoami operation of jboss-cli results in "result" => {"identity" => {"username" => "anonymous"}} In the other case "result" => {"identity" => { "username" => "$local", "realm" => "ManagementRealm" }}
I have to say, I'm not sure what isn't working? The environment in miburman-jon33 is working just fine with the TwoWAY-SSL? I went to the EAP and created the management-user manually and set that information as the connection info. The connection worked fine and my EAP is shown as up. We haven't supported local-authentication before for management properties (only for deployment).
When using Client Certificate Authentication the rhqadmin management-user should not be required. At least it's not required when using legacy security realm (no ssl-context). 1 - legacy security realm (no ssl-context), NO rhqadmin user in mgmt-users.properties -> OK 2 - using ssl-context, NO rhqadmin user in mgmt-users.properties -> KO 3 - using ssl-context, rhqadmin user in mgmt-users.properties -> OK Attaching working configuration (1) host-secured-changed-ok.xml and not working (2) host-secured-changed-ko.xml. So question is if (2) is expected and it's caused by incorrectly configured EAP or if it's issue on JON side. I don't understand why it's working when the security-realm="ManagementRealm" is removed (see comment 2)
Created attachment 1342212 [details] host-secured-changed-ko.xml
Created attachment 1342213 [details] host-secured-changed-ok.xml
Maybe Ondrej can answer that one ? As far as I understood Ondrej's reply, there shouldn't be a difference in authentication if two-way SSL is used, so with legacy either EAP works incorrectly or we do it incorrectly (and by accident use local authentication or something like that). I'm not sure why by default local isn't used by the agent though as default, maybe there's a historical reason for this? If the password and username are set, we issue: httpClient.getCredentialsProvider().setCredentials(new AuthScope(asConnectionParams.getHost(), asConnectionParams.getPort()), credentials); If not, we don't send the information. Our authentication process has no knowledge if SSL is even used or not.
The scenario 2 (using ssl-context, NO rhqadmin user in mgmt-users.properties) works for me as expected (standalone, domain). Where does rhqadmin comes from? Is it CN of the certificate? Is there defined a user in a client process? (e.g. ./bin/jboss-cli.sh -c -u=rhqadmin) What are the results from :whoami operation in the scenarios? I would expect 1 ~ rhqadmin, 2 ~ $local, 3 ~ rhqadmin. AFAIK: There is just $local user available for scenario 2, because Elytron offers just 2-way SSL and the legacy security realm offers just authentication for local anonymous user. In scenarion 3, Elytron offers just 2-way SSL and the legacy SSL offers just authentication for rhqadmin and $local users. In scenarion 1, the legacy realm ensures 2-way SSL and also performs authentication based on client certificate.
The "rhqadmin" is sent as a HTTP authentication header (we don't use jboss-cli). That username/password combination can of course be changed. https://github.com/rhq-project/rhq/blob/master/modules/plugins/wfly-10/src/main/java/org/rhq/modules/plugins/wildfly10/ASConnection.java Lines 166-170 and 202-205 are the ones handling any sort of authentication header. UsernamePasswordCredentials is from Apache Commons HTTP client.