Bug 1248156
| Summary: | [GSS](6.4.z) Remoting Subsystem RemoteOutboundConnectionService is caching ConnectionURI causing issues when DNS changes | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Ricardo Martinelli de Oliveira <rmartine> | ||||||||
| Component: | Remoting | Assignee: | Brad Maxwell <bmaxwell> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jitka Kozana <jkudrnac> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 6.4.4 | CC: | bbaranow, bmaxwell, brian.stansberry, cdewolf, david.lloyd, fspolti, istudens, jbilek, jmartisk, psotirop, tfonteyn | ||||||||
| Target Milestone: | CR1 | Keywords: | Reopened | ||||||||
| Target Release: | EAP 6.4.6 | Flags: | psotirop:
needinfo+
|
||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2017-01-17 11:48:55 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1235746 | ||||||||||
| Attachments: |
|
||||||||||
Adding a test case that shows that the change of EJBClientContext from a context of some Dns host to localhost and back to Dns host works fine : https://github.com/panossot/jboss-eap/commit/b17fdcd5e056647aebfe4a3670d0e020803a736f A workaround to the customers issue could be to have two urls in hosts file -one for test and one for prod server- and change the ejb-client.propetries host url to the one he would like. In another test that automatically changes the DNS url in hosts file to localhost and back to the original url.The change of the DNS url in hosts file, does not have any effect at all if the jvm is not restarted. Tested with eap quickstart ejb-remote example and the change of FQDN in hosts file works fine for 6.4.x. I have started two servers in different ips, loaded some ejb on both of them (different for each server). Then I call the ejb-client. It works for the one server. I change the FQDN in the hosts file to point to the other server. It works. I change it back and it works. I would like to reopen this BZ.
I made some tests and I was able to reproduce it using EAP 6.4.4.
I attached a simple reproducer, How it works:
client -> serverA (outbound connection) -> serverB
I totally disable the DNS caching and by using a code I get the IP from FQDN i was able to identify that the IP changed, but the outbound connection is always using the IP which was resolved during the boot.
How to reproduce:
Install a instance of EAP 6.4.4 (serverA):
-> add a application user ejb/redhat1@
-> add a ejb security-ream, on CLI:
-> /core-service=management/security-realm=ejb-security-realm:add()
-> /core-service=management/security-realm=ejb-security-realm/server-identity=secret:add(value="cmVkaGF0MUA=")
-> Add the outbound connection on remoting subsystem:
<outbound-connections>
<remote-outbound-connection name="ejb-outbound-connection" outbound-socket-binding-ref="remote-server-1" username="ejb" security-realm="ejb-security-realm">
<properties>
<property name="SSL_ENABLED" value="false"/>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
</properties>
</remote-outbound-connection>
</outbound-connections>
-> add the outbound-socket-binding:
<outbound-socket-binding name="remote-server-1">
<remote-destination host="server-1" port="4448"/>
</outbound-socket-binding>
-> add a entry in the /etc/hosts file: 127.0.0.1 server-1
-> Deploy the ejb-server.jar on this EAP instance
-> Start the EAP with the following parameters:
-> -Djboss.node.name=serverA -Dsun.net.inetaddr.ttl=0 -Dsun.net.inetaddr.negative.ttl=0
Install a new EAP 6.4.4 instance(serverB):
-> deploy the file ejb-server1.jar on this instance.
-> start this instance with the following parameters:
-> Djboss.socket.binding.port-offset=1 -Djboss.node.name=serverB -b REAL_HOST_IP
Execute the file ejb-client-jar-with-dependencies.jar (java -jar ...)
At this moment the server-1 is pointing to 127.0.0.1 (the EJB request will fail, but we can see the IP being used and see that no DNS cache is done):
16:01:03,345 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 2) Remote Address server-1/127.0.0.1
16:01:03,346 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 2) EJBServerImpl called, calling server-1
16:01:03,346 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 1) Remote Address server-1/127.0.0.1
16:01:03,352 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 1) EJBServerImpl called, calling server-1
16:01:03,354 INFO [org.jboss.ejb.client] (EJB default - 2) JBoss EJB Client version 1.0.31.Final-redhat-1
16:01:03,368 ERROR [stderr] (EJB default - 2) java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:ejb-server1, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@1da2e06c
Now point server-1 to REAL_HOST_IP and execute the client again:
16:01:57,863 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 3) Remote Address server-1/192.168.200.200
16:01:57,863 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 3) EJBServerImpl called, calling server-1
16:01:57,870 ERROR [stderr] (EJB default - 3) java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:ejb-server1, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@5e69569f
16:01:57,870 ERROR [stderr] (EJB default - 3) at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:747)
16:01:57,870 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 4) Remote Address server-1/192.168.200.200
16:01:57,870 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 4) EJBServerImpl called, calling server-1
Now restart the serverA and execute the client again:
16:03:56,403 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 1) Remote Address server-1/10.96.65.161
16:03:56,403 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 2) Remote Address server-1/10.96.65.161
16:03:56,403 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 1) EJBServerImpl called, calling server-1
16:03:56,403 INFO [com.exmaples.ejb.server.EJBServerImpl] (EJB default - 2) EJBServerImpl called, calling server-1
16:03:56,408 INFO [org.jboss.ejb.client] (EJB default - 2) JBoss EJB Client version 1.0.31.Final-redhat-1
The EJB call will work.
Created attachment 1090752 [details]
ejb client
Created attachment 1090753 [details]
ejb-server.jar (serverA)
Created attachment 1090754 [details]
ejb-server1.jar (serverB)
PR upstream: https://github.com/wildfly/wildfly-core/pull/1229 Verified with EAP 6.4.6.CP.CR2. Jan Martiska <jmartisk> updated the status of jira JBEAP-1960 to Reopened I am reopening this issue, because scenario described in [1] is not working. [1] https://issues.jboss.org/browse/JBEAP-1960?focusedCommentId=13151827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13151827 Brad Maxwell <bmaxwell> updated the status of jira JBEAP-1960 to Coding In Progress Brad Maxwell <bmaxwell> updated the status of jira JBEAP-1960 to Open Jan Martiska <jmartisk> updated the status of jira JBEAP-1960 to Resolved Jiri Pallich <jpallich> updated the status of jira JBEAP-1960 to Closed Retroactively bulk-closing issues from released EAP 6.4 cumulative patches. |
Description of problem: EJB CLient was configured with DNS FQDN to configure access to a remote EJB. If run a simple test adding an entry in /etc/hosts file pointing that FQDN to localhost for tests everything works. However, after finish the tests and remove the entry, the client still connects to localhost instead of resolve the new IP address. Even adding networkaddress.cache.ttl=30 inside security settings didn't work too. Version-Release number of selected component (if applicable): JBoss EAP 6.1.0 How reproducible: Everytime you use DNS names to connect to a remote EJB. Steps to Reproduce: 1. Configure a simple client that connects to a remote EJB using dns name 2. add an entry in /etc/hosts mapping the dns name to localhost 3. run the client code 4. remove the entry in /etc/hosts 5. run the client code again Actual results: EJB remote is still reached from localhost Expected results: After changing DNS record EJB will be reached in this new address Additional info: This code is in the jboss as code, it is a Service which reads the standalone*.xml outbound connection info, remoting/src/main/java/org/jboss/as/remoting/RemoteOutboundConnectionService.java#getConnectionURI() @Override public IoFuture<Connection> connect() throws IOException { final URI uri; try { // we lazily generate the URI on first request to connect() instead of on start() of the service // in order to delay resolving the destination address. No point trying to resolve that address // if nothing really wants to create a connection out of it. uri = this.getConnectionURI(); ... return endpoint.connect(uri, builder.getMap(), callbackHandler, sslContext); } This method is caching it, there is a question though, is connect() called more than once? At startup it would try to connect, and it would stay connected and there is probably a reconnect task somewhere, so there is a question if it calls this connect() method above again or it the endpoint just tries to reconnect. private synchronized URI getConnectionURI() throws IOException, URISyntaxException { if (this.connectionURI != null) { return this.connectionURI; } final OutboundSocketBinding destinationOutboundSocket = this.destinationOutboundSocketBindingInjectedValue.getValue(); final InetAddress destinationAddress = destinationOutboundSocket.getDestinationAddress(); final int port = destinationOutboundSocket.getDestinationPort(); this.connectionURI = new URI(REMOTE_URI_SCHEME + NetworkUtils.formatPossibleIpv6Address( destinationAddress.getHostAddress()) + ":" + port); return this.connectionURI; }