Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1117821

Summary: [QE] (6.3.0) Intermittent fail of org.jboss.as.test.manualmode.web.ssl.HTTPSWebConnectorTestCase due to connection refused
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Pavel Jelinek <pjelinek>
Component: WebAssignee: Emmanuel Hugonnet (ehsavoie) <ehugonne>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Jelinek <pjelinek>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3.0CC: cdewolf, dosoudil, fbogyai, jason.greene, jcacek, kkhan, mbabacek, ozizka, pslavice, rhatlapa, rmaucher
Target Milestone: ER10Keywords: Regression
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-06 14:39:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
server log
none
Patch that seems to be fixing the issue
none
simple_reproducer_start-stop_script none

Description Pavel Jelinek 2014-07-09 12:57:45 UTC
Created attachment 916767 [details]
server log

See: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-AS-Testsuite/job/eap-6x-as-testsuite-RHEL-matrix-OracleJDK6/178/jdk=java16_default,label_exp=RHEL6%26%26x86_64/testReport/org.jboss.as.test.manualmode.web.ssl/HTTPSWebConnectorTestCase/testVerifyingConnector/

Stacktrace:
org.apache.http.conn.HttpHostConnectException: Connection to https://10.16.92.123:18445 refused
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at org.jboss.as.test.manualmode.web.ssl.HTTPSWebConnectorTestCase.makeCall(HTTPSWebConnectorTestCase.java:393)
	at org.jboss.as.test.manualmode.web.ssl.HTTPSWebConnectorTestCase.testVerifyingConnector(HTTPSWebConnectorTestCase.java:344)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:529)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:570)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:549)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)

Standard Output:

***
found key for : cn=client
chain [0] = [
[
  Version: V3
  Subject: CN=client
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 19275201693583380366390581728081420897594415509017664232691879883549791870126660733475084628823320199464163485196841372610798472971243898539778676749327879682779371366245861648887202409159965785463561014974356194190977510223629679918192352070891756660522591633245695318615844664959336971731666861584583536138467041915128668454660380878624869830807951787836551418450934698214291982929149214629484251867847675201278263723498056527090311443727600366109633720909681679248167464473614157893247306098755518140612071049776094265836280377719238729953112424115862599386922486594189809084374330687836934643153498264735360728021
  public exponent: 65537
  Validity: [From: Wed Oct 30 06:06:27 EDT 2013,
               To: Tue Oct 25 06:06:27 EDT 2033]
  Issuer: CN=client
  SerialNumber: [    7c875a85]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: C4 3B 93 A4 21 40 8B E1   C7 FF 12 61 1C F8 87 6D  .;..!@.....a...m
0010: 06 82 8B DA                                        ....
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 30 F6 50 AC DA 08 98 2B   8D 10 B5 10 89 03 58 AC  0.P....+......X.
0010: 3A 0D D0 27 5C E0 D2 1B   0A 87 FA F6 A8 0F C7 A2  :..'\...........
0020: 98 35 FE 6D B3 E9 FA 2A   C5 7C A5 E4 A7 A3 B3 8A  .5.m...*........
0030: 2F 9B F7 5B A7 B0 40 F0   F7 5F 92 21 9E D3 49 60  /..[..@.._.!..I`
0040: AB 6D 5E 23 C4 EF 77 6A   53 E9 06 02 C4 B3 EE 13  .m^#..wjS.......
0050: B4 AE 10 8D 91 5C DF A4   40 B7 A4 EE 1F 2E D1 5A  .....\..@......Z
0060: A3 1B 98 59 95 2E 3A 58   9A 77 0C 91 37 02 BF B7  ...Y..:X.w..7...
0070: 33 2B 6F A6 8A 8C 44 56   58 0D 43 D5 3D 1B 3E D1  3+o...DVX.C.=.>.
0080: E0 CA 1C 7F D1 4D A2 02   F2 56 45 8E B7 CA 90 0A  .....M...VE.....
0090: D1 D2 29 15 0C 9E 20 E3   B4 FA E3 E0 FC 13 C2 ED  ..)... .........
00A0: 9A D9 FF 64 26 B6 5F D3   7E 1A 77 46 C1 14 61 F4  ...d&._...wF..a.
00B0: 61 08 3A EB B3 EA 5B 68   74 92 72 BD 25 A3 B3 B4  a.:...[ht.r.%...
00C0: C5 85 12 1E 56 B4 72 CD   7A 74 BF 6B 41 23 5C 26  ....V.r.zt.kA#\&
00D0: F9 0E 45 6C E4 FD 99 80   7B 58 2E 9E 36 EA 9E 44  ..El.....X..6..D
00E0: A2 41 B7 6F DA 09 AD 80   30 69 3D 8B 85 3F A1 67  .A.o....0i=..?.g
00F0: 0E 8F ED EF 91 44 FF 0D   1E B6 A1 4E 21 5F C2 3C  .....D.....N!_.<

]
***
adding as trusted cert:
  Subject: CN=server
  Issuer:  CN=server
  Algorithm: RSA; Serial number: 0x396cd2e3
  Valid from Wed Oct 30 06:06:26 EDT 2013 until Tue Oct 25 06:06:26 EDT 2033

trigger seeding of SecureRandom
done seeding SecureRandom
***
found key for : cn=untrusted
chain [0] = [
[
  Version: V3
  Subject: CN=untrusted
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 19445191487012648213312464639091435511249333318726029756847359625391451687790767780671251128596358476220673709080080808306038417021915427787230219398620746737239051058719692538708507319277556018535033011453974186010852551921617826407224818034158407954021007699813342964074033817650569521283069969792975929934886236223424708524095677748856918052061150371504631542043113817572610101520450498747989962536755556696226567733908603104955051486986341569117215662758269125304716938596786940410910016338424790386262186761319612541263732847313450580682774080752452034876743483288926246187481051303134116108574121084135657490887
  public exponent: 65537
  Validity: [From: Wed Oct 30 06:06:28 EDT 2013,
               To: Tue Oct 25 06:06:28 EDT 2033]
  Issuer: CN=untrusted
  SerialNumber: [    6dda3832]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4D 16 E7 6F 02 53 42 5C   1D 51 5D F1 93 75 0D A7  M..o.SB\.Q]..u..
0010: DC 5B 21 F7                                        .[!.
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 4B D3 43 F9 0E 88 F7 2D   E0 32 F8 FF 69 2F 24 F2  K.C....-.2..i/$.
0010: 11 D3 61 77 C6 99 EA D2   8C D4 E7 D3 A8 31 F5 44  ..aw.........1.D
0020: 11 CB E7 21 A2 C0 56 98   A0 8D 8C 8F 67 D1 22 60  ...!..V.....g."`
0030: 08 1A 8F 6A D9 E4 F7 1A   AA 2F 37 0F 71 1F 5E F0  ...j...../7.q.^.
0040: 81 20 4D 39 66 54 B9 09   AA 74 50 F5 59 F1 84 8B  . M9fT...tP.Y...
0050: 92 2D F2 8E 33 E4 9D 22   10 9C D0 C4 60 BC 1C 67  .-..3.."....`..g
0060: 8A 07 58 30 0F F3 88 A8   5D 71 48 D5 6C 09 44 E6  ..X0....]qH.l.D.
0070: 97 75 DD EB C4 64 76 0B   41 33 57 D9 68 7D 79 D4  .u...dv.A3W.h.y.
0080: 85 AE 20 AD 47 56 8C 17   A9 C3 D7 2B 58 08 E4 68  .. .GV.....+X..h
0090: D4 F4 C0 AF D1 09 BC 91   9E 48 74 B5 35 07 6D 2D  .........Ht.5.m-
00A0: 91 DE 7C B7 A4 4B 63 50   41 DE D1 68 03 48 53 09  .....KcPA..h.HS.
00B0: 97 F7 5A 69 8E FA A5 AC   6C A1 6B 36 59 B0 D7 92  ..Zi....l.k6Y...
00C0: 15 05 7E A8 97 F7 E0 64   64 93 76 3A 21 07 A8 72  .......dd.v:!..r
00D0: 7A 00 3C C3 C2 C2 58 87   76 23 81 52 05 DE 85 FD  z.<...X.v#.R....
00E0: 5C D3 88 C1 2F 3F 65 9E   B8 71 75 45 72 6A C2 66  \.../?e..quErj.f
00F0: BC C0 83 12 A3 FD 0D 13   80 78 EF 9B 06 70 79 3D  .........x...py=

]
***
adding as trusted cert:
  Subject: CN=server
  Issuer:  CN=server
  Algorithm: RSA; Serial number: 0x396cd2e3
  Valid from Wed Oct 30 06:06:26 EDT 2013 until Tue Oct 25 06:06:26 EDT 2033

trigger seeding of SecureRandom
done seeding SecureRandom
main, setSoTimeout(0) called
main, called close()
main, called closeInternal(true)
main, called closeSocket(selfInitiated)

Comment 1 Josef Cacek 2014-07-09 13:24:24 UTC
Kabir, I think it could be related to the new commit related to default cipher suites in ER9:
https://github.com/jbossas/jboss-eap/commit/ace8bf98962180b1a4c879edbf8073d538fb90a2
Could you have a look?


Attached server log contains:
05:28:34,537 WARN  [org.apache.tomcat.util.net.jsse] (MSC service thread 1-3) JBWEB009002: Unknown element: eNULL
05:28:34,537 WARN  [org.apache.tomcat.util.net.jsse] (MSC service thread 1-12) JBWEB009002: Unknown element: eNULL
05:28:34,536 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.web.connector.https-verify-true: org.jboss.msc.service.StartException in service jboss.web.connector.https-verify-true: JBAS018007: Error starting web connector
	at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:376)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
Caused by: LifecycleException:  JBWEB000023: Protocol handler initialization failed
	at org.apache.catalina.connector.Connector.init(Connector.java:987)
	at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:318)
	... 5 more

Comment 2 Pavel Jelinek 2014-07-10 07:20:09 UTC
I have tried run this test several times from ER9 source against ER7 and ER9 bits. 
When running against ER7 bits it didn't occure.
See: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-as-testsuite-RHEL-matrix-OracleJDK6-HTTPSWebConnectorTestCase/
When running against ER9 bits it occured in 25% cases. 
See: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-as-testsuite-RHEL-matrix-OracleJDK6-HTTPSWebConnectorTestCase-ER9bits/

Comment 5 Emmanuel Hugonnet (ehsavoie) 2014-07-10 13:44:26 UTC
Created attachment 917082 [details]
Patch that seems to be fixing the issue

synchronizing to avoid multiple calls to the init method.

Comment 6 Michal Karm Babacek 2014-07-10 13:56:55 UTC
Created attachment 917098 [details]
simple_reproducer_start-stop_script

Comment 7 Michal Karm Babacek 2014-07-10 13:58:08 UTC
The reproducer script [1] is a very simple one, the issue seems to be best hit on a multiprocessor system [2] with quite slow execution while having multiple https connectors configured. Some additional exceptions produced by the script [3].

What the script does:
  1. it expects the EAP is running
  2. access https connectors
  3. shutdown
  4. start
  5. goto 2.


[1] attachment 917098 [details]
[2] i.e. RHEL6.5 x86_64 KVM virtualized, 1 CPU: 0 hits in an hour, 2 CPUs: several hits in ~40 minutes, Oracle JDK 1.6.0_45.
[3] http://pastebin.test.redhat.com/220733

Comment 8 Rémy Maucherat 2014-07-10 14:08:40 UTC
Committed as r2474.

Comment 9 Pavel Jelinek 2014-07-10 14:13:34 UTC
When running this single test on dev106.mw.lab.eng.bos.redhat.com it fails in cca 80% cases. With jbossweb.jar which Emmanuel sent me this test passed all 5 consequential runs.

Comment 10 Pavel Jelinek 2014-07-10 14:39:42 UTC
To clarify: with original jbossweb the test had failed in 80% cases on that machine. 
With the new jbossweb sent by Emmanuel the test didn't fail any more.

Comment 11 Pavel Jelinek 2014-07-10 15:43:29 UTC
I didn't run all regression tests with this fix. I ran just this single testcase many times on this dev106.mw.lab.eng.bos.redhat.com.

Comment 13 Pavel Jelinek 2014-07-22 15:08:04 UTC
Fixed in EAP 6.3.0.ER10.