Bug 900198 (JBPAPP6-556)
| Summary: | Huge number of tests fail in Hibernate testsuite after upgrade to 4.1.2 due to closed Connection or too many open connections across most of certified databases | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Madhumita Sadhukhan <msadhukh> | ||||
| Component: | Hibernate | Assignee: | Strong Liu <stliu> | ||||
| Status: | CLOSED NEXTRELEASE | QA Contact: | |||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 6.0.0 | CC: | msadhukh, pslavice, steve, stliu, zroubali | ||||
| Target Milestone: | --- | ||||||
| Target Release: | EAP 6.0.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| URL: | http://jira.jboss.org/jira/browse/JBPAPP6-556 | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2012-05-02 08:52:32 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 900296 | ||||||
| Attachments: |
|
||||||
|
Description
Madhumita Sadhukhan
2012-04-12 14:35:08 UTC
Madhumita, are you planning further investigation on this? I am also having this assigned to Strong, so he's aware of the regression from the JIRA notification. Rajesh,
yes I planned to assign it to Strong only after some more further investigation as the failures give different messages on different databases but I feel that this happens due to too many connections opened by some testcase added/modified in ER5.
However I still need to investigate the failures to find if there are multiple reasons why these fail as different number and different sets of tests fail on different DBs.
Also I was waiting for re-run against ER4.1 to complete to ensure that this is not an environmental(dballocator/DB server) issue at all and it already looks good
https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Hibernate/job/eap6-hibernate-core-master-testsuite/178/
Strong,
I started a separate run on few databases(DB2,Sybase,mysql55) with ER5 bits and the gradle patch but minus the LockModeTest(which was only hanging on DB2 and Sybase until ER4.1).Surprisingly this yielded successful testsuite run with few failures without above specified connection issues.
http://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Hibernate/job/eap6-hibernate-core-master-testsuite-testDB2Sybase/13/
Could you please confirm if something was modified in this test?
Hope this helps in nailing down the problem.
Seems like the culprit is indeed LockModeTest Here comes a much stable run across all certified DBs with ER5 minus above test http://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Hibernate/job/eap6-hibernate-core-master-testsuite-testDB2Sybase/14/ I just do not see how LockModeTest results in the behavior you describe here. The only thing I can think of is some kind of strange environment issue. I am running Oracle Java 6 for Linux here and that test exhibits exactly the behavior it should. What kind of environment are you using in these runs? I guess it is possible that if some of these databases do not like the attempt to rollback the transaction from {{org.hibernate.test.locking.LockModeTest#nowAttemptToUpdateRow}} you could potentially see this problem (because closing the session would get skipped and therefore closing the connection would never happen). Seems pretty far fetched, but I've seen JDBC drivers do worse. We could try adding protections for that situation and see if it helps.
Steve, seems like the _org.hibernate.testing.jta.TestingJtaBootstrap_ is the culprit, it opens too many connections but never close them I tried run _org.hibernate.test.nonflushedchanges.MergeTest_ on my local with MySQL, and this only test case fails due to too many connections ------------------------------------------ ``` mysql> show processlist; +------+-----------+-----------------+-----------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------+-----------------+-----------+---------+------+-------+------------------+ | 897 | root | localhost | NULL | Query | 0 | NULL | show processlist | | 900 | hibernate | localhost:61620 | hibernate | Sleep | 16 | | NULL | | 901 | hibernate | localhost:61621 | hibernate | Sleep | 16 | | NULL | | 902 | hibernate | localhost:61622 | hibernate | Sleep | 16 | | NULL | | 903 | hibernate | localhost:61623 | hibernate | Sleep | 16 | | NULL | | 904 | hibernate | localhost:61624 | hibernate | Sleep | 16 | | NULL | | 905 | hibernate | localhost:61625 | hibernate | Sleep | 14 | | NULL | | 906 | hibernate | localhost:61626 | hibernate | Sleep | 15 | | NULL | | 907 | hibernate | localhost:61627 | hibernate | Sleep | 14 | | NULL | | 908 | hibernate | localhost:61628 | hibernate | Sleep | 14 | | NULL | | 909 | hibernate | localhost:61629 | hibernate | Sleep | 14 | | NULL | | 910 | hibernate | localhost:61630 | hibernate | Sleep | 14 | | NULL | | 911 | hibernate | localhost:61631 | hibernate | Sleep | 14 | | NULL | | 912 | hibernate | localhost:61632 | hibernate | Sleep | 14 | | NULL | | 913 | hibernate | localhost:61633 | hibernate | Sleep | 14 | | NULL | | 914 | hibernate | localhost:61634 | hibernate | Sleep | 14 | | NULL | | 915 | hibernate | localhost:61635 | hibernate | Sleep | 14 | | NULL | | 916 | hibernate | localhost:61636 | hibernate | Sleep | 14 | | NULL | | 917 | hibernate | localhost:61637 | hibernate | Sleep | 14 | | NULL | | 918 | hibernate | localhost:61638 | hibernate | Sleep | 14 | | NULL | | 919 | hibernate | localhost:61639 | hibernate | Sleep | 14 | | NULL | | 920 | hibernate | localhost:61640 | hibernate | Sleep | 14 | | NULL | | 921 | hibernate | localhost:61641 | hibernate | Sleep | 14 | | NULL | | 922 | hibernate | localhost:61642 | hibernate | Sleep | 14 | | NULL | | 923 | hibernate | localhost:61643 | hibernate | Sleep | 14 | | NULL | | 924 | hibernate | localhost:61644 | hibernate | Sleep | 14 | | NULL | | 925 | hibernate | localhost:61645 | hibernate | Sleep | 14 | | NULL | | 926 | hibernate | localhost:61646 | hibernate | Sleep | 14 | | NULL | | 927 | hibernate | localhost:61647 | hibernate | Sleep | 14 | | NULL | | 928 | hibernate | localhost:61648 | hibernate | Sleep | 14 | | NULL | | 929 | hibernate | localhost:61649 | hibernate | Sleep | 14 | | NULL | | 930 | hibernate | localhost:61650 | hibernate | Sleep | 14 | | NULL | | 931 | hibernate | localhost:61651 | hibernate | Sleep | 14 | | NULL | | 932 | hibernate | localhost:61652 | hibernate | Sleep | 14 | | NULL | | 936 | hibernate | localhost:61656 | hibernate | Sleep | 7 | | NULL | | 937 | hibernate | localhost:61657 | hibernate | Sleep | 7 | | NULL | | 938 | hibernate | localhost:61658 | hibernate | Sleep | 7 | | NULL | | 939 | hibernate | localhost:61659 | hibernate | Sleep | 7 | | NULL | | 940 | hibernate | localhost:61660 | hibernate | Sleep | 7 | | NULL | | 941 | hibernate | localhost:61661 | hibernate | Sleep | 7 | | NULL | | 942 | hibernate | localhost:61662 | hibernate | Sleep | 7 | | NULL | | 943 | hibernate | localhost:61663 | hibernate | Sleep | 6 | | NULL | | 944 | hibernate | localhost:61664 | hibernate | Sleep | 7 | | NULL | | 945 | hibernate | localhost:61665 | hibernate | Sleep | 7 | | NULL | | 946 | hibernate | localhost:61666 | hibernate | Sleep | 7 | | NULL | | 947 | hibernate | localhost:61667 | hibernate | Sleep | 7 | | NULL | | 948 | hibernate | localhost:61668 | hibernate | Sleep | 7 | | NULL | | 949 | hibernate | localhost:61669 | hibernate | Sleep | 7 | | NULL | | 950 | hibernate | localhost:61670 | hibernate | Sleep | 7 | | NULL | | 951 | hibernate | localhost:61671 | hibernate | Sleep | 7 | | NULL | | 952 | hibernate | localhost:61672 | hibernate | Sleep | 7 | | NULL | | 953 | hibernate | localhost:61673 | hibernate | Sleep | 6 | | NULL | | 954 | hibernate | localhost:61674 | hibernate | Sleep | 6 | | NULL | | 955 | hibernate | localhost:61675 | hibernate | Sleep | 6 | | NULL | | 956 | hibernate | localhost:61676 | hibernate | Sleep | 6 | | NULL | | 957 | hibernate | localhost:61677 | hibernate | Sleep | 6 | | NULL | | 958 | hibernate | localhost:61678 | hibernate | Sleep | 6 | | NULL | | 959 | hibernate | localhost:61679 | hibernate | Sleep | 6 | | NULL | | 960 | hibernate | localhost:61680 | hibernate | Sleep | 6 | | NULL | | 961 | hibernate | localhost:61681 | hibernate | Sleep | 6 | | NULL | | 962 | hibernate | localhost:61682 | hibernate | Sleep | 6 | | NULL | | 963 | hibernate | localhost:61683 | hibernate | Sleep | 6 | | NULL | | 964 | hibernate | localhost:61684 | hibernate | Sleep | 6 | | NULL | | 965 | hibernate | localhost:61685 | hibernate | Sleep | 6 | | NULL | | 966 | hibernate | localhost:61686 | hibernate | Sleep | 6 | | NULL | | 967 | hibernate | localhost:61687 | hibernate | Sleep | 6 | | NULL | | 968 | hibernate | localhost:61688 | hibernate | Sleep | 6 | | NULL | | 969 | hibernate | localhost:61689 | hibernate | Sleep | 6 | | NULL | | 970 | hibernate | localhost:61690 | hibernate | Sleep | 6 | | NULL | | 971 | hibernate | localhost:61691 | hibernate | Sleep | 6 | | NULL | | 972 | hibernate | localhost:61692 | hibernate | Sleep | 6 | | NULL | | 973 | hibernate | localhost:61693 | hibernate | Sleep | 6 | | NULL | | 974 | hibernate | localhost:61694 | hibernate | Sleep | 6 | | NULL | | 975 | hibernate | localhost:61695 | hibernate | Sleep | 6 | | NULL | | 976 | hibernate | localhost:61696 | hibernate | Sleep | 6 | | NULL | | 977 | hibernate | localhost:61697 | hibernate | Sleep | 6 | | NULL | | 978 | hibernate | localhost:61698 | hibernate | Sleep | 6 | | NULL | | 979 | hibernate | localhost:61699 | hibernate | Sleep | 6 | | NULL | | 980 | hibernate | localhost:61700 | hibernate | Sleep | 6 | | NULL | | 981 | hibernate | localhost:61701 | hibernate | Sleep | 6 | | NULL | | 982 | hibernate | localhost:61702 | hibernate | Sleep | 6 | | NULL | | 983 | hibernate | localhost:61703 | hibernate | Sleep | 5 | | NULL | | 984 | hibernate | localhost:61704 | hibernate | Sleep | 6 | | NULL | | 985 | hibernate | localhost:61705 | hibernate | Sleep | 6 | | NULL | | 986 | hibernate | localhost:61706 | hibernate | Sleep | 5 | | NULL | | 987 | hibernate | localhost:61707 | hibernate | Sleep | 5 | | NULL | | 988 | hibernate | localhost:61708 | hibernate | Sleep | 5 | | NULL | | 989 | hibernate | localhost:61709 | hibernate | Sleep | 5 | | NULL | | 990 | hibernate | localhost:61710 | hibernate | Sleep | 5 | | NULL | | 991 | hibernate | localhost:61711 | hibernate | Sleep | 5 | | NULL | | 992 | hibernate | localhost:61712 | hibernate | Sleep | 5 | | NULL | | 993 | hibernate | localhost:61713 | hibernate | Sleep | 5 | | NULL | | 994 | hibernate | localhost:61714 | hibernate | Sleep | 5 | | NULL | | 995 | hibernate | localhost:61715 | hibernate | Sleep | 5 | | NULL | | 996 | hibernate | localhost:61716 | hibernate | Sleep | 5 | | NULL | | 997 | hibernate | localhost:61717 | hibernate | Sleep | 5 | | NULL | | 998 | hibernate | localhost:61718 | hibernate | Sleep | 5 | | NULL | | 999 | hibernate | localhost:61719 | hibernate | Sleep | 5 | | NULL | | 1000 | hibernate | localhost:61720 | hibernate | Sleep | 5 | | NULL | | 1001 | hibernate | localhost:61721 | hibernate | Sleep | 5 | | NULL | | 1002 | hibernate | localhost:61722 | hibernate | Sleep | 5 | | NULL | | 1003 | hibernate | localhost:61723 | hibernate | Sleep | 5 | | NULL | | 1004 | hibernate | localhost:61724 | hibernate | Sleep | 5 | | NULL | | 1005 | hibernate | localhost:61725 | hibernate | Sleep | 5 | | NULL | | 1006 | hibernate | localhost:61726 | hibernate | Sleep | 5 | | NULL | | 1007 | hibernate | localhost:61727 | hibernate | Sleep | 5 | | NULL | | 1008 | hibernate | localhost:61728 | hibernate | Sleep | 5 | | NULL | | 1009 | hibernate | localhost:61729 | hibernate | Sleep | 5 | | NULL | | 1010 | hibernate | localhost:61730 | hibernate | Sleep | 4 | | NULL | | 1011 | hibernate | localhost:61731 | hibernate | Sleep | 5 | | NULL | | 1012 | hibernate | localhost:61732 | hibernate | Sleep | 5 | | NULL | | 1013 | hibernate | localhost:61733 | hibernate | Sleep | 5 | | NULL | | 1014 | hibernate | localhost:61734 | hibernate | Sleep | 5 | | NULL | | 1015 | hibernate | localhost:61735 | hibernate | Sleep | 5 | | NULL | | 1016 | hibernate | localhost:61736 | hibernate | Sleep | 5 | | NULL | | 1017 | hibernate | localhost:61737 | hibernate | Sleep | 5 | | NULL | | 1018 | hibernate | localhost:61738 | hibernate | Sleep | 5 | | NULL | | 1019 | hibernate | localhost:61740 | hibernate | Sleep | 4 | | NULL | | 1020 | hibernate | localhost:61741 | hibernate | Sleep | 4 | | NULL | | 1021 | hibernate | localhost:61742 | hibernate | Sleep | 4 | | NULL | | 1022 | hibernate | localhost:61743 | hibernate | Sleep | 4 | | NULL | | 1023 | hibernate | localhost:61744 | hibernate | Sleep | 4 | | NULL | | 1024 | hibernate | localhost:61745 | hibernate | Sleep | 4 | | NULL | | 1025 | hibernate | localhost:61746 | hibernate | Sleep | 4 | | NULL | | 1026 | hibernate | localhost:61747 | hibernate | Sleep | 4 | | NULL | | 1027 | hibernate | localhost:61748 | hibernate | Sleep | 4 | | NULL | | 1028 | hibernate | localhost:61749 | hibernate | Sleep | 4 | | NULL | | 1029 | hibernate | localhost:61750 | hibernate | Sleep | 4 | | NULL | | 1030 | hibernate | localhost:61751 | hibernate | Sleep | 4 | | NULL | | 1031 | hibernate | localhost:61752 | hibernate | Sleep | 4 | | NULL | | 1032 | hibernate | localhost:61753 | hibernate | Sleep | 4 | | NULL | | 1033 | hibernate | localhost:61754 | hibernate | Sleep | 4 | | NULL | | 1034 | hibernate | localhost:61755 | hibernate | Sleep | 4 | | NULL | | 1035 | hibernate | localhost:61756 | hibernate | Sleep | 4 | | NULL | | 1036 | hibernate | localhost:61757 | hibernate | Sleep | 3 | | NULL | | 1037 | hibernate | localhost:61758 | hibernate | Sleep | 3 | | NULL | | 1038 | hibernate | localhost:61759 | hibernate | Sleep | 3 | | NULL | | 1039 | hibernate | localhost:61760 | hibernate | Sleep | 3 | | NULL | | 1040 | hibernate | localhost:61761 | hibernate | Sleep | 3 | | NULL | | 1041 | hibernate | localhost:61762 | hibernate | Sleep | 3 | | NULL | | 1042 | hibernate | localhost:61763 | hibernate | Sleep | 3 | | NULL | | 1043 | hibernate | localhost:61764 | hibernate | Sleep | 3 | | NULL | | 1044 | hibernate | localhost:61765 | hibernate | Sleep | 3 | | NULL | | 1045 | hibernate | localhost:61766 | hibernate | Sleep | 3 | | NULL | | 1046 | hibernate | localhost:61767 | hibernate | Sleep | 3 | | NULL | | 1047 | hibernate | localhost:61768 | hibernate | Sleep | 3 | | NULL | | 1048 | hibernate | localhost:61769 | hibernate | Sleep | 3 | | NULL | | 1049 | hibernate | localhost:61770 | hibernate | Sleep | 3 | | NULL | | 1050 | hibernate | localhost:61771 | hibernate | Sleep | 3 | | NULL | | 1051 | hibernate | localhost:61772 | hibernate | Sleep | 3 | | NULL | | 1052 | hibernate | localhost:61773 | hibernate | Sleep | 3 | | NULL | | 1053 | hibernate | localhost:61774 | hibernate | Sleep | 3 | | NULL | +------+-----------+-----------------+-----------+---------+------+-------+------------------+ ``` Madhumita please give this patch a try (do not remove LockModeTest) Attachment: Added: JBPAPP-8695.patch even with my patch, there are even two connections living (see the time column)
{quote}
mysql> show processlist;
+------+-----------+-----------------+-----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------+-----------------+-----------+---------+------+-------+------------------+
| 897 | root | localhost | NULL | Query | 0 | NULL | show processlist |
| 1849 | hibernate | localhost:63146 | hibernate | Sleep | 1991 | | NULL |
| 1968 | hibernate | localhost:63300 | hibernate | Sleep | 1810 | | NULL |
| 2841 | hibernate | localhost:65172 | hibernate | Query | 0 | NULL | commit |
+------+-----------+-----------------+-----------+---------+------+-------+------------------+
4 rows in set (0.00 sec)
{quote}
Attachment: Removed: JBPAPP-8695.patch Attachment: Added: JBPAPP-8695.patch Madhumita why do you use 38a6c426ff1dc7a70f3eaec8ec90e744d02e1ae3 instead of the 4.1.2.Final? Strong ,
I am using 4.1.2.Final tag as you specified in both the jobs above.Will try your patch.
That patch wont work Strong. Without it, connections are not bound to the JTA transactions. There is a reason I added that ;) So we need to find out how to get the Enhydra DataSource to actually release connections in this case. That, or get IronJacamar actually figure out how to get Iron Jacamar to work outside AS and use it for DataSource Steve/Strong, Results after application of above patch as follows: https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Hibernate/job/eap6-hibernate-core-master-testsuite-testDB2Sybase/18/ Do not see above mass Connection failures Apart from DB2 which hangs on LockModetest(already known issue) and oracle11gR1RAC which has dballocation issues(will be fixed soon) https://hibernate.onjira.com/browse/HHH-7253 for those tests failing on oracle Steve, I haven't found a fix for this, can you give some help? according to the hudson result, now there is only two tests fail, so I'm closing this CI reports: 1 test fails on mssql, oracle, postgresql 1 more test fails on postgresql and that test hangs on DB2 issue -------------------------- org.hibernate.envers.test.integration.inheritance.tableperclass.ParentAuditing is one of failing tests currently ( fails on ms sql server, postgresql ) this fails on the second run of the test (e.g. with property "org.hibernate.envers.audit_strategy" set to "org.hibernate.envers.strategy.ValidityAuditStrategy") and in the method _org.hibernate.envers.strategy.ValidityAuditStrategy#perform_, it applies a UPGRADE lock to the query List<Object> l = qb.toQuery(session).setLockOptions(LockOptions.UPGRADE).list(); which then runs into this jira https://hibernate.onjira.com/browse/HHH-3298 postgresql also has another failure, org.hibernate.envers.test.integration.inheritance.joined.ParentAuditing.initData we had this before, see org.hibernate.test.legacy.FooBarTest#testQueryLockMode and org.hibernate.test.legacy.ParentChildTest#testComplexCriteria in these two tests, we have this code: LockMode lockMode = supportsLockingNullableSideOfJoin( getDialect() ) ? LockMode.UPGRADE : LockMode.READ; protected boolean supportsLockingNullableSideOfJoin(Dialect dialect) { // db2 and pgsql do *NOT* return ! ( DB2Dialect.class.isInstance( dialect ) || PostgreSQLDialect.class.isInstance( dialect ) ); } Link: Added: This issue is a dependency of JBPAPP-8827 We are able to run Hibernate testsuite with EAP6 ER6 without above mass failures due to too many open connections. https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Hibernate/job/eap6-hibernate-core-master-testsuite/185/ Docs QE Status: Removed: NEW |