Bug 1630533

Summary: Critical Bug in omniORB 4.2.0 distributed with EPEL
Product: [Fedora] Fedora EPEL Reporter: Scott M. <smahoney>
Component: omniORBAssignee: Sandro Mani <manisandro>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: epel7CC: beenje, dwille, karlthered, manisandro
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: omniORB-4.2.4-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-21 04:15:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Thread stack trace showing deadlock. none

Description Scott M. 2018-09-18 21:03:08 UTC
Created attachment 1484501 [details]
Thread stack trace showing deadlock.

Description of problem:

When under heavy load applications using omniORB will hang.  An example stack trace is attached that shows that multiple threads are waiting on the same 3 locks which they have locked in separate orders.  The locks never become available.

After conversing with the omniORB maintainer, it became apparent that this was a known problem and that the problem had been addressed by the nightly 4.2-latest snapshot.

Version-Release number of selected component (if applicable):

Running a CentOS 7.5.1804 base OS with the following omniORB RPMs:
  omniORB-devel-4.2.0-3.el7.x86_64
  python-omniORB-4.2.0-6.el7.centos.noarch
  omniORB-utils-4.2.0-3.el7.x86_64
  omniORB-doc-4.2.0-3.el7.x86_64
  omniORBpy-devel-4.2.0-6.el7.centos.noarch
  omniORB-4.2.0-3.el7.x86_64
  omniORBpy-libs-4.2.0-6.el7.centos.x86_64
  omniORB-servers-4.2.0-3.el7.x86_64


How reproducible:

Difficult to reproduce as the race condition that causes this is unknown.  It happens on all systems on which we use omniORB/omniORBpy but completely aperiodically.

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:


Additional info:
After running for over 24 days with the omniORB-4.2-latest and omniORBpy-4.2-latest snapshots no deadlock has been observed.  

Request that the 4.2-latest updates be added to EPEL.  I am not sure what specific commit in the omniORB and omniORBpy source repos fixed the issue.  Only that the 4.2-latest snapshots from approximately August 17th, 2018 address the bug I'm describing.

Comment 1 Sandro Mani 2018-09-18 21:05:51 UTC
So is 4.2.2 also affected? Can the upstream maintainer give more info about which commit fixes the issue?

Comment 2 Scott M. 2018-09-19 02:34:02 UTC
Checking with upstream maintainer to determine exact commit at which the bug was addressed.  Will update as soon as possible.  Thank you for the quick reply.

Comment 3 Benjamin Bertrand 2020-12-15 14:05:41 UTC
Hello,
There is a bug due to race condition fixed in 4.2.2 that I'd also like to see in EPEL 7: https://sourceforge.net/p/tango-cs/bugs/794/
It's probably the same bug described in this issue.

The fix is mentioned in https://sourceforge.net/p/omniorb/svn/HEAD/tree/branches/4_2/omniORB/ReleaseNotes.txt:

 - Fixed a lock order violation that could lead to a deadlock if
   multiple connections were being opened and closed simultaneously in
   thread pool mode.

Is it possible to get 4.2.2 or greater (4.2.4 is in Fedora 33) in EPEL 7?

Comment 4 Fedora Update System 2023-01-11 17:59:25 UTC
FEDORA-EPEL-2023-fea0a556d5 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-fea0a556d5

Comment 5 Fedora Update System 2023-01-12 02:25:53 UTC
FEDORA-EPEL-2023-fea0a556d5 has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-fea0a556d5

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2023-01-21 04:15:59 UTC
FEDORA-EPEL-2023-fea0a556d5 has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.