Bug 1271330

Summary: nunc-stans: Attempt to release connection that is not acquired
Product: Red Hat Enterprise Linux 7 Reporter: Viktor Ashirov <vashirov>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: high Docs Contact:
Priority: high    
Version: 7.2CC: jkurik, mreynolds, msauton, nkinder, rmeggins, vashirov
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.5.2-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1279573 (view as bug list) Environment:
Last Closed: 2016-11-03 20:36:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1279573    
Attachments:
Description Flags
ccpp-2015-10-26-10:10:58-23716.tar.gz none

Description Viktor Ashirov 2015-10-13 16:24:42 UTC 
Description of problem:
DS with nunc stans enabled produces lots of messages like
[13/Oct/2015:11:29:24 -0400] connection - conn=98 fd=161 Attempt to release connection that is not acquired

Version-Release number of selected component (if applicable):
389-ds-base-1.3.4.0-19.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. Enable nunc stans and restart the server
nsslapd-enable-nunc-stans: on

2. Add 1k users with ldclt
ldclt -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 -f cn=MrXXXXXX -b "ou=people,dc=example,dc=com" -e add,person,incr,noloop,commoncounter -r0 -R1000

3. Check errors log for "Attempt to release connection that is not acquired"

Actual results:
grep "Attempt to release connection that is not acquired" -c /var/log/dirsrv/slapd-instance/errors
1024

Expected results:


Additional info:
Comment 1 Rich Megginson 2015-10-13 16:28:40 UTC 
Upstream ticket:
https://fedorahosted.org/389/ticket/48311
Comment 2 mreynolds 2015-10-23 15:12:24 UTC 
I can not reproduce any errors on 1.3.4 or 1.3.5.  I even tried adding 10000 entries.  

I am testing this on f22:

libtevent-devel-0.9.25-1.fc22.x86_64
libevent-2.0.21-6.fc22.x86_64

What version of libtevent/libevent were you testing with?

Are there any other steps I might be missing?
Comment 4 mreynolds 2015-10-23 15:44:34 UTC 
I was able to reproduce this on f22 using an optimized build of DS.  The debug build worked fine though. 

Just doing a single search triggers the error, so it's not a race condition.  Continuing investigation...
Comment 5 mreynolds 2015-10-23 20:46:23 UTC 
Fixed upstream
Comment 6 Viktor Ashirov 2015-10-26 14:24:07 UTC 
Created attachment 1086527 [details]
ccpp-2015-10-26-10:10:58-23716.tar.gz

With the scratch build that contains a fix [1], I had another crash [2] in quickinstall test with nunc-stans enabled.

[1] https://brewweb.devel.redhat.com/taskinfo?taskID=10008266
[2] http://faf-report.itos.redhat.com/reports/12302/
Comment 7 mreynolds 2015-10-26 15:04:25 UTC 
[2]  This crash is new, and it's happening at server startup(this is odd).  Does it happen every time you start the server?  

If it does crash every time you start the server:

    - Perhaps the scratch build was not created correctly?  I noticed that you build using --with-nunc-stans=../nunc-stans-0.1.5

    - Does the following library already exist?

        /usr/lib64/dirsrv/libnunc-stans.so

    - If this library exists, can you remove "--with-nunc-stans=../nunc-stans-0.1.5" from your scratch build configure command,a nd try and rebuild it
Comment 8 Viktor Ashirov 2015-10-26 15:29:33 UTC 
(In reply to mreynolds from comment #7)
> [2]  This crash is new, and it's happening at server startup(this is odd). 
> Does it happen every time you start the server?  
> 
Crash happens every time I start the server with 'nsslapd-enable-nunc-stans: on' in dse.ldif
> If it does crash every time you start the server:
> 
>     - Perhaps the scratch build was not created correctly?  I noticed that
> you build using --with-nunc-stans=../nunc-stans-0.1.5
> 
>     - Does the following library already exist?
> 
>         /usr/lib64/dirsrv/libnunc-stans.so
Yes, it's there:
ls /usr/lib64/dirsrv/libnunc-stans.so -la
lrwxrwxrwx. 1 root root 22 Oct 26 11:13 /usr/lib64/dirsrv/libnunc-stans.so -> libnunc-stans.so.0.0.0
> 
>     - If this library exists, can you remove
> "--with-nunc-stans=../nunc-stans-0.1.5" from your scratch build configure
> command,a nd try and rebuild it

I used to run this command to build 389-ds-base
$ brew build --scratch rhel-7.2-candidate ./389-ds-base-1.3.4.0-19.99.el7.src.rpm
And spec file was modified only to contain new fix, no other changes were made.

I modified spec file to compile without this line, but build failed: 
http://download.devel.redhat.com/brewroot/work/tasks/6745/10016745/build.log
Comment 16 Mike McCune 2016-03-28 23:12:48 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 18 Viktor Ashirov 2016-06-08 15:54:24 UTC 
Build tested: 
389-ds-base-1.3.5.4-1.el7.x86_64

After repeating steps in the description:
# grep "Attempt to release connection that is not acquired" -c /var/log/dirsrv/slapd-rhel7ds/errors
0

Message no longer appears in the logs.

Marking as VERIFIED.
Comment 19 Nathan Kinder 2016-06-17 16:33:18 UTC 
*** Bug 1274377 has been marked as a duplicate of this bug. ***
Comment 23 errata-xmlrpc 2016-11-03 20:36:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2594.html