Bug 1026931

Summary:	1.2.11.29 crash when removing entries from cache
Product:	Red Hat Enterprise Linux 7	Reporter:	Nathan Kinder <nkinder>
Component:	389-ds-base	Assignee:	Rich Megginson <rmeggins>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Sankar Ramalingam <sramling>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	amsharma, nhosoi
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	389-ds-base-1.3.1.6-8.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1026956 (view as bug list)		Environment:
Last Closed:	2014-06-13 11:48:55 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1026956

Description Nathan Kinder 2013-11-05 16:44:25 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47577

Steps:
1) setup up 2 way MMR
2) make sure the entry cache is full, so that search requests must evict old entries from the cache to make room for new entries
3) generate an ADD load against one of the masters
4) repeatedly search for (objectclass=nsTombstone) from the base suffix

Comment 1 Rich Megginson 2013-11-05 17:44:26 UTC

Steps to reproduce:

1) setup one instance of directory server with suffix dc=example,dc=com
2) Use ldclt like this to add a lot of entries:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e add,person,incr,commoncounter,noloop \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
3) sleep 120 to allow some entries to search/mod/del
4) Use ldclt like this to do a lot of modifies:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e incr,commoncounter,attreplace=sn:valueXXXXXXXXXX \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
5) sleep 30 to allow mods to progress
6) Use ldclt like this to do a lot of deletes:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e delete,incr,commoncounter,noloop \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
7) use ldclt like this to do a lot of searches:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e esearch,incr,commoncounter \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &

You should see a crash within 10 minutes or so

Comment 3 Amita Sharma 2014-02-03 09:26:52 UTC

Followed https://bugzilla.redhat.com/show_bug.cgi?id=1026931#c1 ::

Use ldclt like this to add a lot of entries:
going to sleep
ldclt version 4.23
ldclt[28008]: Starting at Mon Feb  3 14:02:06 2014
ldclt[28008]: Average rate:   48.12/thr  (  38.50/sec), total:    385
.
.
.
.
ldclt[28008]: Average rate:   45.00/thr  (  36.00/sec), total:    360
ldclt[28008]: Average rate:   35.75/thr  (  28.60/sec), total:    286
Use ldclt like this to do a lot of modifies:
going to sleep
ldclt version 4.23
ldclt[28051]: Starting at Mon Feb  3 14:04:06 2014

ldclt[28008]: Average rate:   45.75/thr  (  36.60/sec), total:    366
.
.
.
.
ldclt[28008]: Average rate:   24.50/thr  (  19.60/sec), total:    196
ldclt[28051]: Average rate:   15.25/thr  (  12.20/sec), total:    122
Use ldclt like this to do a lot of deletes:
use ldclt like this to do a lot of searches:
[root@dhcp201-149 export]# ldclt[28008]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt version 4.23
Error: illegal option -e search
ldclt version 4.23
ldclt[28179]: Starting at Mon Feb  3 14:04:36 2014

ldclt[28178]: Ending at Mon Feb  3 14:04:36 2014
ldclt[28178]: Exit status 2 - Error in parameters.
ldclt[28008]: Average rate:    4.62/thr  (   3.70/sec), total:     37
.
.
.
.

ldclt[28051]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28179]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt[28179]: T004: Cannot ldap_delete (CN=0002619,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   14.25/thr  (  11.40/sec), total:    114
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   13.38/thr  (  10.70/sec), total:    107
ldclt[28179]: T006: Cannot ldap_delete (CN=0002832,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   17.62/thr  (  14.10/sec), total:    141
ldclt[28051]: Average rate:   17.62/thr  (  14.10/sec), total:    141
ldclt[28179]: Average rate:   17.50/thr  (  14.00/sec), total:    140
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28051]: Average rate:   15.50/thr  (  12.40/sec), total:    124
ldclt[28179]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28008]: Average rate:   14.25/thr  (  11.40/sec), total:    114
ldclt[28051]: Average rate:   14.75/thr  (  11.80/sec), total:    118
ldclt[28179]: Average rate:   14.12/thr  (  11.30/sec), total:    113
ldclt[28008]: Average rate:   16.75/thr  (  13.40/sec), total:    134
ldclt[28051]: Average rate:   16.75/thr  (  13.40/sec), total:    134
ldclt[28179]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28179]: T007: Cannot ldap_delete (CN=0003261,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   15.62/thr  (  12.50/sec), total:    125
ldclt[28051]: Average rate:   16.12/thr  (  12.90/sec), total:    129
ldclt[28179]: Average rate:   16.12/thr  (  12.90/sec), total:    129
ldclt[28179]: T000: Cannot ldap_delete (CN=0003487,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   18.62/thr  (  14.90/sec), total:    149
ldclt[28051]: Average rate:   19.38/thr  (  15.50/sec), total:    155
ldclt[28179]: Average rate:   18.38/thr  (  14.70/sec), total:    147
ldclt[28179]: T006: Cannot ldap_delete (CN=0003554,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28051]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28179]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28008]: Average rate:   15.75/thr  (  12.60/sec), total:    126
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   15.62/thr  (  12.50/sec), total:    125
ldclt[28051]: Average rate:   15.75/thr  (  12.60/sec), total:    126
ldclt[28008]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28179]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: T005: Cannot ldap_delete (CN=0008140,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28179]: T005: Cannot ldap_delete (CN=0008196,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   15.88/thr  (  12.70/sec), total:    127
ldclt[28051]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt[28179]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28008]: Average rate:   14.75/thr  (  11.80/sec), total:    118
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28008]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28051]: Average rate:   15.88/thr  (  12.70/sec), total:    127
ldclt[28179]: Average rate:   15.12/thr  (  12.10/sec), total:    121
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28051]: Average rate:   15.50/thr  (  12.40/sec), total:    124
ldclt[28179]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28179]: T003: Cannot ldap_delete (CN=0008668,DC=EXAMPLE,DC=com), error=1 (Operations error)

ldclt[28008]: Global average rate: 1808.75/thr  ( 16.08/sec), total:  14470
ldclt[28008]: Global number times "no activity" reports: never
ldclt[28008]: Global no error occurs during this session.

ldclt[28179]: Global average rate: 1374.75/thr  ( 12.22/sec), total:  10998
ldclt[28179]: Global number times "no activity" reports: never
ldclt[28179]: Global error  1 (Operations error) occurs    11 times

NOTE :: even after an hr ds does not crash ::
systemctl status dirsrv
dirsrv - 389 Directory Server dhcp201-149.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled)
   Active: active (running) since Mon 2014-02-03 13:59:36 IST; 32min ago
 Main PID: 27761 (ns-slapd)
   CGroup: /system.slice/system-dirsrv.slice/dirsrv
           └─27761 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dhcp201-149 -i /var/run/dirsrv/slapd-dhcp201-149.pid -w /var/run/dirsrv/slapd-dhcp201-149.startpid

Feb 03 13:59:36 dhcp201-149.englab.pnq.redhat.com systemd[1]: Started 389 Directory Server dhcp201-149..

Hence marking bug as VERIFIED.

Comment 4 Ludek Smid 2014-06-13 11:48:55 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.