Bug 1026931 - 1.2.11.29 crash when removing entries from cache
1.2.11.29 crash when removing entries from cache
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Rich Megginson
Sankar Ramalingam
:
Depends On:
Blocks: 1026956
  Show dependency treegraph
 
Reported: 2013-11-05 11:44 EST by Nathan Kinder
Modified: 2014-06-17 23:01 EDT (History)
2 users (show)

See Also:
Fixed In Version: 389-ds-base-1.3.1.6-8.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1026956 (view as bug list)
Environment:
Last Closed: 2014-06-13 07:48:55 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nathan Kinder 2013-11-05 11:44:25 EST
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47577

Steps:
1) setup up 2 way MMR
2) make sure the entry cache is full, so that search requests must evict old entries from the cache to make room for new entries
3) generate an ADD load against one of the masters
4) repeatedly search for (objectclass=nsTombstone) from the base suffix
Comment 1 Rich Megginson 2013-11-05 12:44:26 EST
Steps to reproduce:

1) setup one instance of directory server with suffix dc=example,dc=com
2) Use ldclt like this to add a lot of entries:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e add,person,incr,commoncounter,noloop \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
3) sleep 120 to allow some entries to search/mod/del
4) Use ldclt like this to do a lot of modifies:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e incr,commoncounter,attreplace=sn:valueXXXXXXXXXX \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
5) sleep 30 to allow mods to progress
6) Use ldclt like this to do a lot of deletes:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e delete,incr,commoncounter,noloop \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &
7) use ldclt like this to do a lot of searches:
    ldclt -h $HOST -p $PORT -D "$LDCLTDN" -w "$LDCLTPW" \
        -e esearch,incr,commoncounter \
        -r1 -R1000000 \
        -n8 \
        -f CN=XXXXXXX -b DC=EXAMPLE,DC=com \
        -q &

You should see a crash within 10 minutes or so
Comment 3 Amita Sharma 2014-02-03 04:26:52 EST
Followed https://bugzilla.redhat.com/show_bug.cgi?id=1026931#c1 ::

Use ldclt like this to add a lot of entries:
going to sleep
ldclt version 4.23
ldclt[28008]: Starting at Mon Feb  3 14:02:06 2014
ldclt[28008]: Average rate:   48.12/thr  (  38.50/sec), total:    385
.
.
.
.
ldclt[28008]: Average rate:   45.00/thr  (  36.00/sec), total:    360
ldclt[28008]: Average rate:   35.75/thr  (  28.60/sec), total:    286
Use ldclt like this to do a lot of modifies:
going to sleep
ldclt version 4.23
ldclt[28051]: Starting at Mon Feb  3 14:04:06 2014

ldclt[28008]: Average rate:   45.75/thr  (  36.60/sec), total:    366
.
.
.
.
ldclt[28008]: Average rate:   24.50/thr  (  19.60/sec), total:    196
ldclt[28051]: Average rate:   15.25/thr  (  12.20/sec), total:    122
Use ldclt like this to do a lot of deletes:
use ldclt like this to do a lot of searches:
[root@dhcp201-149 export]# ldclt[28008]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt version 4.23
Error: illegal option -e search
ldclt version 4.23
ldclt[28179]: Starting at Mon Feb  3 14:04:36 2014

ldclt[28178]: Ending at Mon Feb  3 14:04:36 2014
ldclt[28178]: Exit status 2 - Error in parameters.
ldclt[28008]: Average rate:    4.62/thr  (   3.70/sec), total:     37
.
.
.
.

ldclt[28051]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28179]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt[28179]: T004: Cannot ldap_delete (CN=0002619,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   14.25/thr  (  11.40/sec), total:    114
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   13.38/thr  (  10.70/sec), total:    107
ldclt[28179]: T006: Cannot ldap_delete (CN=0002832,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   17.62/thr  (  14.10/sec), total:    141
ldclt[28051]: Average rate:   17.62/thr  (  14.10/sec), total:    141
ldclt[28179]: Average rate:   17.50/thr  (  14.00/sec), total:    140
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28051]: Average rate:   15.50/thr  (  12.40/sec), total:    124
ldclt[28179]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28008]: Average rate:   14.25/thr  (  11.40/sec), total:    114
ldclt[28051]: Average rate:   14.75/thr  (  11.80/sec), total:    118
ldclt[28179]: Average rate:   14.12/thr  (  11.30/sec), total:    113
ldclt[28008]: Average rate:   16.75/thr  (  13.40/sec), total:    134
ldclt[28051]: Average rate:   16.75/thr  (  13.40/sec), total:    134
ldclt[28179]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28179]: T007: Cannot ldap_delete (CN=0003261,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   15.62/thr  (  12.50/sec), total:    125
ldclt[28051]: Average rate:   16.12/thr  (  12.90/sec), total:    129
ldclt[28179]: Average rate:   16.12/thr  (  12.90/sec), total:    129
ldclt[28179]: T000: Cannot ldap_delete (CN=0003487,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   18.62/thr  (  14.90/sec), total:    149
ldclt[28051]: Average rate:   19.38/thr  (  15.50/sec), total:    155
ldclt[28179]: Average rate:   18.38/thr  (  14.70/sec), total:    147
ldclt[28179]: T006: Cannot ldap_delete (CN=0003554,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28051]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28179]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28008]: Average rate:   15.75/thr  (  12.60/sec), total:    126
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   15.62/thr  (  12.50/sec), total:    125
ldclt[28051]: Average rate:   15.75/thr  (  12.60/sec), total:    126
ldclt[28008]: Average rate:   15.25/thr  (  12.20/sec), total:    122
ldclt[28179]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: T005: Cannot ldap_delete (CN=0008140,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28179]: T005: Cannot ldap_delete (CN=0008196,DC=EXAMPLE,DC=com), error=1 (Operations error)
ldclt[28008]: Average rate:   15.88/thr  (  12.70/sec), total:    127
ldclt[28051]: Average rate:   16.38/thr  (  13.10/sec), total:    131
ldclt[28179]: Average rate:   16.62/thr  (  13.30/sec), total:    133
ldclt[28008]: Average rate:   14.75/thr  (  11.80/sec), total:    118
ldclt[28051]: Average rate:   15.38/thr  (  12.30/sec), total:    123
ldclt[28179]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28008]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28051]: Average rate:   15.88/thr  (  12.70/sec), total:    127
ldclt[28179]: Average rate:   15.12/thr  (  12.10/sec), total:    121
ldclt[28008]: Average rate:   14.88/thr  (  11.90/sec), total:    119
ldclt[28051]: Average rate:   15.50/thr  (  12.40/sec), total:    124
ldclt[28179]: Average rate:   15.00/thr  (  12.00/sec), total:    120
ldclt[28179]: T003: Cannot ldap_delete (CN=0008668,DC=EXAMPLE,DC=com), error=1 (Operations error)

ldclt[28008]: Global average rate: 1808.75/thr  ( 16.08/sec), total:  14470
ldclt[28008]: Global number times "no activity" reports: never
ldclt[28008]: Global no error occurs during this session.

ldclt[28179]: Global average rate: 1374.75/thr  ( 12.22/sec), total:  10998
ldclt[28179]: Global number times "no activity" reports: never
ldclt[28179]: Global error  1 (Operations error) occurs    11 times

NOTE :: even after an hr ds does not crash ::
systemctl status dirsrv@dhcp201-149.service
dirsrv@dhcp201-149.service - 389 Directory Server dhcp201-149.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled)
   Active: active (running) since Mon 2014-02-03 13:59:36 IST; 32min ago
 Main PID: 27761 (ns-slapd)
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@dhcp201-149.service
           └─27761 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dhcp201-149 -i /var/run/dirsrv/slapd-dhcp201-149.pid -w /var/run/dirsrv/slapd-dhcp201-149.startpid

Feb 03 13:59:36 dhcp201-149.englab.pnq.redhat.com systemd[1]: Started 389 Directory Server dhcp201-149..

Hence marking bug as VERIFIED.
Comment 4 Ludek Smid 2014-06-13 07:48:55 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.