RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1150225 - dirsrv not running after restraint job restarts it
Summary: dirsrv not running after restraint job restarts it
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-07 17:45 UTC by Scott Poore
Modified: 2020-09-13 21:20 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-17 17:08:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
restart test rpm (4.07 KB, application/x-rpm)
2014-10-07 17:48 UTC, Scott Poore 		no flags Details
strace output (1.83 MB, application/x-gzip)
2014-10-07 17:52 UTC, Scott Poore 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 1345 0 None None None 2020-09-13 21:20:35 UTC

Description Scott Poore 2014-10-07 17:45:57 UTC 
Description of problem:

I'm trying to restart dirsrv in test automation.  This is failing when the automation is run by restraint:

http://restraint.readthedocs.org/en/latest/

It appears to be running during the job but, after the job finishes and exits, dirsrv is stopped.


Version-Release number of selected component (if applicable):
389-ds-base-1.2.11.15-47.el6.x86_64

How reproducible:
Always in RHEL6.6 and 6.5  I haven't seen this yet in RHEL7.


Steps to Reproduce:
1.  install IPA

ipa-server-install --setup-dns --forwarder=<FORWARDER IP> \
    --hostname=<HOSTNAME> --ip-address=<IP_ADDRESS> -n <DNS DOMAIN> \
    -r <REALM NAME> -a <ADMIN_PASSWD> -p <ADMIN_PASSWD> -U

2. install restraint

cat > /etc/yum.repos.d/restraint.repo <<EOF
[restraint]
name=Restraint
baseurl=http://bpeck.fedorapeople.org/restraint/el6eso/
enabled=1
gpgcheck=0
EOF

yum -y remove rhts-python
yum -y install restraint-rhts beakerlib-redhat staf restraint restraint-client
yum -y update

3.  install restart task rpm

mkdir /root/yum.local.d
cd /root/yum.local.d
# Copy ipa-tests-ipa-server-rhel70-ipa-restart RPM to /root/yum.local.d
yum -y install createrepo
createrepo .
cat > /etc/yum.repos.d/mylocal.repo <<EOF
[mylocal]
name=mylocal
baseurl=file:///root/yum.local.d
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOF

4.  Setup restraint job xml

cd /root

cat > ipa-restart.xml <<EOF
<?xml version="1.0"?>
<job>
  <recipeSet>
   <recipe role="RECIPE_MEMBERS" id="1">
    <task name="/ipa-server/rhel70/ipa-restart">
        <rpm name="ipa-tests-ipa-server-rhel70-ipa-restart" path="/mnt/tests/ipa-server/rhel70/ipa-restart"/>
        <params>
            <param name="RECIPE_MEMBERS" value="hostname1"/>
            <param name="MASTER" value="hostname1"/>
            <param name="BEAKERMASTER" value="hostname1"/>
            <param name="TIER" value="1"/>
        </params>
    </task>
   </recipe>
  </recipeSet>
</job>
EOF

sed -i "s/hostname1/$(hostname)/g" ipa-restart.xml

5.  run restraint job

service restraintd start
ps -ef|grep restraintd
strace -vtfo /tmp/restraintd.strace -p 1470 &
restraint --job ipa-restart.xml --host 1=$(hostname) -v -v -v
fg
<break out of strace>


Actual results:

dirsrv is not running after job:

[root@rhel6-2 ~]# service dirsrv status
dirsrv EXAMPLE-COM dead but pid file exists
dirsrv PKI-IPA dead but pid file exists


Expected results:

Should be running:

[root@rhel6-2 ~]# service dirsrv status
dirsrv EXAMPLE-COM (pid 10249) is running...
dirsrv PKI-IPA (pid 10323) is running...

Additional info:

Will attach strace and rpm
Comment 1 Scott Poore 2014-10-07 17:48:31 UTC 
Created attachment 944702 [details]
restart test rpm

needed for testing restraint
Comment 2 Scott Poore 2014-10-07 17:52:59 UTC 
Created attachment 944704 [details]
strace output

This strace output is from a RHEL6.5 test case.  I had issues capturing output with strace against restraintd on RHEL6.6.
Comment 4 Noriko Hosoi 2014-10-07 18:01:22 UTC 
Hi Scott,

Could you please attach the Directory Server logs (/var/log/dirsrv/slapd-ID/{access,errors}) to the bug?

I'm wondering "not running" means 1) the server was not launched, 2) the server was shutdown, or 3) the server died?

Thanks,
--noriko
Comment 5 Noriko Hosoi 2014-10-07 18:45:58 UTC 
(In reply to Noriko Hosoi from comment #4)
> Hi Scott,
> 
> Could you please attach the Directory Server logs
> (/var/log/dirsrv/slapd-ID/{access,errors}) to the bug?
> 
> I'm wondering "not running" means 1) the server was not launched, 2) the
> server was shutdown, or 3) the server died?
> 
> Thanks,
> --noriko

Never mind...  "what stopped dirsrv" already talked about it (and more).
Comment 6 Scott Poore 2014-10-07 18:57:30 UTC 
Right, and I don't really see much in the logs other than it's being stopped:

[07/Oct/2014:12:35:49 -0500] - slapd shutting down - signaling operation threads
[07/Oct/2014:12:35:49 -0500] - slapd shutting down - closing down internal subsystems and plugins
[07/Oct/2014:12:35:49 -0500] - Waiting for 4 database threads to stop
[07/Oct/2014:12:35:50 -0500] - All database threads now stopped
[07/Oct/2014:12:35:50 -0500] - slapd stopped.


A summary of findings was:

restraind is running make run to execute the runtest.sh which runs the ipactl commands.  when that finishes, ns-slapd is being hit with a SIGHUP

And the following worked to prevent dirsrv from being stopped when restraint finished the task:

setsid nohup ipactl start

It should be noted that the original failure that led here was ipactl start was running but, only dirsrv was down afterwards.  I just removed it from the reproducer to try to simplify things.
Comment 7 Scott Poore 2014-10-08 22:27:00 UTC 
FYI, more info and maybe an easier test.  

After some troubleshooting with Bill this afternoon, we got a workaround that replaced an exec from one of the restraint plugins with setsid to handle execution.  This seemed to detatch the process so it didn't get killed when the restraint task ended.

Now, looking at the execution via restraint, I think a simple reproducer is just

ssh to host and run:

service dirsrv stop

exec service dirsrv start

At this point you're logged out as expected.  But, when I log back in, it's not running.

Or to confirm it's still up before logged out:

cat > exectest <<EOF
service dirsrv start
service dirsrv status
EOF
chmod 755 exectest
service dirsrv stop
exec ./exectest

Then log back in and check.
Comment 11 Noriko Hosoi 2015-01-28 01:16:05 UTC 
Upstream ticket:
https://fedorahosted.org/389/ticket/48014
Comment 12 Noriko Hosoi 2015-03-06 18:51:57 UTC 
Per 389-ds-base ticket triage, put to post 1.3.5.
Comment 16 Noriko Hosoi 2015-03-06 23:13:36 UTC 
I cannot figure out how to disable setting the target version...
This bug is supposed to be set post rhel-7.3.0 (7.4.0?), but it's not allowed.
Setting 7.3.0 for now.
Note You need to log in before you can comment on or make changes to this bug.