Bug 883492

Summary: Cron cartridge does not prevent subsequent runs of particular frequency if a job is already running for that frequency
Product: OKD Reporter: Paul Morie <pmorie>
Component: ContainersAssignee: Paul Morie <pmorie>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: jhou
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-19 19:26:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paul Morie 2012-12-04 17:39:55 UTC 
Description of problem:
The cron cartridge does not implement any type of locking.  Therefore, if a job for frequency X is already running and it's time to start another job, the second job will be started.  The solution is to implement locking for each frequency, and skip runs where the lock cannot be obtained.

Version-Release number of selected component (if applicable):
2.0.20

How reproducible:
Always

Steps to Reproduce:
1. Create an application and embed the cron cartridge.
2. Create a minutely job that runs for 2 minutes and marks its end in a log file in the application's dir (sample code below).

  
Actual results:

End of job should be marked in log file.


Expected results:

End of job is not marked in log file; subsequent runs overwrite log file.

Additional info:
Comment 1 Paul Morie 2012-12-04 17:43:12 UTC 
Test job code (designed to work with a ruby app):

==========

#!/bin/bash

date >> $OPENSHIFT_RUBY_LOG_DIR/test.log
sleep 120
echo 'test job complete!' >> $OPENSHIFT_RUBY_LOG_DIR/test.log

==========

This should go in the app's .openshift/cron/minutely and have the executable bit set.
Comment 2 Paul Morie 2012-12-04 18:53:51 UTC 
I've introduced locking that will result in a skipped run for any frequency if a job for that frequency is already running.  A skipped run will result in a message like so in /var/log/messages:

:SKIPPED: minutely cron run for openshift user '53e8fde8be0742d1bb42cad10da24345'
Comment 3 Paul Morie 2012-12-05 14:19:30 UTC 
Fix is present in devenv AMI 2556.
Comment 4 Jianwei Hou 2012-12-06 02:44:34 UTC 
Verified on devenv_2558

Steps:
1. Create one app and embed cron cartridge
2. Create a minutely job that runs for 2 minutes and marks its end in a log file in the application's dir (sample code below).

#!/bin/bash

date >> $OPENSHIFT_PHP_LOG_DIR/test.log
sleep 120
echo 'test job complete!' >> $OPENSHIFT_PHP_LOG_DIR/test.log

3. Git push

Check locking is enabled for cron cartridge
On app's gear:
[php1-2558t.dev.rhcloud.com logs]\> tailf test.log 
Wed Dec  5 21:39:03 EST 2012
test job complete!

On instance, a skipped run is logged
Dec  5 21:43:02 ip-10-151-4-201 root[31265]: user-cron-jobs Wed Dec  5 21:43:02 EST 2012: :START: minutely cron run for openshift user 'ef037b3bb9494ffca77423bfdae2fc84'
Dec  5 21:43:02 ip-10-151-4-201 root[31275]: user-cron-jobs Wed Dec  5 21:43:02 EST 2012: :SKIPPED: minutely cron run for openshift user 'ef037b3bb9494ffca77423bfdae2fc84'