Bug 645030

iptables workflow.

After the job lands on a startd (of IP address $STARTDADDR) I run the following on the schedd (of IP address $SCHEDDADDR):

iptables -I INPUT 1 -s $STARTDADDR/32 -j DROP
iptables -I OUTPUT 1 -d $STARTDADDR/32 -j DROP

...and run the following on the startd:

iptables -I INPUT 1 -s $SCHEDDADDR/32 -j DROP
iptables -I OUTPUT 1 -d $SCHEDDADDR/32 -j DROP

After I'm done testing I just do a "service iptables restart" on each node to clear the iptables.

[root@box1 ~]# cat /etc/condor/config.d/zBZ645030-cm.config
EXEC = box2.lab.bos.redhat.com

FILESYSTEM_DOMAIN = fake
DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD
ALLOW_WRITE = $(FULL_HOSTNAME), $(EXEC)
ALLOW_ADMINISTRATOR = $(FULL_HOSTNAME)
STARTD_SENDS_ALIVES = TRUE

[root@box2 ~]# cat /etc/condor/config.d/zBZ645030-exec.config
CM = box1.lab.bos.redhat.com

FILESYSTEM_DOMAIN = fake
COLLECTOR_HOST = $(CM)
DAEMON_LIST = MASTER, STARTD
ALLOW_WRITE = $(FULL_HOSTNAME), $(CM)
ALLOW_ADMINISTRATOR = $(FULL_HOSTNAME)
STARTD_SENDS_ALIVES = TRUE

[BZ645030@box1 ~]$ echo -e 'cmd=/bin/sleep\nargs=1d\nrequirements=Arch=!=UNDEFINED\ninitial_dir=/tmp\n+JobLeaseDuration=120\nqueue' | condor_submit

After jobs start, restricting access between box1 and box2 results in job being killed off on box2 and the shadow remains on box1 (job appears running).

STARTD_SENDS_ALIVES=TRUE means the Startd periodically, based on JobLeaseDuration, initiates a TCP connection to the Schedd to send the alive.

The Schedd maintains information about the alives in jobs. It does not check for expiration.

The expiration is delegated to the Shadow, who relies on a broken connection to the Starter, because the check for expiration is only done during reconnect attempts. The "alive" the Shadow uses is the last time the Starter contacted it, which is problematic because Starter->Shadow communication is keyed off STARTER_UPDATE_INTERVAL instead of JobLeaseDuration.

The only way for the Shadow to notice the lease expiration is for the persistent Starter<->Shadow connection to be broken, typically on a 2 hour TCP keepalive boundary.

Some options -

 1. Schedd checks for expiration and terminates Shadow
 2. Startd sends alives to Shadow, which gains a timer to detect expiration
 3. Shadow gains a timer to detect expiration and relies on last Starter communication, which gets keyed off minimum of STARTER_UPDATE_INTERVAL and JobLeaseDuration

The Startd->Schedd connection initiation may be expensive. A timer in the Schedd to walk the queue for expiration may be expensive.

1. Con - the Schedd is undertaking a potentially expensive operation; Pro - simplest change from existing communication patterns
2. Con - no other Startd->Shadow communication happens, will have firewall implications; Pro - conceptually makes sense since the Shadow is responsible for noticing expiration
3. Cons - important for Shadow to frequently update Schedd (no worse than 1's load), Starter must change its update interval semantics; Pro - reduces connection load on Schedd, no need for new Schedd timer/check, reuses existing connection

Separate concern, it may be desirable to put enforce a minimum JobLeaseDuration, as it is user controllable.

FYI, from TCP(7)

       tcp_keepalive_intvl (integer; default: 75; since Linux 2.4)
              The number of seconds between TCP keep-alive probes.

       tcp_keepalive_probes (integer; default: 9; since Linux 2.2)
              The maximum number of TCP keep-alive probes to send before  giv-
              ing  up  and  killing  the connection if no response is obtained
              from the other end.

       tcp_keepalive_time (integer; default: 7200; since Linux 2.2)
              The number of seconds a connection needs to be idle  before  TCP
              begins sending out keep-alive probes.  Keep-alives are only sent
              when the SO_KEEPALIVE socket option  is  enabled.   The  default
              value  is  7200 seconds (2 hours).  An idle connection is termi-
              nated after approximately an additional 11 minutes (9 probes  an
              interval of 75 seconds apart) when keep-alive is enabled.

              Note that underlying connection tracking mechanisms and applica-
              tion timeouts may be much shorter.

So the 2 hour wisdom is more like 2 hours 11 minutes and 15 seconds.

Waiting for the connection to timeout appears to work...

11/17/10 14:28:11 DaemonCore: Command Socket at <192.168.3.153:42185>
11/17/10 14:28:11 Setting maximum accepts per cycle 4.
11/17/10 14:28:11 Initializing a VANILLA shadow for job 10036.0
11/17/10 14:28:11 (10036.0) (14787): Request to run on f12.local <192.168.122.54:44944> was ACCEPTED
11/17/10 16:39:34 (10036.0) (14787): condor_read() failed: recv() returned -1, errno = 110 Connection timed out, reading 5 bytes from startd f12.local.
11/17/10 16:39:34 (10036.0) (14787): IO: Failed to read packet header
11/17/10 16:39:34 (10036.0) (14787): Can no longer talk to condor_starter <192.168.122.54:44944>
11/17/10 16:39:34 (10036.0) (14787): Trying to reconnect to disconnected job
11/17/10 16:39:34 (10036.0) (14787): LastJobLeaseRenewal: 1290022099 Wed Nov 17 14:28:19 2010
11/17/10 16:39:34 (10036.0) (14787): JobLeaseDuration: 30 seconds
11/17/10 16:39:34 (10036.0) (14787): JobLeaseDuration remaining: EXPIRED!
11/17/10 16:39:34 (10036.0) (14787): Reconnect FAILED: Job disconnected too long: JobLeaseDuration (30 seconds) expired
11/17/10 16:39:34 (10036.0) (14787): **** condor_shadow (condor_SHADOW) pid 14787 EXITING WITH STATUS 107

f12.local is a VM that is paused after it starts running the job.

Observing the jobs go back to idle...

Wed Nov 17 16:38:59 EST 2010
-- Submitter: eeyore.local : <192.168.3.153:51686> : eeyore.local
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
10036.0   matt           11/17 14:27   0+02:10:48 R  0   0.0  sleep             

1 jobs; 0 idle, 1 running, 0 held

Wed Nov 17 16:39:59 EST 2010
-- Submitter: eeyore.local : <192.168.3.153:51686> : eeyore.local
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
10036.0   matt           11/17 14:27   0+02:11:23 I  0   0.0  sleep             

1 jobs; 1 idle, 0 running, 0 held

Adjusting keepalive params to allow for disconnection of at most a minute...

$ sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_time = 30
net.ipv4.tcp_keepalive_intvl = 5
net.ipv4.tcp_keepalive_probes = 6

Resulting ShadowLog -
 . job was picked up when VM was unpaused
 . VM was re-paused
 . job back in idle after about a minute

11/17/10 16:51:10 DaemonCore: Command Socket at <192.168.3.153:57989>
11/17/10 16:51:10 Setting maximum accepts per cycle 4.
11/17/10 16:51:10 Initializing a VANILLA shadow for job 10036.0
11/17/10 16:51:10 (10036.0) (3253): Request to run on f12.local <192.168.122.54:54480> was ACCEPTED
11/17/10 16:52:19 (10036.0) (3253): condor_read() failed: recv() returned -1, errno = 110 Connection timed out, reading 5 bytes from startd f12.local.
11/17/10 16:52:19 (10036.0) (3253): IO: Failed to read packet header
11/17/10 16:52:19 (10036.0) (3253): Can no longer talk to condor_starter <192.168.122.54:54480>
11/17/10 16:52:19 (10036.0) (3253): Trying to reconnect to disconnected job
11/17/10 16:52:19 (10036.0) (3253): LastJobLeaseRenewal: 1290030679 Wed Nov 17 16:51:19 2010
11/17/10 16:52:19 (10036.0) (3253): JobLeaseDuration: 30 seconds
11/17/10 16:52:19 (10036.0) (3253): JobLeaseDuration remaining: EXPIRED!
11/17/10 16:52:19 (10036.0) (3253): Reconnect FAILED: Job disconnected too long: JobLeaseDuration (30 seconds) expired
11/17/10 16:52:19 (10036.0) (3253): **** condor_shadow (condor_SHADOW) pid 3253 EXITING WITH STATUS 107

Please use the workaround of reducing the tcp-keepalive window.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
C: The condor_shadow does not actively detect disconnected jobs
C: The shadow must wait for a transport level reset before detecting the disconnection. That reset will take just over two hours to present itself.
W: Use system parameters to reduce the TCP keepalive timeout.
R: The time for disconnect detection can be reduced to a desired value.

Reproduced on RHEL 5.6/4.9 x i386/x86_64 with condor-7.4.4-0.17 and it didn't work properly.

Tested on RHEL 5.6 x i386 with condor-7.4.5-0.6.el5 as CM and RHEL 5.6/4.9 x i386/x86_64 with condor-7.4.5-0.6.el5 as execute nodes.

$ sysctl -a | grep net.ipv4.tcp_keepalive
net.ipv4.tcp_keepalive_intvl = 5
net.ipv4.tcp_keepalive_probes = 6
net.ipv4.tcp_keepalive_time = 30

STARTD_SENDS_ALIVES is TRUE on all machines.

I've submited jobs as in comment #2.

After drop tcp connection by iptables as in comment #1 and ~2 minutes, I see by condor_q that jobs have changed their status from Run to Idle. But slots are still in Claimed state in condor_status. Is this right? I expect that if CM has no info about nodes more than UPDATE_INTERVAL(default 5 minutes) and tcp_keepalive has configuration above, it change status of slots to unclaimed(There is START = TRUE on all machines).

It would be acceptable if the slot remains Claimed/Busy all the way up to the claim lease duration.

http://www.cs.wisc.edu/condor/manual/v7.5/3_5Policy_Configuration.html#25439

You can use condor_status -direct to skip the Collector and ask the Startd its status directly.

The CM does not manage slot states, it just reports what it was last told by the Startd.

Ok, anyway it was verified. --> VERIFIED

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1 @@
-C: The condor_shadow does not actively detect disconnected jobs
+Previously, condor_shadow did not actively detect disconnected jobs. Due to this lack, The shadow waited for a transport level reset before detecting the disconnection. That reset took more than two hours to present itself. This update uses the system parameters to reduce the Transmission Control Protocol (TCP) keepalive timeout. Now, The time for disconnect detection can be reduced to the desired value.-C: The shadow must wait for a transport level reset before detecting the disconnection. That reset will take just over two hours to present itself.
-W: Use system parameters to reduce the TCP keepalive timeout.
-R: The time for disconnect detection can be reduced to a desired value.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0217.html