Created attachment 586586 [details]
Script for checking intervals between samples in each collection.

Description of problem:
  Samples for some collections in Plumage are captured in strange changing intervals.


Version-Release number of selected component (if applicable):
  # rpm -qa | grep condor
   condor-plumage-7.6.8-0.1.el6.i686
   condor-7.6.8-0.1.el6.i686
   condor-classads-7.6.8-0.1.el6.i686


How reproducible:
  100%


Steps to Reproduce:
1. Prepare fresh installation of condor with Plumage.
  # yum install @mrg-grid condor-plumage mongodb-server
  # service mongod start
  # service condor start

2. Submit few simple job.
  # cat /tmp/vanilla.job 
    universe = vanilla
    cmd = /bin/sleep
    args = 3600
    should_transfer_files = no
    requirements = (FileSystemDomain =!= UNDEFINED && Arch =!= UNDEFINED)
    queue 1

  # runuser -s /bin/bash -l test1 -c "condor_submit /tmp/vanilla.job"
  # runuser -s /bin/bash -l test1 -c "condor_submit /tmp/vanilla.job"
  # runuser -s /bin/bash -l test1 -c "condor_submit /tmp/vanilla.job"
  # runuser -s /bin/bash -l test2 -c "condor_submit /tmp/vanilla.job"
  # runuser -s /bin/bash -l test2 -c "condor_submit /tmp/vanilla.job"
  # runuser -s /bin/bash -l test2 -c "condor_submit /tmp/vanilla.job"
  
3. Wait some time (one hour?) and then check intervals between samples in database via attached python script (second column of output is time interval).
  # python plumage_check_samples_time.py
    ...
    < example output attached in next comment >

Actual results:
  Intervals for collections samples.machine and samples.scheduler are not consistent.
  Intervals for samples.machine: 300...,60...,300...,60...
  Intervals for samples.scheduler: 60, 60, 60, 120, 60, 60, 60, 120,...

Expected results:
  Intervals between samples are the same or similar (for one collection).

Additional info:
  When I set UPDATE_INTERVAL (= 20) data for samples.scheduler are completely missing.