Bug 1880392

Compression seems to be run in line 1446. However, "Warning: no archive files found to compress" message appears in pmlogger_daily.log, so no file exists in $tmp/list, and the problematic compression would be run elsewhere.

[/usr/libexec/pcp/bin/pmlogger_daily]

   1430                     if [ -s $tmp/list ]
   1431                     then
   1432                         if $VERBOSE
   1433                         then
   1434                             if [ "$COMPRESSAFTER" -eq 0 ]
   1435                             then
   1436                                 echo "Archive files being compressed ..."
   1437                             else
   1438                                 echo "Archive files older than $COMPRESSAFTER days being compressed ..."
   1439                             fi
   1440                             fmt <$tmp/list | sed -e 's/^/    /'
   1441                         fi
   1442                         if $SHOWME
   1443                         then
   1444                             cat $tmp/list | xargs echo + $COMPRESS
   1445                         else
   1446                             cat $tmp/list | xargs $COMPRESS      <<<<<----------
   1447                         fi
   1448                     else
   1449                         $VERY_VERBOSE && echo "$prog: Warning: no archive files found to compress"     <<<<<----------
   1450                     fi

Yuki, it might be the pmlogger_daily -K that gets launched by pmlogger_check (see _compress_now() in /usr/libexec/pcp/bin/pmlogger_check). It's passing -K to pmlogger_daily, but not the -x flag on the cmdline, and so pmlogger_daily would be using $PCP_COMPRESSAFTER or if not set, falling back to $COMPRESSAFTER_DEFAULT.

The PCP_COMPRESSAFTER setting in the control file should still be honoured though. Uncomment #$PCP_COMPRESSAFTER=3 in /etc/pcp/pmlogger/control.d/local (or whatever control file you're using if not the one mentioned).

Hi Mark,

Thank you for suggesting a workaround. Confirmed that archive files are NOT compressed by uncommenting PCP_COMPRESSAFTER variable.

BTW, regarding the comment 4, is pmlogger_check launched when pmlogger_daily is run manually. This issue is reproducible even if pmlogger_daily is run manually like below.

# /usr/libexec/pcp/bin/pmlogger_daily -x 7

Hi Yuki, yes pmlogger_daily uses pmlogger_check to restart pmlogger. And pmlogger_check then calls pmlogger_daily -K (right at the end, see below). This doesn't sound optimal due to the recursion, however note that pmlogger_daily -K ONLY does compression, which is kind of a stand-alone script in itself.

pmlogger_check :
   
    163 _compress_now()
    164 {
    165     # If $PCP_COMPRESSAFTER=0 in the control file(s), compress archives now.
    166     # Invoked just before exit when this script has finished successfully.
    167     $PCP_BINADM_DIR/pmlogger_daily -K $daily_args
    168 }
...

    985 # Prior to exiting we compress existing logs, if any. See pmlogger_daily -K
    986 # Do not compress on a virgin install - it confuses systemd's Type=forking
    987 # exit status checks. There is nothing to compress anyway. See RHBZ#1721223.
    988 [ -f "$PCP_LOG_DIR/pmlogger/pmlogger_daily.stamp" ] && _compress_now
    989 
    990 [ -f $tmp/err ] && status=1
    991 exit

So I guess one possible fix would be to ensure _compress_now in pmlogger_check included -x in the $daily_args passed to pmlogger_daily -K.

Given you now have a work-around, I do not think this would qualify for RHEL7 zstream - so I would propose to address this issue upstream in the next PCP release (pcp-5.2.2), which would then find it's way back into RHEL8 y-stream via rebase. Does that sound OK to you and the customer?

Thanks

Related:

commit 27a818f9937a9c074a339f63e0ff36b849f5cd0e
Author: Ken McDonell <kenj.au>
Date:   Tue Jul 7 08:22:26 2020 +1000

    pmlogger_daily: -k 0 fixes

Also related:


commit 6628ad55cf111d1e1b1de59f76bf024f53469752
Author: Mark Goodwin <mgoodwin>
Date:   Tue Sep 4 13:25:50 2018 +1000

    pmlogger_check: run periodic compression when exiting early
    
    We always want to run the compression logic for sub-volumes
    even if we are exiting early - otherwise pathological cases
    (e.g. via selinux) can cause this logic to never happen.

Moving to RHEL-8 as this is not a critical fix meeting the RHEL-7 requirements.

Hi Mark, Nathan,

It's fine to move this BZ to RHEL8. RHEL7 zstream is not necessary because the workaround works fine.

Thanks.

After more thought, have concluded this is a usage issue, not a bug. The pmlogger_daily -x option can be used to induce early compression of archive volumes when the $PCP_COMPRESSAFTER environment variable has been explicitly set in the pmlogger control file to a value greater than zero. If that environment variable has not been set, then the default compression interval is zero days (i.e. all except the current volume) and -x has no effect. Note also in particular, -x cannot be used to prevent compression. As has been explained earlier, the correct way to prevent compression is to set $PCP_COMPRESSAFTER in the control file to a desired value greater than zero. For deleting/culling archive volumes, the same principles also apply to the $PCP_CULLAFTER environment variable, and the corresponding -k command line option.

The pmlogger_daily(1) man page has been updated to clarify this, with the following upstream commit for pcp-5.2.2 :

commit ffcdd814435aad7f7c9d86357224c563e26f6371
Author: Mark Goodwin <mgoodwin>
Date:   Thu Oct 22 09:20:49 2020 +1100

    docs: update pmlogger_daily(1) for -x, -K, -k options and $PCP_COMPRESSAFTER
    
    Update pmlogger_daily(1) to clarify -x, -k, -K and use of the
    $PCP_COMPRESSAFTER environment variable in pmlogger control files.
    
    Resolves: RHBZ#1880392

Thanks Mark,

So we should not use PMLOGGER_DAILY_PARAMS in /etc/sysconfig/pmlogger_timers to configure -x or -k option of pmlogger_daily. Is that right?

  ## Type:        string
  ## Default:     ""
  #
  # Run the pmlogger_daily binary, via pmlogger_daily.service, with the
  # following parameters.
  #
  #PMLOGGER_DAILY_PARAMS=""

In that case, we need to update https://access.redhat.com/solutions/2209361 to use $PCP_COMPRESSAFTER in a control file, instead of PMLOGGER_DAILY_PARAMS in /etc/sysconfig/pmlogger_timers.

Hi Yuki, sorry for the delay - I thought I had replied to this.

yes that document will need clarifying. Note it describes the *maximum* time before compression and culling occurs. The $PCP_COMPRESSAFTER (default 0 days) and $PCP_CULLAFTER (default 14 days) variables defined in the control file (or environment) override the command line options. Having so many ways to specify these parameters is confusing, but on-going, the preferred method is to edit the variables in the control file, e.g. /etc/pcp/pmlogger/control.d/local. This can be done for individual pmloggers (just use separate control files as necessary).

I've ended up here because all I want to do is limit pmlogger so it only keeps data from the last 7 days or so.

Of course, all of Red Hat's documentation on how to achieve this is utter nonsense.

I've tried setting:

1) PCP_CULLAFTER=7 in /etc/sysconfig/pmlogger.
2) $PCP_CULLAFTER=7 and export it, in /etc/pcp/pmlogger/control
3) $PCP_CULLAFTER=7 and export it, in /etc/pcp/pmlogger/control.d/local

None of them do any good.

The only thing that works is set PMLOGGER_DAILY_PARAMS="-k 7" in /etc/sysconfig/pmlogger_timers.

Well done Red Hat, it's just incredible how you manage to make the simplest thing as difficult as possible.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1754

"Mark Goodwin  2020-11-04 11:14:19 UTC
Hi Yuki, sorry for the delay - I thought I had replied to this.

yes that document will need clarifying."


HAVE YOU CLARIFIED THAT DOCUMENT YET MARK?