Bug 1690273

Summary: Making "docker system df" optional
Product: Red Hat Enterprise Linux 7 Reporter: Ranjith Rajaram <rrajaram>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Miroslav HradĂ­lek <mhradile>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.7CC: agk, bmr, dornelas, gavin, jhunsaker, mhradile, plambri, sbradley
Target Milestone: rcKeywords: OtherQA, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: sos-3.7-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1690999 (view as bug list) Environment:
Last Closed: 2019-08-06 13:15:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1690999    

Description Ranjith Rajaram 2019-03-19 08:29:38 UTC
Description of problem:

Request to remove "docker system df" from sosreport docker plugin

As per update to the upstream docs(https://github.com/moby/moby/commit/914f0f4b5d6388b187dd161aa9fbe0e28b9f392c),  `system df` command can be very resource-intensive. 

Running sosreport on a OCP node with docker plugin enabled, can result in un-expected behavior if the node has many pods running on them and each pods have a good number of volumes.


Version-Release number of selected component (if applicable):
sos-3.6-16.el7_6.noarch

How reproducible:
Always

Steps to Reproduce:
1. run 'docker system df' on a busy node
2.
3.

Actual results:
'docker system df' is slow and is resource intensive. 

Expected results:
Remove 'docker system df' from the docker plugin

Additional info:


[root@localhost sos]# diff -Naur sos/plugins/docker.py sos/plugins/docker-new.py 
--- sos/plugins/docker.py       2019-03-19 13:54:37.177961308 +0530
+++ sos/plugins/docker-new.py   2019-03-19 13:55:41.095957272 +0530
@@ -49,7 +49,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]

Comment 1 Bryn M. Reeves 2019-03-19 09:52:11 UTC
This was added last year by one of the contributors who maintains the plugin:

  commit d22f226c642053b30c4bf9520e8fdd5611c7dc33
  Author: Jake Hunsaker <jhunsake>
  Date:   Mon Jun 4 16:55:51 2018 -0400

    [docker] Collect events and storage information
    
    Adds collection of the previous day's worth of events handled by the
    daemon, a summarized output of storage needs, and a listing of volumes
    known to the daemon.
    
    Resolves: #1328
    
    Signed-off-by: Jake Hunsaker <jhunsake>
    Signed-off-by: Bryn M. Reeves <bmr>

The question is, does this ever have diagnostic value? If that's the case then the normal approach is to make it optional, so that a user would have to run:

  # sosreport -k docker.system_df

In order to execute the command. We can also mark the option as "slow" in the option metadata, so that users browsing the sos options list will have this warning.

Comment 2 Ranjith Rajaram 2019-03-19 10:57:50 UTC
It does collect information which may be useful to understand or account disk usage.

Making it optional should also help.

Tested it with the following patch



--- sos/plugins/docker.py       2019-03-19 15:53:19.724450968 +0530
+++ sos/plugins/docker-new.py   2019-03-19 16:13:11.961341475 +0530
@@ -24,7 +24,8 @@
             "that have terminated", 'fast', False),
         ("logs", "capture logs for running containers",
             'fast', False),
-        ("size", "capture image sizes for docker ps", 'slow', False)
+        ("size", "capture image sizes for docker ps", 'slow', False),
+        ("system_df", "captures information regarding the amount of disk space used by the docker daemon. This can be very resource-intensive", 'slow', False)
     ]
 
     def setup(self):
@@ -49,7 +50,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]
@@ -61,6 +61,9 @@
         if self.get_option('size'):
             self.add_cmd_output('docker ps -as')
 
+        if self.get_option('system_df'):
+            self.add_cmd_output('docker system df')
+
         nets = self.get_command_output('docker network ls')
 
         if nets['status'] == 0:

Comment 4 Pavel Moravec 2019-03-19 16:20:04 UTC
devel_ack+ for 7.8, assuming there is / will be a common agreement (e.g. with Jake as the author of orig.enhancement) on the optional call.

Comment 5 Jake Hunsaker 2019-03-19 16:55:19 UTC
Upstream PR 1580 - https://github.com/sosreport/sos/pull/1580 - puts 'docker system df' behind the 'size' option, which I think is better than an additional plugin option.

Comment 7 Pavel Moravec 2019-03-19 17:11:36 UTC
ok, devel_ack+ to 7.7 and also for 7.6.z

Either whole patch from https://github.com/sosreport/sos/pull/1580 will be backported or just the relevant part like:


@@ -49,7 +50,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]
@@ -61,6 +61,9 @@
         if self.get_option('size'):
             self.add_cmd_output('docker ps -as')
+            self.add_cmd_output('docker system df')
 
        nets = self.get_command_output('docker network ls')
 
         if nets['status'] == 0:


I.e. -k docker.size=yes will enable the command (to be aligned with upstream on the option name).

Comment 8 Pavel Moravec 2019-03-20 15:14:13 UTC
committed to dist-git for 7.7

Comment 11 Pavel Moravec 2019-03-27 20:21:30 UTC
Hello,
could you please verify the BZ against below build? Thanks in advance.


A yum repository for the build of sos-3.7-1.el7 (task 20748546) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/sos-3.7-1.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.noarch.rpm

Build output will be available for the next 21 days.

Comment 12 Jake Hunsaker 2019-04-17 13:52:40 UTC
Verified 'docker system df' is now properly gated by the docker.size option.

Comment 15 errata-xmlrpc 2019-08-06 13:15:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295