1690273 – Making "docker system df" optional

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1690273 - Making "docker system df" optional

Summary: Making "docker system df" optional

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	sos
Sub Component:
Version:	7.7
Hardware:	All
OS:	Linux
Priority:	high
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Pavel Moravec
QA Contact:	Miroslav Hradílek
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1690999
TreeView+	depends on / blocked

Reported:	2019-03-19 08:29 UTC by Ranjith Rajaram
Modified:	2019-08-06 13:16 UTC (History)
CC List:	8 users (show)
Fixed In Version:	sos-3.7-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1690999 (view as bug list)
Environment:
Last Closed:	2019-08-06 13:15:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	sosreport sos pull 1580	0	'None'	closed	[docker] Correct commands, collect volume info, gate system df	2020-07-04 03:20:04 UTC
Red Hat Product Errata	RHEA-2019:2295	0	None	None	None	2019-08-06 13:16:10 UTC

Description Ranjith Rajaram 2019-03-19 08:29:38 UTC

Description of problem:

Request to remove "docker system df" from sosreport docker plugin

As per update to the upstream docs(https://github.com/moby/moby/commit/914f0f4b5d6388b187dd161aa9fbe0e28b9f392c),  `system df` command can be very resource-intensive. 

Running sosreport on a OCP node with docker plugin enabled, can result in un-expected behavior if the node has many pods running on them and each pods have a good number of volumes.


Version-Release number of selected component (if applicable):
sos-3.6-16.el7_6.noarch

How reproducible:
Always

Steps to Reproduce:
1. run 'docker system df' on a busy node
2.
3.

Actual results:
'docker system df' is slow and is resource intensive. 

Expected results:
Remove 'docker system df' from the docker plugin

Additional info:


[root@localhost sos]# diff -Naur sos/plugins/docker.py sos/plugins/docker-new.py 
--- sos/plugins/docker.py       2019-03-19 13:54:37.177961308 +0530
+++ sos/plugins/docker-new.py   2019-03-19 13:55:41.095957272 +0530
@@ -49,7 +49,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]

Comment 1 Bryn M. Reeves 2019-03-19 09:52:11 UTC

This was added last year by one of the contributors who maintains the plugin:

  commit d22f226c642053b30c4bf9520e8fdd5611c7dc33
  Author: Jake Hunsaker <jhunsake>
  Date:   Mon Jun 4 16:55:51 2018 -0400

    [docker] Collect events and storage information
    
    Adds collection of the previous day's worth of events handled by the
    daemon, a summarized output of storage needs, and a listing of volumes
    known to the daemon.
    
    Resolves: #1328
    
    Signed-off-by: Jake Hunsaker <jhunsake>
    Signed-off-by: Bryn M. Reeves <bmr>

The question is, does this ever have diagnostic value? If that's the case then the normal approach is to make it optional, so that a user would have to run:

  # sosreport -k docker.system_df

In order to execute the command. We can also mark the option as "slow" in the option metadata, so that users browsing the sos options list will have this warning.

Comment 2 Ranjith Rajaram 2019-03-19 10:57:50 UTC

It does collect information which may be useful to understand or account disk usage.

Making it optional should also help.

Tested it with the following patch



--- sos/plugins/docker.py       2019-03-19 15:53:19.724450968 +0530
+++ sos/plugins/docker-new.py   2019-03-19 16:13:11.961341475 +0530
@@ -24,7 +24,8 @@
             "that have terminated", 'fast', False),
         ("logs", "capture logs for running containers",
             'fast', False),
-        ("size", "capture image sizes for docker ps", 'slow', False)
+        ("size", "capture image sizes for docker ps", 'slow', False),
+        ("system_df", "captures information regarding the amount of disk space used by the docker daemon. This can be very resource-intensive", 'slow', False)
     ]
 
     def setup(self):
@@ -49,7 +50,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]
@@ -61,6 +61,9 @@
         if self.get_option('size'):
             self.add_cmd_output('docker ps -as')
 
+        if self.get_option('system_df'):
+            self.add_cmd_output('docker system df')
+
         nets = self.get_command_output('docker network ls')
 
         if nets['status'] == 0:

Comment 4 Pavel Moravec 2019-03-19 16:20:04 UTC

devel_ack+ for 7.8, assuming there is / will be a common agreement (e.g. with Jake as the author of orig.enhancement) on the optional call.

Comment 5 Jake Hunsaker 2019-03-19 16:55:19 UTC

Upstream PR 1580 - https://github.com/sosreport/sos/pull/1580 - puts 'docker system df' behind the 'size' option, which I think is better than an additional plugin option.

Comment 7 Pavel Moravec 2019-03-19 17:11:36 UTC

ok, devel_ack+ to 7.7 and also for 7.6.z

Either whole patch from https://github.com/sosreport/sos/pull/1580 will be backported or just the relevant part like:


@@ -49,7 +50,6 @@
             'ps',
             'ps -a',
             'stats --no-stream',
-            'system df',
             'version',
             'volume ls'
         ]
@@ -61,6 +61,9 @@
         if self.get_option('size'):
             self.add_cmd_output('docker ps -as')
+            self.add_cmd_output('docker system df')
 
        nets = self.get_command_output('docker network ls')
 
         if nets['status'] == 0:


I.e. -k docker.size=yes will enable the command (to be aligned with upstream on the option name).

Comment 8 Pavel Moravec 2019-03-20 15:14:13 UTC

committed to dist-git for 7.7

Comment 11 Pavel Moravec 2019-03-27 20:21:30 UTC

Hello,
could you please verify the BZ against below build? Thanks in advance.


A yum repository for the build of sos-3.7-1.el7 (task 20748546) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/sos-3.7-1.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/1.el7/noarch/sos-3.7-1.el7.noarch.rpm

Build output will be available for the next 21 days.

Comment 12 Jake Hunsaker 2019-04-17 13:52:40 UTC

Verified 'docker system df' is now properly gated by the docker.size option.

Comment 15 errata-xmlrpc 2019-08-06 13:15:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295

Note You need to log in before you can comment on or make changes to this bug.