Bug 1190912

Summary: add /proc/net/softnet_stat counters to linux PMDA
Product: Red Hat Enterprise Linux 7 Reporter: Mark Goodwin <mgoodwin>
Component: pcpAssignee: Mark Goodwin <mgoodwin>
Status: CLOSED ERRATA QA Contact: qe-baseos-tools-bugs
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.1CC: brolley, fche, jmaxwell, lberk, mcermak, mgoodwin, mprchlik, nathans
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 11:53:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mark Goodwin 2015-02-10 00:39:27 UTC
RFE to add /proc/net/softnet_stat counters to the network PMNS subtree. These per-CPU counters (one line per-CPU) are indicative of network issues (Jon Maxwell and I have been discussing, he uses these counters a lot with customer networking issues).

$ cat /proc/net/softnet_stat 
0009ffea 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00060bc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0032e18c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000c3afa 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Kernel code :

struct softnet_data {
        struct Qdisc            *output_queue;
        struct Qdisc            **output_queue_tailp;
        struct list_head        poll_list;
        struct sk_buff          *completion_queue;
        struct sk_buff_head     process_queue;

        /* stats */
        unsigned int            processed;
        unsigned int            time_squeeze;
        unsigned int            cpu_collision;
        unsigned int            received_rps;

#ifdef CONFIG_RPS
        struct softnet_data     *rps_ipi_list;

        /* Elements below can be accessed between CPUs for RPS */
        struct call_single_data csd ____cacheline_aligned_in_smp;
        struct softnet_data     *rps_ipi_next;
        unsigned int            cpu;
        unsigned int            input_queue_head;
        unsigned int            input_queue_tail;
#endif
        unsigned int            dropped;
        struct sk_buff_head     input_pkt_queue;
        struct napi_struct      backlog;
};


And the procfs show function using the above structure :
static int softnet_seq_show(struct seq_file *seq, void *v)
{
        struct softnet_data *sd = v;

        seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
                   sd->processed, sd->dropped, sd->time_squeeze, 0,
                   0, 0, 0, 0, /* was fastroute */
                   sd->cpu_collision, sd->received_rps);
        return 0;
}

Comment 2 Mark Goodwin 2015-02-10 23:08:43 UTC
This'll go upstream first of course

Comment 7 Mark Goodwin 2015-07-23 03:13:18 UTC
upstream linux has added flow_limit_count ..

        seq_printf(seq,
                   "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
                   sd->processed, sd->dropped, sd->time_squeeze, 0,
                   0, 0, 0, 0, /* was fastroute */
                   sd->cpu_collision, sd->received_rps, flow_limit_count);

Comment 8 Mark Goodwin 2015-07-28 01:40:32 UTC
posted upstream for review :

Changes committed to git://pcp.io/markgw/pcp/pcp.git master

commit b4633e2f3792caa359269144cd671d604ff0ec89
Author: Mark Goodwin <mgoodwin>
Date:   Tue Jul 28 11:16:40 2015 +1000

    pmdalinux - add network.softnet metrics from /proc/net/softnet_stat.

    /proc/net/softnet_stat exports per-CPU statistics of various interesting
    metrics related to congestion in the Linux network stack. See the help
    text for details. This commit adds support for these metrics summed over
    all CPUs as network.softnet.*

    In the future we may extend this to also export the per-cpu metrics as
    network.softnet.percpu.* over the CPU instance domain - that's future work.

        modified:   src/pmdas/linux/GNUmakefile
        modified:   src/pmdas/linux/clusters.h
        modified:   src/pmdas/linux/help
        modified:   src/pmdas/linux/pmda.c
        new file:   src/pmdas/linux/proc_net_softnet.c
        new file:   src/pmdas/linux/proc_net_softnet.h
        modified:   src/pmdas/linux/root_linux

Comment 9 Nathan Scott 2015-08-05 02:47:37 UTC
(included in pcp-3.10.6, along with QA test...)

commit 965ba2469423a594bf1ba2aa643359fd6bb94ef3
Author: Nathan Scott <nathans>
Date:   Tue Jul 28 14:10:35 2015 +1000

    pmdalinux: update softnet counters error handling, add qa
    
    Add test qa/1030 to exercise various forms of proc/net/softnet_stat
    being handled by pmdalinux.  Also ensures that we return the usual
    code (PM_ERR_APPVERSION) for unsupported data instead of zero.

Comment 12 Miloš Prchlík 2015-10-14 14:12:09 UTC
Verified for build pcp-3.10.6-2.el7.

Comment 13 errata-xmlrpc 2015-11-19 11:53:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2096.html