Bug 1314610 - snmpd complaining twice "Cannot statfs net:[********]#***: No such file or directory" every 10 minutes [NEEDINFO]
snmpd complaining twice "Cannot statfs net:[********]#***: No such file or di...
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: net-snmp (Show other bugs)
7.2
Unspecified Unspecified
high Severity low
: rc
: ---
Assigned To: Josef Ridky
BaseOS QE Security Team
: Reopened
: 1318087 (view as bug list)
Depends On:
Blocks: 1420851 1473733 1380364
  Show dependency treegraph
 
Reported: 2016-03-03 23:25 EST by Shinobu KINJO
Modified: 2017-12-07 20:52 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-15 00:34:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
sbroz: needinfo? (skinjo)


Attachments (Terms of Use)

  None (edit)
Description Shinobu KINJO 2016-03-03 23:25:59 EST
Description of problem:

The snmpd keeps reading out the following logs.

 ## Log ###########
 Mar  1 13:44:01 osp7-director snmpd[1139]: 
 Cannot statfs net:[4026532264]#012: 
 No such file or directory
 Mar  1 13:44:01 osp7-director snmpd[1139]: 
 Cannot statfs net:[4026532264]#012: 
 No such file or directory
 ## Log ###########

 ## Frequency #####
 Twice / 10 minutes

 $ sudo grep "Cannot statfs net" /var/log/messages |\
 grep "Mar  1" | wc -l
 288
 ## Frequency #####

There is ZERO impact caused by this behaviour.
But it's better to mitigate effectively, because it never stops -;

What's confusing me is setting of "/etc/sysconfig/snmpd".

  $ cat /etc/sysconfig/snmpd 
  ###
  ### File managed by Puppet
  ###
  # snmpd command line options
  OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
                [1] ^^^^^^^^^^^^^

I am wondering if [1] is working or not properly.
If it's working properly, how can we mitigate this noisy behaviour?

Version-Release number of selected component (if applicable):

##################
~$ lsb_release -s
:core-4.1-amd64:core-4.1-noarch
[stack@osp7-director ~]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release:	7.2
Codename:	Maipo
##################

##################
~$ uname -a
Linux osp7-director.fullstack.go 3.10.0-327.4.5.el7.x86_64 
#1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
##################

How reproducible:


Steps to Reproduce:
1. Build RHEL
2. Deploy undercloud using rdo manager
   IMHO I do not think it matters.
3.

Actual results:


Expected results:


Additional info:

##################
$ snmpd -v

NET-SNMP version:  5.7.2
Web:               http://www.net-snmp.org/
Email:             net-snmp-coders@lists.sourceforge.net
##################

##################
Comment 3 Shinobu KINJO 2016-03-04 05:38:38 EST
Yeah, I got a quick reply.

(In reply to Jan Safranek from comment #2)
> (In reply to Shinobu KINJO from comment #0)
> > What's confusing me is setting of "/etc/sysconfig/snmpd".
> > 
> >   $ cat /etc/sysconfig/snmpd 
> >   OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
> >                 [1] ^^^^^^^^^^^^^
> > I am wondering if [1] is working or not properly.
> > If it's working properly, how can we mitigate this noisy behaviour?
> 
> It is working properly, "-LS0-6d" configures logging of log levels from 0
> (error) to 6 (info/debug) to syslog and therefore the message appears in
> /var/log/messages.
> 
> Looking into the code, "Cannot statfs %s" is logged with "error" level, so
> you need to disable the logging completely to get rid of it. I don't think
> it's a good idea, still you can just remove -LS0-6d from the options if you
> really need to.
> 
> I can fix snmpd to log the error only once, that's trivial.

Thank you!

> 
> 
> Now what causes the error. Looking at customer's sosreport and strace, I can
> see that the error is written when processing /proc/mtab and calling
> statfs() for each line in it. It stumbles upon this line:
> 
>     proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0
> 
> Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If
> I read the line correctly, "procfs" is mounted to a network socket. Is it
> even possible? Or what does the line mean?
> 

That was poc case. So it's already gone -; Now it's production.

> 
> Btw, good thinking, strace is always the best thing for debugging Net-SNMP
> and I would ask for it anyway!

Yes, of course. It's already there.
That's a friend of mine.

Again thank you for your really really quick response!!
Comment 5 Shinobu KINJO 2016-03-13 23:24:35 EDT
Update?
Comment 6 Jan Safranek 2016-03-14 05:15:29 EDT
I am not sure what to report here... I'll update snmpd to report "Cannot statfs net:[4026532264]#012:" only once and with better error message in the nearest Net-SNMP update. From comment #3 it seems to me that the customer is quite happy with pruned /proc/mounts.
Comment 7 Shinobu KINJO 2016-03-14 06:43:36 EDT
(In reply to Shinobu KINJO from comment #3)
> Yeah, I got a quick reply.
> 
> (In reply to Jan Safranek from comment #2)
> > (In reply to Shinobu KINJO from comment #0)
> > > What's confusing me is setting of "/etc/sysconfig/snmpd".
> > > 
> > >   $ cat /etc/sysconfig/snmpd 
> > >   OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
> > >                 [1] ^^^^^^^^^^^^^
> > > I am wondering if [1] is working or not properly.
> > > If it's working properly, how can we mitigate this noisy behaviour?
> > 
> > It is working properly, "-LS0-6d" configures logging of log levels from 0
> > (error) to 6 (info/debug) to syslog and therefore the message appears in
> > /var/log/messages.
> > 
> > Looking into the code, "Cannot statfs %s" is logged with "error" level, so
> > you need to disable the logging completely to get rid of it. I don't think
> > it's a good idea, still you can just remove -LS0-6d from the options if you
> > really need to.
> > 
> > I can fix snmpd to log the error only once, that's trivial.

Ι was expecting that you were going to suggest any kind of way to get rid of that noise.

> 
> Thank you!
> 
> > 
> > 
> > Now what causes the error. Looking at customer's sosreport and strace, I can
> > see that the error is written when processing /proc/mtab and calling
> > statfs() for each line in it. It stumbles upon this line:
> > 
> >     proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0
> > 
> > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If
> > I read the line correctly, "procfs" is mounted to a network socket. Is it
> > even possible? Or what does the line mean?

Probably I misunderstood what you said here. It would be more help, if you elaborate on that.

I will have a closer look more and get back to you.

> > 
> 
> That was poc case. So it's already gone -; Now it's production.
> 
> > 
> > Btw, good thinking, strace is always the best thing for debugging Net-SNMP
> > and I would ask for it anyway!
> 
> Yes, of course. It's already there.
> That's a friend of mine.
> 
> Again thank you for your really really quick response!!
Comment 8 Jan Safranek 2016-03-14 06:53:28 EDT
> Ι was expecting that you were going to suggest any kind of way to get rid of that noise.

No, there is no easy workaround. Either remove "net:[4026532288]" from /proc/mounts or fix Net-SNMP via errata.
Comment 9 Shinobu KINJO 2016-03-14 07:11:01 EDT
(In reply to Jan Safranek from comment #8)
> > Ι was expecting that you were going to suggest any kind of way to get rid of that noise.
> 
> No, there is no easy workaround. Either remove "net:[4026532288]" from
> /proc/mounts or fix Net-SNMP via errata.

Can you elaborate on why **there is no easy workaround**?
Comment 10 Jan Safranek 2016-03-14 07:31:23 EDT
Because as I wrote in comment #2,  snmpd reads /proc/mounts and runs statfs on each entry there. If any statfs call fails it logs an error. So, either stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or snmpd must be fixed to log something more useful and only once.
Comment 11 Shinobu KINJO 2016-03-14 07:45:01 EDT
(In reply to Jan Safranek from comment #10)
> Because as I wrote in comment #2,  snmpd reads /proc/mounts 

Thank you for this point.

> and runs statfs
> on each entry there. If any statfs call fails it logs an error. So, either
> stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or
> snmpd must be fixed to log something more useful and only once.
Comment 13 Tasos Papaioannou 2016-05-05 16:16:12 EDT
Hi,

I'm re-opening this BZ, because I don't think it's NOTABUG. OpenStack nodes or other machines using kernel namespaces will have these namespaces listed under /etc/mtab -> /proc/self/mounts. The second field is the namespace type:[inode], and isn't a valid path to pass to statfs(). Can net-snmp be fixed to silently namespace entries in /etc/mtab? Otherwise any system that implements them will see these "errors".
Comment 14 Jan Safranek 2016-05-06 08:09:57 EDT
> Can net-snmp be fixed to silently namespace entries in /etc/mtab?

Of course it can, it should be a simple change.
Comment 15 Shinobu KINJO 2016-10-05 15:33:17 EDT
*** Bug 1318087 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.