Bug 1314610

Summary: snmpd complaining twice "Cannot statfs net:[********]#***: No such file or directory" every 10 minutes
Product: Red Hat Enterprise Linux 7 Reporter: Shinobu KINJO <skinjo>
Component: net-snmpAssignee: Josef Ridky <jridky>
Status: CLOSED ERRATA QA Contact: David Jež <djez>
Severity: low Docs Contact:
Priority: high    
Version: 7.2CC: bschmaus, djez, fkrska, glen, jridky, jsafrane, ovasik, rfreire, sbroz
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: net-snmp-5.7.2-39.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 13:08:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1420851, 1473733, 1660473, 1380364, 1630919    
Attachments:
Description Flags
Patch with possible solution none

Description Shinobu KINJO 2016-03-04 04:25:59 UTC
Description of problem:

The snmpd keeps reading out the following logs.

 ## Log ###########
 Mar  1 13:44:01 osp7-director snmpd[1139]: 
 Cannot statfs net:[4026532264]#012: 
 No such file or directory
 Mar  1 13:44:01 osp7-director snmpd[1139]: 
 Cannot statfs net:[4026532264]#012: 
 No such file or directory
 ## Log ###########

 ## Frequency #####
 Twice / 10 minutes

 $ sudo grep "Cannot statfs net" /var/log/messages |\
 grep "Mar  1" | wc -l
 288
 ## Frequency #####

There is ZERO impact caused by this behaviour.
But it's better to mitigate effectively, because it never stops -;

What's confusing me is setting of "/etc/sysconfig/snmpd".

  $ cat /etc/sysconfig/snmpd 
  ###
  ### File managed by Puppet
  ###
  # snmpd command line options
  OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
                [1] ^^^^^^^^^^^^^

I am wondering if [1] is working or not properly.
If it's working properly, how can we mitigate this noisy behaviour?

Version-Release number of selected component (if applicable):

##################
~$ lsb_release -s
:core-4.1-amd64:core-4.1-noarch
[stack@osp7-director ~]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release:	7.2
Codename:	Maipo
##################

##################
~$ uname -a
Linux osp7-director.fullstack.go 3.10.0-327.4.5.el7.x86_64 
#1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
##################

How reproducible:


Steps to Reproduce:
1. Build RHEL
2. Deploy undercloud using rdo manager
   IMHO I do not think it matters.
3.

Actual results:


Expected results:


Additional info:

##################
$ snmpd -v

NET-SNMP version:  5.7.2
Web:               http://www.net-snmp.org/
Email:             net-snmp-coders@lists.sourceforge.net
##################

##################

Comment 3 Shinobu KINJO 2016-03-04 10:38:38 UTC
Yeah, I got a quick reply.

(In reply to Jan Safranek from comment #2)
> (In reply to Shinobu KINJO from comment #0)
> > What's confusing me is setting of "/etc/sysconfig/snmpd".
> > 
> >   $ cat /etc/sysconfig/snmpd 
> >   OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
> >                 [1] ^^^^^^^^^^^^^
> > I am wondering if [1] is working or not properly.
> > If it's working properly, how can we mitigate this noisy behaviour?
> 
> It is working properly, "-LS0-6d" configures logging of log levels from 0
> (error) to 6 (info/debug) to syslog and therefore the message appears in
> /var/log/messages.
> 
> Looking into the code, "Cannot statfs %s" is logged with "error" level, so
> you need to disable the logging completely to get rid of it. I don't think
> it's a good idea, still you can just remove -LS0-6d from the options if you
> really need to.
> 
> I can fix snmpd to log the error only once, that's trivial.

Thank you!

> 
> 
> Now what causes the error. Looking at customer's sosreport and strace, I can
> see that the error is written when processing /proc/mtab and calling
> statfs() for each line in it. It stumbles upon this line:
> 
>     proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0
> 
> Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If
> I read the line correctly, "procfs" is mounted to a network socket. Is it
> even possible? Or what does the line mean?
> 

That was poc case. So it's already gone -; Now it's production.

> 
> Btw, good thinking, strace is always the best thing for debugging Net-SNMP
> and I would ask for it anyway!

Yes, of course. It's already there.
That's a friend of mine.

Again thank you for your really really quick response!!

Comment 5 Shinobu KINJO 2016-03-14 03:24:35 UTC
Update?

Comment 6 Jan Safranek 2016-03-14 09:15:29 UTC
I am not sure what to report here... I'll update snmpd to report "Cannot statfs net:[4026532264]#012:" only once and with better error message in the nearest Net-SNMP update. From comment #3 it seems to me that the customer is quite happy with pruned /proc/mounts.

Comment 7 Shinobu KINJO 2016-03-14 10:43:36 UTC
(In reply to Shinobu KINJO from comment #3)
> Yeah, I got a quick reply.
> 
> (In reply to Jan Safranek from comment #2)
> > (In reply to Shinobu KINJO from comment #0)
> > > What's confusing me is setting of "/etc/sysconfig/snmpd".
> > > 
> > >   $ cat /etc/sysconfig/snmpd 
> > >   OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"
> > >                 [1] ^^^^^^^^^^^^^
> > > I am wondering if [1] is working or not properly.
> > > If it's working properly, how can we mitigate this noisy behaviour?
> > 
> > It is working properly, "-LS0-6d" configures logging of log levels from 0
> > (error) to 6 (info/debug) to syslog and therefore the message appears in
> > /var/log/messages.
> > 
> > Looking into the code, "Cannot statfs %s" is logged with "error" level, so
> > you need to disable the logging completely to get rid of it. I don't think
> > it's a good idea, still you can just remove -LS0-6d from the options if you
> > really need to.
> > 
> > I can fix snmpd to log the error only once, that's trivial.

Ι was expecting that you were going to suggest any kind of way to get rid of that noise.

> 
> Thank you!
> 
> > 
> > 
> > Now what causes the error. Looking at customer's sosreport and strace, I can
> > see that the error is written when processing /proc/mtab and calling
> > statfs() for each line in it. It stumbles upon this line:
> > 
> >     proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0
> > 
> > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If
> > I read the line correctly, "procfs" is mounted to a network socket. Is it
> > even possible? Or what does the line mean?

Probably I misunderstood what you said here. It would be more help, if you elaborate on that.

I will have a closer look more and get back to you.

> > 
> 
> That was poc case. So it's already gone -; Now it's production.
> 
> > 
> > Btw, good thinking, strace is always the best thing for debugging Net-SNMP
> > and I would ask for it anyway!
> 
> Yes, of course. It's already there.
> That's a friend of mine.
> 
> Again thank you for your really really quick response!!

Comment 8 Jan Safranek 2016-03-14 10:53:28 UTC
> Ι was expecting that you were going to suggest any kind of way to get rid of that noise.

No, there is no easy workaround. Either remove "net:[4026532288]" from /proc/mounts or fix Net-SNMP via errata.

Comment 9 Shinobu KINJO 2016-03-14 11:11:01 UTC
(In reply to Jan Safranek from comment #8)
> > Ι was expecting that you were going to suggest any kind of way to get rid of that noise.
> 
> No, there is no easy workaround. Either remove "net:[4026532288]" from
> /proc/mounts or fix Net-SNMP via errata.

Can you elaborate on why **there is no easy workaround**?

Comment 10 Jan Safranek 2016-03-14 11:31:23 UTC
Because as I wrote in comment #2,  snmpd reads /proc/mounts and runs statfs on each entry there. If any statfs call fails it logs an error. So, either stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or snmpd must be fixed to log something more useful and only once.

Comment 11 Shinobu KINJO 2016-03-14 11:45:01 UTC
(In reply to Jan Safranek from comment #10)
> Because as I wrote in comment #2,  snmpd reads /proc/mounts 

Thank you for this point.

> and runs statfs
> on each entry there. If any statfs call fails it logs an error. So, either
> stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or
> snmpd must be fixed to log something more useful and only once.

Comment 13 Tasos Papaioannou 2016-05-05 20:16:12 UTC
Hi,

I'm re-opening this BZ, because I don't think it's NOTABUG. OpenStack nodes or other machines using kernel namespaces will have these namespaces listed under /etc/mtab -> /proc/self/mounts. The second field is the namespace type:[inode], and isn't a valid path to pass to statfs(). Can net-snmp be fixed to silently namespace entries in /etc/mtab? Otherwise any system that implements them will see these "errors".

Comment 14 Jan Safranek 2016-05-06 12:09:57 UTC
> Can net-snmp be fixed to silently namespace entries in /etc/mtab?

Of course it can, it should be a simple change.

Comment 15 Shinobu KINJO 2016-10-05 19:33:17 UTC
*** Bug 1318087 has been marked as a duplicate of this bug. ***

Comment 22 Josef Ridky 2017-10-12 08:57:38 UTC
Created attachment 1337615 [details]
Patch with possible solution

Comment 28 Elan Ruusamäe 2018-05-25 07:42:22 UTC
> > I can fix snmpd to log the error only once, that's trivial.

or

> > Can net-snmp be fixed to silently namespace entries in /etc/mtab?
> Of course it can, it should be a simple change.

is there patch for any of that? so i could apply and build myself modified snmpd...

Comment 29 Josef Ridky 2018-05-29 12:43:57 UTC
Patch is already prepared, but isn't accepted by upstream yet, so I am not able to deliver it to RHEL.

Comment 30 Elan Ruusamäe 2018-06-04 07:55:32 UTC
thanks for unhiding the patch. applied it to my distro: v5.7.3

https://github.com/pld-linux/net-snmp/commit/c3c768527e0fa7110927eb149e2df13f64e2e4d2

needed to adjust to remove `\n` in it. did you apply on top of other patches?

Comment 31 Josef Ridky 2018-06-04 08:23:37 UTC
Yes, this patch has been prepared on top of all patches net-snmp distro in RHEL.

Comment 32 Elan Ruusamäe 2018-06-05 16:30:49 UTC
can you link to upstream tracker, so any interested person arriving here can subscribe there too?

Comment 33 Josef Ridky 2018-06-07 10:50:19 UTC
Net-snmp upstream is migrating from sourceforge to github. Hope this bugreport will not be omitted during project migration.

https://sourceforge.net/p/net-snmp/bugs/2870/

Comment 39 errata-xmlrpc 2019-08-06 13:08:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2239