Hide Forgot
Description of problem: The snmpd keeps reading out the following logs. ## Log ########### Mar 1 13:44:01 osp7-director snmpd[1139]: Cannot statfs net:[4026532264]#012: No such file or directory Mar 1 13:44:01 osp7-director snmpd[1139]: Cannot statfs net:[4026532264]#012: No such file or directory ## Log ########### ## Frequency ##### Twice / 10 minutes $ sudo grep "Cannot statfs net" /var/log/messages |\ grep "Mar 1" | wc -l 288 ## Frequency ##### There is ZERO impact caused by this behaviour. But it's better to mitigate effectively, because it never stops -; What's confusing me is setting of "/etc/sysconfig/snmpd". $ cat /etc/sysconfig/snmpd ### ### File managed by Puppet ### # snmpd command line options OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" [1] ^^^^^^^^^^^^^ I am wondering if [1] is working or not properly. If it's working properly, how can we mitigate this noisy behaviour? Version-Release number of selected component (if applicable): ################## ~$ lsb_release -s :core-4.1-amd64:core-4.1-noarch [stack@osp7-director ~]$ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.2 (Maipo) Release: 7.2 Codename: Maipo ################## ################## ~$ uname -a Linux osp7-director.fullstack.go 3.10.0-327.4.5.el7.x86_64 #1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux ################## How reproducible: Steps to Reproduce: 1. Build RHEL 2. Deploy undercloud using rdo manager IMHO I do not think it matters. 3. Actual results: Expected results: Additional info: ################## $ snmpd -v NET-SNMP version: 5.7.2 Web: http://www.net-snmp.org/ Email: net-snmp-coders.net ################## ##################
Yeah, I got a quick reply. (In reply to Jan Safranek from comment #2) > (In reply to Shinobu KINJO from comment #0) > > What's confusing me is setting of "/etc/sysconfig/snmpd". > > > > $ cat /etc/sysconfig/snmpd > > OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" > > [1] ^^^^^^^^^^^^^ > > I am wondering if [1] is working or not properly. > > If it's working properly, how can we mitigate this noisy behaviour? > > It is working properly, "-LS0-6d" configures logging of log levels from 0 > (error) to 6 (info/debug) to syslog and therefore the message appears in > /var/log/messages. > > Looking into the code, "Cannot statfs %s" is logged with "error" level, so > you need to disable the logging completely to get rid of it. I don't think > it's a good idea, still you can just remove -LS0-6d from the options if you > really need to. > > I can fix snmpd to log the error only once, that's trivial. Thank you! > > > Now what causes the error. Looking at customer's sosreport and strace, I can > see that the error is written when processing /proc/mtab and calling > statfs() for each line in it. It stumbles upon this line: > > proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0 > > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If > I read the line correctly, "procfs" is mounted to a network socket. Is it > even possible? Or what does the line mean? > That was poc case. So it's already gone -; Now it's production. > > Btw, good thinking, strace is always the best thing for debugging Net-SNMP > and I would ask for it anyway! Yes, of course. It's already there. That's a friend of mine. Again thank you for your really really quick response!!
Update?
I am not sure what to report here... I'll update snmpd to report "Cannot statfs net:[4026532264]#012:" only once and with better error message in the nearest Net-SNMP update. From comment #3 it seems to me that the customer is quite happy with pruned /proc/mounts.
(In reply to Shinobu KINJO from comment #3) > Yeah, I got a quick reply. > > (In reply to Jan Safranek from comment #2) > > (In reply to Shinobu KINJO from comment #0) > > > What's confusing me is setting of "/etc/sysconfig/snmpd". > > > > > > $ cat /etc/sysconfig/snmpd > > > OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" > > > [1] ^^^^^^^^^^^^^ > > > I am wondering if [1] is working or not properly. > > > If it's working properly, how can we mitigate this noisy behaviour? > > > > It is working properly, "-LS0-6d" configures logging of log levels from 0 > > (error) to 6 (info/debug) to syslog and therefore the message appears in > > /var/log/messages. > > > > Looking into the code, "Cannot statfs %s" is logged with "error" level, so > > you need to disable the logging completely to get rid of it. I don't think > > it's a good idea, still you can just remove -LS0-6d from the options if you > > really need to. > > > > I can fix snmpd to log the error only once, that's trivial. Ι was expecting that you were going to suggest any kind of way to get rid of that noise. > > Thank you! > > > > > > > Now what causes the error. Looking at customer's sosreport and strace, I can > > see that the error is written when processing /proc/mtab and calling > > statfs() for each line in it. It stumbles upon this line: > > > > proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0 > > > > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If > > I read the line correctly, "procfs" is mounted to a network socket. Is it > > even possible? Or what does the line mean? Probably I misunderstood what you said here. It would be more help, if you elaborate on that. I will have a closer look more and get back to you. > > > > That was poc case. So it's already gone -; Now it's production. > > > > > Btw, good thinking, strace is always the best thing for debugging Net-SNMP > > and I would ask for it anyway! > > Yes, of course. It's already there. > That's a friend of mine. > > Again thank you for your really really quick response!!
> Ι was expecting that you were going to suggest any kind of way to get rid of that noise. No, there is no easy workaround. Either remove "net:[4026532288]" from /proc/mounts or fix Net-SNMP via errata.
(In reply to Jan Safranek from comment #8) > > Ι was expecting that you were going to suggest any kind of way to get rid of that noise. > > No, there is no easy workaround. Either remove "net:[4026532288]" from > /proc/mounts or fix Net-SNMP via errata. Can you elaborate on why **there is no easy workaround**?
Because as I wrote in comment #2, snmpd reads /proc/mounts and runs statfs on each entry there. If any statfs call fails it logs an error. So, either stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or snmpd must be fixed to log something more useful and only once.
(In reply to Jan Safranek from comment #10) > Because as I wrote in comment #2, snmpd reads /proc/mounts Thank you for this point. > and runs statfs > on each entry there. If any statfs call fails it logs an error. So, either > stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or > snmpd must be fixed to log something more useful and only once.
Hi, I'm re-opening this BZ, because I don't think it's NOTABUG. OpenStack nodes or other machines using kernel namespaces will have these namespaces listed under /etc/mtab -> /proc/self/mounts. The second field is the namespace type:[inode], and isn't a valid path to pass to statfs(). Can net-snmp be fixed to silently namespace entries in /etc/mtab? Otherwise any system that implements them will see these "errors".
> Can net-snmp be fixed to silently namespace entries in /etc/mtab? Of course it can, it should be a simple change.
*** Bug 1318087 has been marked as a duplicate of this bug. ***
Created attachment 1337615 [details] Patch with possible solution
> > I can fix snmpd to log the error only once, that's trivial. or > > Can net-snmp be fixed to silently namespace entries in /etc/mtab? > Of course it can, it should be a simple change. is there patch for any of that? so i could apply and build myself modified snmpd...
Patch is already prepared, but isn't accepted by upstream yet, so I am not able to deliver it to RHEL.
thanks for unhiding the patch. applied it to my distro: v5.7.3 https://github.com/pld-linux/net-snmp/commit/c3c768527e0fa7110927eb149e2df13f64e2e4d2 needed to adjust to remove `\n` in it. did you apply on top of other patches?
Yes, this patch has been prepared on top of all patches net-snmp distro in RHEL.
can you link to upstream tracker, so any interested person arriving here can subscribe there too?
Net-snmp upstream is migrating from sourceforge to github. Hope this bugreport will not be omitted during project migration. https://sourceforge.net/p/net-snmp/bugs/2870/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2239