Bug 1314610
| Summary: | snmpd complaining twice "Cannot statfs net:[********]#***: No such file or directory" every 10 minutes | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Shinobu KINJO <skinjo> | ||||
| Component: | net-snmp | Assignee: | Josef Ridky <jridky> | ||||
| Status: | CLOSED ERRATA | QA Contact: | David Jež <djez> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.2 | CC: | bschmaus, djez, fkrska, glen, jridky, jsafrane, ovasik, rfreire, sbroz | ||||
| Target Milestone: | rc | Keywords: | Reopened | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | net-snmp-5.7.2-39.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-08-06 13:08:42 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1380364, 1420851, 1473733, 1630919, 1660473 | ||||||
| Attachments: |
|
||||||
Yeah, I got a quick reply. (In reply to Jan Safranek from comment #2) > (In reply to Shinobu KINJO from comment #0) > > What's confusing me is setting of "/etc/sysconfig/snmpd". > > > > $ cat /etc/sysconfig/snmpd > > OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" > > [1] ^^^^^^^^^^^^^ > > I am wondering if [1] is working or not properly. > > If it's working properly, how can we mitigate this noisy behaviour? > > It is working properly, "-LS0-6d" configures logging of log levels from 0 > (error) to 6 (info/debug) to syslog and therefore the message appears in > /var/log/messages. > > Looking into the code, "Cannot statfs %s" is logged with "error" level, so > you need to disable the logging completely to get rid of it. I don't think > it's a good idea, still you can just remove -LS0-6d from the options if you > really need to. > > I can fix snmpd to log the error only once, that's trivial. Thank you! > > > Now what causes the error. Looking at customer's sosreport and strace, I can > see that the error is written when processing /proc/mtab and calling > statfs() for each line in it. It stumbles upon this line: > > proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0 > > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If > I read the line correctly, "procfs" is mounted to a network socket. Is it > even possible? Or what does the line mean? > That was poc case. So it's already gone -; Now it's production. > > Btw, good thinking, strace is always the best thing for debugging Net-SNMP > and I would ask for it anyway! Yes, of course. It's already there. That's a friend of mine. Again thank you for your really really quick response!! Update? I am not sure what to report here... I'll update snmpd to report "Cannot statfs net:[4026532264]#012:" only once and with better error message in the nearest Net-SNMP update. From comment #3 it seems to me that the customer is quite happy with pruned /proc/mounts. (In reply to Shinobu KINJO from comment #3) > Yeah, I got a quick reply. > > (In reply to Jan Safranek from comment #2) > > (In reply to Shinobu KINJO from comment #0) > > > What's confusing me is setting of "/etc/sysconfig/snmpd". > > > > > > $ cat /etc/sysconfig/snmpd > > > OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" > > > [1] ^^^^^^^^^^^^^ > > > I am wondering if [1] is working or not properly. > > > If it's working properly, how can we mitigate this noisy behaviour? > > > > It is working properly, "-LS0-6d" configures logging of log levels from 0 > > (error) to 6 (info/debug) to syslog and therefore the message appears in > > /var/log/messages. > > > > Looking into the code, "Cannot statfs %s" is logged with "error" level, so > > you need to disable the logging completely to get rid of it. I don't think > > it's a good idea, still you can just remove -LS0-6d from the options if you > > really need to. > > > > I can fix snmpd to log the error only once, that's trivial. Ι was expecting that you were going to suggest any kind of way to get rid of that noise. > > Thank you! > > > > > > > Now what causes the error. Looking at customer's sosreport and strace, I can > > see that the error is written when processing /proc/mtab and calling > > statfs() for each line in it. It stumbles upon this line: > > > > proc net:[4026532288] proc rw,nosuid,nodev,noexec,relatime 0 0 > > > > Where statfs("net:[4026532288]") fails and snmpd writes error to the log. If > > I read the line correctly, "procfs" is mounted to a network socket. Is it > > even possible? Or what does the line mean? Probably I misunderstood what you said here. It would be more help, if you elaborate on that. I will have a closer look more and get back to you. > > > > That was poc case. So it's already gone -; Now it's production. > > > > > Btw, good thinking, strace is always the best thing for debugging Net-SNMP > > and I would ask for it anyway! > > Yes, of course. It's already there. > That's a friend of mine. > > Again thank you for your really really quick response!! > Ι was expecting that you were going to suggest any kind of way to get rid of that noise.
No, there is no easy workaround. Either remove "net:[4026532288]" from /proc/mounts or fix Net-SNMP via errata.
(In reply to Jan Safranek from comment #8) > > Ι was expecting that you were going to suggest any kind of way to get rid of that noise. > > No, there is no easy workaround. Either remove "net:[4026532288]" from > /proc/mounts or fix Net-SNMP via errata. Can you elaborate on why **there is no easy workaround**? Because as I wrote in comment #2, snmpd reads /proc/mounts and runs statfs on each entry there. If any statfs call fails it logs an error. So, either stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or snmpd must be fixed to log something more useful and only once. (In reply to Jan Safranek from comment #10) > Because as I wrote in comment #2, snmpd reads /proc/mounts Thank you for this point. > and runs statfs > on each entry there. If any statfs call fails it logs an error. So, either > stafs must not fail (i.e. no "net:[4026532288]" entries in /proc/mounts) or > snmpd must be fixed to log something more useful and only once. Hi, I'm re-opening this BZ, because I don't think it's NOTABUG. OpenStack nodes or other machines using kernel namespaces will have these namespaces listed under /etc/mtab -> /proc/self/mounts. The second field is the namespace type:[inode], and isn't a valid path to pass to statfs(). Can net-snmp be fixed to silently namespace entries in /etc/mtab? Otherwise any system that implements them will see these "errors". > Can net-snmp be fixed to silently namespace entries in /etc/mtab?
Of course it can, it should be a simple change.
*** Bug 1318087 has been marked as a duplicate of this bug. *** Created attachment 1337615 [details]
Patch with possible solution
> > I can fix snmpd to log the error only once, that's trivial. or > > Can net-snmp be fixed to silently namespace entries in /etc/mtab? > Of course it can, it should be a simple change. is there patch for any of that? so i could apply and build myself modified snmpd... Patch is already prepared, but isn't accepted by upstream yet, so I am not able to deliver it to RHEL. thanks for unhiding the patch. applied it to my distro: v5.7.3 https://github.com/pld-linux/net-snmp/commit/c3c768527e0fa7110927eb149e2df13f64e2e4d2 needed to adjust to remove `\n` in it. did you apply on top of other patches? Yes, this patch has been prepared on top of all patches net-snmp distro in RHEL. can you link to upstream tracker, so any interested person arriving here can subscribe there too? Net-snmp upstream is migrating from sourceforge to github. Hope this bugreport will not be omitted during project migration. https://sourceforge.net/p/net-snmp/bugs/2870/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2239 |
Description of problem: The snmpd keeps reading out the following logs. ## Log ########### Mar 1 13:44:01 osp7-director snmpd[1139]: Cannot statfs net:[4026532264]#012: No such file or directory Mar 1 13:44:01 osp7-director snmpd[1139]: Cannot statfs net:[4026532264]#012: No such file or directory ## Log ########### ## Frequency ##### Twice / 10 minutes $ sudo grep "Cannot statfs net" /var/log/messages |\ grep "Mar 1" | wc -l 288 ## Frequency ##### There is ZERO impact caused by this behaviour. But it's better to mitigate effectively, because it never stops -; What's confusing me is setting of "/etc/sysconfig/snmpd". $ cat /etc/sysconfig/snmpd ### ### File managed by Puppet ### # snmpd command line options OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid" [1] ^^^^^^^^^^^^^ I am wondering if [1] is working or not properly. If it's working properly, how can we mitigate this noisy behaviour? Version-Release number of selected component (if applicable): ################## ~$ lsb_release -s :core-4.1-amd64:core-4.1-noarch [stack@osp7-director ~]$ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.2 (Maipo) Release: 7.2 Codename: Maipo ################## ################## ~$ uname -a Linux osp7-director.fullstack.go 3.10.0-327.4.5.el7.x86_64 #1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux ################## How reproducible: Steps to Reproduce: 1. Build RHEL 2. Deploy undercloud using rdo manager IMHO I do not think it matters. 3. Actual results: Expected results: Additional info: ################## $ snmpd -v NET-SNMP version: 5.7.2 Web: http://www.net-snmp.org/ Email: net-snmp-coders.net ################## ##################