Bug 955902
Summary: | /usr/bin/munin-cron: not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866 | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Edgar Hoch <edgar.hoch> |
Component: | munin | Assignee: | d. johnson <drjohnson1> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 20 | CC: | 1o5g4r8o, andy, customercare, drjohnson1, edgar.hoch, gfauvie, hlovdal, ingvar, jorti, jvanek, kevin, M8R-7fin56, mehmet, opensource, swehack |
Target Milestone: | --- | Keywords: | Reopened |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-03-07 04:57:12 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Edgar Hoch
2013-04-24 03:26:57 UTC
Can you provide the output of: ls -ld /var/lib/munin /var/lib/munin/plugin-state /var/lib/munin/plugin-state/nobody rpm -qa munin\* Then try this if the last one does not exist? sudo install -d -m 0755 -o nobody -g root /var/lib/munin/plugin-state/nobody Hello, Same problem on a RHEL 6.4. The outputs you ask are: [root@DBSP_H5_MigV data]# ls -ld /var/lib/munin /var/lib/munin/plugin-state /var/lib/munin/plugin-state/nobody drwxr-xr-x 3 munin munin 4096 Apr 24 15:35 /var/lib/munin drwxrwxr-x 5 root munin 4096 Mar 24 04:42 /var/lib/munin/plugin-state drwxr-xr-x 2 nobody root 4096 Apr 24 14:30 /var/lib/munin/plugin-state/nobody [root@DBSP_H5_MigV data]# rpm -qa munin\* munin-2.0.12-2.el6.noarch munin-node-2.0.12-2.el6.noarch munin-common-2.0.12-2.el6.noarch The command you gave (install -d -m 0755 -o nobody -g root /var/lib/munin/plugin-state/nobody) seems not fixing the problem. Best Regards, Output of a manual launch of munin in debug mode: -bash-4.1$ /usr/bin/munin-cron --debug 2013/04/24 16:31:42 [DEBUG] Creating new lock file /var/run/munin/munin-update.lock 2013/04/24 16:31:42 [DEBUG] Creating lock : /var/run/munin/munin-update.lock succeeded 2013/04/24 16:31:42 [INFO]: Starting munin-update 2013/04/24 16:31:42 [DEBUG] Starting work 2013/04/24 16:31:42 [DEBUG] Active workers: 0/16 2013/04/24 16:31:42 [DEBUG] Active workers: 1/16 2013/04/24 16:31:42 [DEBUG] Starting Munin::Master::UpdateWorker<localhost;localhost> 2013/04/24 16:31:42 [DEBUG] Creating new lock file /var/run/munin/munin-localhost-localhost.lock 2013/04/24 16:31:42 [DEBUG] Creating lock : /var/run/munin/munin-localhost-localhost.lock succeeded 2013/04/24 16:31:42 [DEBUG] Reading state for localhost-localhost in /var/lib/munin/state-localhost-localhost.storable 2013/04/24 16:31:42 [INFO] starting work in 1310 for localhost/127.0.0.1:4949. 2013/04/24 16:31:42 TLS set to "disabled". 2013/04/24 16:31:42 [DEBUG] Negotiating capabilities 2013/04/24 16:31:42 [DEBUG] Writing to socket: "cap multigraph dirtyconfig ". 2013/04/24 16:31:42 [DEBUG] Node says /cap multigraph dirtyconfig/ 2013/04/24 16:31:42 [DEBUG] Writing to socket: "list DBSP_H5_MigV ". 2013/04/24 16:31:42 [WARNING] Config node localhost listed no services for DBSP_H5_MigV. Please see http://munin-monitoring.org/wiki/FAQ_no_graphs for further information. 2013/04/24 16:31:42 [DEBUG] Writing to socket: "quit ". 2013/04/24 16:31:42 [DEBUG] quit: 0.000204 sec on localhost/127.0.0.1/4949 2013/04/24 16:31:42 [DEBUG] Everything went smoothly. 2013/04/24 16:31:42 [DEBUG] Writing state for localhost-localhost in /var/lib/munin/state-localhost-localhost.storable 2013/04/24 16:31:42 [DEBUG] Exit status 0 for worker Munin::Master::UpdateWorker<localhost;localhost> 2013/04/24 16:31:42 [INFO]: Munin-update finished for node localhost;localhost (0.01 sec) 2013/04/24 16:31:42 [INFO] Reaping Munin::Master::UpdateWorker<localhost;localhost>. Exit value/signal: 0/0 2013/04/24 16:31:42 [DEBUG] Work done 2013/04/24 16:31:42 [DEBUG] Creating new lock file /var/run/munin/munin-datafile.lock 2013/04/24 16:31:42 [DEBUG] Creating lock : /var/run/munin/munin-datafile.lock succeeded 2013/04/24 16:31:42 [DEBUG] Writing state to /var/lib/munin/datafile.storable 2013/04/24 16:31:42 [INFO]: Munin-update finished (0.02 sec) 2013/04/24 16:31:42 [INFO] Starting munin-limits, getting lock /var/run/munin/munin-limits.lock 2013/04/24 16:31:42 [DEBUG] Creating new lock file /var/run/munin/munin-limits.lock 2013/04/24 16:31:42 [DEBUG] Creating lock : /var/run/munin/munin-limits.lock succeeded 2013/04/24 16:31:42 [DEBUG] munin_readconfig: found Storable version of /var/lib/munin/limits.storable, using it 2013/04/24 16:31:42 [DEBUG] Set default "contacts" to "" 2013/04/24 16:31:42 [DEBUG] Writing state to /var/lib/munin/limits 2013/04/24 16:31:42 [DEBUG] Closing filehandle "/var/lib/munin/limits"... 2013/04/24 16:31:42 [DEBUG] Writing state to /var/lib/munin/limits.storable 2013/04/24 16:31:42 [INFO] munin-limits finished (0.00 sec) 2013/04/24 16:31:43 [DEBUG] Determined that cgiurl_graph is /munin-cgi/munin-cgi-graph 2013/04/24 16:31:43 [DEBUG] munin_readconfig: found Storable version of /var/lib/munin/limits.storable, using it 2013/04/24 16:31:43 [DEBUG] Writing state to /var/lib/munin/htmlconf.storable 2013/04/24 16:31:43 [PERL WARNING] Use of uninitialized value in list assignment at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 134. 2013/04/24 16:31:43 [PERL WARNING] data is null at at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 947 Munin::Master::Utils::munin_write_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 964 Munin::Master::Utils::munin_writeconfig_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 181 Munin::Master::HTMLOld::get_config(0) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 191 Munin::Master::HTMLOld::html_main() called at /usr/share/munin/munin-html line 40 not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 950 Hope I will help you. Not sure; I cannot reproduce the issue you have above: munin-2.0.12-2.fc18.noarch munin-node-2.0.12-2.fc18.noarch munin-common-2.0.12-2.fc18.noarch munin-async-2.0.12-2.fc18.noarch munin-2.0.12-2.fc17.noarch munin-async-2.0.12-2.fc17.noarch munin-node-2.0.12-2.fc17.noarch munin-common-2.0.12-2.fc17.noarch munin-common-2.0.12-2.el6.noarch munin-2.0.12-2.el6.noarch munin-node-2.0.12-2.el6.noarch munin-async-2.0.12-2.el6.noarch munin-java-plugins-2.0.12-2.el6.noarch munin-common-2.0.12-2.el5 munin-java-plugins-2.0.12-2.el5 munin-ruby-plugins-2.0.12-2.el5 munin-node-2.0.12-2.el5 munin-cgi-2.0.12-2.el5 munin-2.0.12-2.el5 munin-async-2.0.12-2.el5 Most recent I can find is from Date: Fri, 25 Jan 2013 16:59:38 -0700 (Shortly after 2.0.11-3) This may take a while to sort out. (In reply to comment #1) > Can you provide the output of: > > ls -ld /var/lib/munin /var/lib/munin/plugin-state > /var/lib/munin/plugin-state/nobody drwxr-xr-x. 3 munin munin 4096 24. Apr 05:17 /var/lib/munin drwxrwxr-x. 5 root munin 4096 24. Mär 04:42 /var/lib/munin/plugin-state drwxr-xr-x. 2 nobody root 4096 24. Apr 00:18 /var/lib/munin/plugin-state/nobody > rpm -qa munin\* munin-node-2.0.12-2.fc18.noarch munin-common-2.0.12-2.fc18.noarch munin-2.0.12-2.fc18.noarch > Then try this if the last one does not exist? > > sudo install -d -m 0755 -o nobody -g root /var/lib/munin/plugin-state/nobody The directory exists already. I did a fresh install of Fedora 18 on a virtual machine, with no modifications regarding munin. yum-updatesd have done an update to munin-common-2.0.13-1.fc18.noarch munin-node-2.0.13-1.fc18.noarch munin-2.0.13-1.fc18.noarch Now I get cron messages every five minutes with mail subject "Cron <munin@rauhfussbussard> test -x /usr/bin/munin-cron && /usr/bin/munin-cron" and mail content not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866. I don't know if one reason for the errer may be that I didn't change the configuration of munin. The munin packages are just installed, without a change. When a package is installed, it should not create errors. I think, it should do nothing, it should not run cron without explicit configuration by the user (it is possible to check for a flag set by the user or set / changed by a command (like munin-activate or eventually with systemctl?). Here a debug output: munin$ /usr/bin/munin-cron --debug 2013/05/09 22:34:03 [DEBUG] Creating new lock file /var/run/munin/munin-update.lock 2013/05/09 22:34:03 [DEBUG] Creating lock : /var/run/munin/munin-update.lock succeeded 2013/05/09 22:34:03 [INFO]: Starting munin-update 2013/05/09 22:34:03 [DEBUG] Starting work 2013/05/09 22:34:03 [DEBUG] Active workers: 0/16 2013/05/09 22:34:03 [DEBUG] Active workers: 1/16 2013/05/09 22:34:03 [DEBUG] Starting Munin::Master::UpdateWorker<localhost;localhost> 2013/05/09 22:34:03 [DEBUG] Lock /var/run/munin/munin-localhost-localhost.lock already exists, checking process 2013/05/09 22:34:03 [DEBUG] Lock contained pid '26412' 2013/05/09 22:34:03 [INFO] Process 26412 is dead, stealing lock, removing file 2013/05/09 22:34:03 [DEBUG] Creating new lock file /var/run/munin/munin-localhost-localhost.lock 2013/05/09 22:34:03 [DEBUG] Creating lock : /var/run/munin/munin-localhost-localhost.lock succeeded 2013/05/09 22:34:03 [DEBUG] Reading state for localhost-localhost in /var/lib/munin/state-localhost-localhost.storable 2013/05/09 22:34:03 [INFO] starting work in 26657 for localhost/127.0.0.1:4949. 2013/05/09 22:34:03 [FATAL] Socket read from localhost failed. Terminating process. at /usr/share/perl5/vendor_perl/Munin/Master/UpdateWorker.pm line 254. 2013/05/09 22:34:03 [ERROR] Munin::Master::UpdateWorker<localhost;localhost> died with '[FATAL] Socket read from localhost failed. Terminating process. at /usr/share/perl5/vendor_perl/Munin/Master/UpdateWorker.pm line 254. ' 2013/05/09 22:34:03 [DEBUG] Exit status 18 for worker Munin::Master::UpdateWorker<localhost;localhost> 2013/05/09 22:34:13 [INFO] Remaining workers: localhost;localhost 2013/05/09 22:34:13 [DEBUG] Got error from 26657: 4608. Handling error 2013/05/09 22:34:13 [DEBUG] In exception handler for failed worker localhost;localhost 2013/05/09 22:34:13 [INFO] Reaping Munin::Master::UpdateWorker<localhost;localhost>. Exit value/signal: 18/0 2013/05/09 22:34:13 [DEBUG] Active workers: 0/16 2013/05/09 22:34:13 [DEBUG] Work done 2013/05/09 22:34:13 [DEBUG] Creating new lock file /var/run/munin/munin-datafile.lock 2013/05/09 22:34:13 [DEBUG] Creating lock : /var/run/munin/munin-datafile.lock succeeded 2013/05/09 22:34:13 [INFO] No old data available for failed worker localhost;localhost. This node will disappear from the html web page hierarchy 2013/05/09 22:34:13 [DEBUG] Writing state to /var/lib/munin/datafile.storable 2013/05/09 22:34:13 [INFO]: Munin-update finished (10.01 sec) 2013/05/09 22:34:13 [INFO] Starting munin-limits, getting lock /var/run/munin/munin-limits.lock 2013/05/09 22:34:13 [DEBUG] Creating new lock file /var/run/munin/munin-limits.lock 2013/05/09 22:34:13 [DEBUG] Creating lock : /var/run/munin/munin-limits.lock succeeded 2013/05/09 22:34:13 [DEBUG] munin_readconfig: found Storable version of /var/lib/munin/limits.storable, using it 2013/05/09 22:34:13 [DEBUG] Set default "contacts" to "" 2013/05/09 22:34:13 [DEBUG] Writing state to /var/lib/munin/limits 2013/05/09 22:34:13 [DEBUG] Closing filehandle "/var/lib/munin/limits"... 2013/05/09 22:34:13 [DEBUG] Writing state to /var/lib/munin/limits.storable 2013/05/09 22:34:13 [INFO] munin-limits finished (0.00 sec) 2013/05/09 22:34:13 [DEBUG] munin_readconfig: found Storable version of /var/lib/munin/limits.storable, using it 2013/05/09 22:34:13 [DEBUG] Writing state to /var/lib/munin/htmlconf.storable 2013/05/09 22:34:13 [PERL WARNING] data is null at at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 863. Munin::Master::Utils::munin_write_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 877 Munin::Master::Utils::munin_writeconfig_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 182 Munin::Master::HTMLOld::get_config(0) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 192 Munin::Master::HTMLOld::html_main() called at /usr/share/munin/munin-html line 40 not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866. Hello, Ok, I think that I solved the problem. In my case (munin 2.0.13 via EPEL) on RHEL 6.4, the origin of the problem was "_" (underscore) in the hostname. Munin uses underscore to pass parameters to the plugins. Try to change your hostname (and check munin-node.conf and munin.conf). It should work. If yes, I think that the status can be change to "solved". Best Regards, Gilles Fauvie - opendbteam.com This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. On a fresh install of Red Hat 6.4 using munin 2.0.12-2 or 2.0.14-1 from EPEL Testing still getting the same errors on root mail except line number changed to 866 from 950. I don't have an "_" on my hostname. I believe the cause is munin-node is not running. I did the following which fixed the error message. $ rm -f /var/lib/munin/{datafile,datafile.storable,htmlconf.storable} $ /etc/init.d/munin-node start Not sure if the first command is necessary. After running once, stoping munin-node did not cause any further error messages. Hello, And in my case (munin-2.0.14-1.el6.noarch via EPEL) on CentOS 6.0, the origin of the problem was "_" (underscore) in the hostname. After change hostname, munin work. We have no "_" in the hostname, but a "-" in the domainname. I don't know if this the origin of the problem, but we cannot change the domainname. "-" is a common character in domainnames (and hostnames), so this must not be a problem. If you are still having this problem, can you try something? in /etc/munin/munin-node.conf, can you add a line like "host_name localhost" (with the name you have in /etc/hosts if the node is not on the master) Here the bug still appears on a fresh F19 install on a system with a dash in the hostname and "host_name whatever" in munin-node.conf does not seem to help. Is there an upstream bug report about this? Worked around with http://munin-monitoring.org/ticket/1285 by hardcoding localhost. munin 2.0.16-1 on centos 5.9 I had a same problem related to address 127.0.0.1 in my host tree in my /etc/munin/munin.conf. I switched first two lines in /etc/hosts, so 127.0.0.1 is now before ::1, and it solved my issue. (CentOS 6.5) This message is a reminder that Fedora 18 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 18. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '18'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 18's end of life. Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 18 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 18's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Do not close this bug just because it was created for Fedora 18, I got this problem (emails every 5 minutes with "not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866") on a freshly installed Fedora 20 system. Munin is installed but not configured by me in any way. There is no non-alphabet characters in the hostname. $ ls -ld /var/lib/munin /var/lib/munin/plugin-state /var/lib/munin/plugin-state/nobody drwxr-xr-x. 4 munin munin 4096 des. 31 12:30 /var/lib/munin drwxrwxr-x. 5 root munin 4096 des. 8 08:28 /var/lib/munin/plugin-state drwxr-xr-x. 2 nobody root 4096 des. 29 19:09 /var/lib/munin/plugin-state/nobody You have new mail in /home/hlovdal/Maildir $ rpm -qa munin\* munin-common-2.0.19-1.fc20.noarch munin-node-2.0.19-1.fc20.noarch munin-2.0.19-1.fc20.noarch $ What do you have in your config file for host_name ? If you look in your /etc/hosts file, do you see an exact match for host_name (above) ? After installing, set host_name to something found in /etc/hosts and restart munin-node. tell me if the problem continues. so far, every case i've seen is resolved by this, including fresh installs. The dbus / NM / systemd magic isnt working for hostname, so setting it to $HOSTNAME results in the error you see. Using something listed in /etc/hosts ensures you do not see this message. Thanks for your answer. munin-2.0.19-1.fc20.noarch still sends an error mail to root every five minutes: Subject: Cron <munin@rauhfussbussard> test -x /usr/bin/munin-cron && /usr/bin/munin-cron Mail body: not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866. I have not modified any of the munin files. I have just installed the packages. The answer from "hostname" and from "uname -n" does not match any entry in /etc/hosts. I don't want to enter the hostname in /etc/hosts because we use dns and nis or ldap. The (default) two lines for localhost are the only ones which are in every /etc/hosts. Doesn't munin check the hostname depending on /etc/nsswitch.conf (e.g. use system functions to resolve the hostname)??? Why do you think that the hostname must match an entry in /etc/hosts? I think munin should do NOTHING if I just install the packages but not configure or enable it. It SHOULD NOT even run cron jobs every five minutes! I just want to install the same list of packages on all computers independed of using it on a host. If I want to use it, I have to configure and enable it. Either the cron job should not be installed, or the cron job script of munin should check something (e.g. a variable in a config file) if munin is enabled or not, und do more work only if it es enabled. I currently don't have munin running, but I want to configure it for some hosts some time. At the moment I don't have the time to configure munin (maybe probably in the next weeks or months), but I think if enabled it should do some default jobs for the local host (using the hostname returned from "hostname" or "uname -n", resolved using the getent* or similar functions). > The answer from "hostname" and from "uname -n" does not match any entry in /etc/hosts. ^^ That is why you get the warning every 5mins. See comment 19 for why. This error is not a munin bug at all. This is related to the way your NSS config is setup. It is an error if a package needs manual intervention to prevent flooding the mailbox with error mails. Package installation must run unattended and work error free out of the box. It is an error if a package depends on an matching entry in /etc/hosts when nearly all other packages have no problem with his configuration. It seems to me that munin uses an improper methode. It must not crash because an entry in /etc/hosts is missing. It is an error if perl crashes. The error message is a perl error message in a munin perl program - it is obvious that this is an error of munin. A program must not crash because of predictable situations - it has to handle this situation! For munin: "not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866." is a perl error message in a perl program of munin. So this is a munin error. I can understand that munin does not crash when there is a matching entry in /etc/hosts. But it is not my fault that munin crashes when this condition is not fulfilled - it's the problem of munin! Don't give me the fault of munin's program code! (In reply to d. johnson from comment #21) > See comment 19 for why. This error is not a munin bug at all. This is > related to the way your NSS config is setup. You require that users / administrators should add the hostname (contained in /etc/hostname) to /etc/hosts. But his is a very bad requirement because it may break networking in some configurations / scenarios. Consider the usual case where the network address is assiged dynamically using dhcp. /etc/hosts requires an ip address and a hostname. But with dhcp the ip address is not fixed - it may change on each boot. You cannot enter the right ip address to /etc/hosts - it may not match the assigned ip address of the network interface. Setting network configuration to static assignment of ip address and other network parameters would solve this problem - but this would require that munin mist only be installed on hosts with static address assignment? This could not be a requirement that you _really_ think of! Please note that the default of Fedora is that /etc/hosts only contains two entries for localhost (one for ipv4, one for ipv6), but not an entry for the hostname. An entry for the hostname was used long time ago, I am not sure, it may be until Fedora 10 oder Fedora 12. Then Fedora has changed the installation to use full qualified hostnames (hostnames with dns domain) and to not add it to /etc/hosts by default. Please note another problem: Suppose /etc/hostname contains "myhost.example.com". Think of a scenario where the assigned ip address is changed in the dns server. If the hostname would be in /etc/hosts too, then the host may use a wrong ip address if /etc/hosts is not changed at the same time (there may be different administrators) even if the host uses dhcp. But dhcp and dns was designed for central managing of network addresses, and to avoid that each host must be changed by hand too. Munin must be configured to not create any errors after a package installation or update. Munin must be compatible with the defaults of Fedora! It is not true that Fedora must be changed to satisfy the requirements of munin. The munin package is in an inacceptable state because the program crashes and floods the mailbox of the administrator. Nobody can install the munin package without doing something to avoid the error mails - otherwise the mailbox or the partition containing it may be filled over time until all space is exhausted. And most of them have to search for instructions how to do it... - and they will be annoyed about this package that makes trouble. Not every user and administrator writes a bug report... This is a bug in the way your system is configured for NSS. It is not responding correctly to the hostname queries. (In reply to d. johnson from comment #24) > This is a bug in the way your system is configured for NSS. It is not > responding correctly to the hostname queries. Can you please specify what is wrong? It seems to me that you say that Fedora's defaults are wrong. -Doctor, my arm hurts when I do this. -Well don't do that then! Till, That is exactly what I have said. There is a service that is *supposed* to handle hostname requests, as part of systemd, that fails to respond properly. You could write a simple testcase based on the comments above or the 3 lines of perl that exemplify this issue. It is unique to this distro. This problem does not occur on other distros. Arch, Debian, Ubuntu, Gentoo to name a few. You can see this issue by adding $HOSTNAME to /etc/hosts which circumvents the NSS problem entirely by making a manual entry. Not ideal, but does work, everytime. I have checked the files of munin-2.0.19-1.fc20.noarch and found the following: 1. /etc/munin/munin.conf contains the lines (unmodified from the package): [localhost] address 127.0.0.1 use_node_name yes There are matching lines in /etc/hosts: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 It seems that your requirement are fulfilled but the error occures. 2. I tried a modified file /etc/munin/munin.conf: I added a domain to the hostname: [localhost.localdomain] address 127.0.0.1 use_node_name yes This hostname is also matched in /etc/hosts. In this case the error occures too. 3. It seems to me that the default configuration provided by the installation package does not use the hostname. (???) 4. I run each of the commands in /etc/cron.d/munin and found that the error is caused by /usr/share/munin/munin-html . 5. In /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm function get_config (line 153) returns an undefined value ("undef"). The reason is that line 165 "$htmlconfig = generate_config($use_cache);" returns an undefined value ("undef"). 6. In /usr/share/perl5/vendor_perl/Munin/Master/HTMLConfig.pm function generate_config (line 34) returns an undefined value because in line 60 "$cache = get_group_tree($config);" returns an undefined value. 7. In /usr/share/perl5/vendor_perl/Munin/Master/HTMLConfig.pm function get_group_tree (line 137) returns a null value because in line 191 "push( @$groups, grep {defined $_} get_group_tree($child, $base.munin_get_node_name($child) . "/"));" the @$groups array remain empty. 8. I am not sure if functions munin_get_sorted_children (/usr/share/perl5/vendor_perl/Munin/Master/HTMLConfig.pm line 381) and munin_get_children (/usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 481) work right. They modify the data structure (the hash in the array) so that I am not sure if the references are correct. 9. In /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm I am wondering about the code in function munin_parse_config (line 310) for parsing a hostname (lines 350-352). For example, for an entry "[localhost]" in /etc/munin/munin.conf it will create a $prefix value of "localhost;localhost:". For "[localhost.localdomain]" it will create "localdomain;localhost.localdomain:". For "[myhost.example.com]" it will create "example.com;myhost.example.com:". This values will be used to create a hierarchical structure in the hash (variable $config in some parts of the code). I am wondering about the structure, for example about the localhost - localhost hierarchy (and in general all hostnamewithoutdomain - hostnamewithoutdomain combination). Until now I have not found the excact part of the perl code which causes the problem and how it can be fixed. But until now I have not found system calls that causes the error. For localhost there are matching entries in /etc/hosts. Maybe someone can found the problematic code in munin using the list of problematic code above? I changed the summary to match the error message of Fedora 20. Edgar, wrong section. in /etc/munin/munin-node.conf, set host_name to something found in /etc/hosts and restart munin-node. (In reply to d. johnson from comment #30) > Edgar, wrong section. Please tell us the reason what is wrong with my statements. Please don't repeat always the same statement without a comprehensible reason. > in /etc/munin/munin-node.conf, set host_name to something found in > /etc/hosts and restart munin-node. I have tried both "host_name localhost" and "host_name localhost.localdomain" in the file /etc/munin/munin-node.conf in the line after "#host_name localhost.localdomain". But the error message still occured. A call of (as root) "runuser -u munin -- perl /usr/share/munin/munin-html" prints to the shell: "not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866." I still cannot see what the host_name entry has to do with the problems listed in comment #28. I found three open munin tickets at the munin website that seems to describe the same or at least a similar problem - not on Fedora, but Debian. http://munin-monitoring.org/ticket/1257 http://munin-monitoring.org/ticket/1219 http://munin-monitoring.org/ticket/1154 re/ #c31 - Did you restart munin-node ? (If you did not restart it, then the change will not take effect.) The debian bug reports you pointed to are quite outdated with no config details. See also, "We should have a better error message, but since the workaround is just to wait, i'm pushing it to 3.0, and lower its priority." As for your specifics Edgar, Please run "rpm -Va munin\*" and post the output along with your config files. I have tested the defaults *many* times with only changing the host_name parameter and nothing else, with 100% success. (In reply to d. johnson from comment #33) > re/ #c31 - Did you restart munin-node ? (If you did not restart it, then > the change will not take effect.) How do I restart munin-node? It was not running at any time before. I just installed the package (the same list of packages on all our hosts) - so I _can_ configure and run it on _some_ of our host! # systemctl status munin-node munin-node.service - Munin Node Server. Loaded: loaded (/usr/lib/systemd/system/munin-node.service; disabled) Active: inactive (dead) Docs: man:munin-node The only process that was running was /usr/bin/munin-cron. I have disabled the munin cron job (using comment sign in /etc/cron.d/munin) to avoid the every five minutes mail during the testing period. > The debian bug reports you pointed to are quite outdated with no config > details. See also, "We should have a better error message, but since the > workaround is just to wait, i'm pushing it to 3.0, and lower its priority." Yes, they are a bit outdated, but there is no notice that the problem described there is solved. And our error message is similar - not excact the same - but of course, it _may_ be solved in the meentime but not noted in the tickets. > As for your specifics Edgar, > > Please run "rpm -Va munin\*" and post the output along with your config > files. [root@rauhfussbussard ~]# rpm -Va munin\* S.5....T. c /etc/munin/munin-node.conf S.5....T. c /etc/cron.d/munin .......T. /var/www/html/munin/static/definitions.html .......T. /var/www/html/munin/static/dynazoom.html .......T. /var/www/html/munin/static/favicon.ico .......T. /var/www/html/munin/static/formatdate.js .......T. /var/www/html/munin/static/logo-h.png .......T. /var/www/html/munin/static/logo.png .......T. /var/www/html/munin/static/querystring.js .......T. /var/www/html/munin/static/style-1.2.css .......T. /var/www/html/munin/static/style-new.css .......T. /var/www/html/munin/static/style.css .......T. /var/www/html/munin/static/zoom.js The modification of /etc/munin/munin-node.conf is the following (for testing): # diff /etc/munin/munin-node.conf.old /etc/munin/munin-node.conf 37a38 > hostname localhost.localdomain The only change in /etc/cron.d/munin is the comment sign: #*/5 * * * * munin test -x /usr/bin/munin-cron && /usr/bin/munin-cron The other files are unmodified, only in the state that the package installation leaves. You will need munin-node active and running for it to work. Example: $ systemctl status munin-node munin-node.service - Munin Node Server. Loaded: loaded (/usr/lib/systemd/system/munin-node.service; enabled) Active: active (running) (In reply to d. johnson from comment #35) > You will need munin-node active and running for it to work. > > Example: > > $ systemctl status munin-node > munin-node.service - Munin Node Server. > Loaded: loaded (/usr/lib/systemd/system/munin-node.service; enabled) > Active: active (running) You know that I just want to install the package, not to run the software (at least not on all host, only on some of them)? Maybe that the error does not occur when munin-node is running. But this is not the point. The problem is that the package sends error mails when the package is installed. This is a behaviour that a new installed package must not do! But behind of that: I think it is still an error if the code does not handle null and undefined values in a proper manner (e.g., no error even if no data is available...). Have any of you tried setting selinux to permissive mode and then running munin-cron? I had the same problem with reference error from munin-cron and I tried permissive mode temporarily and everything worked. I've already had to write my own policy module to get munin-node working with selinux and it's clear to me now that the policy included in the EPEL package does not work. I'm no pro at selinux but if I get my policy working I'll post it here so people who are better than me can scrutinize it. Stefan, if you attach the AVC's you are seeing, the correct labels can be applied and the policy can be adjusted to fix whatever is wrong with it. The policy in EPEL should just work if things are labeled correctly. Because this bug report does not relate to your AVC issue, you may be asked to open another ticket under the selinux-policy component. munin-node-2.0.19-1.fc20.noarch: 2014/03/13 10:26:02 [PERL WARNING] data is null at at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 863. Munin::Master::Utils::munin_write_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 877 Munin::Master::Utils::munin_writeconfig_storable('/var/lib/munin/htmlconf.storable', undef) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 182 Munin::Master::HTMLOld::get_config(0) called at /usr/share/perl5/vendor_perl/Munin/Master/HTMLOld.pm line 192 Munin::Master::HTMLOld::html_main() called at /usr/share/munin/munin-html line 40 Happens since i upgraded the system from fc19 with A WORKING munin setup. munin.config on the system has not changed from fc19 to fc20 . SELINUX is not involved here at all. Bug identified: get_group_tree(0) shall return the htmlconfig , but it doesn't and it can't: "HTMLConfig.pm" line 203: return unless $visible; The callerfunction expects it to return something: "HTMLOld.pm" [readonly] line 165: $htmlconfig = generate_config($use_cache); ... munin_writeconfig_storable($cachefile, $htmlconfig); with no config in $htmlconfig, writeconfig fails. this happens, because there are no "children" to loop over, so $visible never gets set and the function has no chance to return usefull data. I checked the fc19 mainserver with generates our pages and $visible was set there to "1" sometimes. Attention: It is possible, that the fc20 client on our server should not generate htmlpages, because our main system does that. BUT, it should not crash with a stacktrace at any time. A crash always means, there something broken. customercare@resellerdesktop, in /etc/munin/munin-node.conf, set host_name to something found in /etc/hosts and restart munin-node. The bug was not caused by a wrong hostname. Explanation: /etc/munin/plugins/ contains normaly symlinks to "/usr/share/munin/plugins/pluginname" As munin was upgraded via yum in 2012 ( yes we didn't notice it, for quite some time :) ) the plugins directorystructure was switched to those symlinks. But that did not work on all active installations. Some never got the symlinks. The bug happens, when there are no children to work with ( see comment #40 ). Is can be caused by a wrong hostname in munin.conf, but it also happens, when there are no services in the plugins directory. FIX: fill your plugins directory with these symlinks : lrwxrwxrwx 1 root root 28 4. Feb 2013 cpu -> /usr/share/munin/plugins/cpu lrwxrwxrwx 1 root root 27 4. Feb 2013 df -> /usr/share/munin/plugins/df lrwxrwxrwx 1 root root 33 4. Feb 2013 df_inode -> /usr/share/munin/plugins/df_inode ...etc..etc.. lrwxrwxrwx 1 root root 31 4. Feb 2013 vmstat -> /usr/share/munin/plugins/vmstat Restart munin-node and bug is gone. ATTENTION: no plugins = no services : leads to a complete munin failure, because none of the tools produces anything of value. This is a resonable bevavior if your not telling munin what to test, obviosly. This should be fixed by a test, if there are any plugins enabled! and a correct intuitive error message, because it does not make ANY SENSE to have no services at all :) It would be nice, if this feature request makes it to the dev of munin. Can we close this now ? (In reply to customercare from comment #43) > Can we close this now ? No - because the problem still exists. Some minutes ago I have installed munin ("yum install munin") on a Fedora 19 and a Fedora 20 machine to see what happens. Now, on both machines, root gets e-mails everv five minutes from cron with the following content: not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866. Please note that after installation /etc/munin/plugins contains symbolic links as suggested in comment #42: /etc/munin/plugins: insgesamt 0 lrwxrwxrwx. 1 root root 28 25. Jul 12:43 cpu -> /usr/share/munin/plugins/cpu lrwxrwxrwx. 1 root root 27 25. Jul 12:43 df -> /usr/share/munin/plugins/df lrwxrwxrwx. 1 root root 33 25. Jul 12:43 df_inode -> /usr/share/munin/plugins/df_inode lrwxrwxrwx. 1 root root 34 25. Jul 12:43 diskstats -> /usr/share/munin/plugins/diskstats lrwxrwxrwx. 1 root root 32 25. Jul 12:43 entropy -> /usr/share/munin/plugins/entropy lrwxrwxrwx. 1 root root 39 25. Jul 12:43 exim_mailqueue -> /usr/share/munin/plugins/exim_mailqueue lrwxrwxrwx. 1 root root 30 25. Jul 12:43 forks -> /usr/share/munin/plugins/forks lrwxrwxrwx. 1 root root 37 25. Jul 12:43 fw_conntrack -> /usr/share/munin/plugins/fw_conntrack lrwxrwxrwx. 1 root root 43 25. Jul 12:43 fw_forwarded_local -> /usr/share/munin/plugins/fw_forwarded_local lrwxrwxrwx. 1 root root 35 25. Jul 12:43 fw_packets -> /usr/share/munin/plugins/fw_packets lrwxrwxrwx. 1 root root 41 25. Jul 12:43 hddtemp_smartctl -> /usr/share/munin/plugins/hddtemp_smartctl lrwxrwxrwx. 1 root root 28 25. Jul 12:43 if_brnetboot -> /usr/share/munin/plugins/if_ lrwxrwxrwx. 1 root root 32 25. Jul 12:43 if_err_brnetboot -> /usr/share/munin/plugins/if_err_ lrwxrwxrwx. 1 root root 32 25. Jul 12:43 if_err_netboot -> /usr/share/munin/plugins/if_err_ lrwxrwxrwx. 1 root root 32 25. Jul 12:43 if_err_virbr0 -> /usr/share/munin/plugins/if_err_ lrwxrwxrwx. 1 root root 32 25. Jul 12:43 if_err_virbr0-nic -> /usr/share/munin/plugins/if_err_ lrwxrwxrwx. 1 root root 28 25. Jul 12:43 if_netboot -> /usr/share/munin/plugins/if_ lrwxrwxrwx. 1 root root 28 25. Jul 12:43 if_virbr0 -> /usr/share/munin/plugins/if_ lrwxrwxrwx. 1 root root 28 25. Jul 12:43 if_virbr0-nic -> /usr/share/munin/plugins/if_ lrwxrwxrwx. 1 root root 35 25. Jul 12:43 interrupts -> /usr/share/munin/plugins/interrupts lrwxrwxrwx. 1 root root 33 25. Jul 12:43 irqstats -> /usr/share/munin/plugins/irqstats lrwxrwxrwx. 1 root root 29 25. Jul 12:43 load -> /usr/share/munin/plugins/load lrwxrwxrwx. 1 root root 31 25. Jul 12:43 lpstat -> /usr/share/munin/plugins/lpstat lrwxrwxrwx. 1 root root 31 25. Jul 12:43 memory -> /usr/share/munin/plugins/memory lrwxrwxrwx. 1 root root 32 25. Jul 12:43 netstat -> /usr/share/munin/plugins/netstat lrwxrwxrwx. 1 root root 36 25. Jul 12:43 nfs4_client -> /usr/share/munin/plugins/nfs4_client lrwxrwxrwx. 1 root root 35 25. Jul 12:43 nfs_client -> /usr/share/munin/plugins/nfs_client lrwxrwxrwx. 1 root root 29 25. Jul 12:43 nfsd -> /usr/share/munin/plugins/nfsd lrwxrwxrwx. 1 root root 30 25. Jul 12:43 nfsd4 -> /usr/share/munin/plugins/nfsd4 lrwxrwxrwx. 1 root root 35 25. Jul 12:43 open_files -> /usr/share/munin/plugins/open_files lrwxrwxrwx. 1 root root 36 25. Jul 12:43 open_inodes -> /usr/share/munin/plugins/open_inodes lrwxrwxrwx. 1 root root 42 25. Jul 12:43 postfix_mailqueue -> /usr/share/munin/plugins/postfix_mailqueue lrwxrwxrwx. 1 root root 43 25. Jul 12:43 postfix_mailvolume -> /usr/share/munin/plugins/postfix_mailvolume lrwxrwxrwx. 1 root root 34 25. Jul 12:43 processes -> /usr/share/munin/plugins/processes lrwxrwxrwx. 1 root root 33 25. Jul 12:43 proc_pri -> /usr/share/munin/plugins/proc_pri lrwxrwxrwx. 1 root root 43 25. Jul 12:43 sendmail_mailqueue -> /usr/share/munin/plugins/sendmail_mailqueue lrwxrwxrwx. 1 root root 43 25. Jul 12:43 sendmail_mailstats -> /usr/share/munin/plugins/sendmail_mailstats lrwxrwxrwx. 1 root root 45 25. Jul 12:43 sendmail_mailtraffic -> /usr/share/munin/plugins/sendmail_mailtraffic lrwxrwxrwx. 1 root root 29 25. Jul 12:43 swap -> /usr/share/munin/plugins/swap lrwxrwxrwx. 1 root root 32 25. Jul 12:43 threads -> /usr/share/munin/plugins/threads lrwxrwxrwx. 1 root root 31 25. Jul 12:43 uptime -> /usr/share/munin/plugins/uptime lrwxrwxrwx. 1 root root 30 25. Jul 12:43 users -> /usr/share/munin/plugins/users lrwxrwxrwx. 1 root root 31 25. Jul 12:43 vmstat -> /usr/share/munin/plugins/vmstat I did nothing but installing munin with yum. It is not acceptable that just installing a package floats a mail account with error mails every five minutes! Please solve it in a way that after installation munin is inactive and produces no error mails and doesn't fill the log files or other files if it was not manually activated by the administrator of the machine. Here the current package versions: munin-node-2.0.21-1.fc19.noarch munin-common-2.0.21-1.fc19.noarch munin-2.0.21-1.fc19.noarch munin-common-2.0.21-1.fc20.noarch munin-node-2.0.21-1.fc20.noarch munin-2.0.21-1.fc20.noarch Edgar, Your issue will vanish if you set the hostname in the config file. See #c11 for example. (In reply to d. johnson from comment #46) > Edgar, Your issue will vanish if you set the hostname in the config file. > See #c11 for example. No. I told you why I couldn't do that. See comments #22 and #23 and the others. For short: - Package installation must not require manual intervention to prevent the package from doing something bad, like flooding the mailbox with error messages. It should work with the files provided by the distribution without errors. It is ok if it does nothing if it is not configured and activated manually, but simple package installation must leave the system in a save state. - Each package must be installable in a enterprise environment. For example, it should work with dns, ldap, nis as nameservices. This requires that it must not be neccessary to enter a hostname to /etc/hosts other than what is done while system installation, e.g. localhost in case of Fedora 19 and 20. - A package should handle the case that hostname is not manually set in the config file in another way than flooding the system mailbox with error mails. In your case, you didn't handle the case at all because perl prints an internal error ("not a reference"...). Your code should check if a value is a reference before using it as reference... (I had not looked at your code, but the error message supposes this cause.) Please, can you modify munin packages so that no manual intervention is required after package installation to prevent munin from doing something bad. Bug reporter: The problem still exists, just installing the rpm triggers it. Bug assignee: Well don't just install the rpm. Take some additional, (non-obivious) action. Patient: My arm hurts when I do this. Doctor: Well don't do that then! Can you please explain how you think these two dialouges are different? I started to pull my hair out over this too after suddenly getting the dreaded error filling up my mailbox a few days ago: not a reference at /usr/share/perl5/vendor_perl/Munin/Master/Utils.pm line 866. Nothing above worked, and there was nothing particularly useful in the munin logs. Let's try running a plugin: # munin-run --debug load # Processing plugin configuration from /etc/munin/plugin-conf.d/df # Processing plugin configuration from /etc/munin/plugin-conf.d/fw_ # Processing plugin configuration from /etc/munin/plugin-conf.d/hddtemp_smartctl # Processing plugin configuration from /etc/munin/plugin-conf.d/munin-node # Processing plugin configuration from /etc/munin/plugin-conf.d/postfix # Processing plugin configuration from /etc/munin/plugin-conf.d/sendmail # Setting /rgid/ruid/ to /99/99/ # Setting /egid/euid/ to /99 99/99/ # Setting up environment Segmentation fault Other plugins also faulted. /var/log/audit/audit.log had lots of these lines: type=ANOM_ABEND msg=audit(1409068502.721:235158): auid=4294967295 uid=0 gid=99 ses=4294967295 pid=23138 comm="/usr/sbin/munin" exe="/usr/bin/perl" sig=11 Then tried this: # rpm -Va perl* prelink: /usr/lib64/libperl.so.5.18.2: prelinked file was modified S.?...... /usr/lib64/libperl.so.5.18.2 Then this: # prelink /usr/lib64/libperl.so.5.18.2 # rpm -Va perl* prelink: /usr/lib64/libperl.so.5.18.2: prelinked file was modified S.?...... /usr/lib64/libperl.so.5.18.2 Still wouldn't prelink. So tried all: # prelink -av -mR Running kernel was kernel-3.15.9-200.fc20.x86_64 Had installed kernel-3.15.10-200.fc20.x86_64 the day before the errors started, but hadn't booted into it yet. Rebooted and plugins are working again, but I can't be sure exactly why munin-node was segfaulting. The current version should set the default hostname and avoid this issue. |