Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Grafana does not start and reports several permissions issues.
Version-Release number of selected component (if applicable):
grafana-6.7.3-1.el8
How reproducible:
Always
Steps to Reproduce:
1. On a fresh RHEL-8.3 compose install grafana server
# yum install -y grafana
2. Set the admin password from command line before the very first start of grafana server
# grafana-cli admin reset-admin-password superSeCrEtPsWD
3. Start the grafana-server servis
# systemctl start grafana-server
Actual results:
The output of the "systemctl start grafana-server" is as follows:
# systemctl restart grafana-server
Job for grafana-server.service failed because the control process exited with error code.
See "systemctl status grafana-server.service" and "journalctl -xe" for details.
# systemctl status -l grafana-server.service
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-06-02 13:29:50 EDT; 39s ago
Docs: http://docs.grafana.org
Process: 35021 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (code=exited, status=1/FAILURE)
Main PID: 35021 (code=exited, status=1/FAILURE)
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Stopped Grafanainstance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Start request repeated too quickly.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.
Expected results:
Grafana server starts without any issue
Additional info:
Digging a bit deeper and starting the Grafana manually (instead of using systemd) it reviles issues with permissions of several directories:
# . /etc/sysconfig/grafana-server
# cd ${GRAFANA_HOME}
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
Failed to start grafana. error: failed to initialize file handler: open /var/log/grafana/grafana.log: permission denied
# chown grafana:grafana /var/log/grafana/grafana.log
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:37:14] Failed to verify pid directory logger=server error="mkdir /var/run/grafana: permission denied"
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:12:16] Problem reading image dir logger=cleanup error="open /var/lib/grafana/png: permission denied"
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png
To sum up: running the following set of commands before the very first run of grafana-server fixes the issue. IMO these files/directories need to have the correct permissions and ownership before grafana-server starts - as such it should be part of grafana SPEC file (%post):
# chown grafana:grafana /var/log/grafana/grafana.log
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png
One more note related to the ownership of "/var/log/grafana/grafana.log" file: Following the reproduction scenario above the file is owned by root:root. However if "grafana-server" is started before "grafana-cli" is used then the ownership is grafana:grafana.
Comment 2Andreas Gerstmayr
2020-06-04 15:03:39 UTC
Comment 3Andreas Gerstmayr
2020-06-05 18:43:38 UTC
Technically this bug is related to #1805472
As this bug is also present in upstream packaging, I've opened https://github.com/grafana/grafana/issues/25367 and will wait how they'll resolve it.
It only occurs in a very specific situation:
- grafana-cli is run as root *before* the first start of grafana-server
I'd say most people are not aware of grafana-cli anyway, and if they want to change the admin's password, they can also do it in the web interface.
If upstream comes up with a solution in time for RHEL 8.3 I'll create a -2 build of 6.7.4.
Comment 4Andreas Gerstmayr
2020-06-10 10:50:00 UTC
Looks like there won't be a response from upstream in time for RHEL 8.3, moving to 8.4 now.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: grafana security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2021:1859