Bug 1843170

Summary: grafana may not start due to permission issues
Product: Red Hat Enterprise Linux 8 Reporter: Jan Kurik <jkurik>
Component: grafanaAssignee: Andreas Gerstmayr <agerstmayr>
Status: CLOSED ERRATA QA Contact: Jan Kurik <jkurik>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.3CC: agerstmayr, jkurik, mgoodwin, nathans, tis
Target Milestone: rcKeywords: Bugfix, Triaged
Target Release: 8.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: grafana-7.3.4-1.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1850471    
Bug Blocks:    

Description Jan Kurik 2020-06-02 18:07:03 UTC
Description of problem:
Grafana does not start and reports several permissions issues.

Version-Release number of selected component (if applicable):
grafana-6.7.3-1.el8

How reproducible:
Always

Steps to Reproduce:
1. On a fresh RHEL-8.3 compose install grafana server
# yum install -y grafana
2. Set the admin password from command line before the very first start of grafana server
# grafana-cli admin reset-admin-password superSeCrEtPsWD
3. Start the grafana-server servis
# systemctl start grafana-server

Actual results:
The output of the "systemctl start grafana-server" is as follows:

# systemctl restart grafana-server
Job for grafana-server.service failed because the control process exited with error code.
See "systemctl status grafana-server.service" and "journalctl -xe" for details.
# systemctl status -l grafana-server.service
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-06-02 13:29:50 EDT; 39s ago
Docs: http://docs.grafana.org
Process: 35021 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (code=exited, status=1/FAILURE)
Main PID: 35021 (code=exited, status=1/FAILURE)

Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Stopped Grafanainstance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Start request repeated too quickly.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.

Expected results:
Grafana server starts without any issue

Additional info:
Digging a bit deeper and starting the Grafana manually (instead of using systemd) it reviles issues with permissions of several directories:

# . /etc/sysconfig/grafana-server
# cd ${GRAFANA_HOME}
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -

Failed to start grafana. error: failed to initialize file handler: open /var/log/grafana/grafana.log: permission denied
# chown grafana:grafana /var/log/grafana/grafana.log
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:37:14] Failed to verify pid directory           logger=server error="mkdir /var/run/grafana: permission denied"
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:12:16] Problem reading image dir                logger=cleanup error="open /var/lib/grafana/png: permission denied"
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png


To sum up: running the following set of commands before the very first run of grafana-server fixes the issue. IMO these files/directories need to have the correct permissions and ownership before grafana-server starts - as such it should be part of grafana SPEC file (%post):
# chown grafana:grafana /var/log/grafana/grafana.log
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png

Comment 1 Jan Kurik 2020-06-03 04:07:04 UTC
One more note related to the ownership of "/var/log/grafana/grafana.log" file: Following the reproduction scenario above the file is owned by root:root. However if "grafana-server" is started before "grafana-cli" is used then the ownership is grafana:grafana.

Comment 2 Andreas Gerstmayr 2020-06-04 15:03:39 UTC
opened upstream bug https://github.com/grafana/grafana/issues/25367 with a list of possible solutions

Comment 3 Andreas Gerstmayr 2020-06-05 18:43:38 UTC
Technically this bug is related to #1805472
As this bug is also present in upstream packaging, I've opened https://github.com/grafana/grafana/issues/25367 and will wait how they'll resolve it.

It only occurs in a very specific situation:
- grafana-cli is run as root *before* the first start of grafana-server

I'd say most people are not aware of grafana-cli anyway, and if they want to change the admin's password, they can also do it in the web interface.

If upstream comes up with a solution in time for RHEL 8.3 I'll create a -2 build of 6.7.4.

Comment 4 Andreas Gerstmayr 2020-06-10 10:50:00 UTC
Looks like there won't be a response from upstream in time for RHEL 8.3, moving to 8.4 now.

Comment 5 Nathan Scott 2020-06-23 04:24:35 UTC
*** Bug 1805472 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2021-05-18 15:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: grafana security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1859