RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1843170 - grafana may not start due to permission issues
Summary: grafana may not start due to permission issues
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: grafana
Version: 8.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.3
Assignee: Andreas Gerstmayr
QA Contact: Jan Kurik
URL:
Whiteboard:
: 1805472 (view as bug list)
Depends On: 1850471
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-02 18:07 UTC by Jan Kurik
Modified: 2021-09-17 12:44 UTC (History)
5 users (show)

Fixed In Version: grafana-7.3.4-1.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:51:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:1859 0 None None None 2021-05-18 15:51:33 UTC

Description Jan Kurik 2020-06-02 18:07:03 UTC
Description of problem:
Grafana does not start and reports several permissions issues.

Version-Release number of selected component (if applicable):
grafana-6.7.3-1.el8

How reproducible:
Always

Steps to Reproduce:
1. On a fresh RHEL-8.3 compose install grafana server
# yum install -y grafana
2. Set the admin password from command line before the very first start of grafana server
# grafana-cli admin reset-admin-password superSeCrEtPsWD
3. Start the grafana-server servis
# systemctl start grafana-server

Actual results:
The output of the "systemctl start grafana-server" is as follows:

# systemctl restart grafana-server
Job for grafana-server.service failed because the control process exited with error code.
See "systemctl status grafana-server.service" and "journalctl -xe" for details.
# systemctl status -l grafana-server.service
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-06-02 13:29:50 EDT; 39s ago
Docs: http://docs.grafana.org
Process: 35021 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (code=exited, status=1/FAILURE)
Main PID: 35021 (code=exited, status=1/FAILURE)

Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Stopped Grafanainstance.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Start request repeated too quickly.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jun 02 13:29:50 ci-vm-10-0-137-55.hosted.upshift.rdu2.redhat.com systemd[1]: Failed to startGrafana instance.

Expected results:
Grafana server starts without any issue

Additional info:
Digging a bit deeper and starting the Grafana manually (instead of using systemd) it reviles issues with permissions of several directories:

# . /etc/sysconfig/grafana-server
# cd ${GRAFANA_HOME}
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -

Failed to start grafana. error: failed to initialize file handler: open /var/log/grafana/grafana.log: permission denied
# chown grafana:grafana /var/log/grafana/grafana.log
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:37:14] Failed to verify pid directory           logger=server error="mkdir /var/run/grafana: permission denied"
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# su -c "/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}" -s /bin/bash grafana -
...
EROR[06-02|13:12:16] Problem reading image dir                logger=cleanup error="open /var/lib/grafana/png: permission denied"
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png


To sum up: running the following set of commands before the very first run of grafana-server fixes the issue. IMO these files/directories need to have the correct permissions and ownership before grafana-server starts - as such it should be part of grafana SPEC file (%post):
# chown grafana:grafana /var/log/grafana/grafana.log
# mkdir /var/run/grafana
# chown grafana:grafana /var/run/grafana
# mkdir /var/lib/grafana/png
# chown grafana:grafana /var/lib/grafana/png

Comment 1 Jan Kurik 2020-06-03 04:07:04 UTC
One more note related to the ownership of "/var/log/grafana/grafana.log" file: Following the reproduction scenario above the file is owned by root:root. However if "grafana-server" is started before "grafana-cli" is used then the ownership is grafana:grafana.

Comment 2 Andreas Gerstmayr 2020-06-04 15:03:39 UTC
opened upstream bug https://github.com/grafana/grafana/issues/25367 with a list of possible solutions

Comment 3 Andreas Gerstmayr 2020-06-05 18:43:38 UTC
Technically this bug is related to #1805472
As this bug is also present in upstream packaging, I've opened https://github.com/grafana/grafana/issues/25367 and will wait how they'll resolve it.

It only occurs in a very specific situation:
- grafana-cli is run as root *before* the first start of grafana-server

I'd say most people are not aware of grafana-cli anyway, and if they want to change the admin's password, they can also do it in the web interface.

If upstream comes up with a solution in time for RHEL 8.3 I'll create a -2 build of 6.7.4.

Comment 4 Andreas Gerstmayr 2020-06-10 10:50:00 UTC
Looks like there won't be a response from upstream in time for RHEL 8.3, moving to 8.4 now.

Comment 5 Nathan Scott 2020-06-23 04:24:35 UTC
*** Bug 1805472 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2021-05-18 15:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: grafana security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1859


Note You need to log in before you can comment on or make changes to this bug.