RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1805472 - grafana-cli is causing grafana-server logins to fail when used before grafana.db is created
Summary: grafana-cli is causing grafana-server logins to fail when used before grafana...
Keywords:
Status: CLOSED DUPLICATE of bug 1843170
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: grafana
Version: 8.2
Hardware: All
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.3
Assignee: Andreas Gerstmayr
QA Contact: Jan Kurik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-20 20:44 UTC by Jan Kurik
Modified: 2021-09-17 12:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-23 04:24:35 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
grafana.log (4.56 KB, text/plain)
2020-02-20 20:44 UTC, Jan Kurik 		no flags Details
View All

Description Jan Kurik 2020-02-20 20:44:30 UTC 
Created attachment 1664492 [details]
grafana.log

Description of problem:
Assumption: Grafana is configured to use sqlite database to store its data (this is the default).

After installation of grafana package there is no /var/lib/grafana/grafana.db database present. This database is created either when grafana-server starts or when grafana-cli utility is used for the first time.

If the database is created by grafana-server after its start, the /var/lib/grafana/grafana.db database is then owned by grafana:grafana user. However if grafana-cli utility is used before grafana-server first start, then the database is created by the grafana-cli utility and it is owned by root:root user.

Having the /var/lib/grafana/grafana.db database owned by root:root user makes the database inaccessible for the grafana-server (which is running under grafana:grafana user). This is causing crash of the grafana-server. The crash generates a report which is attached to this bug (see the grafana.log file).

Running grafana-cli utility as grafana:grafana user (i.e. using "su") does not work as such attempt fails with the following error message:

Error: Could not read user from database. Error: handler not found

Version-Release number of selected component (if applicable):
* grafana-6.3.6-1.el8

How reproducible:
Always

Steps to Reproduce:
1. Install fresh RHEL-8.2 (Beta)
2. Install Grafana
# yum install -y grafana
3. Set admin password using grafana-cli utility
# grafana-cli admin reset-admin-password SuperSecretPassword
4. Start grafana-server
# systemctl start grafana-server
5. Open grafana login page in a web browser
# firefox http://localhost:3000
6. Try to log in using admin/SuperSecretPassword credentials

Actual results:
* The login attempt fails showing "big red box with HTTP Error code 500" and redirects back to the login page
* /var/log/grafana/grafana.log contains the attached invalid memory address trace (see the grafana.log file)

Expected results:
* Login is successful

Additional info:
A simple workaround is to "chown grafana:grafana /var/lib/grafana/grafana.db" after use of grafana-cli utility
Comment 1 Mark Goodwin 2020-02-21 03:16:39 UTC 
Hi Jan, I've verified what you've seen. Since the grafana installation %post scripts create a grafana user and group (if none already exist), a simple work-around is to run grafana-cli as the grafana user, e.g. 

sudo -u grafana grafana-cli admin reset-admin-password SuperSecretPassword

A fix would be in the grafana-cli wrapper to check the user is either root or grafana and then exec the grafana-cli binary via sudo -u grafana (works for both root and the grafana user) as in the above example.

Alternatively, perhaps grafana-cli could check if the grafana service is running (and/or the db already exists) and abort if not.

Given there are work-arounds (either the chown or the sudo method), should we target RHEL8.3 for this fix (via rebase when the time comes), and work with upstream in the mean-time?

Also - the grafana-server doesn't crash - the login UI page reports an error that the db is read-only, then returns to the login prompt. So I've edited the BZ summary to reflect this.
Comment 2 Jan Kurik 2020-02-21 07:15:42 UTC 
Hi Mark,

I had some permission issues with "su" or "sudo" commands and then I have realized the $HOME of grafana is not writeable for grafana user after installation:

# grep grafana /etc/passwd
grafana:x:993:990:grafana user account:/usr/share/grafana:/sbin/nologin

# ls -ld /usr/share/grafana
drwxr-xr-x. 5 root root 43 Feb 21 00:12 /usr/share/grafana

IMO the "/usr/share/grafana" should be owned by grafana:grafana.


Regarding the crash - see the attached grafana.log. There is a clear message:

runtime error: invalid memory address or nil pointer dereference

When I use "strace" to trace the grafana process, I can see that one of the grafana's worker treads gets SIGSEGV signal, and then the worker thread crashes. Signal handler for SIGSEGV signal generates the message in grafana.log file.

The message I see in web browser is as follows (it is shown as HTML gibberish):

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <meta name="viewport" content="width=device-width"> <meta name="theme-color" content="#000"> <title>Grafana - Error</title> <base href="/" /> <link rel="stylesheet" href="public/build/grafana.dark.69fae754221074f118fc.css"> <link rel="icon" type="image/png" href="public/img/fav32.png"> <link rel="mask-icon" href="public/img/grafana_mask_icon.svg" color="#F05A28"> </head> <body class="theme-dark"> <div class="main-view"> <div class="page-container"> <div class="page-header"> <div class="page-header__inner"> <span class="page-header__logo"> <i class="page-header__icon fa fa-frown-o"></i> </span> <div class="page-header__info-block"> <h1 class="page-header__title"> <a class="text-link" href="login">Grafana</a><span> / Server Error</span><span></span> </h1> <div class="page-header__sub-title">Sadly something went wrong</div> </div> </div> </div> </div> <div class="page-container page-body ng-scope" style="padding: 2rem"> <div class="alert"> <div class="alert-icon"><i class="fa fa-exclamation-triangle"></i></div> <div class="alert-body"> <div class="alert-title">Server Error</div> </div> </div> <br /> <div style="padding: 2rem 0 0"> <p>Check the Grafana server logs for the detailed error message.</p> </div> </div> </div> </body> </html> 

Filtering out, it says:

Server Error
Sadly something went wrong
Check the Grafana server logs for the detailed error message.


But anyway, whatever fixes the issue works for me. RHEL-8.3 is fine from my perspective as I do not believe there will be many customers playing with grafana-cli utility before the very first start of grafana-server.
Comment 3 Andreas Gerstmayr 2020-05-05 15:36:21 UTC 
Fixed in the Grafana 6.7.3 rebase

*** This bug has been marked as a duplicate of bug 1807323 ***
Comment 4 Jan Kurik 2020-06-12 05:00:29 UTC 
This is not fixed in grafana-6.7.4-1.el8 build.
Comment 5 Andreas Gerstmayr 2020-06-15 15:10:26 UTC 
The grafana.db should be created with the corrent user + permissions (as part of the CVE fix).

The reason grafana doesn't start are other files/directories afaics, as mentioned here: https://bugzilla.redhat.com/show_bug.cgi?id=1843170
Imho we can mark this bug as duplicate?
Comment 6 Nathan Scott 2020-06-23 04:24:35 UTC 
(In reply to Andreas Gerstmayr from comment #5)
> [...] Imho we can mark this bug as duplicate?

+1

*** This bug has been marked as a duplicate of bug 1843170 ***
Note You need to log in before you can comment on or make changes to this bug.