Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1805472 - grafana-cli is causing grafana-server logins to fail when used before grafana.db is created
Summary: grafana-cli is causing grafana-server logins to fail when used before grafana...
Keywords:
Status: CLOSED DUPLICATE of bug 1843170
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: grafana
Version: 8.2
Hardware: All
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.3
Assignee: Andreas Gerstmayr
QA Contact: Jan Kurik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-20 20:44 UTC by Jan Kurik
Modified: 2020-12-20 05:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-23 04:24:35 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
grafana.log (4.56 KB, text/plain)
2020-02-20 20:44 UTC, Jan Kurik
no flags Details

Description Jan Kurik 2020-02-20 20:44:30 UTC
Created attachment 1664492 [details]
grafana.log

Description of problem:
Assumption: Grafana is configured to use sqlite database to store its data (this is the default).

After installation of grafana package there is no /var/lib/grafana/grafana.db database present. This database is created either when grafana-server starts or when grafana-cli utility is used for the first time.

If the database is created by grafana-server after its start, the /var/lib/grafana/grafana.db database is then owned by grafana:grafana user. However if grafana-cli utility is used before grafana-server first start, then the database is created by the grafana-cli utility and it is owned by root:root user.

Having the /var/lib/grafana/grafana.db database owned by root:root user makes the database inaccessible for the grafana-server (which is running under grafana:grafana user). This is causing crash of the grafana-server. The crash generates a report which is attached to this bug (see the grafana.log file).

Running grafana-cli utility as grafana:grafana user (i.e. using "su") does not work as such attempt fails with the following error message:

Error: Could not read user from database. Error: handler not found

Version-Release number of selected component (if applicable):
* grafana-6.3.6-1.el8

How reproducible:
Always

Steps to Reproduce:
1. Install fresh RHEL-8.2 (Beta)
2. Install Grafana
# yum install -y grafana
3. Set admin password using grafana-cli utility
# grafana-cli admin reset-admin-password SuperSecretPassword
4. Start grafana-server
# systemctl start grafana-server
5. Open grafana login page in a web browser
# firefox http://localhost:3000
6. Try to log in using admin/SuperSecretPassword credentials

Actual results:
* The login attempt fails showing "big red box with HTTP Error code 500" and redirects back to the login page
* /var/log/grafana/grafana.log contains the attached invalid memory address trace (see the grafana.log file)

Expected results:
* Login is successful

Additional info:
A simple workaround is to "chown grafana:grafana /var/lib/grafana/grafana.db" after use of grafana-cli utility

Comment 1 Mark Goodwin 2020-02-21 03:16:39 UTC
Hi Jan, I've verified what you've seen. Since the grafana installation %post scripts create a grafana user and group (if none already exist), a simple work-around is to run grafana-cli as the grafana user, e.g. 

sudo -u grafana grafana-cli admin reset-admin-password SuperSecretPassword

A fix would be in the grafana-cli wrapper to check the user is either root or grafana and then exec the grafana-cli binary via sudo -u grafana (works for both root and the grafana user) as in the above example.

Alternatively, perhaps grafana-cli could check if the grafana service is running (and/or the db already exists) and abort if not.

Given there are work-arounds (either the chown or the sudo method), should we target RHEL8.3 for this fix (via rebase when the time comes), and work with upstream in the mean-time?

Also - the grafana-server doesn't crash - the login UI page reports an error that the db is read-only, then returns to the login prompt. So I've edited the BZ summary to reflect this.

Comment 2 Jan Kurik 2020-02-21 07:15:42 UTC
Hi Mark,

I had some permission issues with "su" or "sudo" commands and then I have realized the $HOME of grafana is not writeable for grafana user after installation:

# grep grafana /etc/passwd
grafana:x:993:990:grafana user account:/usr/share/grafana:/sbin/nologin

# ls -ld /usr/share/grafana
drwxr-xr-x. 5 root root 43 Feb 21 00:12 /usr/share/grafana

IMO the "/usr/share/grafana" should be owned by grafana:grafana.


Regarding the crash - see the attached grafana.log. There is a clear message:

runtime error: invalid memory address or nil pointer dereference

When I use "strace" to trace the grafana process, I can see that one of the grafana's worker treads gets SIGSEGV signal, and then the worker thread crashes. Signal handler for SIGSEGV signal generates the message in grafana.log file.

The message I see in web browser is as follows (it is shown as HTML gibberish):

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <meta name="viewport" content="width=device-width"> <meta name="theme-color" content="#000"> <title>Grafana - Error</title> <base href="/" /> <link rel="stylesheet" href="public/build/grafana.dark.69fae754221074f118fc.css"> <link rel="icon" type="image/png" href="public/img/fav32.png"> <link rel="mask-icon" href="public/img/grafana_mask_icon.svg" color="#F05A28"> </head> <body class="theme-dark"> <div class="main-view"> <div class="page-container"> <div class="page-header"> <div class="page-header__inner"> <span class="page-header__logo"> <i class="page-header__icon fa fa-frown-o"></i> </span> <div class="page-header__info-block"> <h1 class="page-header__title"> <a class="text-link" href="login">Grafana</a><span> / Server Error</span><span></span> </h1> <div class="page-header__sub-title">Sadly something went wrong</div> </div> </div> </div> </div> <div class="page-container page-body ng-scope" style="padding: 2rem"> <div class="alert"> <div class="alert-icon"><i class="fa fa-exclamation-triangle"></i></div> <div class="alert-body"> <div class="alert-title">Server Error</div> </div> </div> <br /> <div style="padding: 2rem 0 0"> <p>Check the Grafana server logs for the detailed error message.</p> </div> </div> </div> </body> </html> 

Filtering out, it says:

Server Error
Sadly something went wrong
Check the Grafana server logs for the detailed error message.


But anyway, whatever fixes the issue works for me. RHEL-8.3 is fine from my perspective as I do not believe there will be many customers playing with grafana-cli utility before the very first start of grafana-server.

Comment 3 Andreas Gerstmayr 2020-05-05 15:36:21 UTC
Fixed in the Grafana 6.7.3 rebase

*** This bug has been marked as a duplicate of bug 1807323 ***

Comment 4 Jan Kurik 2020-06-12 05:00:29 UTC
This is not fixed in grafana-6.7.4-1.el8 build.

Comment 5 Andreas Gerstmayr 2020-06-15 15:10:26 UTC
The grafana.db should be created with the corrent user + permissions (as part of the CVE fix).

The reason grafana doesn't start are other files/directories afaics, as mentioned here: https://bugzilla.redhat.com/show_bug.cgi?id=1843170
Imho we can mark this bug as duplicate?

Comment 6 Nathan Scott 2020-06-23 04:24:35 UTC
(In reply to Andreas Gerstmayr from comment #5)
> [...] Imho we can mark this bug as duplicate?

+1

*** This bug has been marked as a duplicate of bug 1843170 ***


Note You need to log in before you can comment on or make changes to this bug.