Bug 1273943 - [CALAMARI]: Calamari graphs are not generated after a fresh install of the cluster, even after doing I/O
[CALAMARI]: Calamari graphs are not generated after a fresh install of the cl...
Status: ASSIGNED
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Calamari (Show other bugs)
1.3.1
x86_64 Linux
unspecified Severity high
: rc
: 1.3.4
Assigned To: Gregory Meno
ceph-qe-bugs
Bara Ancincova
:
Depends On:
Blocks: ceph131rn 1299303 1372735
  Show dependency treegraph
 
Reported: 2015-10-21 10:35 EDT by shylesh
Modified: 2016-09-19 21:51 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Missing Calamari graphs After installing a new Ceph cluster and initializing the Calamari server, Calamari graphs can be missing. The graphs are missing on any node connected to Calamari after the `calamari-ctl initialize` command was run. To work around this issue, run the `calamari-ctl initialize` and `salt '*' state.highstate` commands after connecting additional Ceph cluster nodes to Calamari. These commands can be run multiple times without any issues.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
calamari logs (225.89 KB, application/x-gzip)
2015-10-21 10:36 EDT, shylesh
no flags Details

  None (edit)
Description shylesh 2015-10-21 10:35:43 EDT
Description of problem:
graphs are not generated on a freshly configured cluster, even after doing I/O

Version-Release number of selected component (if applicable):
1.3.1 Ubuntu

How reproducible:
Always

Steps to Reproduce:
1.Installed and created a 1.3.1 ubuntu cluster through repo based install
2.launched clamari , connected the nodes to the calamari
3.start I/O , and observe calamari graphs (cpus, network, storage etc)

Actual results:
Calamari is not generating any graph

Expected results:

Calamari graph should be generated
Additional info:
Attaching the calamari logs
Comment 2 shylesh 2015-10-21 10:36 EDT
Created attachment 1085169 [details]
calamari logs
Comment 3 Gregory Meno 2015-10-21 14:44:53 EDT
Which graph is not reporting data?

Do you have a system that I can inspect?
Comment 5 Gregory Meno 2015-10-27 18:54:06 EDT
I am seeing IOPS graphed on the first instance. 
I'm not sure what changed.
Comment 6 Harish NV Rao 2015-10-28 12:33:36 EDT
Hi Gregory,

Please let us know if any other logs or info needed for this defect. Any workaround you suggest?

Regards,
Harish
Comment 7 Gregory Meno 2015-10-28 14:27:29 EDT
What I need it a description of the problem that I can observe.
In https://bugzilla.redhat.com/show_bug.cgi?id=1273943#c4 I tried those steps and got IOPS time series data.

Perhaps we need to setup a screen share. I don't understand the issue.
Comment 8 shylesh 2015-10-29 11:07:24 EDT
Hi Greg,

Currently this setup is in use for testing. I will prepare another setup and let you know in case if I hit this issue again
Comment 11 Gregory Meno 2015-11-09 15:19:13 EST
well there are not logs that help me there.

After running "sudo calamari-ctl initialize" the graphs work like I expect so that can be a work-around. 

I am a little confused about the order of operations when you set this up.
Is there anywhere I can see that?
Comment 12 shylesh 2015-11-10 02:42:41 EST
Hi Greg,

This is the doc we are following https://gitlab.cee.redhat.com/ngoswami/red-hat-ceph-storage-installation-guide-ubuntu/blob/devel/calamari.adoc. Calamri commands are at the end of the page.

Let me know if you need more info.
Comment 13 shylesh 2015-11-10 04:54:08 EST
(In reply to shylesh from comment #12)
> Hi Greg,
> 
> This is the doc we are following
> https://gitlab.cee.redhat.com/ngoswami/red-hat-ceph-storage-installation-
> guide-ubuntu/blob/devel/calamari.adoc. Calamri commands are at the end of
> the page.
> 
> Let me know if you need more info.

Hi Greg,

I already ran "sudo calamari-ctl initialize" before you did, but it was not generating the graph. But after you ran it for the second time on the same setup i think it started working.
Comment 14 shylesh 2015-11-10 05:13:37 EST
Hi Greg,

I had another setup where graphs were not generated, so running "calamari-ctl initialize " second time after connecting all nodes the graphs started appearing.
Comment 15 Harish NV Rao 2015-11-10 08:03:25 EST
Hi Gregory,

Should the additional step of running calamari-ctl initialize be added to installation doc or should this bug be added to known issues in release notes?

Please clarify.

thanks,
Harish
Comment 16 Gregory Meno 2015-11-10 17:15:45 EST
I am leaning toward adding it to the known issues in the release notes.

What I think is happening is something is going wrong with the minion connection yeilding an application of the salt provisioning that installs diamond.

The instance that Shylesh provided doesn't contain logs to support that hypothesis. I need to figure out what to do next.
Comment 17 Gregory Meno 2015-11-10 17:16:51 EST
specific instructions for known issues page would look like:

after connecting cluster nodes to the admin node run sudo calamari-ctl initialize
Comment 18 Ken Dreyer (Red Hat) 2015-11-11 10:26:35 EST
We'll document this under "Known Issues" for RHCS 1.3.1, and re-target this BZ to RHCS 1.3.2.
Comment 20 Gregory Meno 2015-11-11 16:59:23 EST
Bara,

I don't think that really covers the problem correctly.

I've updated the Doc text to more accurately describe it.

What do you think?
Comment 22 Federico Lucifredi 2015-12-10 18:32:05 EST
We need a clean reproduction of this bug, and we will then look at fixing in 1.3.2. 

If cannot repro successfully, please close.
Comment 23 shylesh 2015-12-16 06:31:53 EST
I will update this bug as soon as we get 1.3.2 build.

@Greg,

Please let me know what info has to be collected so that I can take care during installation of calamari itself.
Comment 25 shylesh 2016-01-28 08:58:45 EST
Hi Greg,

After upgrading from 1.3.1 to 1.3.1 Async I ran "salt '*' state.highstate" graphs started appearing.  

This issues and workaround is not working consistently.
Comment 26 Hemanth Kumar 2016-01-28 12:03:00 EST
Hi Greg, 

Similar behavior is seen on ubuntu setup also.
I was unable to see the graphs after 1.3.1-GA was installed. Tried "calamari-ctl initialize" but that did not help me. So, I executed "sudo salt '*' state.highstate" and it worked.,  "calamari-ctl initialize" did not solve the issue here.
Comment 27 Gregory Meno 2016-02-02 14:22:43 EST
I checked the code that should be there and it's just running salt state.highstate 


This is how you 'd identify that is true for your case, you'd see the message:
[INFO] Updating already connected nodes.

when running calamari-ctl initialize

if that is the case there are two possibilities.

1. there is a bug in the invocation, this isn't logged anywhere we can easily get to I can mod the code to expose the error

basically you are modifiying /opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu/calamari_ctl.py

to say 
handler.setLevel(logging.DEBUG)

instead of 
handler.setLevel(logging.INFO)

Then running it again would produce logging on stdout that may shed light on the issue.



2. something else changed
If step on doesn't clear up what's going on I'll need to poke a system where this is happening.
Comment 29 Gregory Meno 2016-02-11 17:57:26 EST
I haven't heard otherwise
Comment 32 Tanay Ganguly 2016-02-22 10:56:41 EST
As per comment #25,#26,#30,#31
The doc text needs a change:

Please change it to:

After installing a new Ceph cluster and initializing the Calamari server, Calamari graphs can be missing. The graphs are missing on any node connected to Calamari after the `sudo calamari-ctl initialize` command was run. To work around this issue, run `sudo calamari-ctl initialize` and salt '*' state.highstate after connecting additional Ceph cluster nodes to Calamari. This command can be run multiple times without any issues.



+ and salt '*' state.highstate
Comment 33 Harish NV Rao 2016-02-22 11:00:46 EST
Also the doc text heading needs to be just: "Missing Calamari graphs" as this issue is seen on both RHEL and Ubuntu

Note You need to log in before you can comment on or make changes to this bug.