Description of problem: graphs are not generated on a freshly configured cluster, even after doing I/O Version-Release number of selected component (if applicable): 1.3.1 Ubuntu How reproducible: Always Steps to Reproduce: 1.Installed and created a 1.3.1 ubuntu cluster through repo based install 2.launched clamari , connected the nodes to the calamari 3.start I/O , and observe calamari graphs (cpus, network, storage etc) Actual results: Calamari is not generating any graph Expected results: Calamari graph should be generated Additional info: Attaching the calamari logs
Created attachment 1085169 [details] calamari logs
Which graph is not reporting data? Do you have a system that I can inspect?
I am seeing IOPS graphed on the first instance. I'm not sure what changed.
Hi Gregory, Please let us know if any other logs or info needed for this defect. Any workaround you suggest? Regards, Harish
What I need it a description of the problem that I can observe. In https://bugzilla.redhat.com/show_bug.cgi?id=1273943#c4 I tried those steps and got IOPS time series data. Perhaps we need to setup a screen share. I don't understand the issue.
Hi Greg, Currently this setup is in use for testing. I will prepare another setup and let you know in case if I hit this issue again
well there are not logs that help me there. After running "sudo calamari-ctl initialize" the graphs work like I expect so that can be a work-around. I am a little confused about the order of operations when you set this up. Is there anywhere I can see that?
Hi Greg, This is the doc we are following https://gitlab.cee.redhat.com/ngoswami/red-hat-ceph-storage-installation-guide-ubuntu/blob/devel/calamari.adoc. Calamri commands are at the end of the page. Let me know if you need more info.
(In reply to shylesh from comment #12) > Hi Greg, > > This is the doc we are following > https://gitlab.cee.redhat.com/ngoswami/red-hat-ceph-storage-installation- > guide-ubuntu/blob/devel/calamari.adoc. Calamri commands are at the end of > the page. > > Let me know if you need more info. Hi Greg, I already ran "sudo calamari-ctl initialize" before you did, but it was not generating the graph. But after you ran it for the second time on the same setup i think it started working.
Hi Greg, I had another setup where graphs were not generated, so running "calamari-ctl initialize " second time after connecting all nodes the graphs started appearing.
Hi Gregory, Should the additional step of running calamari-ctl initialize be added to installation doc or should this bug be added to known issues in release notes? Please clarify. thanks, Harish
I am leaning toward adding it to the known issues in the release notes. What I think is happening is something is going wrong with the minion connection yeilding an application of the salt provisioning that installs diamond. The instance that Shylesh provided doesn't contain logs to support that hypothesis. I need to figure out what to do next.
specific instructions for known issues page would look like: after connecting cluster nodes to the admin node run sudo calamari-ctl initialize
We'll document this under "Known Issues" for RHCS 1.3.1, and re-target this BZ to RHCS 1.3.2.
Bara, I don't think that really covers the problem correctly. I've updated the Doc text to more accurately describe it. What do you think?
We need a clean reproduction of this bug, and we will then look at fixing in 1.3.2. If cannot repro successfully, please close.
I will update this bug as soon as we get 1.3.2 build. @Greg, Please let me know what info has to be collected so that I can take care during installation of calamari itself.
Hi Greg, After upgrading from 1.3.1 to 1.3.1 Async I ran "salt '*' state.highstate" graphs started appearing. This issues and workaround is not working consistently.
Hi Greg, Similar behavior is seen on ubuntu setup also. I was unable to see the graphs after 1.3.1-GA was installed. Tried "calamari-ctl initialize" but that did not help me. So, I executed "sudo salt '*' state.highstate" and it worked., "calamari-ctl initialize" did not solve the issue here.
I checked the code that should be there and it's just running salt state.highstate This is how you 'd identify that is true for your case, you'd see the message: [INFO] Updating already connected nodes. when running calamari-ctl initialize if that is the case there are two possibilities. 1. there is a bug in the invocation, this isn't logged anywhere we can easily get to I can mod the code to expose the error basically you are modifiying /opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu/calamari_ctl.py to say handler.setLevel(logging.DEBUG) instead of handler.setLevel(logging.INFO) Then running it again would produce logging on stdout that may shed light on the issue. 2. something else changed If step on doesn't clear up what's going on I'll need to poke a system where this is happening.
I haven't heard otherwise
As per comment #25,#26,#30,#31 The doc text needs a change: Please change it to: After installing a new Ceph cluster and initializing the Calamari server, Calamari graphs can be missing. The graphs are missing on any node connected to Calamari after the `sudo calamari-ctl initialize` command was run. To work around this issue, run `sudo calamari-ctl initialize` and salt '*' state.highstate after connecting additional Ceph cluster nodes to Calamari. This command can be run multiple times without any issues. + and salt '*' state.highstate
Also the doc text heading needs to be just: "Missing Calamari graphs" as this issue is seen on both RHEL and Ubuntu