Description of problem: With the batch update 1 for RHGSWA, the tendrl-monotoring-integration provides updates to the grafana dashboards. These updates have to be done without any interference to the clusters being managed and the whole WA stack to be re-installed. Version-Release number of selected component (if applicable): tendrl-monitoring-integration-1.6.3-11.el7rhgs.noarch How reproducible: 100 % Procedure: 1.Stop tendrl-monitoring-integration service. 2. Do a yum update to install latest version of the tendrl-monitoring-integration package. This way the settings and configuration files remain preserved and would require no change. 3. Run the upgrade script provided. The script would require 3 inputs from the user : a) grafana admin_user (default : /etc/tendrl/monitoring-integration/grafana/grafana.ini line 143) b) grafana admin_password (default : /etc/tendrl/monitoring-integration/grafana/grafana.ini line 146) c) WA server-node ip. 4. Restart tendrl-monitoring-integration service. Actual results: Expected results: The above procedure should upgrade the tendrl-monitoring-integration package and the grafana dashboards should be updated. Additional info:
Not having user input credentials for upgrade would be ideal. Customer should view it as one upgrade Vs two (RHGS and WA)
(In reply to Anand Paladugu from comment #6) > Not having user input credentials for upgrade would be ideal. Customer > should view it as one upgrade Vs two (RHGS and WA) The user credential inputs have been made optional, and the script would run successfully if the default credentials are not modified by the user. If the user changes the default credentials then the user has to input the grafana username and password.
WA upgrade procedure: On Storage Nodes ---------------- 1. Update the RHGS-WA packages #yum update tendrl-* 2. Restart the RHGS-WA services #systemctl restart tendrl-node-agent #systemctl restart tendrl-gluster-integration On WA Server Node --------------------- 1. Stop the tendrl-monitoring-integration servive #systemctl stop tendrl-monitoring-integration 2. Update the RHGS-WA packages #yum update tendrl-* 3. Run the migration script #tendrl-upgrade --username=username --password=password Note: username and passowrd options are optional if the user has not updated the default grafana credentials. 4. Restart the RHGS-WA services #systemctl restart tendrl-node-agent #systemctl restart tendrl-monitoring-integration #systemctl restart tendrl-notifier #systemctl restart tendrl-api 5. Restart httpd service #systemctl restart httpd Preferably the storage nodes should be upgraded first and then the server node.
(In reply to anmol sachan from comment #8) > 3. Run the migration script > #tendrl-upgrade --username=username --password=password > Note: username and passowrd options are optional if the user has not > updated the default grafana credentials. We should be quite careful with documenting this Note, to avoid confusion "which" username and password is meant (because the only credentials mentioned in WA documentation is for the WA (Tendrl) interface). > Preferably the storage nodes should be upgraded first and then the server > node. If the WA server will be updated after the storage nodes including the core OS update and if it will be necessary to reboot the machine (e.g. because of kernel update) the user might hit Bug 1622461 (tendrl-node-agent (and gluster-integration) services will crash on Storage servers).
One more question. (In reply to Daniel Horák from comment #9) > (In reply to anmol sachan from comment #8) > > 3. Run the migration script > > #tendrl-upgrade --username=username --password=password > > Note: username and passowrd options are optional if the user has not > > updated the default grafana credentials. > > We should be quite careful with documenting this Note, to avoid confusion > "which" > username and password is meant (because the only credentials mentioned in WA > documentation is for the WA (Tendrl) interface). Is there even possibility, that the user will change the Grafana Admin password, without breaking additional functionality? (and without changing the respective value in the related config files)
(In reply to Daniel Horák from comment #9) > (In reply to anmol sachan from comment #8) > > 3. Run the migration script > > #tendrl-upgrade --username=username --password=password > > Note: username and passowrd options are optional if the user has not > > updated the default grafana credentials. > > We should be quite careful with documenting this Note, to avoid confusion > "which" > username and password is meant (because the only credentials mentioned in WA > documentation is for the WA (Tendrl) interface). In the upgrade process, the tendrl-upgrade scripts would need grafana username and password. The default values for which are located in the /etc/tendrl/monitoring-integration/grafana/grafana.ini (line 141 security section). The grafana credentials are not the same credentials as that of WA UI. I agree that it should be documented carefully. > > > > Preferably the storage nodes should be upgraded first and then the server > > node. > > If the WA server will be updated after the storage nodes including the core > OS > update and if it will be necessary to reboot the machine (e.g. because of > kernel > update) the user might hit Bug 1622461 (tendrl-node-agent (and > gluster-integration) services will crash on Storage servers). During the upgrade procedure, there is not requirement of System restart from WA side. If there is a restart from the user similar to the scenario mentioned above, etcd will go down, which will be point of communication failure for services for the storage nodes. In this situation the user shall take an alternate approach. I would need more info on that before providing a upgrade procedure. Adding need info on Sunil for info on what RHGS will be doing in the update procedure and what approach they will take during an OS update which requires reboot, as WA has to be in coherence with RHGS.
(In reply to Daniel Horák from comment #10) > Is there even possibility, that the user will change the Grafana Admin > password, > without breaking additional functionality? (and without changing the > respective > value in the related config files) Grafana UI provides option for user to update the grafana password. This password persists in the grafana.db and not in config files. Thus it is possible and all the services still will work.
(In reply to Daniel Horák from comment #9) > If the WA server will be updated after the storage nodes including the core > OS > update and if it will be necessary to reboot the machine (e.g. because of > kernel > update) the user might hit Bug 1622461 (tendrl-node-agent (and > gluster-integration) services will crash on Storage servers). If the user hit Bug 1622461 workaround is specified in the doc_text. This is something which needs to take care irrespective of WA is getting upgraded or not After the rhel upgrade, admin should make sure that all WA services are up and running and it works as expected. There are three cases you qe needs to test: 1. Rhel 7.5, upgrade WA to the newer version 2. Upgrade Rhel 7.5 to 7.6 and then upgrade WA to the newer version 3. Rhel 7.5, upgrade WA to newer version. After this upgrade WA to the newer version Tn all the above cases, newer version of gluster is a prerequisite To test all the above cases, upgrade scenario mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=1631264#c8, holds good. please report if you find any issues
(In reply to Anmol Sachan from comment #12) > Grafana UI provides option for user to update the grafana password. This > password persists in the grafana.db and not in config files. Thus it is > possible and all the services still will work. I understand, that the password is stored in grafana.db and can be changed via Grafana Web UI, but the password is stored also in Tendrl monitoring- integration configuration file: /etc/tendrl/monitoring-integration/monitoring-integration.conf.yaml and this will not be updated if the password will be changed via Grafana Web. My concern is, what will break on Tendrl site, if the actual admin Grafana password will be different, than what is stored in the Tendrl monitoring- integration configuration file? Also why would user change the admin password, while he is not supposed to login to the Grafana dashboard?
(In reply to Daniel Horák from comment #14) > I understand, that the password is stored in grafana.db and can be changed > via Grafana Web UI, but the password is stored also in Tendrl monitoring- > integration configuration file: > /etc/tendrl/monitoring-integration/monitoring-integration.conf.yaml > and this will not be updated if the password will be changed via Grafana Web. > > My concern is, what will break on Tendrl site, if the actual admin Grafana > password will be different, than what is stored in the Tendrl monitoring- > integration configuration file? The credentials in both grafana.ini and monitoring-integration configuration file are same and are the default credentials for grafana. These same credentials are set for grafana during the tendrl setup (Default).If the password is changed from grafana web ui, it will not break anything and everything should be working fine. > > Also why would user change the admin password, while he is not supposed to > login to the Grafana dashboard? I agree that the user is not supposed to login into grafana, but it is possible. That is why the username and password arguments in the tendrl-upgrade script are optional. Its a fail-safe for the upgrade process for the users that may run into this scenario. If there is no credential change by the user, then there is no point of concern here and the user should be able to upgrade with the tendrl-upgrade script without passing on the credentials.
(In reply to Anmol Sachan from comment #16) > If the password is changed from grafana web ui, it will not break > anything and everything should be working fine. My understanding is, that it will break the tendrl-monitoring-service: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # systemctl status tendrl-monitoring-integration -l ● tendrl-monitoring-integration.service - Monitoring Integration Loaded: loaded (/usr/lib/systemd/system/tendrl-monitoring-integration.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Mon 2018-10-01 11:19:24 CEST; 967ms ago Process: 16904 ExecStart=/usr/bin/tendrl-monitoring-integration (code=exited, status=1/FAILURE) Main PID: 16904 (code=exited, status=1/FAILURE) Oct 01 11:19:24 tendrl-server.example.com systemd[1]: tendrl-monitoring-integration.service: main process exited, code=exited, status=1/FAILURE Oct 01 11:19:24 tendrl-server.example.com systemd[1]: Unit tendrl-monitoring-integration.service entered failed state. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: tendrl-monitoring-integration.service failed. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: tendrl-monitoring-integration.service holdoff time over, scheduling restart. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: Stopped Monitoring Integration. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: start request repeated too quickly for tendrl-monitoring-integration.service Oct 01 11:19:24 tendrl-server.example.com systemd[1]: Failed to start Monitoring Integration. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: Unit tendrl-monitoring-integration.service entered failed state. Oct 01 11:19:24 tendrl-server.example.com systemd[1]: tendrl-monitoring-integration.service failed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With quite explanatory error in /var/log/messages: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Oct 1 11:19:23 tendrl-server journal: 2018-10-01 09:19:23.264273+00:00 - monitoring_integration - /usr/lib/python2.7/site-packages/tendrl/monitoring_integration/grafana/dashboard.py:26 - upload_default_dashboards - ERROR - Invalid username or password ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(In reply to Daniel Horák from comment #18) > (In reply to Anmol Sachan from comment #16) > > If the password is changed from grafana web ui, it will not break > > anything and everything should be working fine. > > My understanding is, that it will break the tendrl-monitoring-service: If above is the case, then we should not expose the grafana username and password fields in the documentation. The upgrade script will still work fine, but the user will end up breaking the monitoring integration if there is change in the credentials. Then let's have the below as the upgrade procedure, provides RHGS : WA upgrade procedure: On Storage Nodes ---------------- 1. Update the RHGS-WA packages #yum update tendrl-* 2. Restart the RHGS-WA services #systemctl restart tendrl-node-agent #systemctl restart tendrl-gluster-integration On WA Server Node --------------------- 1. Stop the tendrl-monitoring-integration servive #systemctl stop tendrl-monitoring-integration 2. Update the RHGS-WA packages #yum update tendrl-* 3. Run the migration script #tendrl-upgrade 4. Restart the RHGS-WA services #systemctl restart tendrl-node-agent #systemctl restart tendrl-monitoring-integration #systemctl restart tendrl-notifier #systemctl restart tendrl-api 5. Restart httpd service #systemctl restart httpd
(In reply to Nishanth Thomas from comment #13) > There are three cases you qe needs to test: > 1. Rhel 7.5, upgrade WA to the newer version > 2. Upgrade Rhel 7.5 to 7.6 and then upgrade WA to the newer version > 3. Rhel 7.5, upgrade WA to newer version. After this upgrade WA to the > newer version I think, that the most probable case is, that all packages will be updated in one step (using general "yum update" command, as it is mentioned for example in RHGS documentation [1]). So I think, we should document it only as "additional steps" to already existing procedure of upgrading RHEL or RHGS machine. [1] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html-single/installation_guide/#Updating_Red_Hat_Storage_in_the_Offline_Mode I know, that it is related to previous version of RHGS, but I'm expecting similar procedure also for the upcoming release.
If I read this right, there is no complete draft of the upgrade procedure, not even for WA alone.
Please note that this BZ depends on BZ 1636187 and as such can't be verified before that BZ is clarified and verified first.
Aligning terminology with RHGS (Upgrade -> Update).
Quick Start Guide - Getting Started with Web Administration contains chapter 4.2. Red Hat Gluster Storage Web Administration 3.4.x to 3.4.y describing update procedure for RHGS WA from version 3.4.0 to 3.4.1. I've tested the procedure multiple times and it works as expected. >> VERIFIED
Question from comment 20 has been already resolved.