Description of problem: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/index doesn't document how to upgrade DWH running on separate host
Section II documents upgrading a remote DB environment. https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/upgrade_guide/upgrading_a_standalone_manager_remote_database_environment What is missing from here? If something specific is missing, I suggest changing the title of the bug to be more specific, or close as NOTABUG.
Here is not just the DB, it's the whole DWH service
Is this the correct flow: 1. Upgrade RHVM to 4.4 [1] 2. Upgrade DWH DB to Postgresql12 [2] 3. Upgrade DWH service (TBD): a. Backup DWH service from remote machine. b. Run engine-setup on the same machine as the DWH DB to install DWH service. c. Connect the DWH service to the RHVM service. Questions: When and how do you install Grafana on the remote DWH machine? [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/upgrade_guide/index#Upgrading_the_Manager_to_4-4_4-3_remote_db [2] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/upgrade_guide/index#Upgrading_Remote_Databases_from_PG10_to_PG12_4-3_remote_db
(In reply to Steve Goodman from comment #3) > Is this the correct flow: > > 1. Upgrade RHVM to 4.4 [1] > 2. Upgrade DWH DB to Postgresql12 [2] > 3. Upgrade DWH service (TBD): > a. Backup DWH service from remote machine. Assuming that the remote machine only had DWH (either new setup or migrated there based on existing docs), it's enough to: engine-backup --file=backupfile (meaning, leave scope=all, which will include in reality files+dwhdb. BTW, the same command, in 4.4.3, following bug 1879373, will include a new scope I add there, grafanadb). > b. Run engine-setup on the same machine as the DWH DB to install DWH > service. Didn't understand. You should basically do what you do on an engine restore: # engine-backup --mode=restore --file=backupfile --provision-all-databases (assuming a local DWH db. If it's remote, that's a different issue. Is it in current bug as well?). > c. Connect the DWH service to the RHVM service. Meaning? Above should be enough, IMO. > > Questions: > > When and how do you install Grafana on the remote DWH machine? No need for a separate step - we decided that dwh will require grafana, so you always have it. > > [1] > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/ > html-single/upgrade_guide/index#Upgrading_the_Manager_to_4-4_4-3_remote_db > [2] > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/ > html-single/upgrade_guide/ > index#Upgrading_Remote_Databases_from_PG10_to_PG12_4-3_remote_db
Sorry, I am not sure above is clear. There are two separate flows here: 1. dwh + dwhdb on same separate machine. For this, this should be enough: 1.1. On old machine: Backup: # engine-backup --file=backupfile Copy backupfile somewhere and keep it. Stop and disable dwhd: # systemctl stop ovirt-engine-dwhd # systemctl disable ovirt-engine-dwhd 1.2. On new machine: Add repos as applicable. Install software: # dnf install ovirt-engine-dwh Copy backupfile to the new machine. Restore: # engine-backup --mode=restore --file=backupfile --provision-all-databases Setup: # engine-setup If that's not enough, I'd consider it a bug. 2. Engine on machine A, DWH on machine B, DWH DB on machine C, using engine-backup: Assuming that A was already upgraded, and that B and C are 4.3/el7. 2.1. On old B: Backup: # engine-backup --file=backupfile Copy backupfile somewhere and keep it Stop and disable dwhd: # systemctl stop ovirt-engine-dwhd # systemctl disable ovirt-engine-dwhd Keep somewhere credentials (db name, user name, password) of DWH DB. You can get them with: grep ^DWH_DB /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/*.conf 2.2. Reinstall C Reinstall machine C with el8. Add repos as applicable. Install postgresql-server, postgresql-contrib, both version 12 (meaning, follow instructions to enable modules as in "Setup Remote DB"). initdb, create db, db user, edit postgresql.conf and pg_hba, etc. - following instructions of "Setup remote DB". Use same credentials as checked/kept in previous step. 2.3. Reinstall B Reinstall B with el8. Add repos as applicable. dnf install ovirt-engine-dwh 2.4. Restore on new B Copy backup file. Restore: # engine-backup --mode=restore --file-backupfile Do not pass '--provision-all-databases', as you want to use the existing DB you created in the previous step, not have engine-backup create it for you (it can't, as it's on a separate machine). Setup: # engine-setup 3. Engine on machine A, DWH on machine B, DWH DB on machine C, using pg tools: This is an alternative to (2.). I think this is what you currently (intend to?) document (didn't check existing docs, and we changed them recently...). I also didn't test it myself, so no idea how smooth it is. As I said in private, I do not recommend this in general, but it might make sense in specific cases. I also do not provide concrete details, but instead copy yours, because it seems that this is what you intended also here: (In reply to Steve Goodman from comment #3) > Is this the correct flow: > > 1. Upgrade RHVM to 4.4 [1] > 2. Upgrade DWH DB to Postgresql12 [2] These seem missing. I think you already did something similar for 4.3, did you also update 4.4 then? - enable PG 12 module before installing it - initdb, create db/user, edit postgresql.conf and pg_hba.conf - Open firewall Perhaps other stuff. Basically, copy most stuff from "Setup remote DB" doc, and add pg_dump/pg_restore where needed. > 3. Upgrade DWH service (TBD): > a. Backup DWH service from remote machine. How? I assume you mean: engine-backup --scope=files --file=backupfile If so, that's ok. > b. Run engine-setup on the same machine as the DWH DB to install DWH > service. Didn't follow. You mean (using my terminology) to run engine-setup on C? Why? > c. Connect the DWH service to the RHVM service. How? Do we have instructions for this in 4.3?
One more item we should add: If you have engine on one machine and dwh on another, and you run 'engine-setup' on the engine machine, and with bug 1894420 fixed, it will automatically stop dwh on its machine, and will tell you in the end to start it there manually, but will never prompt the user asking anything about this, e.g.: ========================================================================================= # engine-setup ... [ INFO ] Stopping websocket-proxy service [ INFO ] Stopping DWH service on host didi-rhel7-dwh.eng.lab.tlv.redhat.com... [ INFO ] Stopped DWH [ INFO ] Stopping service: grafana-server ... --== SUMMARY ==-- [ INFO ] Restarting httpd Web access is enabled at: http://didi-centos8-engine.lab.eng.tlv2.redhat.com:80/ovirt-engine https://didi-centos8-engine.lab.eng.tlv2.redhat.com:443/ovirt-engine Internal CA CB:13:A4:EF:94:1E:A5:98:0F:A8:F2:F4:DC:40:82:E0:AC:A7:76:9A SSH fingerprint: SHA256:pWnO/y/NadPul/cpmnt5uHrZ0F/WYQuS3kViyNr+pm4 [WARNING] Less than 16384MB of memory is available Please start the DWH service on the host 'didi-rhel7-dwh.eng.lab.tlv.redhat.com'. ========================================================================================= So we should better document this - after upgrading such an engine, user should manually start dwh on its machine. Also clarifying that this does not happen automatically. We have, in systemd conf [1]: Restart=on-failure RestartSec=3600 But the automatic stopping is not a failure, so systemd will not try to restart. [1] /usr/lib/systemd/system/ovirt-engine-dwhd.service
In 4.4, we support the following on one machine: DWHDB + DWH service + Grafana. We do NOT support any other configuration. So I am only worried about scenario 1 in comment 5, but not scenario 2. BUT, Martin, do we need to consider the following scenario: BEFORE: RHV 4.3 installed with Engine on machine A, DWH on machine B, DWH DB on machine C END GOAL: RHV 4.4 installed with Engine on machine A, DWH and DWH DB and Grafana on machine B (machine C is no longer required) ?
(In reply to Steve Goodman from comment #7) > In 4.4, we support the following on one machine: DWHDB + DWH service + > Grafana. We do NOT support any other configuration. AFAIU this proposition is expressed only in a single Note at the beginning of [1]. The rest of this section seems mostly identical to 4.3, including considerations for using a DB on its own machine. [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/data_warehouse_guide/installing_and_configuring_data_warehouse_on_a_separate_machine_dwh_admin > So I am only worried > about scenario 1 in comment 5, but not scenario 2. > > BUT, Martin, do we need to consider the following scenario: > > BEFORE: RHV 4.3 installed with Engine on machine A, DWH on machine B, DWH DB > on machine C > END GOAL: RHV 4.4 installed with Engine on machine A, DWH and DWH DB and > Grafana on machine B (machine C is no longer required) > > ? If we are going to stand behind our above decision, it seems like we have no choice, right? We must document how to do such an upgrade. The procedure for it should be mostly as (2.) of comment 5 (or (3.), if people really want, but I'd not repeat this here), with these changes, more-or-less: 2.2. should be replaced with something like 'Decommission C', and its current content should be moved to 2.3 (on machine B) 2.4. The restore command should include --change-dwh-db-credentials, and other options as applicable - at least, probably, '--dwh-db-host=localhost'. In particular: There is currently no way to save the user from having to do all the work in (2.2.) etc. by passing '--provision-dwh-db' - engine-backup will refuse to continue. Feel free to open a bug on engine-backup if we want to add this functionality - it should not be hard. After the restore command, and before engine-setup, we likely need to add something like the two `sed` commands from step 1 of [2]. This needs to be verified. [2] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/data_warehouse_guide/Migrating_Data_Warehouse_to_a_Separate_Machine_DWH_admin#configuring_new_machine That said, IMO there is no problem in continuing supporting 3 machines as in comment 5's (2.). We already have lots of documentation for separate DB for the engine, and for DWH, no need to rethink all of this and actually remove it (which is what I think we should do if we do not intend to support it). Instead, I'd replace the mentioned Note in [1] with something like this: =========================================================================== The 'engine-setup' and 'engine-backup' utilities support a large set of combinations with regards to the placement of the various components - the engine itself, its database, DWH, its database, Grafana (and some others). In our testing, we found that the biggest performance improvements were achieved by separating them to two machines: one with the engine and its DB, another with the rest (DWH, its DB, grafana). Further separation to more machines did not improve performance significantly, and in some cases resulted in lower performance. Red Hat recommends to have all of them on a single machine, and if performance is expected to be problematic, separate to two machines as above. Other, more complex setups, involving separation to more than two machines, will likely work without problems, but will also likely be more complex in migration or restore scenarios - so should be thoroughly verified. ===========================================================================
(In reply to Steve Goodman from comment #7) I agree with didi in comment #8
Returned to backlog due to writing resource constraints.
Before addressing this, please first make sure bug 1919973 is addressed. It's an easy fix.also address bug
*** Bug 1846279 has been marked as a duplicate of this bug. ***
(In reply to Yedidyah Bar David from comment #4) > BTW, > the same command, in 4.4.3, following bug 1879373, will include a new scope > I add there, grafanadb). If I remember correctly, In 4.3, we didn't use Grafana. So why does this new scope exist in 4.3? And if we didn't use Grafana in 4.3, then how does this matter for upgrading from 4.3 to 4.4?
Didi, In comment 5, you back up the DWH service, but you don't back up the DWH db. Is that intentional? In other words, do you not need to back up the old DWH database? From comment 6: > So we should better document this - after upgrading such an engine, user should manually start dwh on its machine. What is the command for this? Is it: # systemctl start ovirt-engine-dwhd.service Also, we don't manually start this service anywhere else--not in the Installation Guide [1] or the Data Warehouse Guide [2]. Why is that? And why do we need to do it here? [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/installing_red_hat_virtualization_as_a_standalone_manager_with_remote_databases/index#Installing_and_Configuring_Data_Warehouse_on_a_Separate_Machine_install_RHVM [2] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/data_warehouse_guide/installing_and_configuring_data_warehouse_on_a_separate_machine_dwh_admin
(In reply to Steve Goodman from comment #14) > Didi, > > In comment 5, you back up the DWH service, but you don't back up the DWH db. > Is that intentional? In other words, do you not need to back up the old > DWH database? > > From comment 6: > > So we should better document this - after upgrading such an engine, user should manually > start dwh on its machine. > > What is the command for this? Is it: > > # systemctl start ovirt-engine-dwhd.service Yes. > > Also, we don't manually start this service anywhere else--not in the > Installation Guide [1] or the Data Warehouse Guide [2]. Why is that? And why > do we need to do it here? Let me clarify. The context of comment 6 is: 1. You have engine and dwh on separate machines (with dwh db on same machine as dwh, or different one, does not matter). 2. You run engine-setup on the _engine_ machine. Normally you do this on routine z-stream upgrades, but you might have other reasons to do that. One such reason is as part of current procedure - before you start it, you upgrade the engine machine, and this includes running 'engine-setup'. In this case, your DWH machine is 4.3, and your engine machine is already 4.4, but I think the following will still work. When you do this, engine-setup "asks" dwh, through the DB, even though it runs on a separate machine, to stop. If all goes well, it will stop there, and engine-setup will continue, with output as in comment 6. It will also tell you to start dwh on its machine. Then, the user should manually start dwh, with the above command. Now: For current procedure, that's not really that important/relevant, because after you upgrade the engine machine, you most likely continue and upgrade the DWH machine. So you should not care that much if dwh service is up or down. But in principle, you might want/need to have a delay between these two parts - keep dwh on 4.3 for some more time after upgrading the engine - and then this point does apply. Bottom line: we should probably open another bug for this... I think it's best to add a note somewhere, after running engine-setup, in the guide for installing dwh on separate machine, or in the procedure to migrate dwh to a separate machine, or both, and also to current bug, after upgrading the engine machine. I now noticed that this is actually missing here - "4.4. Upgrading the Red Hat Virtualization Manager from 4.3 to 4.4" finishes with 'engine-backup --mode=restore', without running engine-setup. Please add it and then the note. Can be in a separate bug/patch, though.
(In reply to Yedidyah Bar David from comment #16) > > Also, we don't manually start this service anywhere else--not in the > > Installation Guide [1] or the Data Warehouse Guide [2]. Why is that? And why > > do we need to do it here? > > Let me clarify. The context of comment 6 is: > > 1. You have engine and dwh on separate machines (with dwh db on same machine > as dwh, or different one, does not matter). > > 2. You run engine-setup on the _engine_ machine. > > Normally you do this on routine z-stream upgrades, but you might have other > reasons to do that. > > One such reason is as part of current procedure - before you start it, you > upgrade the engine machine, and this includes running 'engine-setup'. In > this case, your DWH machine is 4.3, and your engine machine is already 4.4, > but I think the following will still work. > > When you do this, engine-setup "asks" dwh, through the DB, even though it > runs on a separate machine, to stop. If all goes well, it will stop there, > and engine-setup will continue, with output as in comment 6. It will also > tell you to start dwh on its machine. > > Then, the user should manually start dwh, with the above command. > > Now: For current procedure, that's not really that important/relevant, > because after you upgrade the engine machine, you most likely continue and > upgrade the DWH machine. So you should not care that much if dwh service is > up or down. But in principle, you might want/need to have a delay between > these two parts - keep dwh on 4.3 for some more time after upgrading the > engine - and then this point does apply. > > Bottom line: we should probably open another bug for this... I think it's > best to add a note somewhere, after running engine-setup, in the guide for > installing dwh on separate machine, or in the procedure to migrate dwh to a > separate machine, or both, and also to current bug, after upgrading the > engine machine. Can you please suggest wording for the note you're talking about? It's not clear to me. > I now noticed that this is actually missing here - "4.4. > Upgrading the Red Hat Virtualization Manager from 4.3 to 4.4" finishes with > 'engine-backup --mode=restore', without running engine-setup. Please add it > and then the note. Can be in a separate bug/patch, though. I added another step to the topic to run engine-setup. Please approve.
(In reply to Steve Goodman from comment #19) > Can you please suggest wording for the note you're talking about? It's not > clear to me. The note should be something like: ======================================================================= 'engine-setup' in the last step should also stop DWH on its machine. If you intend to postpone the next parts of this procedure, you should start dwh on the dwh machine: # systemctl start ovirt-engine-dwhd.service ======================================================================= Now, where to add it? That's slightly more complex: 1. In the procedure for the current bug, I think after the sentence: 'Now you need to upgrade the remote databases in your environment'. 2. In the upgrade guide, there are very few mentions of a remote DWH machine, and none of them are around "Updates between minor releases", AFAICT. The section "Upgrading the Manager from 4.2 to 4.3" has [1]. We should add something like [1] also in other places, including "Updates between minor releases". There, we should also add the above note, after 'engine-setup' on the engine machine. If you decide to try to update all applicable places (basically every place with two machines where engine-setup is used to update stuff), then do not add the above note as-is for places that deal with versions < 4.4, as the behavior was different then. [1] Follow these same steps when upgrading any of the following: - the Red Hat Virtualization Manager - a remote machine with the Data Warehouse service > > > I now noticed that this is actually missing here - "4.4. > > Upgrading the Red Hat Virtualization Manager from 4.3 to 4.4" finishes with > > 'engine-backup --mode=restore', without running engine-setup. Please add it > > and then the note. Can be in a separate bug/patch, though. > > I added another step to the topic to run engine-setup. Please approve. Thanks, looks good. I'll also comment on the MR.
Moving to ON_QA. Gui, please test this. Although this is targeting 4.4.8, this is left-over from 4.4.7, so if you are able to, please prioritize this.
Peer review completed.
Gui just to be clear, the new procedure to test is in this MR: Please review the merge request: https://gitlab.cee.redhat.com/rhci-documentation/docs-Red_Hat_Enterprise_Virtualization/-/merge_requests/2012 Preview is here [1]. Please see both how new topic fits into the flow [1], and also review the new topic itself: "4.5. Upgrading the remote Data Warehouse service and database" [2]. [1] https://jenkins.dxp.redhat.com/job/CCS/job/ccs-mr-preview/4606/artifact/doc-Upgrade_Guide/preview/index.html#Remote_Upgrading_from_4-3 [2] https://jenkins.dxp.redhat.com/job/CCS/job/ccs-mr-preview/4604/artifact/doc-Upgrade_Guide/preview/index.html#proc_upgrading-the-remote-data-warehouse-service-and-database_4-3_remote_db
The steps of the preview from comment #24 for the upgrading scenarios 1 of comment #5 are tested and validated So verifying