Created attachment 1555596 [details] ovirt engine logs Description of problem: Can't login to web admin portal after ovirt update 4.3.2 to 4.3.3.5 Version-Release number of selected component (if applicable): [root@ovirt-eng ovirt-engine]# rpm -qa --last|more ovirt-release43-4.3.3-1.el7.noarch Tue 16 Apr 2019 05:46:40 PM MSK v2v-conversion-host-ansible-1.12.1-1.el7.noarch Tue 16 Apr 2019 05:46:39 PM MSK ovirt-ansible-hosted-engine-setup-1.0.17-1.el7.noarch Tue 16 Apr 2019 05:46:39 PM MSK ovirt-engine-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:38 PM MSK ovirt-engine-dbscripts-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:36 PM MSK ovirt-engine-tools-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:34 PM MSK ovirt-engine-webadmin-portal-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:33 PM MSK ovirt-engine-ui-extensions-1.0.4-1.el7.noarch Tue 16 Apr 2019 05:46:14 PM MSK ovirt-engine-restapi-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:12 PM MSK ovirt-engine-backend-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:10 PM MSK ovirt-engine-setup-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:07 PM MSK ovirt-engine-setup-plugin-ovirt-engine-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:06 PM MSK ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:05 PM MSK ovirt-engine-vmconsole-proxy-helper-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:04 PM MSK ovirt-engine-setup-plugin-cinderlib-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:03 PM MSK ovirt-ansible-cluster-upgrade-1.1.13-1.el7.noarch Tue 16 Apr 2019 05:46:03 PM MSK ovirt-web-ui-1.5.2-1.el7.noarch Tue 16 Apr 2019 05:46:02 PM MSK ovirt-engine-extensions-api-impl-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:46:00 PM MSK ovirt-engine-metrics-1.3.0.2-1.el7.noarch Tue 16 Apr 2019 05:45:59 PM MSK ovirt-iso-uploader-4.3.1-1.el7.noarch Tue 16 Apr 2019 05:45:58 PM MSK python-ovirt-engine-sdk4-4.3.1-2.el7.x86_64 Tue 16 Apr 2019 05:45:56 PM MSK ovirt-engine-websocket-proxy-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:55 PM MSK ovirt-engine-tools-backup-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:54 PM MSK ovirt-engine-setup-plugin-websocket-proxy-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:54 PM MSK ovirt-engine-setup-plugin-ovirt-engine-common-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:53 PM MSK ovirt-engine-setup-base-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:52 PM MSK python2-ovirt-engine-lib-4.3.3.5-1.el7.noarch Tue 16 Apr 2019 05:45:51 PM MSK How reproducible: install updates and login to https://ovirt-eng.domain.ru Steps to Reproduce: 1. 2. 3. Actual results: Can't login to web admin portal. Expected results: success login to web admin portal Additional info: I think root cause is "Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select * from getallfromvds(?, ?)]; nested exception is org.postgresql.util.PSQLException: The column name supported_domain_versions was not found in this ResultSet." in engine.log. DB schema was not updated?
plus, every 10min ovirt engine host automatically reboots
Was engine-setup run after updating the packages? If not please run engine-setup
[root@ovirt-eng ~]# date Tue Apr 23 10:54:11 MSK 2019 [root@ovirt-eng ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423105211-3mndsd.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] ovirt-provider-ovn already installed, skipping. --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] No product updates found --== NETWORK CONFIGURATION ==-- Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. NOTICE: iptables is deprecated and will be removed in future releases Do you want Setup to configure the firewall? (Yes, No) [Yes]: [ INFO ] firewalld will be configured as firewall manager. --== DATABASE CONFIGURATION ==-- The detected DWH database size is 80 MB. Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete. If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: Perform full vacuum on the oVirt engine history database ovirt_engine_history@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html (Yes, No) [No]: Yes --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: Yes --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423105210-3mndsd.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190423105348-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed [root@ovirt-eng ~]
Created attachment 1557456 [details] engine-setup logs
Please follow instructions here https://www.ovirt.org/documentation/self-hosted/chap-Maintenance_and_Upgrading_Resources.html to set global maintenance mode before upgrading.
# On this host "ovirt-engine" running [root@vsrvlab02-2 ~]# hosted-engine --set-maintenance --mode=global [root@vsrvlab02-2 ~]# # !!! default answers [root@ovirt-eng ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423154528-i7cdyg.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] ovirt-provider-ovn already installed, skipping. --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] No product updates found --== NETWORK CONFIGURATION ==-- Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. NOTICE: iptables is deprecated and will be removed in future releases Do you want Setup to configure the firewall? (Yes, No) [Yes]: [ INFO ] firewalld will be configured as firewall manager. --== DATABASE CONFIGURATION ==-- The detected DWH database size is 80 MB. Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete. If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: Perform full vacuum on the oVirt engine history database ovirt_engine_history@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html (Yes, No) [No]: --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423154528-i7cdyg.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190423154726-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed [root@ovirt-eng ~]# # "Yes" to all questions [root@ovirt-eng ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423154812-xvlclb.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] ovirt-provider-ovn already installed, skipping. --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] No product updates found --== NETWORK CONFIGURATION ==-- Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. NOTICE: iptables is deprecated and will be removed in future releases Do you want Setup to configure the firewall? (Yes, No) [Yes]: [ INFO ] firewalld will be configured as firewall manager. --== DATABASE CONFIGURATION ==-- The detected DWH database size is 80 MB. Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete. If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: Perform full vacuum on the oVirt engine history database ovirt_engine_history@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html (Yes, No) [No]: Yes --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: Yes --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190423154812-xvlclb.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190423154857-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed [root@ovirt-eng ~]#
Created attachment 1557652 [details] engine-setup logs "default answers"
Created attachment 1557653 [details] engine-setup logs "Yes" to any questions
Sandro, could you please take a look why setting global maintenance didn't work?
(In reply to Martin Perina from comment #9) > Sandro, could you please take a look why setting global maintenance didn't > work? Redirecting the question to Simone. Not sure that the engine has seen the global maintenance flag being set on the storage if engine-setup has been executed immediately after setting the maintenance to global.
Yes, global-maintenance flag is immediately set by the host on the shared storage. engine-setup instead checks a copy of if in the the engine DB from the info periodically polled from the host so we can have a 20-30 seconds delay to get the info there.
Maintenance state over 1 week. What should I do? [root@vsrvlab02-1 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host vsrvlab02-1.domain.ru (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : vsrvlab02-1.domain.ru Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 41073213 local_conf_timestamp : 2341857 Host timestamp : 2341856 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2341856 (Tue May 7 17:29:34 2019) host-id=1 score=3400 vm_conf_refresh_time=2341857 (Tue May 7 17:29:35 2019) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False --== Host vsrvlab02-2.domain.ru (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : vsrvlab02-2.domain.ru Host ID : 2 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : fd57c08a local_conf_timestamp : 2394370 Host timestamp : 2394370 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2394370 (Tue May 7 17:29:26 2019) host-id=2 score=3400 vm_conf_refresh_time=2394370 (Tue May 7 17:29:26 2019) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! [root@vsrvlab02-1 ~]# [root@ovirt-eng ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20190507173343-m045ve.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] ovirt-provider-ovn already installed, skipping. --== PACKAGES ==-- [ INFO ] Checking for product updates... Setup needs to install or update the following packages: [updated] ovirt-engine-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-backend-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-backend-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-dbscripts-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-dbscripts-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-extensions-api-impl-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-extensions-api-impl-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-restapi-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-restapi-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-base-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-base-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-plugin-cinderlib-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-plugin-cinderlib-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-plugin-ovirt-engine-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-plugin-ovirt-engine-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-plugin-ovirt-engine-common-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-plugin-ovirt-engine-common-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-setup-plugin-websocket-proxy-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-setup-plugin-websocket-proxy-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-tools-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-tools-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-tools-backup-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-tools-backup-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-vmconsole-proxy-helper-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-vmconsole-proxy-helper-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-webadmin-portal-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-webadmin-portal-4.3.3.7-1.el7.noarch is an update [updated] ovirt-engine-websocket-proxy-4.3.3.6-1.el7.noarch will be updated [update] ovirt-engine-websocket-proxy-4.3.3.7-1.el7.noarch is an update [updated] python2-ovirt-engine-lib-4.3.3.6-1.el7.noarch will be updated [update] python2-ovirt-engine-lib-4.3.3.7-1.el7.noarch is an update Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages. Do you wish to update them now? (Yes, No) [Yes]: no [ ERROR ] Failed to execute stage 'Environment customization': Aborted, packages must be updated. You can pass "--offline" to prevent checking for updates. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190507173343-m045ve.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190507173408-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed [root@ovirt-eng ~]# [root@ovirt-eng ~]# [root@ovirt-eng ~]# engine-setup --offline [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20190507173411-45yooe.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] ovirt-provider-ovn already installed, skipping. --== PACKAGES ==-- --== NETWORK CONFIGURATION ==-- Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. NOTICE: iptables is deprecated and will be removed in future releases Do you want Setup to configure the firewall? (Yes, No) [Yes]: [ INFO ] firewalld will be configured as firewall manager. --== DATABASE CONFIGURATION ==-- The detected DWH database size is 61 MB. Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete. If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: Perform full vacuum on the oVirt engine history database ovirt_engine_history@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html (Yes, No) [No]: --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190507173411-45yooe.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190507173421-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed [root@ovirt-eng ~]#
Starting the engine and waiting a couple of minutes to have it updating its DB should be enough. If you cannot start the engine, you can execute: engine-setup --otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True to completely skip that check.
(In reply to Simone Tiraboschi from comment #13) > Starting the engine and waiting a couple of minutes to have it updating its > DB should be enough. > > If you cannot start the engine, you can execute: > > engine-setup > --otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True > > to completely skip that check. This works for me, thanks!
Let's document the command line for skipping the test on the error message we output when we detect we are not in global maintenance letting the user know the test can be skipped if the system has been moved to global maintenance but not yet known by the engine itself.
Moved out to 4.4 being low severity
(In reply to Sandro Bonazzola from comment #17) > Let's document the command line for skipping the test on the error message > we output when we detect we are not in global maintenance letting the user > know the test can be skipped if the system has been moved to global > maintenance but not yet known by the engine itself. Where do you want to document this? This isn't a "command line option" but an "answer file item", and we do not have a document for these. Creating one was rejected in the past, in bug 1414915. I agree that comment 13 is not really in the scope of that bug, because it's not "answering a question" - you can't generate an answer file that includes this item by running engine-setup manually and using its generated file. Perhaps it's about time we do document it. The most important reason for not writing such a doc, IMO, is that it's quite likely to not be updated, and having wrong docs is probably worse than no docs.
> (In reply to Sandro Bonazzola from comment #17) > Where do you want to document this? This isn't a "command line option" but > an "answer file item", and we do not have a document for these. Creating one > was rejected in the past, in bug 1414915. I think we can add a note in the engine-setup man page (https://github.com/oVirt/ovirt-engine/blob/master/packaging/man/man8/engine-setup.8) which also completely miss --otopi-environment flag. Or perhaps extend the error text: "[ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data." By adding something like: "If the system is in Global Maintenance mode but the tool is still reporting it as not in maintenance, you can skip this check re-running engine-setup with an additional option: `--otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True`
(In reply to Sandro Bonazzola from comment #20) > > I think we can add a note in the engine-setup man page > (https://github.com/oVirt/ovirt-engine/blob/master/packaging/man/man8/engine- > setup.8) Decided not to, see below > which also completely miss > --otopi-environment flag. Indeed. I wonder if we should document it, and especially how to do this. We already have: https://www.ovirt.org/develop/developer-guide/engine/engine-setup.html https://www.ovirt.org/develop/developer-guide/engine/otopi.html Perhaps we should have added also the request for current bug to one of them instead of patching the code, nvm. If we do want to do this properly, we should probably: 1. Make help/man texts generated somehow, to make it easier to keep them up-to-date 2. Make the option parsing code partially common between the tools, so that we do not have to duplicate e.g. '--otopi-environment' code/help/man among the tools > > > Or perhaps extend the error text: > "[ ERROR ] It seems that you are running your engine inside of the > hosted-engine VM and are not in "Global Maintenance" mode. > In that case you should put the system into the "Global > Maintenance" mode before running engine-setup, or the hosted-engine HA agent > might kill the machine, which might corrupt your data." > > By adding something like: > > "If the system is in Global Maintenance mode but the tool is still reporting > it as not in maintenance, you can skip this check re-running engine-setup > with an additional option: > `--otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True` 118403 does the latter, slightly changing the suggested text. I verified that this check runs also in other tools, e.g. engine-cleanup, and that's why I decided to not change help/man (too much work) nor mention 'engine-setup' in the error (misleading/confusing).
With PR 136 merged, we have: # engine-setup --help | grep environment --show-environment-documentation Show documentation about used environment keys. And # engine-setup --show-environment-documentation | grep -B1 maintenance * OVESETUP_CONFIG/continueSetupOnHEVM - doc_text : Allow overriding/ignoring the check for hosted-engine global maintenance. Use with care! This is the only key (other than OVESETUP_DOCS/showDocText which was added for this feature) that has doc_text right now. It should be easy to add more later, if wished. I do not promise the output of '--show-environment-documentation' to be stable. I only spent a short time styling it, and it might be changed in the future. Please do not rely on it for automation. In particular, it's not clear how well it will look if we want longer doc texts (more than one line per a specific item).
This bugzilla is included in oVirt 4.5.0 release, published on April 20th 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.