Bug 1295178

Summary: Engine setup fails due to pending RestoreAllSnapshots tasks (type 224)
Product: [oVirt] ovirt-engine Reporter: Nikolai Sednev <nsednev>
Component: Setup.EngineAssignee: Yedidyah Bar David <didi>
Status: CLOSED DUPLICATE QA Contact: Gonza <grafuls>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.1.3CC: bugs, derez, didi, lveyde, mlipchuk, nsednev, oourfali, rmartins, sbonazzo, stirabos, tnisan, ylavi
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
pstehlik: testing_ack+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
URL: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88WVhFLTlfSHVodUU/view?usp=sharing
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-13 12:49:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1290528    
Bug Blocks: 1294361    
Attachments:
Description Flags
logs from the engine setup failure.tar.gz none

Description Nikolai Sednev 2016-01-03 09:42:58 UTC
Description of problem:
Engine setup fails with an [ ERROR ] Failed to execute stage 'Setup validation': '224'.
# yum update rhevm* -y
Loaded plugins: product-id, security, subscription-manager, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Repository jboss-eap-6.4.Z is listed more than once in the configuration
Repository jboss-eap-6.4.Z is listed more than once in the configuration
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package rhevm-dwh-setup.noarch 0:3.6.1-1.el6ev will be updated
---> Package rhevm-dwh-setup.noarch 0:3.6.2-1.el6ev will be an update
---> Package rhevm-extensions-api-impl.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-extensions-api-impl.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-lib.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-lib.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-sdk-python.noarch 0:3.6.0.3-1.el6ev will be updated
---> Package rhevm-sdk-python.noarch 0:3.6.2.0-1.el6ev will be an update
---> Package rhevm-setup.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-setup-base.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup-base.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-setup-plugin-ovirt-engine.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup-plugin-ovirt-engine.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-setup-plugin-ovirt-engine-common.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup-plugin-ovirt-engine-common.noarch 0:3.6.2-0.1.el6 will be an update
--> Processing Dependency: ovirt-setup-lib for package: rhevm-setup-plugin-ovirt-engine-common-3.6.2-0.1.el6.noarch
---> Package rhevm-setup-plugin-vmconsole-proxy-helper.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup-plugin-vmconsole-proxy-helper.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-setup-plugin-websocket-proxy.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-setup-plugin-websocket-proxy.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-vmconsole-proxy-helper.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-vmconsole-proxy-helper.noarch 0:3.6.2-0.1.el6 will be an update
---> Package rhevm-websocket-proxy.noarch 0:3.6.1.3-0.1.el6 will be updated
---> Package rhevm-websocket-proxy.noarch 0:3.6.2-0.1.el6 will be an update
--> Running transaction check
---> Package ovirt-setup-lib.noarch 0:1.0.1-1.el6ev will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================================
 Package                                                  Arch                  Version                         Repository                   Size
==================================================================================================================================================
Updating:
 rhevm-dwh-setup                                          noarch                3.6.2-1.el6ev                   rhev-3.6.2-3                 65 k
 rhevm-extensions-api-impl                                noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 38 k
 rhevm-lib                                                noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 22 k
 rhevm-sdk-python                                         noarch                3.6.2.0-1.el6ev                 rhev-3.6.2-3                517 k
 rhevm-setup                                              noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                2.5 k
 rhevm-setup-base                                         noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 98 k
 rhevm-setup-plugin-ovirt-engine                          noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                168 k
 rhevm-setup-plugin-ovirt-engine-common                   noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 75 k
 rhevm-setup-plugin-vmconsole-proxy-helper                noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 19 k
 rhevm-setup-plugin-websocket-proxy                       noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 20 k
 rhevm-vmconsole-proxy-helper                             noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 11 k
 rhevm-websocket-proxy                                    noarch                3.6.2-0.1.el6                   rhev-3.6.2-3                 11 k
Installing for dependencies:
 ovirt-setup-lib                                          noarch                1.0.1-1.el6ev                   rhev-3.6.2-3                 18 k

Transaction Summary
==================================================================================================================================================
Install       1 Package(s)
Upgrade      12 Package(s)

Total download size: 1.0 M
Downloading Packages:
(1/13): ovirt-setup-lib-1.0.1-1.el6ev.noarch.rpm                                                                           |  18 kB     00:00     
(2/13): rhevm-dwh-setup-3.6.2-1.el6ev.noarch.rpm                                                                           |  65 kB     00:00     
(3/13): rhevm-extensions-api-impl-3.6.2-0.1.el6.noarch.rpm                                                                 |  38 kB     00:00     
(4/13): rhevm-lib-3.6.2-0.1.el6.noarch.rpm                                                                                 |  22 kB     00:00     
(5/13): rhevm-sdk-python-3.6.2.0-1.el6ev.noarch.rpm                                                                        | 517 kB     00:00     
(6/13): rhevm-setup-3.6.2-0.1.el6.noarch.rpm                                                                               | 2.5 kB     00:00     
(7/13): rhevm-setup-base-3.6.2-0.1.el6.noarch.rpm                                                                          |  98 kB     00:00     
(8/13): rhevm-setup-plugin-ovirt-engine-3.6.2-0.1.el6.noarch.rpm                                                           | 168 kB     00:00     
(9/13): rhevm-setup-plugin-ovirt-engine-common-3.6.2-0.1.el6.noarch.rpm                                                    |  75 kB     00:00     
(10/13): rhevm-setup-plugin-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch.rpm                                                |  19 kB     00:00     
(11/13): rhevm-setup-plugin-websocket-proxy-3.6.2-0.1.el6.noarch.rpm                                                       |  20 kB     00:00     
(12/13): rhevm-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch.rpm                                                             |  11 kB     00:00     
(13/13): rhevm-websocket-proxy-3.6.2-0.1.el6.noarch.rpm                                                                    |  11 kB     00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                             3.7 MB/s | 1.0 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : rhevm-lib-3.6.2-0.1.el6.noarch                                                                                                1/25 
  Updating   : rhevm-setup-base-3.6.2-0.1.el6.noarch                                                                                         2/25 
  Installing : ovirt-setup-lib-1.0.1-1.el6ev.noarch                                                                                          3/25 
  Updating   : rhevm-setup-plugin-ovirt-engine-common-3.6.2-0.1.el6.noarch                                                                   4/25 
  Updating   : rhevm-setup-plugin-ovirt-engine-3.6.2-0.1.el6.noarch                                                                          5/25 
  Updating   : rhevm-setup-plugin-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch                                                                6/25 
  Updating   : rhevm-setup-plugin-websocket-proxy-3.6.2-0.1.el6.noarch                                                                       7/25 
  Updating   : rhevm-websocket-proxy-3.6.2-0.1.el6.noarch                                                                                    8/25 
  Updating   : rhevm-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch                                                                             9/25 
  Updating   : rhevm-setup-3.6.2-0.1.el6.noarch                                                                                             10/25 
  Updating   : rhevm-dwh-setup-3.6.2-1.el6ev.noarch                                                                                         11/25 
  Updating   : rhevm-sdk-python-3.6.2.0-1.el6ev.noarch                                                                                      12/25 
  Updating   : rhevm-extensions-api-impl-3.6.2-0.1.el6.noarch                                                                               13/25 
  Cleanup    : rhevm-websocket-proxy-3.6.1.3-0.1.el6.noarch                                                                                 14/25 
  Cleanup    : rhevm-vmconsole-proxy-helper-3.6.1.3-0.1.el6.noarch                                                                          15/25 
  Cleanup    : rhevm-setup-plugin-vmconsole-proxy-helper-3.6.1.3-0.1.el6.noarch                                                             16/25 
  Cleanup    : rhevm-setup-plugin-websocket-proxy-3.6.1.3-0.1.el6.noarch                                                                    17/25 
  Cleanup    : rhevm-setup-3.6.1.3-0.1.el6.noarch                                                                                           18/25 
  Cleanup    : rhevm-setup-plugin-ovirt-engine-3.6.1.3-0.1.el6.noarch                                                                       19/25 
  Cleanup    : rhevm-dwh-setup-3.6.1-1.el6ev.noarch                                                                                         20/25 
  Cleanup    : rhevm-setup-plugin-ovirt-engine-common-3.6.1.3-0.1.el6.noarch                                                                21/25 
  Cleanup    : rhevm-setup-base-3.6.1.3-0.1.el6.noarch                                                                                      22/25 
  Cleanup    : rhevm-lib-3.6.1.3-0.1.el6.noarch                                                                                             23/25 
  Cleanup    : rhevm-sdk-python-3.6.0.3-1.el6ev.noarch                                                                                      24/25 
  Cleanup    : rhevm-extensions-api-impl-3.6.1.3-0.1.el6.noarch                                                                             25/25 
  Verifying  : rhevm-setup-base-3.6.2-0.1.el6.noarch                                                                                         1/25 
  Verifying  : rhevm-lib-3.6.2-0.1.el6.noarch                                                                                                2/25 
  Verifying  : rhevm-setup-plugin-ovirt-engine-3.6.2-0.1.el6.noarch                                                                          3/25 
  Verifying  : rhevm-extensions-api-impl-3.6.2-0.1.el6.noarch                                                                                4/25 
  Verifying  : rhevm-setup-plugin-websocket-proxy-3.6.2-0.1.el6.noarch                                                                       5/25 
  Verifying  : rhevm-setup-plugin-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch                                                                6/25 
  Verifying  : ovirt-setup-lib-1.0.1-1.el6ev.noarch                                                                                          7/25 
  Verifying  : rhevm-dwh-setup-3.6.2-1.el6ev.noarch                                                                                          8/25 
  Verifying  : rhevm-setup-3.6.2-0.1.el6.noarch                                                                                              9/25 
  Verifying  : rhevm-sdk-python-3.6.2.0-1.el6ev.noarch                                                                                      10/25 
  Verifying  : rhevm-setup-plugin-ovirt-engine-common-3.6.2-0.1.el6.noarch                                                                  11/25 
  Verifying  : rhevm-vmconsole-proxy-helper-3.6.2-0.1.el6.noarch                                                                            12/25 
  Verifying  : rhevm-websocket-proxy-3.6.2-0.1.el6.noarch                                                                                   13/25 
  Verifying  : rhevm-setup-3.6.1.3-0.1.el6.noarch                                                                                           14/25 
  Verifying  : rhevm-websocket-proxy-3.6.1.3-0.1.el6.noarch                                                                                 15/25 
  Verifying  : rhevm-setup-plugin-ovirt-engine-common-3.6.1.3-0.1.el6.noarch                                                                16/25 
  Verifying  : rhevm-vmconsole-proxy-helper-3.6.1.3-0.1.el6.noarch                                                                          17/25 
  Verifying  : rhevm-lib-3.6.1.3-0.1.el6.noarch                                                                                             18/25 
  Verifying  : rhevm-setup-plugin-ovirt-engine-3.6.1.3-0.1.el6.noarch                                                                       19/25 
  Verifying  : rhevm-setup-base-3.6.1.3-0.1.el6.noarch                                                                                      20/25 
  Verifying  : rhevm-setup-plugin-vmconsole-proxy-helper-3.6.1.3-0.1.el6.noarch                                                             21/25 
  Verifying  : rhevm-sdk-python-3.6.0.3-1.el6ev.noarch                                                                                      22/25 
  Verifying  : rhevm-extensions-api-impl-3.6.1.3-0.1.el6.noarch                                                                             23/25 
  Verifying  : rhevm-setup-plugin-websocket-proxy-3.6.1.3-0.1.el6.noarch                                                                    24/25 
  Verifying  : rhevm-dwh-setup-3.6.1-1.el6ev.noarch                                                                                         25/25 

Dependency Installed:
  ovirt-setup-lib.noarch 0:1.0.1-1.el6ev                                                                                                          

Updated:
  rhevm-dwh-setup.noarch 0:3.6.2-1.el6ev                                    rhevm-extensions-api-impl.noarch 0:3.6.2-0.1.el6                      
  rhevm-lib.noarch 0:3.6.2-0.1.el6                                          rhevm-sdk-python.noarch 0:3.6.2.0-1.el6ev                             
  rhevm-setup.noarch 0:3.6.2-0.1.el6                                        rhevm-setup-base.noarch 0:3.6.2-0.1.el6                               
  rhevm-setup-plugin-ovirt-engine.noarch 0:3.6.2-0.1.el6                    rhevm-setup-plugin-ovirt-engine-common.noarch 0:3.6.2-0.1.el6         
  rhevm-setup-plugin-vmconsole-proxy-helper.noarch 0:3.6.2-0.1.el6          rhevm-setup-plugin-websocket-proxy.noarch 0:3.6.2-0.1.el6             
  rhevm-vmconsole-proxy-helper.noarch 0:3.6.2-0.1.el6                       rhevm-websocket-proxy.noarch 0:3.6.2-0.1.el6                          

Complete!
[root@Upgrade_setup ~]# rhevm-dwh-setup
-bash: rhevm-dwh-setup: command not found
[root@Upgrade_setup ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-dwh.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-packaging-rhevm-reports.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20160103112652-bjikek.log
          Version: otopi-1.4.0 (otopi-1.4.0-1.el6ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Stage: Environment customization
         
          Welcome to the RHEV 3.6 setup/upgrade.
          Please read the RHEV 3.6 install guide
          https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/index.html.
          Please refer to the RHEV Upgrade Helper application
          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.
          Would you like to proceed? (Yes, No) [Yes]: 
         
          --== PRODUCT OPTIONS ==--
         
          Configure VM Console Proxy on this host (Yes, No) [Yes]: 
         
          --== PACKAGES ==--
         
[ INFO  ] Checking for product updates...
          Setup has found updates for some packages:
          PACKAGE: [updated] rhevm-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-backend-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-backend-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-dbscripts-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-dbscripts-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-dwh-3.6.1-1.el6ev.noarch
          PACKAGE: [update] rhevm-dwh-3.6.2-1.el6ev.noarch
          PACKAGE: [updated] rhevm-restapi-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-restapi-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-tools-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-tools-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-userportal-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-userportal-3.6.2-0.1.el6.noarch
          PACKAGE: [updated] rhevm-webadmin-portal-3.6.1.3-0.1.el6.noarch
          PACKAGE: [update] rhevm-webadmin-portal-3.6.2-0.1.el6.noarch
          do you wish to update them now? (Yes, No) [Yes]: 
[ INFO  ] Checking for an update for Setup...
         
          --== ALL IN ONE CONFIGURATION ==--
         
         
          --== NETWORK CONFIGURATION ==--
         
[WARNING] Host name Upgrade_setup has no domain suffix
[WARNING] Failed to resolve Upgrade_setup using DNS, it can be resolved only locally
          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]: 
[ INFO  ] iptables will be configured as firewall manager.
[WARNING] Host name Upgrade_setup has no domain suffix
[WARNING] Failed to resolve Upgrade_setup using DNS, it can be resolved only locally
         
          --== DATABASE CONFIGURATION ==--
         
          The detected DWH database size is 1.4 GB.
          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.
          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.
          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: 
         
          --== OVIRT ENGINE CONFIGURATION ==--
         
         
          --== STORAGE CONFIGURATION ==--
         
         
          --== PKI CONFIGURATION ==--
         
         
          --== APACHE CONFIGURATION ==--
         
         
          --== SYSTEM CONFIGURATION ==--
         
         
          --== MISC CONFIGURATION ==--
         
         
          --== END OF CONFIGURATION ==--
         
[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
          Generated iptables rules are different from current ones.
          Do you want to review them? (Yes, No) [No]: 
[ INFO  ] Cleaning stale zombie tasks and commands
         
          --== CONFIGURATION PREVIEW ==--
         
          Firewall manager                        : iptables
          Update Firewall                         : True
          Host FQDN                               : Upgrade_setup
          Upgrade packages                        : True
          Engine database secured connection      : False
          Engine database host                    : localhost
          Engine database user name               : engine
          Engine database name                    : engine
          Engine database port                    : 5432
          Engine database host name validation    : False
          Engine installation                     : True
          PKI organization                        : Test
          DWH installation                        : True
          DWH database secured connection         : False
          DWH database host                       : localhost
          DWH database user name                  : ovirt_engine_history
          DWH database name                       : ovirt_engine_history
          Backup DWH database                     : True
          DWH database port                       : 5432
          DWH database host name validation       : False
          Reports installation                    : False
          Configure local Reports database        : False
          Engine Host FQDN                        : Upgrade_setup
          Configure VMConsole Proxy               : True
          Configure WebSocket Proxy               : False
         
          Please confirm installation settings (OK, Cancel) [OK]: 
[WARNING] Setup found legacy kerberos/ldap directory integration in use (added by engine-manage-domains). This provider is obsoleted and will be removed in future. Please migrate to ovirt-engine-extension-aaa-ldap provider or contact support for assistance.
[ INFO  ] Cleaning async tasks and compensations
[ ERROR ] Failed to execute stage 'Setup validation': '224'
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160103112652-bjikek.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160103112931-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed


Version-Release number of selected component (if applicable):
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
rhevm-3.6.1.3-0.1.el6.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.4-1.el6ev.noarch
ovirt-vmconsole-1.0.0-1.el6ev.noarch
rhevm-reports-3.5.5-2.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch


How reproducible:
100%

Steps to Reproduce:
1.Install the 3.5.6 engine with DWH reports but don't set DWH reports (answer no during the engine-setup). 
2.Upgrade to rhevm-3.6.1.3-0.1.el6->rhev-release-3.6.2-2-001.noarch
3.Now try upgrading to rhevm-3.6.1.3-0.1.el6.noarch

Actual results:
After successful rhevm yum update, engine-setup fails with "[ ERROR ] Failed to execute stage 'Setup validation': '224' ".

Expected results:
engine-setup should pass OK.

Additional info:
Logs from engine attached.

Comment 1 Nikolai Sednev 2016-01-03 09:46:53 UTC
Created attachment 1111119 [details]
logs from the engine setup failure.tar.gz

Comment 2 Yedidyah Bar David 2016-01-03 09:49:17 UTC
likely a duplicate of bug 1290528

Comment 3 Nikolai Sednev 2016-01-03 09:51:14 UTC
Adding the sosreport from the engine https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88WVhFLTlfSHVodUU/view?usp=sharing

Comment 4 Yedidyah Bar David 2016-01-05 14:35:06 UTC
Oved, verified patch [1], see description there. But this will not fix the "root problem", will just make engine-setup fail more nicely. What can cause these 224 lines to appear there? Since Nikolai opened this bug, I see on his system two more lines, and he claims he did nothing related to snapshots (224 is RestoreAllSnapshots in VdcActionType.java). What's that? Can we safely make taskcleaner.sh clean them? Perhaps another bug/RFE is needed for that.

Comment 5 Yedidyah Bar David 2016-01-05 14:44:56 UTC
[1] https://gerrit.ovirt.org/51230

Comment 6 Oved Ourfali 2016-01-05 15:06:16 UTC
(In reply to Yedidyah Bar David from comment #4)
> Oved, verified patch [1], see description there. But this will not fix the
> "root problem", will just make engine-setup fail more nicely. What can cause
> these 224 lines to appear there? Since Nikolai opened this bug, I see on his
> system two more lines, and he claims he did nothing related to snapshots
> (224 is RestoreAllSnapshots in VdcActionType.java). What's that? Can we
> safely make taskcleaner.sh clean them? Perhaps another bug/RFE is needed for
> that.

I have no idea.
Worth asking the storage team.
Derez?

Comment 7 Tal Nisan 2016-01-06 09:57:05 UTC
QE contact for this week is Maor, he will have a look

Comment 8 Daniel Erez 2016-01-06 10:23:38 UTC
(In reply to Yedidyah Bar David from comment #4)
> Oved, verified patch [1], see description there. But this will not fix the
> "root problem", will just make engine-setup fail more nicely. What can cause
> these 224 lines to appear there? Since Nikolai opened this bug, I see on his
> system two more lines, and he claims he did nothing related to snapshots
> (224 is RestoreAllSnapshots in VdcActionType.java). What's that? Can we
> safely make taskcleaner.sh clean them? Perhaps another bug/RFE is needed for
> that.

RestoreAllSnapshots command handles undo and commit snapshot operation. You can try and check if there's an hanging task on vdsm (using 'vdsClient getAllTasks'). Or, from the webadmin, see if any snapshot is in status locked. Cleaning the tasks by taskcleaner.sh would affect the locked snapshot and fail the undo/commit operation (and it might damage the associated VM).

Comment 9 Yedidyah Bar David 2016-01-07 07:31:48 UTC
(In reply to Daniel Erez from comment #8)
> RestoreAllSnapshots command handles undo and commit snapshot operation. You
> can try and check if there's an hanging task on vdsm (using 'vdsClient
> getAllTasks'). Or, from the webadmin, see if any snapshot is in status
> locked. Cleaning the tasks by taskcleaner.sh would affect the locked
> snapshot and fail the undo/commit operation (and it might damage the
> associated VM).

Ran 'vdsClient -s 0 getAllTasks' on all of Nikolai's hosts and the output is empty for all of them. How to continue?

Changing the subject for now, instead of merely closing as duplicate of bug 1290528. The new subject assumes bug 1290528 is solved.

Tal - please take over. Nikolai can provide access to machines.

Comment 10 Tal Nisan 2016-01-12 08:44:05 UTC
Liron, you're this week's QE contact, can you have a look please?

Comment 11 Nikolai Sednev 2016-01-12 12:26:55 UTC
I've eventually succeeded upgrading the engine, by migrating it's VM to another host and then powering OFF/ON the engine's VM. I think that engine's VM was running on host with over 81% memory load, this badly influenced the engine's VM and caused for these problems. It looks like as more performance issue.
I don't have this problem any more on my setup now.

Comment 12 Yaniv Lavi 2016-01-13 12:49:46 UTC
This is by design. It is risky to clean this task and we should wait and fail on timeout.

Comment 13 Yedidyah Bar David 2016-01-13 13:01:15 UTC
(In reply to Yaniv Dary from comment #12)
> This is by design. It is risky to clean this task and we should wait and
> fail on timeout.

Isn't it possible that due to some bug or something, such a task will remain hanging? I looked at Nikolai's setup over more than a day and some were not cleared.

Also, we currently do not timeout afaik, but try endlessly (if user says 'yes').

Comment 14 Yaniv Lavi 2016-01-13 13:02:58 UTC
(In reply to Yedidyah Bar David from comment #13)
> (In reply to Yaniv Dary from comment #12)
> > This is by design. It is risky to clean this task and we should wait and
> > fail on timeout.
> 
> Isn't it possible that due to some bug or something, such a task will remain
> hanging? I looked at Nikolai's setup over more than a day and some were not
> cleared.
> 
> Also, we currently do not timeout afaik, but try endlessly (if user says
> 'yes').

Then we need a different bug on storage to cover that.

Comment 15 Yedidyah Bar David 2016-01-13 13:26:42 UTC
(In reply to Yaniv Dary from comment #14)
> Then we need a different bug on storage to cover that.

Very well. Nikolai - can you please open one? Thanks.

*** This bug has been marked as a duplicate of bug 1290528 ***

Comment 16 Nikolai Sednev 2016-01-13 14:48:39 UTC
It's a bit problematic to open a new bug on storage as I can't provide reproduction for this as it happened during very complicated upgrade on our upgrade setup, there could be many triggers for snapshot was running on background and then other tasks were running on the background. Should logs provided within this bug be sufficient for opening a new bug?

Comment 17 Yedidyah Bar David 2016-01-13 14:59:50 UTC
(In reply to Nikolai Sednev from comment #16)
> It's a bit problematic to open a new bug on storage as I can't provide
> reproduction for this as it happened during very complicated upgrade on our
> upgrade setup, there could be many triggers for snapshot was running on
> background and then other tasks were running on the background. Should logs
> provided within this bug be sufficient for opening a new bug?

I have no idea, I guess you'll have to simply try :-)

If the storage people will need more information that you'll not be able to supply, they can always close insufficient_data.

You do not have any other logs? engine/vdsm?

Comment 18 Nikolai Sednev 2016-01-13 15:19:57 UTC
(In reply to Yedidyah Bar David from comment #17)
> (In reply to Nikolai Sednev from comment #16)
> > It's a bit problematic to open a new bug on storage as I can't provide
> > reproduction for this as it happened during very complicated upgrade on our
> > upgrade setup, there could be many triggers for snapshot was running on
> > background and then other tasks were running on the background. Should logs
> > provided within this bug be sufficient for opening a new bug?
> 
> I have no idea, I guess you'll have to simply try :-)
> 
> If the storage people will need more information that you'll not be able to
> supply, they can always close insufficient_data.
> 
> You do not have any other logs? engine/vdsm?

No other logs are available except those attached to this bug.