Bug 782176

Summary: [engine-cleanup] engine-cleanup should work even when there are active connections to DB
Product: [Retired] oVirt Reporter: zvi <zfridler>
Component: ovirt-engine-installerAssignee: Alex Lourie <alourie>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: high    
Version: unspecifiedCC: acathrow, alourie, bazulay, hateya, iheim, jkt, obasan, oschreib, sgordon, yeylon
Target Milestone: ---Keywords: Reopened
Target Release: 3.2   
Hardware: x86_64   
OS: Linux   
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-25 20:46:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 760640, 888027    

Description zvi 2012-01-16 18:35:39 UTC
Description of problem:
An engine-cleanup fails when there are active connections to DB (pgADMIN)  


How reproducible:
Always

Steps to Reproduce:
1.Clean engine DB via "engine-cleanup" when there is at least one connection to the DB (pgADMIN)

2.
3.
  
Actual results:
engine-cleanup fails:

2012-01-16 16:46:47::DEBUG::common_utils::205::root:: stderr = dropdb: database removal failed: ERROR:  database "engine" is being accessed by other users

 DETAIL:  There are 1 other session(s) using the database.

 2012-01-16 16:46:47::DEBUG::common_utils::206::root:: retcode = 1

 2012-01-16 16:46:47::ERROR::engine-cleanup::347::root:: Traceback (most recent call last):

  File "/usr/bin/engine-cleanup", line 341, in runFunc

   funcName()

   File "/usr/bin/engine-cleanup", line 282, in drop

    output, rc = utils.execCmd(cmd, None, True, MSG_ERROR_DROP_DB, [])

   File "/usr/share/ovirt-engine/scripts/common_utils.py", line 209, in execCmd

     raise Exception(msg)

 Exception: Error: Database drop failed


Expected results:
engine-cleanup should success  

Additional info:
A -F flag can be added to engine-cleanup - to force disconnection of all active db connections

Comment 1 Alex Lourie 2012-02-07 10:11:12 UTC
*** Bug 773723 has been marked as a duplicate of this bug. ***

Comment 3 Ofer Schreiber 2012-05-08 09:37:33 UTC
This one was partially solved:
1. If the DB is local - services should be restarted
2. If the DB is remote - there's nothing we can do.

Comment 4 Itamar Heim 2012-08-09 08:05:19 UTC
closing ON_QA bugs as oVirt 3.1 was released:
http://www.ovirt.org/get-ovirt/

Comment 5 Stephen Gordon 2012-12-19 17:13:15 UTC
I'm re-opening this as I did an installation using the 3.2 alpha bits and selected local DB but when I tried to clean-up (my install failed on iptables setup after creating the database because well iptables doesn't exist in F18) I got this output:

# engine-cleanup 
WARNING: Executing oVirt Engine cleanup utility.
This utility will wipe all existing data including configuration settings, certificates and database.
In addition, all existing DB connections will be closed.
Would you like to proceed? (yes|no): yes

Stopping ovirt-engine service...                         [ DONE ]
Removing Database...                                    [ ERROR ]
Removing CA...                                           [ DONE ]
Stopping engine-notifierd service...                     [ DONE ]

Cleanup finished with errors, please see log file
Error: failed to clear active DB connections
Cleanup log available at /var/log/ovirt-engine/engine-cleanup_2012_12_19_12_05_10.log

The cleanup log shows the cause of failure:

# tail /var/log/ovirt-engine/engine-cleanup_2012_12_19_12_05_10.log
2012-12-19 12:05:12::DEBUG::engine-cleanup::261::root:: CA Remove completed successfully
2012-12-19 12:05:12::DEBUG::engine-cleanup::389::root:: stoping engine-notifierd service.
2012-12-19 12:05:12::DEBUG::common_utils::1163::root:: stopping engine-notifierd
2012-12-19 12:05:12::DEBUG::common_utils::1200::root:: executing action engine-notifierd on service stop
2012-12-19 12:05:12::DEBUG::common_utils::390::root:: Executing command --> '/sbin/service engine-notifierd stop'
2012-12-19 12:05:12::DEBUG::common_utils::428::root:: output = Stopping engine-notifierd (via systemctl):  [  OK  ]

2012-12-19 12:05:12::DEBUG::common_utils::429::root:: stderr = 
2012-12-19 12:05:12::DEBUG::common_utils::430::root:: retcode = 0
2012-12-19 12:05:12::INFO::engine-cleanup::414::root:: Error: failed to clear active DB connections

Attempting to re-run engine-setup at this point results in:

Creating Database...                                  [ ERROR ]
Database creation failed
Please check log file /var/log/ovirt-engine/engine-setup_2012_12_19_12_08_02.log for more information
[root@sgordon-ovirt-engineusersysredhatcom ~]# tail /var/log/ovirt-engine/engine-setup_2012_12_19_12_08_02.log
  File "/usr/share/ovirt-engine/scripts/setup_sequences.py", line 154, in run
    step.run()
  File "/usr/share/ovirt-engine/scripts/setup_sequences.py", line 60, in run
    function()
  File "/usr/bin/engine-setup", line 969, in _createDB
    output, rc = utils.execCmd(cmdList=cmd, failOnError=True, msg=output_messages.ERR_DB_CREATE_FAILED, maskList=masked_value_set)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 433, in execCmd
    raise Exception(msg)
Exception: Database creation failed

So even though this is a local database, the active connections weren't cleared and as a result the database wasn't removed.

Comment 6 Stephen Gordon 2012-12-19 17:15:51 UTC
Oh in addition at this point re-running engine-cleanup results in this:

# engine-cleanup 
WARNING: Executing oVirt Engine cleanup utility.
This utility will wipe all existing data including configuration settings, certificates and database.
In addition, all existing DB connections will be closed.
Would you like to proceed? (yes|no): yes

Stopping ovirt-engine service...                         [ DONE ]

Error: Couldn't connect to the database server.Check that connection is working and rerun the cleanup utility
Error: Cleanup failed.
please check log at /var/log/ovirt-engine/engine-cleanup_2012_12_19_12_13_48.log

# tail /var/log/ovirt-engine/engine-cleanup_2012_12_19_12_15_09.log
2012-12-19 12:15:23::DEBUG::common_utils::430::root:: retcode = 2
2012-12-19 12:15:23::ERROR::engine-cleanup::476::root:: Traceback (most recent call last):
  File "/usr/bin/engine-cleanup", line 471, in <module>
    main(options)
  File "/usr/bin/engine-cleanup", line 433, in main
    if db.exists() and options.drop_db:
  File "/usr/bin/engine-cleanup", line 366, in exists
    raise Exception(MSG_ERROR_CONNECT_DB)
Exception: Error: Couldn't connect to the database server.Check that connection is working and rerun the cleanup utility

Comment 7 Alex Lourie 2012-12-20 09:19:40 UTC
Stephen

Could you please attach the complete log file? It would help me to understand the problem.

Thanks.

Comment 8 Stephen Gordon 2012-12-20 13:12:20 UTC
(In reply to comment #7)
> Stephen
> 
> Could you please attach the complete log file? It would help me to
> understand the problem.
> 
> Thanks.

I don't have the log from that exact run anymore but I just tried it on another fresh install of the 3.2 alpha bits and got the same result. Will attach.

Comment 10 Alex Lourie 2012-12-23 13:20:15 UTC
Thank you Stephen, now I see what the problem is.

Comment 11 Alex Lourie 2012-12-26 15:16:53 UTC
The fix is posted for review: http://gerrit.ovirt.org/#/c/10389/

Comment 12 Alex Lourie 2012-12-27 12:02:22 UTC
*** Bug 886019 has been marked as a duplicate of this bug. ***