Bug 1384864

Summary: Clearing asynchronous task event launched on a per-minute basis
Product: [oVirt] ovirt-engine Reporter: nicolas
Component: Backend.CoreAssignee: Ravi Nori <rnori>
Status: CLOSED NOTABUG QA Contact: meital avital <mavital>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.0.3CC: bugs, mperina, nicolas
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-18 19:11:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
ovirt-engine logs none

Description nicolas 2016-10-14 10:00:43 UTC
Created attachment 1210454 [details]
ovirt-engine logs

Description of problem:

We've recently had a problem migrating a few disks from one iSCSI-based datastore to another, as the task seemed stalled after 2 days having in consideration these were small disks, around 30 GB each. (we found nothing relevant in the logs about why did it stall).

We found a utility called: unlock_entity.sh, which we used to unblock the tasks from the Events tab:

  PGPASSWORD=dbpass /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -v -u engine -d engine -t disk <disk-uuid>

This seemed to work, but now we have lots of events like this in the events list:

  Clearing asynchronous task Unknown that started at Tue Oct 11 07:46:40 WEST 2016.

This message seems to appear in an once per minute interval. This is not a serious issue, but it's very annoying. A restart of the ovirt-engine did not help.

I'm attaching ovirt-engine logs.

Version-Release number of selected component (if applicable):

4.0.3-1

Comment 1 Martin Perina 2016-10-17 10:33:09 UTC
Ravi, could you please investigate?

Comment 2 Ravi Nori 2016-10-18 15:39:34 UTC
Since the issue is being caused by zombie tasks, unlocking the entity is not sufficient. The zombie task needs to be cleaned up. Cleaning the zombie task should stop the log messages. 

export PGPASSWORD="${ENGINE_DB_PASSWORD}"

cd /usr/share/ovirt-engine/setup/dbutils

./taskcleaner.sh -u "${ENGINE_DB_USER}" -d "${ENGINE_DB_DATABASE}"

./unlock_entity.sh -u "${ENGINE_DB_USER}" -d "${ENGINE_DB_DATABASE}" -t all

Comment 3 nicolas 2016-10-18 18:40:31 UTC
I confirm the 'taskcleaner.sh' command along with 'unlock_entity.sh' indeed cleared the hung events and it stopped showing up in the task list.

Thanks!