1441563 – [RFE] Indicate domain job kind in job stats

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1441563 - [RFE] Indicate domain job kind in job stats

Summary: [RFE] Indicate domain job kind in job stats

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Jiri Denemark
QA Contact:	yafu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-04-12 08:45 UTC by Milan Zamazal
Modified:	2017-09-22 05:42 UTC (History)
CC List:	13 users (show)
Fixed In Version:	libvirt-3.2.0-4.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-02 00:05:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2017:1846	0	normal	SHIPPED_LIVE	libvirt bug fix and enhancement update	2017-08-01 18:02:50 UTC

Description Milan Zamazal 2017-04-12 08:45:50 UTC

When a domain job is running on a VM, there is currently no way to find out what kind of job it is, i.e. whether the VM is migrating to another host, creating a snapshot, etc. While virDomainGetJobStats and VIR_DOMAIN_EVENT_ID_JOB_COMPLETED event provide various information about a job, it's not possible to identify the particular job operation from them.

It would be very useful to have a new field identifying the job kind in virDomainGetJobStats and VIR_DOMAIN_EVENT_ID_JOB_COMPLETED event or having other means to find out the job kind, even from processes that didn't start the job.

See https://www.redhat.com/archives/libvir-list/2017-April/msg00540.html and other e-mails in the thread for related discussion.

Comment 2 Michal Skrivanek 2017-04-12 09:09:39 UTC

if it's not too difficult a backport would be great, otherwise we have to add workarounds and pollute the metadata section with this information

Comment 3 Jiri Denemark 2017-04-26 15:47:22 UTC

Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-April/msg01292.html

Comment 4 Jiri Denemark 2017-04-27 13:36:48 UTC

This is now implemented upstream by

commit b1c79d785f9385eb4ebd5e71d1d44e67f744f4d9
Refs: v3.2.0-281-gb1c79d785
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Apr 26 11:59:59 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 15:06:40 2017 +0200

    Add VIR_DOMAIN_JOB_OPERATION typed parameter

    The parameter is reported by virDomainGetJobStats API and
    VIR_DOMAIN_EVENT_ID_JOB_COMPLETED event and it can be used to identify
    the operation (migration, snapshot, ...) to which the reported
    statistics belong.

    https://bugzilla.redhat.com/show_bug.cgi?id=1441563

    Signed-off-by: Jiri Denemark <jdenemar>

commit 2a978269fc9dc31c354a7c9132cb62d52e7613ef
Refs: v3.2.0-282-g2a978269f
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Apr 26 12:00:09 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 15:08:12 2017 +0200

    qemu: Report VIR_DOMAIN_JOB_OPERATION

    Not all async jobs are visible via virDomainGetJobStats (either they are
    too fast or getting the stats is not allowed during the job), but
    forcing all of them to advertise the operation is easier than hunting
    the jobs for which fetching statistics is allowed. And we won't need to
    think about this when we add support for getting stats for more jobs.

    https://bugzilla.redhat.com/show_bug.cgi?id=1441563

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 9 yafu 2017-05-27 09:48:02 UTC

Verify with libvirt-3.2.0-6.virtcov.el7.x86_64.

Test steps:
1.Test for start guest:
# virsh start rhel7.3-min & virsh domjobinfo rhel7.3-min
[12] 27872
Job type:         Unbounded   
Operation:        Start       
Time elapsed:     307          ms


2.Test for save & restore:
(1)save
# virsh save rhel7.3-min /tmp/save & sleep 0.3 ; virsh domjobinfo rhel7.3-min
[11] 25225
Job type:         Unbounded   
Operation:        Save        
Time elapsed:     344          ms
Data processed:   1.130 MiB
Data remaining:   5.059 GiB
Data total:       5.063 GiB
Memory processed: 1.130 MiB
Memory remaining: 5.059 GiB
Memory total:     5.063 GiB
Dirty rate:       0            pages/s
Iteration:        1           
Constant pages:   768         
Normal pages:     287         
Normal data:      1.121 MiB
Expected downtime: 300          ms
Setup time:       45           ms

(2)restore
# virsh restore /tmp/save & sleep 0.3 ; virsh domjobinfo rhel7.3-min
[12] 10804
Job type:         Unbounded   
Operation:        Restore     
Time elapsed:     131          ms

(3)Check the --completed domjob info after guest restored:
# virsh domjobinfo rhel7.3-min --completed
Job type:         Completed   
Operation:        Save        
Time elapsed:     1151         ms
Data processed:   657.641 MiB
Data remaining:   0.000 B
Data total:       5.063 GiB
Memory processed: 657.641 MiB
Memory remaining: 0.000 B
Memory total:     5.063 GiB
Memory bandwidth: 710.475 MiB/s
Dirty rate:       0            pages/s
Iteration:        3           
Constant pages:   1161836     
Normal pages:     165480      
Normal data:      646.406 MiB
Total downtime:   1142         ms
Setup time:       57           ms


3.Test for dump:
# virsh dump rhel7.3-min /tmp/dump & sleep 0.1 ; virsh domjobinfo rhel7.3-min
[4] 20177

[3]+  Stopped                 virsh dump rhel7.3-min /tmp/dump
Job type:         Unbounded   
Operation:        Dump        
Time elapsed:     142          ms
Data processed:   259.271 KiB
Data remaining:   5.060 GiB
Data total:       5.063 GiB
Memory processed: 259.271 KiB
Memory remaining: 5.060 GiB
Memory total:     5.063 GiB
Dirty rate:       0            pages/s
Iteration:        1           
Constant pages:   8099386     
Normal pages:     1192657     
Normal data:      4.550 GiB
Expected downtime: 300          ms
Setup time:       39           ms

The domjob kind is also "Dump" when get with '--completed"  after the dump completed.

4.Test for snapshot:
(1)internal snapshot
# virsh snapshot-create rhel7.3-min & sleep 0.3 ; virsh domjobinfo rhel7.3-min
[13] 12232

error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainSnapshotCreateXML)

[13]   Exit 1                  virsh snapshot-create rhel7.3-min

There is a bug about this issue: 
https://bugzilla.redhat.com/show_bug.cgi?id=1448306

(2)external snapshot
# virsh snapshot-create-as rhel7.3-min s8 --memspec /tmp/test & sleep 1 ; virsh domjobinfo rhel7.3-min
[12] 15361
Job type:         Unbounded   
Operation:        Snapshot    
Time elapsed:     1001         ms
Data processed:   57.981 MiB
Data remaining:   645.770 MiB
Data total:       5.063 GiB
Memory processed: 57.981 MiB
Memory remaining: 645.770 MiB
Memory total:     5.063 GiB
Memory bandwidth: 37.666 MiB/s
Dirty rate:       0            pages/s
Iteration:        1           
Constant pages:   6954365     
Normal pages:     844213      
Normal data:      3.220 GiB
Expected downtime: 300          ms
Setup time:       35           ms

The domain job kind is also "Snapshot" with --completed.

5.Test for snapshot-revert(only internal snapshot supported):
# virsh snapshot-revert rhel7.3-min 1495871240 & sleep 0.2 ; virsh domjobinfo rhel7.3-min
[12] 18279
Job type:         Unbounded   
Operation:        Snapshot revert
Time elapsed:     918          ms


6.Test for migration:
(1)Do migration:
virsh migrate rhel7.3-min qemu+ssh://10.66.4.251/system --live --verbose 

(2)Open another migration to get domain job info
#virsh domjobinfo
# virsh domjobinfo rhel7.3-min
Job type:         Unbounded   
Operation:        Outgoing migration
Time elapsed:     7069         ms
Data processed:   28.041 MiB
Data remaining:   1.045 GiB
Data total:       5.063 GiB
Memory processed: 28.041 MiB
Memory remaining: 1.045 GiB
Memory total:     5.063 GiB
Memory bandwidth: 5.010 MiB/s
Dirty rate:       0            pages/s
Iteration:        1           
Constant pages:   1048517     
Normal pages:     4865        
Normal data:      19.004 MiB
Expected downtime: 300          ms
Setup time:       42           ms

(3)After migration completed, get domain job info with --completed on source host:
# virsh domjobinfo rhel7.3-min --completed
Job type:         Completed   
Operation:        Outgoing migration
Time elapsed:     4240         ms
Time elapsed w/o network: 4236         ms
Data processed:   226.817 MiB
Data remaining:   0.000 B
Data total:       5.063 GiB
Memory processed: 226.817 MiB
Memory remaining: 0.000 B
Memory total:     5.063 GiB
Memory bandwidth: 87.136 MiB/s
Dirty rate:       0            pages/s
Iteration:        4           
Constant pages:   1293990     
Normal pages:     55114       
Normal data:      215.289 MiB
Total downtime:   262          ms
Downtime w/o network: 258          ms
Setup time:       58           ms


(4)After migration completed, get domain job info with --completed on source host:
# virsh domjobinfo rhel7.3-min --completed
Job type:         Completed   
Operation:        ****Unknown****     
Time elapsed:     4220         ms
Time elapsed w/o network: 4216         ms
Data processed:   226.817 MiB
Data remaining:   0.000 B
Data total:       5.063 GiB
Memory processed: 226.817 MiB
Memory remaining: 0.000 B
Memory total:     5.063 GiB
Memory bandwidth: 87.136 MiB/s
Dirty rate:       0            pages/s
Iteration:        4           
Constant pages:   1293990     
Normal pages:     55114       
Normal data:      215.289 MiB
Total downtime:   262          ms
Downtime w/o network: 258          ms
Setup time:       58           ms

Comment 10 yafu 2017-05-27 10:00:52 UTC

Hi, Jiri,

If trying to get the domjob info on target host when doing migration, it will show 3 kinds of info:
1.Execute "virsh domjobinfo" immediately after the migration start:
#virsh domjobinfo rhel7.3-min
Job type:         Unbounded   
Operation:        Incoming migration
Time elapsed:     105       ms

2.Execute "virsh domjobinfo" after migration start about few seconds:
#virsh domjobinfo rhel7.3-min 
error: Operation not supported: migration statistics are available only on the s
ource host

3.Execute "virsh domjobinfo --completed" after the migration completed:
# virsh domjobinfo rhel7.3-min --completed
Job type:         Completed   
Operation:        ****Unknown****     
Time elapsed:     4220         ms
Time elapsed w/o network: 4216         ms
Data processed:   226.817 MiB
Data remaining:   0.000 B
Data total:       5.063 GiB
Memory processed: 226.817 MiB
Memory remaining: 0.000 B
Memory total:     5.063 GiB
Memory bandwidth: 87.136 MiB/s
Dirty rate:       0            pages/s
Iteration:        4           
Constant pages:   1293990     
Normal pages:     55114       
Normal data:      215.289 MiB
Total downtime:   262          ms
Downtime w/o network: 258          ms
Setup time:       58           ms

The domain job kind shows "Unknown" on target host after the migration completed.

Would you help to check the issue please? Thanks a lot.

Comment 11 Jiri Denemark 2017-05-29 08:03:15 UTC

Oh, looks like I forgot to update the operation in completed stats on the destination. Checking completed stats on the source host should work though. And it should also be reported in the migration event sent by the source daemon.

Comment 12 yafu 2017-05-31 03:19:19 UTC

(In reply to Jiri Denemark from comment #11)
> Oh, looks like I forgot to update the operation in completed stats on the
> destination. Checking completed stats on the source host should work though.
> And it should also be reported in the migration event sent by the source
> daemon.

Thanks. I filed a separate bug to track this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1457052

And I found the operation shows the digit not name in the event, does that work as expected?

Test stesps:
1.#virsh event --all --loop

2.Open another terminal, do migration:
migrate rhel7.3-min qemu+ssh://10.66.4.251/system --live --verbose 
Migration: [100 %]

3.Check the outputs of event:
...
event 'migration-iteration' for domain rhel7.3-min: iteration: '1'
event 'migration-iteration' for domain rhel7.3-min: iteration: '2'
event 'migration-iteration' for domain rhel7.3-min: iteration: '3'
event 'lifecycle' for domain rhel7.3-min: Suspended Migrated
event 'migration-iteration' for domain rhel7.3-min: iteration: '4'
event 'lifecycle' for domain rhel7.3-min: Stopped Migrated
event 'job-completed' for domain rhel7.3-min:
	operation: ****5*****
	time_elapsed: 8550
	time_elapsed_net: 8547
	downtime: 501
	downtime_net: 498
	setup_time: 38
	data_total: 5436686336
	data_processed: 702045654
	data_remaining: 0
	memory_total: 5436686336
	memory_processed: 702045654
	memory_remaining: 0
	memory_bps: 99365940
	memory_constant: 1161186
	memory_normal: 168517
	memory_normal_bytes: 690245632
	memory_dirty_rate: 0
	memory_iteration: 4
	disk_total: 0
	disk_processed: 0
	disk_remaining: 0

Dump/save/snapshot also show digit operation in event outputs.

Comment 13 Jiri Denemark 2017-06-05 11:32:57 UTC

Yeah, the job-completed event printing is very simple and it just prints raw data from libvirt without translating them into human readable format.

Comment 14 Jiri Denemark 2017-06-06 07:40:40 UTC

Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-June/msg00220.html

Comment 15 Jiri Denemark 2017-06-06 07:44:05 UTC

(In reply to Jiri Denemark from comment #14)
> Patch sent upstream for review:
> https://www.redhat.com/archives/libvir-list/2017-June/msg00220.html

Oops, this obviously applies to another bz.

Comment 16 yafu 2017-06-12 05:38:43 UTC

(In reply to Jiri Denemark from comment #15)
> (In reply to Jiri Denemark from comment #14)
> > Patch sent upstream for review:
> > https://www.redhat.com/archives/libvir-list/2017-June/msg00220.html
> 
> Oops, this obviously applies to another bz.

Thanks for your reply.

According to comment 9 and comment 12, move this bug to verified.

Comment 17 errata-xmlrpc 2017-08-02 00:05:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 18 errata-xmlrpc 2017-08-02 01:30:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Note You need to log in before you can comment on or make changes to this bug.