Bug 1482537

Summary: [RFE] qemu-img copy-offloading (convert command)
Product: Red Hat Enterprise Linux 7 Reporter: Yaniv Kaul <ykaul>
Component: qemu-kvm-rhevAssignee: Fam Zheng <famz>
Status: CLOSED ERRATA QA Contact: Tingting Mao <timao>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: chayang, coli, dgilbert, famz, hreitz, juzhang, knoel, kwolf, lolyu, michen, mtessun, pingl, qzhang, rjones, timao, virt-maint, wquan, yama, ylavi
Target Milestone: rcKeywords: FutureFeature, Performance
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-7.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1594820 (view as bug list) Environment:
Last Closed: 2018-11-01 11:01:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 912761, 1069557, 1558125, 1594820    

Description Yaniv Kaul 2017-08-17 13:38:55 UTC 
Description of problem:

qemu-img should support copying data, if the target and source support XCOPY, using the XCOPY SCSI command. 
It will significantly reduce CPU and bandwidth consumption on the host and will accelerate the copy.
It may be using ibiscsi, which already has the support for it[1]. Another interesting factor is that the libiscsi maintainer recently joined Red Hat (as kernel CIFS maintainer). 

This feature will benefit both RHV and OpenStack.

[1] https://github.com/sahlberg/libiscsi/commit/4bc5f962e2f0655734d0cc9cfe327d2800b7ac79
Comment 2 Yaniv Kaul 2018-05-16 07:43:00 UTC 
My original request has been somewhat generic 'qemu-img'. That means that any copy command should benefit from it - such as commit, 'dd' (who's using it?!), rebase(?).

Should we open more specific RFEs, especially on commit?
Comment 3 Fam Zheng 2018-05-16 08:13:12 UTC 
Listing the sub-commands is enough IMO. I count 3 of them: commit, rebase and convert.

Another BZ could be opened for block jobs (block-commit, block-stream, drive-backup, drive-mirror and all their blockdev-* variants.
Comment 4 Yaniv Kaul 2018-05-16 08:19:57 UTC 
(In reply to Fam Zheng from comment #3)
> Listing the sub-commands is enough IMO. I count 3 of them: commit, rebase
> and convert.
> 
> Another BZ could be opened for block jobs (block-commit, block-stream,
> drive-backup, drive-mirror and all their blockdev-* variants.

Of course - this was actually my main motivation - faster live storage migration...  opened https://bugzilla.redhat.com/show_bug.cgi?id=1578699
Comment 7 Miroslav Rezanina 2018-07-04 08:12:29 UTC 
Fix included in qemu-kvm-rhev-2.12.0-7.el7
Comment 10 Tingting Mao 2018-07-17 09:32:45 UTC 
Hi Fam,

According the result in comment 9, it seems only the usage of libiscsi(usage: iscsi://) improved the performance. However, for nfs and iscsi(usage: login target and create lvs) backend, the convert time is longer with Qemu-kvm-rhev-2.12.0-7.el7 than it with Qemu-kvm-rhev-2.10.0-21.el7_5.1. So could you please help to check where is wrong.

Thanks.
Comment 11 Fam Zheng 2018-07-17 09:47:22 UTC 
Can you do

strace -e copy_file_range -f qemu-img convert ...

for the nfs case and see if the syscall is invoked?

We can ignore the lv+iscsi for now, it seems host kernel is not optimized.
Comment 12 Tingting Mao 2018-07-17 10:06:46 UTC 
(In reply to Fam Zheng from comment #11)
> Can you do
> 
> strace -e copy_file_range -f qemu-img convert ...
> 
> for the nfs case and see if the syscall is invoked?

Yes, it does.

# time strace -e copy_file_range -f qemu-img convert -f qcow2 -O qcow2 base.qcow2 convert.qcow2 
strace: Process 58483 attached
[pid 58483] copy_file_range(10, [327680], 12, [327680], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [2424832], 12, [2424832], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [4521984], 12, [4521984], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [6619136], 12, [6619136], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [8716288], 12, [8716288], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [10813440], 12, [10813440], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [12910592], 12, [12910592], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [15007744], 12, [15007744], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [17104896], 12, [17104896], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [19202048], 12, [19202048], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [21299200], 12, [21299200], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [23396352], 12, [23396352], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [25493504], 12, [25493504], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [27590656], 12, [27590656], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [29687808], 12, [29687808], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [31784960], 12, [31784960], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [33882112], 12, [33882112], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [35979264], 12, [35979264], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [38076416], 12, [38076416], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [40173568], 12, [40173568], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [42270720], 12, [42270720], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [44367872], 12, [44367872], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [46465024], 12, [46465024], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [48562176], 12, [48562176], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [50659328], 12, [50659328], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [52756480], 12, [52756480], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [54853632], 12, [54853632], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [56950784], 12, [56950784], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [59047936], 12, [59047936], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [61145088], 12, [61145088], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [63242240], 12, [63242240], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [65339392], 12, [65339392], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [67436544], 12, [67436544], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [69533696], 12, [69533696], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [71630848], 12, [71630848], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [73728000], 12, [73728000], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [75825152], 12, [75825152], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [77922304], 12, [77922304], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [80019456], 12, [80019456], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [82116608], 12, [82116608], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [84213760], 12, [84213760], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [86310912], 12, [86310912], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [88408064], 12, [88408064], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [90505216], 12, [90505216], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [92602368], 12, [92602368], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [94699520], 12, [94699520], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [96796672], 12, [96796672], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [98893824], 12, [98893824], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [100990976], 12, [100990976], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [103088128], 12, [103088128], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [105185280], 12, [105185280], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [107282432], 12, [107282432], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [109379584], 12, [109379584], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [111476736], 12, [111476736], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [113573888], 12, [113573888], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [115671040], 12, [115671040], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [117768192], 12, [117768192], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [119865344], 12, [119865344], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [121962496], 12, [121962496], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [124059648], 12, [124059648], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [126156800], 12, [126156800], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [128253952], 12, [128253952], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [130351104], 12, [130351104], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [132448256], 12, [132448256], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [134545408], 12, [134545408], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [136642560], 12, [136642560], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [138739712], 12, [138739712], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [140836864], 12, [140836864], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [142934016], 12, [142934016], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [145031168], 12, [145031168], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [147128320], 12, [147128320], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [149225472], 12, [149225472], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [151322624], 12, [151322624], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [153419776], 12, [153419776], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [155516928], 12, [155516928], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [157614080], 12, [157614080], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [159711232], 12, [159711232], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [161808384], 12, [161808384], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [163905536], 12, [163905536], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [166002688], 12, [166002688], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [168099840], 12, [168099840], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [170196992], 12, [170196992], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [172294144], 12, [172294144], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [174391296], 12, [174391296], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [176488448], 12, [176488448], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [178585600], 12, [178585600], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [180682752], 12, [180682752], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [182779904], 12, [182779904], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [184877056], 12, [184877056], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [186974208], 12, [186974208], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [189071360], 12, [189071360], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [191168512], 12, [191168512], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [193265664], 12, [193265664], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [195362816], 12, [195362816], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [197459968], 12, [197459968], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [199557120], 12, [199557120], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [201654272], 12, [201654272], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [203751424], 12, [203751424], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [205848576], 12, [205848576], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [207945728], 12, [207945728], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [210042880], 12, [210042880], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [212140032], 12, [212140032], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [214237184], 12, [214237184], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [216334336], 12, [216334336], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [218431488], 12, [218431488], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [220528640], 12, [220528640], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [222625792], 12, [222625792], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [224722944], 12, [224722944], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [226820096], 12, [226820096], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [228917248], 12, [228917248], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [231014400], 12, [231014400], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [233111552], 12, [233111552], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [235208704], 12, [235208704], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [237305856], 12, [237305856], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [239403008], 12, [239403008], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [241500160], 12, [241500160], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [243597312], 12, [243597312], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [245694464], 12, [245694464], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [247791616], 12, [247791616], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [249888768], 12, [249888768], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [251985920], 12, [251985920], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [254083072], 12, [254083072], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [256180224], 12, [256180224], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [258277376], 12, [258277376], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [260374528], 12, [260374528], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [262471680], 12, [262471680], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [264568832], 12, [264568832], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [266665984], 12, [266665984], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [268763136], 12, [268763136], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [270860288], 12, [270860288], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [272957440], 12, [272957440], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [275054592], 12, [275054592], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [277151744], 12, [277151744], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [279248896], 12, [279248896], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [281346048], 12, [281346048], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [283443200], 12, [283443200], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [285540352], 12, [285540352], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [287637504], 12, [287637504], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [289734656], 12, [289734656], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [291831808], 12, [291831808], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [293928960], 12, [293928960], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [296026112], 12, [296026112], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [298123264], 12, [298123264], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [300220416], 12, [300220416], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [302317568], 12, [302317568], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [304414720], 12, [304414720], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [306511872], 12, [306511872], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [308609024], 12, [308609024], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [310706176], 12, [310706176], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [312803328], 12, [312803328], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [314900480], 12, [314900480], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [316997632], 12, [316997632], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [319094784], 12, [319094784], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [321191936], 12, [321191936], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [323289088], 12, [323289088], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [325386240], 12, [325386240], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [327483392], 12, [327483392], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [329580544], 12, [329580544], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [331677696], 12, [331677696], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [333774848], 12, [333774848], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [335872000], 12, [335872000], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [337969152], 12, [337969152], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [340066304], 12, [340066304], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [342163456], 12, [342163456], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [344260608], 12, [344260608], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [346357760], 12, [346357760], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [348454912], 12, [348454912], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [350552064], 12, [350552064], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [352649216], 12, [352649216], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [354746368], 12, [354746368], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [356843520], 12, [356843520], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [358940672], 12, [358940672], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [361037824], 12, [361037824], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [363134976], 12, [363134976], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [365232128], 12, [365232128], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [367329280], 12, [367329280], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [369426432], 12, [369426432], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [371523584], 12, [371523584], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [373620736], 12, [373620736], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [375717888], 12, [375717888], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [377815040], 12, [377815040], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [379912192], 12, [379912192], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [382009344], 12, [382009344], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [384106496], 12, [384106496], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [386203648], 12, [386203648], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [388300800], 12, [388300800], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [390397952], 12, [390397952], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [392495104], 12, [392495104], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [394592256], 12, [394592256], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [396689408], 12, [396689408], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [398786560], 12, [398786560], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [400883712], 12, [400883712], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [402980864], 12, [402980864], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [405078016], 12, [405078016], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [407175168], 12, [407175168], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [409272320], 12, [409272320], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [411369472], 12, [411369472], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [413466624], 12, [413466624], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [415563776], 12, [415563776], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [417660928], 12, [417660928], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [419758080], 12, [419758080], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [421855232], 12, [421855232], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [423952384], 12, [423952384], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [426049536], 12, [426049536], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [428146688], 12, [428146688], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [430243840], 12, [430243840], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [432340992], 12, [432340992], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [434438144], 12, [434438144], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [436535296], 12, [436535296], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [438632448], 12, [438632448], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [440729600], 12, [440729600], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [442826752], 12, [442826752], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [444923904], 12, [444923904], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [447021056], 12, [447021056], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [449118208], 12, [449118208], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [451215360], 12, [451215360], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [453312512], 12, [453312512], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [455409664], 12, [455409664], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [457506816], 12, [457506816], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [459603968], 12, [459603968], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [461701120], 12, [461701120], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [463798272], 12, [463798272], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [465895424], 12, [465895424], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [467992576], 12, [467992576], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [470089728], 12, [470089728], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [472186880], 12, [472186880], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [474284032], 12, [474284032], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [476381184], 12, [476381184], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [478478336], 12, [478478336], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [480575488], 12, [480575488], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [482672640], 12, [482672640], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [484769792], 12, [484769792], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [486866944], 12, [486866944], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [488964096], 12, [488964096], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [491061248], 12, [491061248], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [493158400], 12, [493158400], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [495255552], 12, [495255552], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [497352704], 12, [497352704], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [499449856], 12, [499449856], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [501547008], 12, [501547008], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [503644160], 12, [503644160], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [505741312], 12, [505741312], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [507838464], 12, [507838464], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [509935616], 12, [509935616], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [512032768], 12, [512032768], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [514129920], 12, [514129920], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [516227072], 12, [516227072], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [518324224], 12, [518324224], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [520421376], 12, [520421376], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [522518528], 12, [522518528], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [524615680], 12, [524615680], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [526712832], 12, [526712832], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [528809984], 12, [528809984], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [530907136], 12, [530907136], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [533004288], 12, [533004288], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [535101440], 12, [535101440], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [537264128], 12, [537264128], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [539361280], 12, [539361280], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [541458432], 12, [541458432], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [543555584], 12, [543555584], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [545652736], 12, [545652736], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [547749888], 12, [547749888], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [549847040], 12, [549847040], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [551944192], 12, [551944192], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [554041344], 12, [554041344], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [556138496], 12, [556138496], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [558235648], 12, [558235648], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [560332800], 12, [560332800], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [562429952], 12, [562429952], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [564527104], 12, [564527104], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [566624256], 12, [566624256], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [568721408], 12, [568721408], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [570818560], 12, [570818560], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [572915712], 12, [572915712], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [575012864], 12, [575012864], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [577110016], 12, [577110016], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [579207168], 12, [579207168], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [581304320], 12, [581304320], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [583401472], 12, [583401472], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [585498624], 12, [585498624], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [587595776], 12, [587595776], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [589692928], 12, [589692928], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [591790080], 12, [591790080], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [593887232], 12, [593887232], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [595984384], 12, [595984384], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [598081536], 12, [598081536], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [600178688], 12, [600178688], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [602275840], 12, [602275840], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [604372992], 12, [604372992], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [606470144], 12, [606470144], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [608567296], 12, [608567296], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [610664448], 12, [610664448], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [612761600], 12, [612761600], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [614858752], 12, [614858752], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [616955904], 12, [616955904], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [619053056], 12, [619053056], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [621150208], 12, [621150208], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [623247360], 12, [623247360], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [625344512], 12, [625344512], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [627441664], 12, [627441664], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [629538816], 12, [629538816], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [631635968], 12, [631635968], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [633733120], 12, [633733120], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [635830272], 12, [635830272], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [637927424], 12, [637927424], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [640024576], 12, [640024576], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [642121728], 12, [642121728], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [644218880], 12, [644218880], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [646316032], 12, [646316032], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [648413184], 12, [648413184], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [650510336], 12, [650510336], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [652607488], 12, [652607488], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [654704640], 12, [654704640], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [656801792], 12, [656801792], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [658898944], 12, [658898944], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [660996096], 12, [660996096], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [663093248], 12, [663093248], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [665190400], 12, [665190400], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [667287552], 12, [667287552], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [669384704], 12, [669384704], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [671481856], 12, [671481856], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [673579008], 12, [673579008], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [675676160], 12, [675676160], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [677773312], 12, [677773312], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [679870464], 12, [679870464], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [681967616], 12, [681967616], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [684064768], 12, [684064768], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [686161920], 12, [686161920], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [688259072], 12, [688259072], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [690356224], 12, [690356224], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [692453376], 12, [692453376], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [694550528], 12, [694550528], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [696647680], 12, [696647680], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [698744832], 12, [698744832], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [700841984], 12, [700841984], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [702939136], 12, [702939136], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [705036288], 12, [705036288], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [707133440], 12, [707133440], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [709230592], 12, [709230592], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [711327744], 12, [711327744], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [713424896], 12, [713424896], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [715522048], 12, [715522048], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [717619200], 12, [717619200], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [719716352], 12, [719716352], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [721813504], 12, [721813504], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [723910656], 12, [723910656], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [726007808], 12, [726007808], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [728104960], 12, [728104960], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [730202112], 12, [730202112], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [732299264], 12, [732299264], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [734396416], 12, [734396416], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [736493568], 12, [736493568], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [738590720], 12, [738590720], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [740687872], 12, [740687872], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [742785024], 12, [742785024], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [744882176], 12, [744882176], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [746979328], 12, [746979328], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [749076480], 12, [749076480], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [751173632], 12, [751173632], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [753270784], 12, [753270784], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [755367936], 12, [755367936], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [757465088], 12, [757465088], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [759562240], 12, [759562240], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [761659392], 12, [761659392], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [763756544], 12, [763756544], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [765853696], 12, [765853696], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [767950848], 12, [767950848], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [770048000], 12, [770048000], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [772145152], 12, [772145152], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [774242304], 12, [774242304], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [776339456], 12, [776339456], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [778436608], 12, [778436608], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [780533760], 12, [780533760], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [782630912], 12, [782630912], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [784728064], 12, [784728064], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [786825216], 12, [786825216], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [788922368], 12, [788922368], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [791019520], 12, [791019520], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [793116672], 12, [793116672], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [795213824], 12, [795213824], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [797310976], 12, [797310976], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [799408128], 12, [799408128], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [801505280], 12, [801505280], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [803602432], 12, [803602432], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [805699584], 12, [805699584], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [807796736], 12, [807796736], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [809893888], 12, [809893888], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [811991040], 12, [811991040], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [814088192], 12, [814088192], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [816185344], 12, [816185344], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [818282496], 12, [818282496], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [820379648], 12, [820379648], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [822476800], 12, [822476800], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [824573952], 12, [824573952], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [826671104], 12, [826671104], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [828768256], 12, [828768256], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [830865408], 12, [830865408], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [832962560], 12, [832962560], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [835059712], 12, [835059712], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [837156864], 12, [837156864], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [839254016], 12, [839254016], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [841351168], 12, [841351168], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [843448320], 12, [843448320], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [845545472], 12, [845545472], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [847642624], 12, [847642624], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [849739776], 12, [849739776], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [851836928], 12, [851836928], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [853934080], 12, [853934080], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [856031232], 12, [856031232], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [858128384], 12, [858128384], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [860225536], 12, [860225536], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [862322688], 12, [862322688], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [864419840], 12, [864419840], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [866516992], 12, [866516992], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [868614144], 12, [868614144], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [870711296], 12, [870711296], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [872808448], 12, [872808448], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [874905600], 12, [874905600], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [877002752], 12, [877002752], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [879099904], 12, [879099904], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [881197056], 12, [881197056], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [883294208], 12, [883294208], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [885391360], 12, [885391360], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [887488512], 12, [887488512], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [889585664], 12, [889585664], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [891682816], 12, [891682816], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [893779968], 12, [893779968], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [895877120], 12, [895877120], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [897974272], 12, [897974272], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [900071424], 12, [900071424], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [902168576], 12, [902168576], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [904265728], 12, [904265728], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [906362880], 12, [906362880], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [908460032], 12, [908460032], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [910557184], 12, [910557184], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [912654336], 12, [912654336], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [914751488], 12, [914751488], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [916848640], 12, [916848640], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [918945792], 12, [918945792], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [921042944], 12, [921042944], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [923140096], 12, [923140096], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [925237248], 12, [925237248], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [927334400], 12, [927334400], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [929431552], 12, [929431552], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [931528704], 12, [931528704], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [933625856], 12, [933625856], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [935723008], 12, [935723008], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [937820160], 12, [937820160], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [939917312], 12, [939917312], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [942014464], 12, [942014464], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [944111616], 12, [944111616], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [946208768], 12, [946208768], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [948305920], 12, [948305920], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [950403072], 12, [950403072], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [952500224], 12, [952500224], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [954597376], 12, [954597376], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [956694528], 12, [956694528], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [958791680], 12, [958791680], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [960888832], 12, [960888832], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [962985984], 12, [962985984], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [965083136], 12, [965083136], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [967180288], 12, [967180288], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [969277440], 12, [969277440], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [971374592], 12, [971374592], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [973471744], 12, [973471744], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [975568896], 12, [975568896], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [977666048], 12, [977666048], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [979763200], 12, [979763200], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [981860352], 12, [981860352], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [983957504], 12, [983957504], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [986054656], 12, [986054656], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [988151808], 12, [988151808], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [990248960], 12, [990248960], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [992346112], 12, [992346112], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [994443264], 12, [994443264], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [996540416], 12, [996540416], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [998637568], 12, [998637568], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1000734720], 12, [1000734720], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1002831872], 12, [1002831872], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1004929024], 12, [1004929024], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1007026176], 12, [1007026176], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1009123328], 12, [1009123328], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1011220480], 12, [1011220480], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1013317632], 12, [1013317632], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1015414784], 12, [1015414784], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1017511936], 12, [1017511936], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1019609088], 12, [1019609088], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1021706240], 12, [1021706240], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1023803392], 12, [1023803392], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1025900544], 12, [1025900544], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1027997696], 12, [1027997696], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1030094848], 12, [1030094848], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1032192000], 12, [1032192000], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1034289152], 12, [1034289152], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1036386304], 12, [1036386304], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1038483456], 12, [1038483456], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1040580608], 12, [1040580608], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1042677760], 12, [1042677760], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1044774912], 12, [1044774912], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1046872064], 12, [1046872064], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1048969216], 12, [1048969216], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1051066368], 12, [1051066368], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1053163520], 12, [1053163520], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1055260672], 12, [1055260672], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1057357824], 12, [1057357824], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1059454976], 12, [1059454976], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1061552128], 12, [1061552128], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1063649280], 12, [1063649280], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1065746432], 12, [1065746432], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1067843584], 12, [1067843584], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1069940736], 12, [1069940736], 2097152, 0) = 2097152
[pid 58483] copy_file_range(10, [1072037888], 12, [1072037888], 2097152, 0) = 2097152
[pid 58483] +++ exited with 0 +++
+++ exited with 0 +++

real	0m48.354s
user	0m0.108s
sys	0m1.626s
Comment 13 Fam Zheng 2018-07-18 01:21:01 UTC 
I think it is doing what we want. The time may be a result of disk bottleneck.

Can you collect the same for "iscsi host initiator+lv" case?
Comment 14 Tingting Mao 2018-07-18 02:08:04 UTC 
(In reply to Fam Zheng from comment #13)
> I think it is doing what we want. The time may be a result of disk
> bottleneck.

It is still a little hard to understand why the time is unexpectedly longer than the result with qemu-kvm-rhel-2.10 (For the tested images, I did the same operation for them.)

> Can you collect the same for "iscsi host initiator+lv" case?

The result is the same as NFS.

# time strace -e trace=copy_file_range -f qemu-img convert -f raw -O qcow2 /dev/vgtest/source /dev/vgtest/target 
strace: Process 41060 attached
[pid 41060] copy_file_range(10, [0], 12, [327680], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2097152], 12, [2424832], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [4194304], 12, [4521984], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [6291456], 12, [6619136], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [8388608], 12, [8716288], 2097152, 0) = 2097152
......
......
......
[pid 41060] copy_file_range(10, [2134900736], 12, [2135425024], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2136997888], 12, [2137522176], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2139095040], 12, [2139619328], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2141192192], 12, [2141716480], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2143289344], 12, [2143813632], 2097152, 0) = 2097152
[pid 41060] copy_file_range(10, [2145386496], 12, [2145910784], 2096640, 0) = 2096640
[pid 41060] copy_file_range(10, [2147483136], 12, [2148007424], 512, 0) = 512
[pid 41060] +++ exited with 0 +++
+++ exited with 0 +++

real	1m37.363s
user	0m0.072s
sys	0m3.137s
Comment 15 Fam Zheng 2018-07-18 02:49:58 UTC 
(In reply to timao from comment #14)
> (In reply to Fam Zheng from comment #13)
> > I think it is doing what we want. The time may be a result of disk
> > bottleneck.
> 
> It is still a little hard to understand why the time is unexpectedly longer
> than the result with qemu-kvm-rhel-2.10 (For the tested images, I did the
> same operation for them.)

Your disk is slow, but your network is fast. The old QEMU uses read/write and cache, and the new uses copy offloading, which cannot use host page cache, that's why the time is longer. How does it compare if you add "-t none" to the command line?
Comment 16 Tingting Mao 2018-07-24 10:35:03 UTC 
Result of regression test for 'convert':

For raw
new bug:
Bug 1607774 - Target files for 'qemu-img convert' do not support thin_provisoning with iscsi backend

For qcow2
new bugs(2):
Bug 1601871 - Target image needs larger size when executing convert with iscsi backend 
Bug 1605061 - preallocation mode(full and falloc) is not supported when creating qcow2 file with iscsi backend
Comment 17 Tingting Mao 2018-07-24 10:44:09 UTC 
Hi Fam,

According to the comment 9, comment 12, comment 14 and comment 16, is it enough to set the bug as verified? Thanks in advance.

Best,
Tingting Mao
Comment 18 Fam Zheng 2018-07-26 01:21:53 UTC 
Please run qemu-img regression tests as well then this can be verified.
Comment 19 Tingting Mao 2018-08-13 06:16:35 UTC 
Hi Fam,

After run qcow2+nfs(v4, soft), qcow2+libiscsi(iscsi://), qcow2+iscsi(local+lvs), raw+nfs(v4, soft), raw＋libsicsi(iscsi://) and raw+iscsi(local+lvs), besides the new reported bugs in comment 16, there is two new bugs listed below. So is the test enough to set the bug as verified now? Thanks in advance. 

Bug 1613215 - qemu-img shouldn't truncate image with the invalid option value
Bug 1601310 - qemu-img map 'Aborted (core dumped)' when specifying a plain file
Comment 20 Fam Zheng 2018-08-14 07:20:30 UTC 
Yes. Thanks.
Comment 21 Tingting Mao 2018-08-14 07:23:55 UTC 
According comment 9, comment 12, comment 14, comment 16 and comment 19, set the bug as verified.
Comment 22 Yaniv Kaul 2018-08-29 13:06:16 UTC 
(In reply to Tingting Mao from comment #21)
> According comment 9, comment 12, comment 14, comment 16 and comment 19, set
> the bug as verified.

Is the performance better now than before of the qemu-img command?
Comment 23 Tingting Mao 2018-08-29 15:01:12 UTC 
(In reply to Yaniv Kaul from comment #22)
> (In reply to Tingting Mao from comment #21)
> > According comment 9, comment 12, comment 14, comment 16 and comment 19, set
> > the bug as verified.
> 
> Is the performance better now than before of the qemu-img command?

For libiscsi, it indeed improved. While for iscsi(local + lvs) and nfs backend, it seems not. For more details, you could refer to comment 9 and comment 13.
Comment 24 Yaniv Kaul 2018-08-29 15:32:15 UTC 
(In reply to Tingting Mao from comment #23)
> (In reply to Yaniv Kaul from comment #22)
> > (In reply to Tingting Mao from comment #21)
> > > According comment 9, comment 12, comment 14, comment 16 and comment 19, set
> > > the bug as verified.
> > 
> > Is the performance better now than before of the qemu-img command?
> 
> For libiscsi, it indeed improved. While for iscsi(local + lvs) and nfs
> backend, it seems not. For more details, you could refer to comment 9 and
> comment 13.

So we need specific bugs tracking those performance regressions - either we allow to disable this feature, or we fix the performance regression?
Comment 25 Fam Zheng 2018-08-30 01:55:04 UTC 
In bug 1607774 we've disabled it by default. It's done because of behavioral change on sparsifying.
Comment 26 errata-xmlrpc 2018-11-01 11:01:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443