Description of problem: During V2V transfer ovirt-imageio process memory continue to grow unreasonably. All requests are kept historically, every 10s the stats are reported to engine for UI progress bar. This is aggravated by many requests from v2v. Version-Release number of selected component (if applicable): ovirt-imageio-common-1.3.0-0.el7ev.noarch ovirt-imageio-daemon-1.3.0-0.el7ev.noarch How reproducible: Always Steps to Reproduce: 1. Transfer VM from VMware to RHV system 2. On the host monitor ovirt-imageio RES utiliation Actual results: Continuous linear memory growth Expected results: No continuous linear memory growth Additional info: Full logs will be attahced
Created attachment 1454997 [details] Updated graph of 4 VMS transfer
Guy, this issue should be fixed by the attached patch. Can you check that it solve the issue you see?
I tests this also in scale setup now. Before this change we had 2.5G RES (in top) after 20-30 100G virt-v2v imports. With this change, we have constant 100M RES after 10 100G virt-v2v imports. Fixed in: commit b8e83ef58d3b8c57191cd5913fc348153aace094 Author: Nir Soffer <nsoffer> Date: Sat Jul 21 01:15:40 2018 +0300 daemon: Optimize transferred calculation When uploading or downloading big images using tiny chunks, the operations list becomes huge, taking lot of memory and consuming significant cpu time. Optimize the calculation by removing completed operations from the ticket, and keeping a sorted list of transferred ranges. When an operation is removed, the operation range is merged into the completed ranges list, which should be small in typical download or upload flows. When calculating total transferred bytes, we create list of ranges from the ongoing operations, which should be small, and merge the completed and ongoing ranges. Change-Id: Ib33967d9e858353037542eedbb3c68d350bf3ad4 Bug-Url: https://bugzilla.redhat.com/1592847 Signed-off-by: Nir Soffer <nsoffer>
We're releasing today 4.2.6 RC2 including v1.4.3 which is referencing this bug. can you please check this bug status?
(In reply to Sandro Bonazzola from comment #6) 1.4.3 should fix this, but it was not tested yet by QE. We can list this bug as fixes included by this version, but we cannot mark it as verified without testing.
We have a downstream build, moving to ON_QA
Created attachment 1479169 [details] RES of imageio build 1.4.3
From load run on 19.8 with ovirt-imageio-daemon-1.4.3 version with V2V migration of 10 VMS 100GB to FC 66% full disk we see the RES is indeed stable on 95G and not increasing as before, thus verifying the bug, attache the RES image.
Guy, can you add more details how the behavior changed since previous versions during same flows? (e.g. 10x100g import) - On which versions we tested the same flow? - What was the memory usage seen in each tested version?
Created attachment 1479201 [details] 1.4.2 imagio rate
Before comment10 got mixed up, should be MB not GB. Added previous ovirt-imageio-daemon 1.4.2 version RES with the same scenario. So in 1.4.2 the RES increases throughout the test up to 3.6 GB. In 1.4.3 the RES stabilize around 95 MB and does not continue to increase.
Thanks! I did not know it was so bad. any chance to get a graph from the 1.4.3 test run?
1.4.3 test run graph is attached - "RES of imageio build 1.4.3"
Thanks Guy, I think we have all data.