Germano can you look at this https://bugzilla.redhat.com/show_bug.cgi?id=1488892 - this reduced the number of full refreshes and landed in 4.2.1

(In reply to Roy Golan from comment #6)
> Germano can you look at this
> https://bugzilla.redhat.com/show_bug.cgi?id=1488892 - this reduced the
> number of full refreshes and landed in 4.2.1

Hi Roy,

Nice.

But this case has just 2 active and 2 enabled paths and the load spikes are *very* high. There seem to be a big contention of LVM commands because there are 50+ SDs in total, so dropping all the caches and refreshing everything fires a lot of lvm commands. Even if refreshed just once, I think the problem may still be perceptible. So I'm not sure that patch is enough to full solve this. What do you think?

Vdsm does not limit the number of lvm commnads, and for every domain we refresh lvm
caches when performing operations like adding a storage domain. If you have big
amount of storage domains you will have big amount of concurrent lvm commands
in such case.

We discussed this possible issue in the past, but never seen a real case when it
was problematic.

I think this patch can help:
https://gerrit.ovirt.org/c/66932/

I suggest to first reproduce this issue in scale environment, and see if the patch
makes a difference.

Patch https://gerrit.ovirt.org/96422 is only for discussion, eliminating unneeded
lvm refreshes. I think we can work on this for 4.4, since this requires major 
changes in the system.

Patch https://gerrit.ovirt.org/96436 is also only for discussion. I think this is
the direction we should take in 4.4 if we want to improve scaleability.

This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

We limit now the amount of concurrent lvm calls to 10 per host.
This may help to reduce the load.

We also have experimental patches that may improve caching more,
but they require much more work. We will continue to work on this after
we have tests results for the current code.

Daniel, can you reproduce with vdsm 4.30.8, and test with vdsm 4.30.9?

As I see the bug now in modified state we will receive the fix in the coming build.
We added it to our sprint to validate it in the coming build?

Regarding your request of VDSM - What VDSM will be part of RHV 4.3 Beta 2  that will be built today Feb 21

WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

I have noticed that QA ack was missing ( Acked it now)
Is this part of the 4.3.1-2 drop?

Germano, can you define what is "significant load", and how can we measure it?

I think we should first reproduce this issue with vdsm 4.30.8, and then
try the same with vdsm-4.30.9, and compare the results.

nir can you please explain how we can verify your latest change  of concurrent lvm calls to 10 per host.

(In reply to Daniel Gur from comment #18)
> nir can you please explain how we can verify your latest change  of
> concurrent lvm calls to 10 per host.

We can watch the number of lvm processes run by vdsm, and check that we 
never see more than 10 processes while working with 50 storage domains.
In previous versions we could see up to 50 lvm processes (one per domain).

(In reply to Nir Soffer from comment #17)
> Germano, can you define what is "significant load", and how can we measure
> it?
> 
> I think we should first reproduce this issue with vdsm 4.30.8, and then
> try the same with vdsm-4.30.9, and compare the results.

Hi Nir,

I added some examples of what the "significant load" load produces in comment #0. See items 'd' to 'g', which are easier to see from a single host.
On the actual customer environment, which has dozens of hosts, all hosts doing this at the same time impacts the storage performance for the VMs, they get reduced peformance too (IO).

If I understand your patch correctly, we will spread this load over more time to generate less impact by limiting concurrent lvm commands. But it is not doing N SDs at a time and when finished start another SD, it will still do all SDs at the same time, just limiting the amount of lvm commands. So each SD will take much longer to produce than doing N SDs at a time. I'm afraid this will make things like GetVolumeSize block for much longer as producing a SD will now take longer, triggering things like BZ1598266 even more?

Scale Environment:
  Hosted-Engine
  2 Hosts - SuperMicro Server with ~200VMs each

  rhv-release-4.3.1-1-001
  vdsm-4.30.9-1.el7ev.x86_64


Test Preparation:
* Create 60 LUNs on NetApp (FC)

Two Tests were run:
1. Add 50 Storage Domains - Duration: 2:05 Hrs (2019-02-24 17:00-19:00 Israel Time)
2. Add more 10 Storage Domains - Duration ~40min (2019-02-25 17:15-17:55 Israel Time)

During the tests , resource was collected using 'nmon'.(Link bellow)

Results:
1. Hosts CPU - ~12%  , (Sys - 2% , IO wait - 0%)
2. Hosts Disk Busy - 4-5%


VDSM log:

No 'monitor became unresponsive' messages




Link for 'nmon' results:

https://drive.google.com/open?id=1-I8JsjrZLJaE29qndwQmUR5phA6cCwpp

(In reply to Germano Veit Michel from comment #20)
> (In reply to Nir Soffer from comment #17)
 
> If I understand your patch correctly, we will spread this load over more
> time to generate less impact by limiting concurrent lvm commands.

Right. Before this patch, we could have up to 50 lvm commands on every host
running at the same time, reading from remote storage.

With the patch, we can have only 10 lvm commands running at the same time.

> But it is
> not doing N SDs at a time and when finished start another SD, it will still
> do all SDs at the same time, just limiting the amount of lvm commands. So
> each SD will take much longer to produce than doing N SDs at a time.

In my tests running many lvm commands just make every commands slower. We had
see every lvm command takes up to 80 seconds few years ago. So I don't think
limiting the number of commands will slow down anything.

> I'm
> afraid this will make things like GetVolumeSize block for much longer as
> producing a SD will now take longer, triggering things like BZ1598266 even
> more?

I don't think it will effect this, but we need to test this. Basically any
storage domain function can be block for unlimited time because of the shared
domain cache if iscsi/fc rescan is very slow. The real issue is slow rescans
not slow lvm commands.

We could not reproduce the issue in 4.3, and we did not test with earlier version,
so I'm not sure we understand this issue. It is possible that this was improved
by limiting number of concurrent lvm commands in vdsm, but without reproducing
the issue with a version that does include this change we cannot be sure.

Therefore I don't think we can have meaningful doc text.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1077

sync2jira

sync2jira