Bug 1902468

Summary:	Support 50 concurrent add_lockspace operations
Product:	Red Hat Enterprise Linux 8	Reporter:	Nir Soffer <nsoffer>
Component:	sanlock	Assignee:	David Teigland <teigland>
Status:	CLOSED ERRATA	QA Contact:	cluster-qe <cluster-qe>
Severity:	high	Docs Contact:
Priority:	high
Version:	8.3	CC:	aefrat, agk, cluster-maint, cmarthal, jbrassow, mtessun, rhandlin, teigland, vjuranek
Target Milestone:	rc	Keywords:	Triaged, ZStream
Target Release:	8.4	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1912923 (view as bug list)		Environment:
Last Closed:	2021-05-18 15:09:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1845909, 1903358, 1912923

Description Nir Soffer 2020-11-29 11:37:56 UTC

Description of problem:

add_lockspace operation takes lot of time, depending on sanlock io timeout.
According to sanlock timeouts.h header:
https://pagure.io/sanlock/raw/master/f/src/sanlock.h

 *                             fast norm slow
 * io_timeout_seconds             5   10   20
 * delta_acquire_held_max       130  200  340
 * delta_acquire_held_min       100  140  220
 * delta_acquire_free_max        25   50  100
 * delta_acquire_free_min        10   20   40

With few storage domains we can measure that the time to complete async
add_lockspace operations is 30 seconds with 10 seconds io timeout, and 50
seconds with 20 seconds io timeout. After unclean shutdown this operation
takes much more, up to 270 seconds.

However with more storage domains and io timeout of 20 seconds, we measured
2m:20s seconds after clean shutdown, and 13m:50s! after unclean shutdown.

Turns out that sanlock uses a thread pool with 8 workers for serving client
requests, and add_lockspace is blocking one thread in the thread pool for
the entire operation even when used with the SANLK_ADD_ASYNC flag.

I found that changing the number of threads using:

   ExecStart=/usr/sbin/sanlock daemon -t 32

Fixes this issue. With this we run 20 concurent add_lockspace and the time
to complete the operation is 50 after clean shutdown and 270 seconds after
unclean shutdown.

However we cannot use this since it requires modifying sanlock.service file.

We support and test 50 storage domains, and we have a hard limit of 100
storage domains. We would like to avoid unneeded delays when using 50 storage
domains. Using larger domains is not recommended due to other limitations in
RHV.

When host did not complete add_lockspace operation, acquiring sanlock 
resource is not possible, so RHV will not let you run HA VMs with a storage
lease on the host, or perform storage operations that require a storage 
lease (e.g. copy disk). This issue introduces large delays harming the user
experience.

Version-Release number of selected component (if applicable):
sanlock-3.8.2-1.el8.x86_64

How reproducible:
Always

Steps to Reproduce - clean shutdown:
1. Create 50 storage domains in RHV
2. Put host to maintenance
3. Active host
4. Run "watch sanlock client status"

Actual results:
Sanlock completes lockspace addition in ~350 seconds

Expected results:
Sanlock completes lockspace addition in ~50 seconds

Steps to Reproduce - unclean shutdown:
1. Create 50 storage domains in RHV
2. Perform unclean shutdown (cut the power for bare metal, kill qemu for vm)
3. Start host
4. Run "watch sanlock client status"

Actual results:
Sanlock completes lockspace addition in ~30 minutes

Expected results:
Sanlock completes lockspace addition in ~5 minutes

Additional info:

We would like to add the number of worker threads to sanlock configuration
file so we can increase this value in RHV during deployment.

Or better, change the implementation so add_locksapce will be async and 
support running reasonable number of concurrent add_lockspace operations.

Comment 1 Nir Soffer 2020-11-30 20:17:11 UTC

I posted patch to allow configuring max_worker_threads via /etc/sanlock.conf:
https://lists.fedorahosted.org/archives/list/sanlock-devel@lists.fedorahosted.org/thread/X3B2KMSNIRO7SOCRMTTV7CN7AACYLC3G/

This is good enough for RHV to tune this value.

David, can we this change in 8.3.z?

Comment 2 David Teigland 2020-11-30 21:12:30 UTC

pushed to git
https://pagure.io/sanlock/c/e722c69cef883af1f7d60cbb2708f0092dbf2ab0

Putting in zstream is ok with me, and would just need the necessary bug with approvals to be created which I'm not able to do.

Comment 4 David Teigland 2020-12-02 15:26:11 UTC

Since RHV QE will be testing this would they provide the qa_ack?

Comment 5 Nir Soffer 2020-12-07 16:20:18 UTC

(In reply to David Teigland from comment #4)
> Since RHV QE will be testing this would they provide the qa_ack?

Avihay, can you test this bug?

The next RHV build will use this feature
(see https://gerrit.ovirt.org/c/vdsm/+/112498)
So testing should be pretty easy.

Comment 6 Nir Soffer 2020-12-07 16:25:40 UTC

David, can we get a scratch build for testing?

It should include these patches:

32a73a5 sanlock: Report actual max_worker_threads
e722c69 config: Add max_worker_threads
    
It would be nice to have also this patch:

0ff9c1a sanlock: Shrink thread pool when there is no work

Comment 7 David Teigland 2020-12-07 16:51:36 UTC

scratch build from top of current master branch
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=33533874

Comment 8 Nir Soffer 2020-12-10 21:41:01 UTC

I tested this change with vdsm 4.40.39, with 40 storage domains, and it works fine.

Tested configuration:

$ cat /etc/vdsm/vdsm.conf.d/99-foobar.conf 
# Configuration for foobar storage.

[sanlock]
# Set renewal timeout to 160 seconds
# (8 * io_timeout == 160).
io_timeout = 20


$ cat /etc/sanlock/sanlock.conf 
# Configuration for vdsm
our_host_name = 095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
max_worker_threads = 50


$ cat /etc/multipath/conf.d/foobar.conf 
# Configuration for foobar storage.

overrides {
    # Queue I/O for 160 seconds when all paths fail
    # (no_path_retry * polling_interval == 160).
    no_path_retry 32
}


# sanlock client status -D
daemon 095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
    our_host_name=095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
    use_watchdog=1
    high_priority=0
    mlock_level=1
    quiet_fail=1
    debug_renew=0
    debug_clients=0
    debug_cmds=0xfffffffffe06ffff
    renewal_history_size=180
    gid=179
    uid=179
    sh_retries=8
    max_sectors_kb_ignore=0
    max_sectors_kb_align=0
    max_sectors_kb_num=1024
    max_worker_threads=50
    write_init_io_timeout=60
    use_aio=1
    kill_grace_seconds=40
    helper_pid=964
    helper_kill_fd=7
    helper_full_count=0
    helper_last_status=102814
    monotime=102829
    version_str=3.8.2
    version_num=3.8.2
    version_hex=03080200
    smproto_hex=00000001
...


When adding lockspaces, we can see that we add all lockspaces concurrently:

# sanlock client status 
daemon 095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
p -1 helper
p -1 listener
p -1 status
p 3147 
s 1f4e2acd-6224-4345-ae06-1db97c403a0c:1:/rhev/data-center/mnt/storage\:_export_11/1f4e2acd-6224-4345-ae06-1db97c403a0c/dom_md/ids:0 ADD
s b2ff7cd6-64a6-453d-b71e-22fa4b11f16e:1:/rhev/data-center/mnt/storage\:_export_00/b2ff7cd6-64a6-453d-b71e-22fa4b11f16e/dom_md/ids:0 ADD
s e0549942-c08c-478b-9c8d-c116e98b59d4:1:/rhev/data-center/mnt/storage\:_export_09/e0549942-c08c-478b-9c8d-c116e98b59d4/dom_md/ids:0 ADD
s 27afe55d-68d6-49c1-a085-83425b4bdd67:1:/rhev/data-center/mnt/storage\:_export_14/27afe55d-68d6-49c1-a085-83425b4bdd67/dom_md/ids:0 ADD
s 1297b300-a126-45e8-8a7f-c69edcf057f1:1:/rhev/data-center/mnt/storage\:_export_08/1297b300-a126-45e8-8a7f-c69edcf057f1/dom_md/ids:0 ADD
s cd205f88-9b6a-46c6-9993-8569a5dfb8f1:1:/rhev/data-center/mnt/storage\:_export_15/cd205f88-9b6a-46c6-9993-8569a5dfb8f1/dom_md/ids:0 ADD
s 0d38fccd-5585-40f7-9a73-d215b23bf7ed:1:/rhev/data-center/mnt/storage\:_export_18/0d38fccd-5585-40f7-9a73-d215b23bf7ed/dom_md/ids:0 ADD
s 09525896-e9c1-4d24-8e4d-dab18346a274:1:/rhev/data-center/mnt/storage\:_export_07/09525896-e9c1-4d24-8e4d-dab18346a274/dom_md/ids:0 ADD
s a87b183d-df2d-4de3-b370-507c6261c328:1:/rhev/data-center/mnt/storage\:_export_02/a87b183d-df2d-4de3-b370-507c6261c328/dom_md/ids:0 ADD
s 839db77f-fde3-4e13-bfb6-56be604631ed:1:/rhev/data-center/mnt/storage\:_export_01/839db77f-fde3-4e13-bfb6-56be604631ed/dom_md/ids:0 ADD
s 7e25e16e-2cdf-4063-a638-3d61c0427ac6:1:/rhev/data-center/mnt/storage\:_export_19/7e25e16e-2cdf-4063-a638-3d61c0427ac6/dom_md/ids:0 ADD
s 8f870f86-3e4e-44a3-9bc6-c0af6f9d7944:1:/rhev/data-center/mnt/storage\:_export_05/8f870f86-3e4e-44a3-9bc6-c0af6f9d7944/dom_md/ids:0 ADD
s 2b5b232f-4a59-4ace-9e18-3841c63505de:1:/rhev/data-center/mnt/storage\:_export_16/2b5b232f-4a59-4ace-9e18-3841c63505de/dom_md/ids:0 ADD
s 2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3:1:/rhev/data-center/mnt/storage\:_export_04/2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3/dom_md/ids:0 ADD
s d046c651-ac7c-49ca-9541-31e47b7c38ef:1:/rhev/data-center/mnt/storage\:_export_06/d046c651-ac7c-49ca-9541-31e47b7c38ef/dom_md/ids:0 ADD
s 980601ff-1e81-4e10-bf79-a29d2d5b0e9f:1:/rhev/data-center/mnt/storage\:_export_10/980601ff-1e81-4e10-bf79-a29d2d5b0e9f/dom_md/ids:0 ADD
s 82d93a3d-3d88-42ca-a328-f295dfc48ef2:1:/rhev/data-center/mnt/storage\:_export_13/82d93a3d-3d88-42ca-a328-f295dfc48ef2/dom_md/ids:0 ADD
s c345fc52-f82e-4904-9ad4-1cc84b0fc639:1:/rhev/data-center/mnt/storage\:_export_03/c345fc52-f82e-4904-9ad4-1cc84b0fc639/dom_md/ids:0 ADD
s 0e49277b-b13e-49d6-bb56-e3da6d0cc397:1:/rhev/data-center/mnt/storage\:_export_12/0e49277b-b13e-49d6-bb56-e3da6d0cc397/dom_md/ids:0 ADD
s 5bb456c1-bcef-4fb5-adf2-01de35f46f6d:1:/rhev/data-center/mnt/storage\:_export_17/5bb456c1-bcef-4fb5-adf2-01de35f46f6d/dom_md/ids:0 ADD
s c202aeb7-771a-4f8c-b94b-39b805695b35:1:/dev/c202aeb7-771a-4f8c-b94b-39b805695b35/ids:0 ADD
s e673e145-d644-44ac-a416-a8fc14b029ca:1:/dev/e673e145-d644-44ac-a416-a8fc14b029ca/ids:0 ADD
s 06e43bfc-2ffe-419e-843b-59a5d23165e1:1:/dev/06e43bfc-2ffe-419e-843b-59a5d23165e1/ids:0 ADD
s 2e7daa04-2cd4-4e93-bab8-866227a8a37b:1:/dev/2e7daa04-2cd4-4e93-bab8-866227a8a37b/ids:0 ADD
s 31e3675e-302a-4a22-ad9a-402c0c1e9765:1:/dev/31e3675e-302a-4a22-ad9a-402c0c1e9765/ids:0 ADD
s f53d7919-d275-4ad4-ba57-f280000de397:1:/dev/f53d7919-d275-4ad4-ba57-f280000de397/ids:0 ADD
s ebf4cf14-0a33-4c3d-bd83-9f161760fecb:1:/dev/ebf4cf14-0a33-4c3d-bd83-9f161760fecb/ids:0 ADD
s ae5fe4a4-cd9e-421d-b501-93ea1c23bc73:1:/dev/ae5fe4a4-cd9e-421d-b501-93ea1c23bc73/ids:0 ADD
s 2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a:1:/dev/2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a/ids:0 ADD
s 7cf51212-8247-41b0-ab38-da5954059d7a:1:/dev/7cf51212-8247-41b0-ab38-da5954059d7a/ids:0 ADD
s 829b9bed-1b74-4f74-9d18-1006c87ef620:1:/dev/829b9bed-1b74-4f74-9d18-1006c87ef620/ids:0 ADD
s 8ef0cc64-77c5-49e0-a0f5-03cee9bbee20:1:/dev/8ef0cc64-77c5-49e0-a0f5-03cee9bbee20/ids:0 ADD
s f5597bd4-5c4f-4c4c-bd98-e8a33dace830:1:/dev/f5597bd4-5c4f-4c4c-bd98-e8a33dace830/ids:0 ADD
s 84dc4e3c-00fd-4263-84e8-fc246eeee6e9:1:/dev/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/ids:0 ADD
s 103047b0-7650-4d31-89ee-6a39da955804:1:/dev/103047b0-7650-4d31-89ee-6a39da955804/ids:0 ADD
s 301a38ee-f20b-4eaa-9781-722434d8c8a0:1:/dev/301a38ee-f20b-4eaa-9781-722434d8c8a0/ids:0 ADD
s 71702e1e-82d2-41d2-93df-678f2415da2a:1:/dev/71702e1e-82d2-41d2-93df-678f2415da2a/ids:0 ADD
s 6749f0e2-848c-4b54-b206-51ae90109efc:1:/dev/6749f0e2-848c-4b54-b206-51ae90109efc/ids:0 ADD
s 08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1:1:/dev/08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1/ids:0 ADD
s 747ea907-c24c-43ff-a389-660c729703b0:1:/dev/747ea907-c24c-43ff-a389-660c729703b0/ids:0 ADD


Status after all lockspaces were added:

# sanlock client status
daemon 095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
p -1 helper
p -1 listener
p 79833 backup-raw
p 3147 
p -1 status
s 82d93a3d-3d88-42ca-a328-f295dfc48ef2:1:/rhev/data-center/mnt/storage\:_export_13/82d93a3d-3d88-42ca-a328-f295dfc48ef2/dom_md/ids:0
s 7e25e16e-2cdf-4063-a638-3d61c0427ac6:1:/rhev/data-center/mnt/storage\:_export_19/7e25e16e-2cdf-4063-a638-3d61c0427ac6/dom_md/ids:0
s 0d38fccd-5585-40f7-9a73-d215b23bf7ed:1:/rhev/data-center/mnt/storage\:_export_18/0d38fccd-5585-40f7-9a73-d215b23bf7ed/dom_md/ids:0
s 0e49277b-b13e-49d6-bb56-e3da6d0cc397:1:/rhev/data-center/mnt/storage\:_export_12/0e49277b-b13e-49d6-bb56-e3da6d0cc397/dom_md/ids:0
s 2b5b232f-4a59-4ace-9e18-3841c63505de:1:/rhev/data-center/mnt/storage\:_export_16/2b5b232f-4a59-4ace-9e18-3841c63505de/dom_md/ids:0
s 839db77f-fde3-4e13-bfb6-56be604631ed:1:/rhev/data-center/mnt/storage\:_export_01/839db77f-fde3-4e13-bfb6-56be604631ed/dom_md/ids:0
s cd205f88-9b6a-46c6-9993-8569a5dfb8f1:1:/rhev/data-center/mnt/storage\:_export_15/cd205f88-9b6a-46c6-9993-8569a5dfb8f1/dom_md/ids:0
s f53d7919-d275-4ad4-ba57-f280000de397:1:/dev/f53d7919-d275-4ad4-ba57-f280000de397/ids:0
s e673e145-d644-44ac-a416-a8fc14b029ca:1:/dev/e673e145-d644-44ac-a416-a8fc14b029ca/ids:0
s 06e43bfc-2ffe-419e-843b-59a5d23165e1:1:/dev/06e43bfc-2ffe-419e-843b-59a5d23165e1/ids:0
s 301a38ee-f20b-4eaa-9781-722434d8c8a0:1:/dev/301a38ee-f20b-4eaa-9781-722434d8c8a0/ids:0
s c202aeb7-771a-4f8c-b94b-39b805695b35:1:/dev/c202aeb7-771a-4f8c-b94b-39b805695b35/ids:0
s 31e3675e-302a-4a22-ad9a-402c0c1e9765:1:/dev/31e3675e-302a-4a22-ad9a-402c0c1e9765/ids:0
s 6749f0e2-848c-4b54-b206-51ae90109efc:1:/dev/6749f0e2-848c-4b54-b206-51ae90109efc/ids:0
s ebf4cf14-0a33-4c3d-bd83-9f161760fecb:1:/dev/ebf4cf14-0a33-4c3d-bd83-9f161760fecb/ids:0
s 829b9bed-1b74-4f74-9d18-1006c87ef620:1:/dev/829b9bed-1b74-4f74-9d18-1006c87ef620/ids:0
s f5597bd4-5c4f-4c4c-bd98-e8a33dace830:1:/dev/f5597bd4-5c4f-4c4c-bd98-e8a33dace830/ids:0
s 08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1:1:/dev/08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1/ids:0
s 8ef0cc64-77c5-49e0-a0f5-03cee9bbee20:1:/dev/8ef0cc64-77c5-49e0-a0f5-03cee9bbee20/ids:0
s 71702e1e-82d2-41d2-93df-678f2415da2a:1:/dev/71702e1e-82d2-41d2-93df-678f2415da2a/ids:0
s 747ea907-c24c-43ff-a389-660c729703b0:1:/dev/747ea907-c24c-43ff-a389-660c729703b0/ids:0
s ae5fe4a4-cd9e-421d-b501-93ea1c23bc73:1:/dev/ae5fe4a4-cd9e-421d-b501-93ea1c23bc73/ids:0
s 2e7daa04-2cd4-4e93-bab8-866227a8a37b:1:/dev/2e7daa04-2cd4-4e93-bab8-866227a8a37b/ids:0
s 2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a:1:/dev/2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a/ids:0
s 7cf51212-8247-41b0-ab38-da5954059d7a:1:/dev/7cf51212-8247-41b0-ab38-da5954059d7a/ids:0
s 103047b0-7650-4d31-89ee-6a39da955804:1:/dev/103047b0-7650-4d31-89ee-6a39da955804/ids:0
s 84dc4e3c-00fd-4263-84e8-fc246eeee6e9:1:/dev/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/ids:0
s 1f4e2acd-6224-4345-ae06-1db97c403a0c:1:/rhev/data-center/mnt/storage\:_export_11/1f4e2acd-6224-4345-ae06-1db97c403a0c/dom_md/ids:0
s c345fc52-f82e-4904-9ad4-1cc84b0fc639:1:/rhev/data-center/mnt/storage\:_export_03/c345fc52-f82e-4904-9ad4-1cc84b0fc639/dom_md/ids:0
s 980601ff-1e81-4e10-bf79-a29d2d5b0e9f:1:/rhev/data-center/mnt/storage\:_export_10/980601ff-1e81-4e10-bf79-a29d2d5b0e9f/dom_md/ids:0
s e0549942-c08c-478b-9c8d-c116e98b59d4:1:/rhev/data-center/mnt/storage\:_export_09/e0549942-c08c-478b-9c8d-c116e98b59d4/dom_md/ids:0
s 8f870f86-3e4e-44a3-9bc6-c0af6f9d7944:1:/rhev/data-center/mnt/storage\:_export_05/8f870f86-3e4e-44a3-9bc6-c0af6f9d7944/dom_md/ids:0
s a87b183d-df2d-4de3-b370-507c6261c328:1:/rhev/data-center/mnt/storage\:_export_02/a87b183d-df2d-4de3-b370-507c6261c328/dom_md/ids:0
s 1297b300-a126-45e8-8a7f-c69edcf057f1:1:/rhev/data-center/mnt/storage\:_export_08/1297b300-a126-45e8-8a7f-c69edcf057f1/dom_md/ids:0
s 09525896-e9c1-4d24-8e4d-dab18346a274:1:/rhev/data-center/mnt/storage\:_export_07/09525896-e9c1-4d24-8e4d-dab18346a274/dom_md/ids:0
s d046c651-ac7c-49ca-9541-31e47b7c38ef:1:/rhev/data-center/mnt/storage\:_export_06/d046c651-ac7c-49ca-9541-31e47b7c38ef/dom_md/ids:0
s b2ff7cd6-64a6-453d-b71e-22fa4b11f16e:1:/rhev/data-center/mnt/storage\:_export_00/b2ff7cd6-64a6-453d-b71e-22fa4b11f16e/dom_md/ids:0
s 2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3:1:/rhev/data-center/mnt/storage\:_export_04/2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3/dom_md/ids:0
s 27afe55d-68d6-49c1-a085-83425b4bdd67:1:/rhev/data-center/mnt/storage\:_export_14/27afe55d-68d6-49c1-a085-83425b4bdd67/dom_md/ids:0
s 5bb456c1-bcef-4fb5-adf2-01de35f46f6d:1:/rhev/data-center/mnt/storage\:_export_17/5bb456c1-bcef-4fb5-adf2-01de35f46f6d/dom_md/ids:0
r 5bb456c1-bcef-4fb5-adf2-01de35f46f6d:SDM:/rhev/data-center/mnt/storage\:_export_17/5bb456c1-bcef-4fb5-adf2-01de35f46f6d/dom_md/leases:1048576:38 p 3147



When removing lockspaces, we can see that we remove all lockspaces concurrently:

# sanlock client status 
daemon 095bce26-9d5b-4ae2-b3bf-dddfc9cdf541
p -1 helper
p -1 listener
p -1 rem_lockspace
p 3147 
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 rem_lockspace
p -1 status
s 5bb456c1-bcef-4fb5-adf2-01de35f46f6d:1:/rhev/data-center/mnt/storage\:_export_17/5bb456c1-bcef-4fb5-adf2-01de35f46f6d/dom_md/ids:0 REM
s 27afe55d-68d6-49c1-a085-83425b4bdd67:1:/rhev/data-center/mnt/storage\:_export_14/27afe55d-68d6-49c1-a085-83425b4bdd67/dom_md/ids:0 REM
s 2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3:1:/rhev/data-center/mnt/storage\:_export_04/2b5ccfe5-d05e-43d8-ae60-d36e3f7dd2e3/dom_md/ids:0 REM
s b2ff7cd6-64a6-453d-b71e-22fa4b11f16e:1:/rhev/data-center/mnt/storage\:_export_00/b2ff7cd6-64a6-453d-b71e-22fa4b11f16e/dom_md/ids:0 REM
s d046c651-ac7c-49ca-9541-31e47b7c38ef:1:/rhev/data-center/mnt/storage\:_export_06/d046c651-ac7c-49ca-9541-31e47b7c38ef/dom_md/ids:0 REM
s 09525896-e9c1-4d24-8e4d-dab18346a274:1:/rhev/data-center/mnt/storage\:_export_07/09525896-e9c1-4d24-8e4d-dab18346a274/dom_md/ids:0 REM
s 1297b300-a126-45e8-8a7f-c69edcf057f1:1:/rhev/data-center/mnt/storage\:_export_08/1297b300-a126-45e8-8a7f-c69edcf057f1/dom_md/ids:0 REM
s a87b183d-df2d-4de3-b370-507c6261c328:1:/rhev/data-center/mnt/storage\:_export_02/a87b183d-df2d-4de3-b370-507c6261c328/dom_md/ids:0 REM
s 8f870f86-3e4e-44a3-9bc6-c0af6f9d7944:1:/rhev/data-center/mnt/storage\:_export_05/8f870f86-3e4e-44a3-9bc6-c0af6f9d7944/dom_md/ids:0 REM
s e0549942-c08c-478b-9c8d-c116e98b59d4:1:/rhev/data-center/mnt/storage\:_export_09/e0549942-c08c-478b-9c8d-c116e98b59d4/dom_md/ids:0 REM
s 980601ff-1e81-4e10-bf79-a29d2d5b0e9f:1:/rhev/data-center/mnt/storage\:_export_10/980601ff-1e81-4e10-bf79-a29d2d5b0e9f/dom_md/ids:0 REM
s c345fc52-f82e-4904-9ad4-1cc84b0fc639:1:/rhev/data-center/mnt/storage\:_export_03/c345fc52-f82e-4904-9ad4-1cc84b0fc639/dom_md/ids:0 REM
s 1f4e2acd-6224-4345-ae06-1db97c403a0c:1:/rhev/data-center/mnt/storage\:_export_11/1f4e2acd-6224-4345-ae06-1db97c403a0c/dom_md/ids:0 REM
s 84dc4e3c-00fd-4263-84e8-fc246eeee6e9:1:/dev/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/ids:0 REM
s 103047b0-7650-4d31-89ee-6a39da955804:1:/dev/103047b0-7650-4d31-89ee-6a39da955804/ids:0 REM
s 7cf51212-8247-41b0-ab38-da5954059d7a:1:/dev/7cf51212-8247-41b0-ab38-da5954059d7a/ids:0 REM
s 2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a:1:/dev/2f8c68ef-d5c1-4f34-9a6a-f7cb2f0aef0a/ids:0 REM
s 2e7daa04-2cd4-4e93-bab8-866227a8a37b:1:/dev/2e7daa04-2cd4-4e93-bab8-866227a8a37b/ids:0 REM
s ae5fe4a4-cd9e-421d-b501-93ea1c23bc73:1:/dev/ae5fe4a4-cd9e-421d-b501-93ea1c23bc73/ids:0 REM
s 747ea907-c24c-43ff-a389-660c729703b0:1:/dev/747ea907-c24c-43ff-a389-660c729703b0/ids:0 REM
s 71702e1e-82d2-41d2-93df-678f2415da2a:1:/dev/71702e1e-82d2-41d2-93df-678f2415da2a/ids:0 REM
s 8ef0cc64-77c5-49e0-a0f5-03cee9bbee20:1:/dev/8ef0cc64-77c5-49e0-a0f5-03cee9bbee20/ids:0 REM
s 08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1:1:/dev/08c7e615-9fa0-4abd-8b0a-4e7f77bf3bd1/ids:0 REM
s f5597bd4-5c4f-4c4c-bd98-e8a33dace830:1:/dev/f5597bd4-5c4f-4c4c-bd98-e8a33dace830/ids:0 REM
s 829b9bed-1b74-4f74-9d18-1006c87ef620:1:/dev/829b9bed-1b74-4f74-9d18-1006c87ef620/ids:0 REM
s ebf4cf14-0a33-4c3d-bd83-9f161760fecb:1:/dev/ebf4cf14-0a33-4c3d-bd83-9f161760fecb/ids:0 REM
s 6749f0e2-848c-4b54-b206-51ae90109efc:1:/dev/6749f0e2-848c-4b54-b206-51ae90109efc/ids:0 REM
s 31e3675e-302a-4a22-ad9a-402c0c1e9765:1:/dev/31e3675e-302a-4a22-ad9a-402c0c1e9765/ids:0 REM
s c202aeb7-771a-4f8c-b94b-39b805695b35:1:/dev/c202aeb7-771a-4f8c-b94b-39b805695b35/ids:0 REM
s 301a38ee-f20b-4eaa-9781-722434d8c8a0:1:/dev/301a38ee-f20b-4eaa-9781-722434d8c8a0/ids:0 REM
s 06e43bfc-2ffe-419e-843b-59a5d23165e1:1:/dev/06e43bfc-2ffe-419e-843b-59a5d23165e1/ids:0 REM
s e673e145-d644-44ac-a416-a8fc14b029ca:1:/dev/e673e145-d644-44ac-a416-a8fc14b029ca/ids:0 REM
s f53d7919-d275-4ad4-ba57-f280000de397:1:/dev/f53d7919-d275-4ad4-ba57-f280000de397/ids:0 REM
s cd205f88-9b6a-46c6-9993-8569a5dfb8f1:1:/rhev/data-center/mnt/storage\:_export_15/cd205f88-9b6a-46c6-9993-8569a5dfb8f1/dom_md/ids:0 REM
s 839db77f-fde3-4e13-bfb6-56be604631ed:1:/rhev/data-center/mnt/storage\:_export_01/839db77f-fde3-4e13-bfb6-56be604631ed/dom_md/ids:0 REM
s 2b5b232f-4a59-4ace-9e18-3841c63505de:1:/rhev/data-center/mnt/storage\:_export_16/2b5b232f-4a59-4ace-9e18-3841c63505de/dom_md/ids:0 REM
s 0e49277b-b13e-49d6-bb56-e3da6d0cc397:1:/rhev/data-center/mnt/storage\:_export_12/0e49277b-b13e-49d6-bb56-e3da6d0cc397/dom_md/ids:0 REM
s 0d38fccd-5585-40f7-9a73-d215b23bf7ed:1:/rhev/data-center/mnt/storage\:_export_18/0d38fccd-5585-40f7-9a73-d215b23bf7ed/dom_md/ids:0 REM
s 7e25e16e-2cdf-4063-a638-3d61c0427ac6:1:/rhev/data-center/mnt/storage\:_export_19/7e25e16e-2cdf-4063-a638-3d61c0427ac6/dom_md/ids:0 REM
s 82d93a3d-3d88-42ca-a328-f295dfc48ef2:1:/rhev/data-center/mnt/storage\:_export_13/82d93a3d-3d88-42ca-a328-f295dfc48ef2/dom_md/ids:0 REM


Here some performance results:

|------------------------|------------|-----------|------------|
| operation              | io_timeout | 8 workers | 50 workers |
|-------------------------------------------------|------------|
| activate host          |         10 |       123 |         47 |
|                        |         20 |       224 |         71 |
| deactivate host        |         10 |        16 |          8 |
|                        |         20 |        16 |          5 |
|------------------------|------------|-----------|------------|


The time is measured from performing the action in RHV UI, until all lockspaces
are added (ADD is removed in "sanlock client status").

Activate host after clean shutdown took less then 2 minutes. Based on tests
with 20 storage domains and 8 workers, 40 storage domains would take about 9 minutes.

Since RHV QE do not have capacity to test this now, and this is  a tiny change,
I think this is good enough.

Comment 10 Nir Soffer 2021-01-04 13:03:31 UTC

David, do we need to clone this bug to 8.3.z to get this fix in 8.3.z?

Also we need an official build with this fix for testing bug 1903358.

Comment 11 David Teigland 2021-01-04 15:53:34 UTC

(In reply to Nir Soffer from comment #10)
> David, do we need to clone this bug to 8.3.z to get this fix in 8.3.z?

I'm not sure if the zstream process involves manually cloning the bug or if that's automated.

Comment 12 Bob Handlin 2021-01-04 16:56:30 UTC

I don't think PMs do this for RHEL 8...

Bob

Comment 17 errata-xmlrpc 2021-05-18 15:09:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sanlock bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1711