+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1432385 +++ ====================================================================== Description of problem: If there's a large number of LUNs attached to the hypervisor then 'imgbase layout' takes very long time to complete, at the same time 'imgbase check' and basic LVM commands like 'lvs', 'vgs', 'pvs' do not have such problem. Version-Release number of selected component (if applicable): Red Hat Virtualization Host 4.0 (el7.3) Kernel 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux imgbased-0.8.11-0.1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. Install latest RHV-H host 2. Attach 250+ LUNs 3. run 'imgbase layout' Actual results: Command takes ages to complete (most probably because it scans the devices which it should not scan) Expected results: Command completes in shorter time Additional info: note the difference between 'imbase check' and 'imgbase layout' (the latter takes ~70 min to complete): #time imgbase check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK real 0m13.138s user 0m6.414s sys 0m0.812s #time imgbase layout rhvh-4.0-0.20170104.0 +- rhvh-4.0-0.20170104.0+1 rhvh-4.0-0.20170201.0 +- rhvh-4.0-0.20170201.0+1 real 70m19.711s user 5m42.503s sys 11m15.482s It also has been noted by the customer that 'yum' command also runs with huge delay, should we open a separate bug on that? (Originally by Igor Netkachev)
Can you please re-test with 4.0.7 which has been released yesterday? It should have improvements in this area (Originally by Sandro Bonazzola)
Sandro, Customer has just re-tested with the latest RHV-H and the problem is still there: # time imgbase check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK real 0m16.509s user 0m8.052s sys 0m1.818s # time imgbase layout rhvh-4.0-0.20170104.0 +- rhvh-4.0-0.20170104.0+1 rhvh-4.0-0.20170201.0 +- rhvh-4.0-0.20170201.0+1 rhvh-4.0-0.20170307.0 +- rhvh-4.0-0.20170307.0+1 real 95m14.719s user 10m0.454s sys 24m36.242s -- Kind Regards, Igor Netkachev Technical Support Engineer Red Hat Global Support Services (Originally by Igor Netkachev)
Imgbased should just be checking "lvs -oname, tags" and filtering. No scanning of lvm beyond tags is done, though the code may not be very optimized for large sets. What is the runtime when checking tags? Is there a test system I can look at? (Originally by Ryan Barry)
QE attempted to reproduce this issue in-house, the details steps are as follow, and we will keep the environment to Ryan by email for further investigation and debug Since we don't have an environment as complicated as the customer's, we figured out a way to simulate the customer's situation, which is to create 1000 lvs, and then run command "time imgbase layout". Test versions: redhat-virtualization-host-4.0-20170307.1 imgbased-0.8.16-0.1.el7ev.noarch Steps: 1. Install RHVH on the local disk of a machine which has 21 iSCSI luns. 2. Register to engine, and attach the 21 iSCSI luns in one storage domain. 3. Run a script to create 1000 lvs by "lvcreate -L 1500M 78c869fe-4447-4a49-9a8d-5a132c7cb6a1" 4. Run "time imgbase layout" Results: 1. After step4, the cmd failed with errors, took a long time: [root@bootp-73-75-11 ~]# time imgbase layout Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 51, in <module> CliApplication() File "/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication app.hooks.emit("post-arg-parse", args) File "/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", line 172, in post_argparse print(layout.dumps()) File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", line 200, in dumps return self.app.imgbase.layout() File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 138, in layout return self.naming.layout() File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 109, in layout tree = self.tree(lvs) File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 205, in tree names = datasource() File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 91, in list_our_lv_names our_lvs = [lv for lv in lvs if has_our_tag(lv)] File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 89, in has_our_tag for tag in our_tags) File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 89, in <genexpr> for tag in our_tags) File "/usr/lib/python2.7/site-packages/imgbased/lvm.py", line 239, in tags self.lvm_name]).split(",") File "/usr/lib/python2.7/site-packages/imgbased/utils.py", line 331, in lvs return self.call(["lvs"] + args, **kwargs) File "/usr/lib/python2.7/site-packages/imgbased/utils.py", line 385, in call return super(LvmBinary, self).call(*args, stderr=DEVNULL, **kwargs) File "/usr/lib/python2.7/site-packages/imgbased/utils.py", line 326, in call stdout = call(*args, **kwargs) File "/usr/lib/python2.7/site-packages/imgbased/utils.py", line 120, in call return subprocess.check_output(*args, **kwargs).strip() File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['lvs', '--noheadings', '-olv_tags', u'78c869fe-4447-4a49-9a8d-5a132c7cb6a1/lvol779']' returned non-zero exit status 5 real 81m44.297s user 2m56.766s sys 6m51.057s 2. Try to run "imgbase layout" again, this time it succeed without errors, and total time is shorter: [root@bootp-73-75-11 ~]# time imgbase layout rhvh-4.0-0.20170307.0 +- rhvh-4.0-0.20170307.0+1 real 13m43.746s user 1m27.120s sys 0m54.498s Additional info: 1. When running "time imgbase layout", cmd "lvs --noheadings -olv_tags vgname/lvname" will be called for each lv. [root@bootp-73-75-11 ~]# ps -ef |grep lvs root 4489 4346 0 14:51 pts/0 00:00:00 lvs --noheadings -olv_tags 78c869fe-4447-4a49-9a8d-5a132c7cb6a1/lvol111 root 4491 50102 0 14:51 pts/1 00:00:00 grep --color=auto lvs 2. The runtime of "lvs --noheadings -olv_tags vgname/lvname" while running "time imgbase layout" for the first time is: [root@bootp-73-75-11 ~]# time lvs --noheadings -olv_tags 78c869fe-4447-4a49-9a8d-5a132c7cb6a1/lvol139 WARNING: Not using lvmetad because config setting use_lvmetad=0. WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache). real 0m3.081s user 0m0.090s sys 0m0.393s 3. The runtime of "lvs --noheadings -olv_tags vgname/lvname" after "time imgbase layout" finished is: [root@bootp-73-75-11 ~]# time lvs --noheadings -olv_tags 78c869fe-4447-4a49-9a8d-5a132c7cb6a1/lvol139 WARNING: Not using lvmetad because config setting use_lvmetad=0. WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache). real 0m0.252s user 0m0.041s sys 0m0.026s We can see from the results that the time of "lvs --noheadings -olv_tags vgname/lvname" before "time imgbase layout" finished is much longer than the one after "time imgbase layout" finished, and this may be why the first time running "time imgbase layout" consumed much more time than the second time. (Originally by Daijie Guo)
Verified on rhvh-3.6-0.20170424.0 based on the steps in comment 7, below is the details result: [root@dell-per515-02 ~]# imgbase w [INFO] You are on rhvh-3.6-0.20170424.0+1 [root@dell-per515-02 ~]# rpm -q imgbased imgbased-0.8.18-0.1.el7ev.noarch [root@dell-per515-02 ~]# lvs | wc -l 1019 [root@dell-per515-02 ~]# time imgbase w [INFO] You are on rhvh-3.6-0.20170424.0+1 real 0m0.385s user 0m0.222s sys 0m0.164s [root@dell-per515-02 ~]# time imgbase layout rhvh-3.6-0.20170424.0 +- rhvh-3.6-0.20170424.0+1 real 0m1.117s user 0m0.516s sys 0m0.589s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1212