Description of problem: Currently, on each targetcli create call, rtslib will rebuild its bs_cache, so as the /sys/kernel/config/target/core dir gets more entries this takes longer and longer to scan. Hence using repetitive targetcli in the block create for creating iqn, backend, setting acls, setting globals will induce too much delay in block create. As the number of blocks on the node increases, the delay will be too longer. This does not happen if we open targetcli in interactive mode and just do multiple create commands form it, since the bs_cache is build once. Note: The bs_cache build(one time) will still be there. But what we save is multiple bs_cache building. Read More: https://goo.gl/8aYT38 Currently the cli can wait till 300 sec for daemon to return the create rpc request $ for i in {1...1024}; do echo "===== $i ====="; time gluster-block create sample/block$i ha 3 10.16.153.96,10.16.153.99,10.16.153.102 1MiB --json-pretty [...] ===== 429 ===== { "IQN":"iqn.2016-12.org.gluster-block:91dbf2d7-8bbf-44f2-9972-05b8f75b5df9", "PORTAL(S)":[ "10.16.153.96:3260", "10.16.153.99:3260", "10.16.153.102:3260" ], "RESULT":"SUCCESS" } real 4m58.734s user 0m0.005s sys 0m0.001s ===== 430 ===== Did not receive any response from gluster-block daemon. Please check log files to find the reason real 5m0.109s user 0m0.003s sys 0m0.006s [...] How reproducible: 100% Steps to Reproduce: 1. create 1024 block devices in a loop Actual results: Cli should return the response Expected results: Cli times out as the create delay is longer that 300 sec Additional info: Use targetcli interactive mode to avoid delays seen from unwanted bs_cache rebuilding
Gluster-block create request times out after ~300th block. I tried this twice, on two different setups just to be sure, and was able to hit this both the times. Logs/sosreports will be copied at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/<bugnumber>/
At CNS, creation of 350 block devices was successful. These devices are created via gluster-block provisionor --> heketi --> gluster-blockd. There were no timeouts seen. (outut of all pvc's shall be attached shortly) so we are good with CNS perspective. builds used: ------------ cns-deploy-5.0.0-43.el7rhgs.x86_64 gluster-block-0.2.1-12.el7rhgs.x86_64 titbit: ------- 351st device took close to 20 seconds to get created. so we are good w.r.t timedelay for block device creation I suppose. [root@dhcp46-207 ~]# oc get pvc/mongodb-351 NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE mongodb-351 Pending glusterblock 20s [root@dhcp46-207 ~]# oc get pvc/mongodb-351 NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE mongodb-351 Bound pvc-c1db0ec2-9d11-11e7-af38-005056b32785 1Gi RWO glusterblock 21s
Created attachment 1327802 [details] attachment for comment 12, output of 'oc get pvc'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2773