Bug 1482057 - gluster-block: cli times out after ~400 block creates on a node
Summary: gluster-block: cli times out after ~400 block creates on a node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: gluster-block
Version: rhgs-3.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.3.0
Assignee: Prasanna Kumar Kalever
QA Contact: Sweta Anandpara
URL:
Whiteboard:
Depends On:
Blocks: 1417151
TreeView+ depends on / blocked
 
Reported: 2017-08-16 11:39 UTC by Prasanna Kumar Kalever
Modified: 2017-09-21 04:20 UTC (History)
6 users (show)

Fixed In Version: gluster-block-0.2.1-9.el7rhgs
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-21 04:20:54 UTC
Embargoed:


Attachments (Terms of Use)
attachment for comment 12, output of 'oc get pvc' (103.69 KB, text/plain)
2017-09-19 08:19 UTC, krishnaram Karthick
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:2773 0 normal SHIPPED_LIVE new packages: gluster-block 2017-09-21 08:16:22 UTC

Description Prasanna Kumar Kalever 2017-08-16 11:39:21 UTC
Description of problem:

Currently, on each targetcli create call, rtslib will rebuild its bs_cache, so as the /sys/kernel/config/target/core dir gets more entries this takes longer and longer to scan. Hence using repetitive targetcli in the block create for creating iqn, backend, setting acls, setting globals will induce too much delay in block create. As the number of blocks on the node increases, the delay will be too longer.
     
This does not happen if we open targetcli in interactive mode and just do multiple create commands form it, since the bs_cache is build once.

Note: The bs_cache build(one time) will still be there. But what we save is multiple bs_cache building.

Read More:
https://goo.gl/8aYT38                                                       

Currently the cli can wait till 300 sec for daemon to return the create rpc request

$ for i in {1...1024}; do echo "===== $i ====="; time gluster-block create sample/block$i ha 3 10.16.153.96,10.16.153.99,10.16.153.102 1MiB --json-pretty
[...]

===== 429 =====
{
  "IQN":"iqn.2016-12.org.gluster-block:91dbf2d7-8bbf-44f2-9972-05b8f75b5df9",
  "PORTAL(S)":[
    "10.16.153.96:3260",
    "10.16.153.99:3260",
    "10.16.153.102:3260"
  ],
  "RESULT":"SUCCESS"
}

real    4m58.734s
user    0m0.005s
sys     0m0.001s
===== 430 =====
Did not receive any response from gluster-block daemon. Please check log files to find the reason

real    5m0.109s
user    0m0.003s
sys     0m0.006s

[...]

How reproducible:
100%

Steps to Reproduce:
1. create 1024 block devices in a loop

Actual results:
Cli should return the response

Expected results:
Cli times out as the create delay is longer that 300 sec

Additional info:
Use targetcli interactive mode to avoid delays seen from unwanted bs_cache rebuilding

Comment 11 Sweta Anandpara 2017-09-18 00:07:32 UTC
Gluster-block create request times out after ~300th block. I tried this twice, on two different setups just to be sure, and was able to hit this both the times.

Logs/sosreports will be copied at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/<bugnumber>/

Comment 12 krishnaram Karthick 2017-09-19 08:17:26 UTC
At CNS, creation of 350 block devices was successful. These devices are created via gluster-block provisionor --> heketi --> gluster-blockd. There were no timeouts seen. (outut of all pvc's shall be attached shortly)

so we are good with CNS perspective. 

builds used:
------------
cns-deploy-5.0.0-43.el7rhgs.x86_64
gluster-block-0.2.1-12.el7rhgs.x86_64

titbit:
-------
351st device took close to 20 seconds to get created. so we are good w.r.t timedelay for block device creation I suppose.

[root@dhcp46-207 ~]# oc get pvc/mongodb-351
NAME          STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mongodb-351   Pending                                      glusterblock   20s
[root@dhcp46-207 ~]# oc get pvc/mongodb-351
NAME          STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mongodb-351   Bound     pvc-c1db0ec2-9d11-11e7-af38-005056b32785   1Gi        RWO           glusterblock   21s

Comment 13 krishnaram Karthick 2017-09-19 08:19:02 UTC
Created attachment 1327802 [details]
attachment for comment 12, output of 'oc get pvc'

Comment 16 errata-xmlrpc 2017-09-21 04:20:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2773


Note You need to log in before you can comment on or make changes to this bug.