Bug 1784947

Summary: Slowness when creating multiple (30+) NFS-backed volumes
Product: Red Hat OpenStack Reporter: David Vallee Delisle <dvd>
Component: python-os-brickAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: apevec, eharney, jmelvin, jschluet, lhh, ltoscano
Target Milestone: z13Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-os-brick-2.3.9-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-28 18:23:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Vallee Delisle 2019-12-18 19:01:16 UTC 
Description of problem:
When creating multiple nfs-backed volumes (using the netapp driver, in this case, but I believe it doesn't matter), it takes longer and longer to mount them with os_brick [1]. We also end up with defunct processes [2]

I strongly believe it's because the _read_mount() function runs "mount" to generate a list of what's mounted and it looks like we're getting blocked or locked at some point when it's ran concurrently.

We tested with reading /proc/mounts instead and we see great improvement.

Version-Release number of selected component (if applicable):
python2-os-brick-2.3.4-2.el7ost.noarch

How reproducible:
All the time

Steps to Reproduce:
1. configure nfs backed storage
2. create 30 volumes


Actual results:
Takes a while before they get created

Expected results:
Should be faster

Additional info:
[1]
~~~
$ grep -oP " 21:(4[8-9]|5[0-9]):.*mount.*returned: [0-9]+ in \K[0-9\.]+" cinder-volume.log | tr "\n" " "
0.629 0.651 0.642 0.688 0.650 0.730 0.665 0.701 0.726 1.310 0.660 1.361 1.344 1.323 1.307 2.001 2.034 2.060 2.071 2.044 2.677 2.044 3.341 2.687 3.301 3.289 3.326 3.960 3.342 3.343 3.262 2.613 3.240 3.271 4.691 4.790 4.812 5.530 5.565 5.019 3.756 4.435 5.132 5.080 5.082 5.697 5.679 4.877 5.587 4.844 5.514 6.146 6.153 7.560 7.583 7.556 6.963 7.638 6.297 7.745 7.803 7.193 7.880 8.512 8.543 9.249 9.228 8.505 9.221 9.887 9.178 9.805 9.091 9.766 9.797 10.553 10.541 10.599 11.292 9.269 11.325 9.941 10.623 11.207 11.229 12.116 11.461 12.143 12.149 11.984 12.719 13.316 13.313 13.275 12.598 12.619 13.256 11.957 12.652 11.975 13.322 14.037 13.859 13.890 13.946 13.895 13.921 13.980 14.690 14.047 14.732 14.024 14.612 15.422 14.717 15.399 14.064 15.434 14.750 16.144 15.449 15.425 14.689 15.371 16.072 16.205 16.225 15.561 16.230 14.864 16.188 18.131 19.563 18.909 19.604 17.534 19.603 18.906 18.212 19.697 20.422 19.656 20.345 19.702 20.378 19.760 20.446 20.199 19.526 20.186 20.293 20.961 20.958 19.607 20.278 19.563 20.316 18.889 19.607 18.197 20.240 20.969 18.096 19.575 18.792 20.000 20.688 20.064 21.418 20.754 22.060 21.366 21.925 21.259 23.298 22.588 23.303 23.247 24.620 23.330 23.982 24.668 24.618 23.895 23.194 25.247 24.531 23.896 25.121 25.709 24.339 25.664 26.334 25.014 26.342 24.984 27.704 26.324 25.010 27.604 26.248 26.938 27.619 28.245 28.239 27.576 28.863 29.491 29.514 28.842 28.811 29.517 29.423 28.025 28.722 28.651 29.323 29.400 28.752 28.115 28.695 30.092 27.411 28.738 30.057 30.026 29.365 28.722 29.993 28.693 27.392 29.364 28.066 27.914 29.980 27.226 29.291 28.614 26.580 29.956 29.292 29.366 30.014 30.662 31.402 30.705 31.405 30.070 31.364 31.385 30.727 30.638 31.328 31.298 29.960 30.629 30.561 31.209 31.223 30.538 31.241 30.521 29.841 27.908 27.264 28.555 29.203 31.207 30.548 29.886 30.631 31.274 29.924 30.566 29.281 31.242 28.636 29.230 28.569 30.561 31.265 27.908 29.914 31.269 29.908 30.597 29.223 29.896 29.948 29.259 28.666 29.309 29.355 28.697 29.391 30.065 29.415 30.748 29.484 30.111 30.209 30.875 30.914 30.237 30.208 30.944 29.560 28.792 26.863 30.883 27.520 28.150 30.212 29.518 30.899 30.245 28.335 27.694 30.924 30.288 29.634 28.990 30.168 28.849 30.868 29.512 28.202 30.160 29.471 28.804 29.375 30.076 30.000 29.314 30.566 31.232 30.511 29.866 31.219 30.535 29.876 29.282 29.944 29.954 29.258 29.292 29.977 28.589 29.252 27.935 29.966 29.267 28.574 29.959 26.614 27.272 27.919 29.961 29.296 28.654 29.331 27.360 29.997 26.741 28.004 27.601 30.208 28.282 29.578 28.929 30.215 29.569 30.260 29.613 30.311 29.673 29.709 28.441 29.126 27.775 28.652 27.965 28.676 27.899 28.624 27.916 28.603 27.250 29.329 28.672 29.338 28.010 27.342 28.041 26.712 30.065 26.047 28.717 29.379 27.381 29.895 29.163 26.435 28.506 27.127 29.880 27.824 28.298 27.647 26.969 28.927 29.641 29.625 28.955 29.634 28.932 28.926 29.630 29.537 28.750 29.478 29.145 28.485 29.043 28.341 29.024 27.590 28.276 28.936 26.920 28.046 24.644 27.356 25.338 26.674 25.797 26.467 27.136 24.507 25.181 23.877 27.202 25.348 24.678 25.965 26.592 24.058 27.236 24.644 26.035 25.388 26.680 27.373 24.020 25.899 26.561 26.500 26.494 25.197 25.814 27.956 27.330 27.414 28.071 28.168 27.577 28.226 26.381 27.018 28.299 27.662 28.372 27.765 25.854 28.384 27.142 26.485 27.742 28.411 27.123 25.819 26.472 27.683 25.660 27.688 27.014 26.331 25.033 24.356 27.152 26.508 25.238 27.819 25.878 27.868 28.521 29.291 28.639 29.332 27.991 28.643 27.270 27.932 27.913 27.925 27.229 27.926 27.851 27.203 25.860 26.551 27.359 26.719 26.104 27.396 25.421 24.798 26.195 25.573 27.485 26.816 28.257 25.548 26.239 28.982 27.604 26.917 28.283 27.429 28.105 28.801 26.763 25.495 26.143 28.277 27.602 27.627 26.951 26.283 26.920 26.250 26.947 26.881 26.819 27.516 27.424 26.017 26.735 27.427 27.320 25.277 27.285 25.952 24.657 26.616 24.526 25.867 26.538 27.233 26.595 27.286 24.610 23.990 25.906 26.593 25.260 26.545 25.905 25.237 24.559 23.192 23.895 25.084 25.754 26.430 26.422 27.148 27.151 27.852 27.243 27.904 27.319 25.453 26.772 26.116 26.190 23.607 24.238 25.519 26.186 24.233 22.895 24.930 25.617 25.530 24.174 24.851 26.160 25.519 24.232 24.902 23.008 24.290 25.617 22.368 24.982 23.671 24.463 25.803 25.137 25.760 26.380 25.669 24.297 24.982 24.909 24.910 24.205 26.085 25.364 26.062 26.012 25.363 24.714 24.081 24.752 25.419 26.099 24.767 25.452 24.113 23.440 24.123 22.090 22.775 21.328 22.002 24.059 22.690 23.380 23.303 23.996 22.683 24.733 23.369 23.398 22.767 23.427 23.527 22.916 24.210 21.697 22.993 22.342 22.976 22.304 21.636 21.619 20.293 20.960 22.286 22.939 23.578 22.906 22.254 21.582 22.157 20.834 23.522 21.504 22.829 22.074 24.106 23.428 23.401 22.748 24.133 22.778 23.475 22.745 22.079 23.369 22.089 22.733 23.393 23.422 21.412 22.723 20.757 22.068 22.783 19.557 20.194 20.842 19.640 20.943 19.006 20.295 20.455 19.106 19.772 19.135 20.502 21.205 19.816 20.517 20.511 19.843 18.499 19.170 17.867 18.543 17.869 17.196 19.219 18.544 19.144 18.478 19.830 19.139 18.484 17.808 19.177 19.123 19.844 17.095 17.777 18.985 18.344 18.313 18.272 20.825 20.815 20.152 20.816 18.820 20.169 19.556 18.865 18.205 16.139 16.802 17.500 15.534 16.194 14.256 14.957 15.610 17.681 16.419 15.132 16.504 15.818 15.798 15.104 14.484 13.794 13.088 13.118 13.792 13.089 14.412 14.359 13.604 14.356 15.056 15.653 15.019
~~~

[2]
~~~
root      839999  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840004  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840079  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840102  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840142  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840216  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840238  767775  3 21:13 ?        00:00:00 [mount] <defunct>
root      840246  767775  4 21:13 ?        00:00:00 [mount] <defunct>
root      840371  767775  4 21:13 ?        00:00:00 [mount] <defunct>
root      840481  767775  4 21:13 ?        00:00:00 [mount] <defunct>
~~~
Comment 2 Jeremy 2020-07-29 18:29:36 UTC 
*** Bug 1861866 has been marked as a duplicate of this bug. ***
Comment 3 Jeremy 2020-07-29 19:37:33 UTC 
*** Bug 1849821 has been marked as a duplicate of this bug. ***
Comment 19 errata-xmlrpc 2020-10-28 18:23:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4388