Bug 1572215

Summary: [GSS] Volume mismatch and Excessive server load on CNS Gluster nodes
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Damian Wojsław <dwojslaw>
Component: heketiAssignee: John Mulligan <jmulligan>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Rachael <rgeorge>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.3CC: dwojslaw, hchiramm, jmulligan, madam, pprakash, rhs-bugs, rtalur, sankarshan, storage-qa-internal, vinug
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-23 14:04:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1573420, 1641915, 1642792    

Description Damian Wojsław 2018-04-26 12:30:16 UTC 
Description of problem:
Gluster nodes observe 100% CPU load during pv provisioning.
There seems to be some mismatch between actual number of LVM devices and what heketi thinks the number is.

Version-Release number of selected component (if applicable):
atomic-openshift-3.6.173.0.112-1.git.0.17301ab.el7.x86_64   Mon Apr 16 13:10:26 2018
atomic-openshift-clients-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 Mon Apr 16 13:10:18 2018
atomic-openshift-docker-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch Mon Apr 16 10:36:37 2018
atomic-openshift-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch Mon Apr 16 10:36:38 2018
atomic-openshift-node-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 Mon Apr 16 13:10:28 2018
atomic-openshift-sdn-ovs-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 Mon Apr 16 13:10:29 2018
atomic-openshift-utils-3.6.173.0.112-1.git.2.17a27d3.el7.noarch Mon Apr 16 10:36:23 2018
atomic-registries-1.22.1-1.gitd36c015.el7.x86_64            Mon Apr 16 10:34:20 2018

glusterfs-3.8.4-53.el7.x86_64                               Mon Apr 16 10:35:23 2018
glusterfs-client-xlators-3.8.4-53.el7.x86_64                Mon Apr 16 10:34:19 2018
glusterfs-fuse-3.8.4-53.el7.x86_64                          Mon Apr 16 10:36:27 2018
glusterfs-libs-3.8.4-53.el7.x86_64                          Mon Apr 16 10:34:17 2018

heketi-client-5.0.1-1.el7.x86_64                            Mon Apr 16 10:36:40 2018

How reproducible:


Steps to Reproduce:
1. Issue PVC
2. Gluster nodes get high load
3. Writes to pods hang

Actual results:
Gluster nodes observe 100% CPU load during pv provisioning. 

Expected results:
PVC gets successfully done. Writes are unaffected.

Additional info: