Bug 1706009 - OCS upgrade playbook does not check brick capacity before upgrading the OCS cluster
Summary: OCS upgrade playbook does not check brick capacity before upgrading the OCS c...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: cns-ansible
Version: ocs-3.11
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: OCS 3.11.z Async
Assignee: John Mulligan
QA Contact: Nitin Goyal
URL:
Whiteboard:
Depends On:
Blocks: 1703695
TreeView+ depends on / blocked
 
Reported: 2019-05-03 11:49 UTC by Ashmitha Ambastha
Modified: 2020-02-19 16:16 UTC (History)
10 users (show)

Fixed In Version: openshift-ansible-3.11.116
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-19 16:16:44 UTC
Embargoed:


Attachments (Terms of Use)

Description Ashmitha Ambastha 2019-05-03 11:49:31 UTC
Description of problem:
Before performing an upgrade, the upgrade playbook must check whether the bricks are 90% full or not.

Upgrade playbook must ensure that the bricks are not more than 90% full using the following command:

# df -kh | grep -v ^Filesystem | awk '{if($5>"90%") print $0}'

If the bricks are more than 90% full, the upgrade must not be performed and the upgrade playbook should fail with appropriate errors. 

Version-Release number of selected component (if applicable): OCS 3.11 and openshift-ansible 3.11.110 and above. 


How reproducible: always. 


Steps to Reproduce:
1. Deploy a cluster to be upgraded. 
2. Run IOs on the cluster before performing an upgrade using the upgrade playbook. 
3. Run the upgrade playbook, 
# ansible-playbook -i <inv_file> /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/upgrade.yml 
4. the playbook should check and ensure that the bricks are not more than 90% full before performing an upgrade to the cluster. 

Actual results: The upgrade playbook does not check the brick capacity before performing an upgrade and if the bricks are more than 90% full, the upgrade playbook does not fail. 


Expected results: The upgrade playbook should check the brick capacity before performing an upgrade and if the bricks are more than 90% full, the upgrade playbook must fail.

Comment 2 Jose A. Rivera 2019-05-03 19:35:52 UTC
Valid bug, taking this in.

Comment 3 Jose A. Rivera 2019-05-03 19:36:48 UTC
Talur, is the command listed in comment #1 sufficient for this purpose?

Comment 4 Raghavendra Talur 2019-05-04 14:01:24 UTC
I modified the command a little to only check heketi devices. Also, as mentioned in the offline chat, we don't need to be as conservative and have bumped the number to 96 percent.

df -kh | grep -v ^Filesystem | grep "/var/lib/heketi" | awk '{if($5>"96%") print $0}'

Comment 5 Jose A. Rivera 2019-05-07 14:46:07 UTC
PR is upstream: https://github.com/openshift/openshift-ansible/pull/11588

Comment 6 Jose A. Rivera 2019-05-08 14:05:26 UTC
PR is merged.

Comment 12 Jose A. Rivera 2019-05-14 13:20:11 UTC
New PR is merged, build should happen tonight.

Comment 14 RamaKasturi 2019-06-19 11:21:10 UTC
Hello Jose / talur ,

   Can one of you provide a devel_ack for the bug ?

Thanks
kasturi


Note You need to log in before you can comment on or make changes to this bug.