Bug 1732944

Summary: scratch space pvc is created on a different node
Product: Container Native Virtualization (CNV) Reporter: Tzvi Avni <tavni>
Component: StorageAssignee: Adam Litke <alitke>
Status: CLOSED WORKSFORME QA Contact: Natalie Gavrielov <ngavrilo>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.0CC: alitke, cnv-qe-bugs, grajaiya, ngavrilo, ycui
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-28 12:20:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
'kubectl get pvc -o yaml' and 'kubect describe pvc' for both pvc and scratcg-pvc none

Description Tzvi Avni 2019-07-24 19:39:04 UTC 
Created attachment 1593215 [details]
'kubectl get pvc -o yaml' and 'kubect describe pvc' for both pvc and scratcg-pvc

Description of problem:
While importing qcow2 image, importing pvc is created on one node while scratch space pvc is created on a different node. importing fails.
This was found while running scratch_space automation tests.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Running the tests on ssh 10.0.152.170:
ssh cnv-qe-jenkins@10.0.152.170 -i ~/.ssh/cnv-qe-jenkins.key

2.run the test:
pipenv run pytest /home/cnv-qe-jenkins/tzvika/cnv-tests/tests/storage/test_scratch_space.py --tc-file=tests/test-config.yaml --tc-format=yaml --log-level=INFO --log-cli-level=INFO -v --pdb -s

3.

Actual results:
pvc and scratch-space pvc's are created on different nodes


Expected results:
both PVC's should be created on the same node


Additional info:
Comment 2 Ying Cui 2019-07-31 12:22:30 UTC 
Could you check this bug?
Comment 3 Ying Cui 2019-08-07 12:27:20 UTC 
Natalie, could you reproduce it?
Comment 4 Gowrishankar Rajaiyan 2019-08-18 05:07:05 UTC 
This case can be confirmed by Dev and IMO doesn't need to be reproduced unless told otherwise.
Comment 5 Natalie Gavrielov 2019-08-19 08:17:26 UTC 
I tried recreating the situation described with no success.

A few notes:
1. I used the test_scratch_space.py with a few modifications (the script doesn't work without these changes).
2. Used both import and upload scenarios.
3. Tried numerous times to catch the situation described in comment 0 (that the pvcs are created on different nodes) but when there are 2 pv's available, with the right size, on the same node always picks them (and not one here and one there).
4. I did see a situations where one pv is 25Gi and the other is 10Gi (just like in the attachments we see the pvc sizes), but again - they are both on the same node, resulting in a successful upload/import.
5. From the attachments I can't actually tell if the pvcs were created on different nodes, and there isn't any information about the availability of other pvs in the system.
Comment 6 Tzvi Avni 2019-08-19 08:21:28 UTC 
Did you try to have three nodes, each one has 25Gi and 10Gi PV's? This I think was your setup when I encountered this....
Comment 7 Tzvi Avni 2019-08-19 08:31:15 UTC 
You right, I forgot to show the PV's themselves in the attachment and the nodes they are attached to...
If you run the test I describe in my last comment and you do not fail, can you please try another one:
two nodes, each with 10G, and the another node (third) with 10G and 25G. If the import pvc is attached to to one of the PV's that exist on the first or the second node, the scratch pvc should be in pending as the node does not have another PV. if the pvc is attached to one of the PV's on the third node, we have reproduction.
Comment 8 Tzvi Avni 2019-08-20 13:47:14 UTC 
Just noticed that the ip of the machine and description how to reproduce the issue are described in the bug.