Bug 1732944 - scratch space pvc is created on a different node
Summary: scratch space pvc is created on a different node
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 2.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Adam Litke
QA Contact: Natalie Gavrielov
Depends On:
TreeView+ depends on / blocked
Reported: 2019-07-24 19:39 UTC by Tzvi Avni
Modified: 2019-08-28 12:20 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-08-28 12:20:08 UTC
Target Upstream Version:

Attachments (Terms of Use)
'kubectl get pvc -o yaml' and 'kubect describe pvc' for both pvc and scratcg-pvc (10.00 KB, application/x-tar)
2019-07-24 19:39 UTC, Tzvi Avni
no flags Details

Description Tzvi Avni 2019-07-24 19:39:04 UTC
Created attachment 1593215 [details]
'kubectl get pvc -o yaml' and 'kubect describe pvc' for both pvc and scratcg-pvc

Description of problem:
While importing qcow2 image, importing pvc is created on one node while scratch space pvc is created on a different node. importing fails.
This was found while running scratch_space automation tests.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Running the tests on ssh
ssh cnv-qe-jenkins@ -i ~/.ssh/cnv-qe-jenkins.key

2.run the test:
pipenv run pytest /home/cnv-qe-jenkins/tzvika/cnv-tests/tests/storage/test_scratch_space.py --tc-file=tests/test-config.yaml --tc-format=yaml --log-level=INFO --log-cli-level=INFO -v --pdb -s


Actual results:
pvc and scratch-space pvc's are created on different nodes

Expected results:
both PVC's should be created on the same node

Additional info:

Comment 2 Ying Cui 2019-07-31 12:22:30 UTC
Could you check this bug?

Comment 3 Ying Cui 2019-08-07 12:27:20 UTC
Natalie, could you reproduce it?

Comment 4 Gowrishankar Rajaiyan 2019-08-18 05:07:05 UTC
This case can be confirmed by Dev and IMO doesn't need to be reproduced unless told otherwise.

Comment 5 Natalie Gavrielov 2019-08-19 08:17:26 UTC
I tried recreating the situation described with no success.

A few notes:
1. I used the test_scratch_space.py with a few modifications (the script doesn't work without these changes).
2. Used both import and upload scenarios.
3. Tried numerous times to catch the situation described in comment 0 (that the pvcs are created on different nodes) but when there are 2 pv's available, with the right size, on the same node always picks them (and not one here and one there).
4. I did see a situations where one pv is 25Gi and the other is 10Gi (just like in the attachments we see the pvc sizes), but again - they are both on the same node, resulting in a successful upload/import.
5. From the attachments I can't actually tell if the pvcs were created on different nodes, and there isn't any information about the availability of other pvs in the system.

Comment 6 Tzvi Avni 2019-08-19 08:21:28 UTC
Did you try to have three nodes, each one has 25Gi and 10Gi PV's? This I think was your setup when I encountered this....

Comment 7 Tzvi Avni 2019-08-19 08:31:15 UTC
You right, I forgot to show the PV's themselves in the attachment and the nodes they are attached to...
If you run the test I describe in my last comment and you do not fail, can you please try another one:
two nodes, each with 10G, and the another node (third) with 10G and 25G. If the import pvc is attached to to one of the PV's that exist on the first or the second node, the scratch pvc should be in pending as the node does not have another PV. if the pvc is attached to one of the PV's on the third node, we have reproduction.

Comment 8 Tzvi Avni 2019-08-20 13:47:14 UTC
Just noticed that the ip of the machine and description how to reproduce the issue are described in the bug.

Note You need to log in before you can comment on or make changes to this bug.