Bug 2175710
| Summary: | CNV 4.13 nightly | manually increasing the number of virt-api pods does not work | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Boaz <bbenshab> |
| Component: | Virtualization | Assignee: | Barak <bmordeha> |
| Status: | CLOSED ERRATA | QA Contact: | zhe peng <zpeng> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.13.0 | CC: | acardace, akamra, danken, fdeutsch, jhopper |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | 4.13.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | v4.13.3.rhel9-94 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-08-16 14:09:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Taking the severity of the bug and capacity of the team we plan to target this to CNV 4.14 oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
kubevirt.kubevirt.io/jsonpatch='[
{
"op": "add",
"path": "/spec/customizeComponents/patches",
"value": [{
"patch": "[{\"op\":\"replace\",\"path\":\"/spec/replicas\",\"value\":6}]",
"resourceName": "virt-api",
"resourceType": "Deployment",
"type": "json"
}]
}
]'
HCO JSON Patch might be working in few situations and might not be working in some situations.
1) The working examples
a)
oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
kubevirt.kubevirt.io/jsonpatch='[
{
"op": "add",
"path": "/spec/customizeComponents/patches",
"value": [{
"patch": "[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/resources/requests\",\"value\":{\"cpu\": \"10m\",\"memory\": \"400Mi\"}}]",
"resourceName": "virt-controller",
"resourceType": "Deployment",
"type": "json"
}]
}
]'
b) oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{
"op": "add",
"path": "/spec/configuration/developerConfiguration/cpuAllocationRatio",
"value": 20
}]'
2) The not working examples
a) oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
kubevirt.kubevirt.io/jsonpatch='[
{
"op": "add",
"path": "/spec/configuration/additionalGuestMemoryOverheadRatio",
"value": 1.5
}
]'
b) oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
kubevirt.kubevirt.io/jsonpatch='[
{
"op": "add",
"path": "/spec/customizeComponents/patches",
"value": [{
"patch": "[{\"op\":\"replace\",\"path\":\"/spec/replicas\",\"value\":6}]",
"resourceName": "virt-api",
"resourceType": "Deployment",
"type": "json"
}]
}
]'
(In reply to Kedar Bidarkar from comment #4) > HCO JSON Patch might be working in few situations and might not be working > in some situations. > > > 1) The working examples > a) > > oc annotate --overwrite -n openshift-cnv hyperconverged > kubevirt-hyperconverged \ > kubevirt.kubevirt.io/jsonpatch='[ > { > "op": "add", > "path": "/spec/customizeComponents/patches", > "value": [{ > "patch": > "[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/resources/ > requests\",\"value\":{\"cpu\": \"10m\",\"memory\": \"400Mi\"}}]", > "resourceName": "virt-controller", > "resourceType": "Deployment", > "type": "json" > }] > } > ]' > > b) oc annotate --overwrite -n openshift-cnv hyperconverged > kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{ > "op": "add", > "path": > "/spec/configuration/developerConfiguration/cpuAllocationRatio", > "value": 20 > }]' > > 2) The not working examples > > a) oc annotate --overwrite -n openshift-cnv hyperconverged > kubevirt-hyperconverged \ > kubevirt.kubevirt.io/jsonpatch='[ > { > "op": "add", > "path": "/spec/configuration/additionalGuestMemoryOverheadRatio", > "value": 1.5 > } > ]' > > b) oc annotate --overwrite -n openshift-cnv hyperconverged > kubevirt-hyperconverged \ > kubevirt.kubevirt.io/jsonpatch='[ > { > "op": "add", > "path": "/spec/customizeComponents/patches", > "value": [{ > "patch": > "[{\"op\":\"replace\",\"path\":\"/spec/replicas\",\"value\":6}]", > "resourceName": "virt-api", > "resourceType": "Deployment", > "type": "json" > }] > } > ]' Indeed there is a bug with the calculation of virt-api replicas probably by the following Change: https://github.com/kubevirt/kubevirt/pull/7401 The following PR should resolve it: https://github.com/kubevirt/kubevirt/pull/10027 as for > a) oc annotate --overwrite -n openshift-cnv hyperconverged > kubevirt-hyperconverged \ > kubevirt.kubevirt.io/jsonpatch='[ > { > "op": "add", > "path": "/spec/configuration/additionalGuestMemoryOverheadRatio", > "value": 1.5 > } > ]' Please try using a string val it worked for me and i can see the kv has been modified. my command: oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{ "op": "add", "path": "/spec/configuration/additionalGuestMemoryOverheadRatio", "value": "1.5" }]' https://bugzilla.redhat.com/show_bug.cgi?id=2222190 https://bugzilla.redhat.com/show_bug.cgi?id=2222191 created BZ for 4.12 and 4.11 and backported test with build: CNV-v4.13.3.rhel9-160
step :
1. apply patch:
oc patch hco -n openshift-cnv kubevirt-hyperconverged --type=merge -p '{"metadata":{"annotations":{"kubevirt.kubevirt.io/jsonpatch":"[{\"op\": \"add\", \"path\": \"/spec/customizeComponents/patches\", \"value\": [{\"resourceType\": \"Deployment\", \"resourceName\": \"virt-api\", \"type\": \"json\", \"patch\": \"[{\\\"op\\\": \\\"replace\\\", \\\"path\\\": \\\"/spec/replicas\\\", \\\"value\\\": 6}]\"}]}]"}}}'
check:
$oc get deployment virt-api -n openshift-cnv -o yaml
replicas: 6
availableReplicas: 6
readyReplicas: 6
replicas: 6
updatedReplicas: 6
$oc get pod -n openshift-cnv|grep virt-api
$ oc get pods -n openshift-cnv | grep virt-api
virt-api-5689868576-djjwh 1/1 Running 0 4m55s
virt-api-5689868576-hpqdl 1/1 Running 0 4m55s
virt-api-5689868576-jlnz2 1/1 Running 0 4m55s
virt-api-5689868576-mphkx 1/1 Running 0 4m55s
virt-api-5689868576-tzl8w 1/1 Running 0 2d3h
virt-api-5689868576-xfltr 1/1 Running 0 2d3h
2: apply patch:
oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{
"op": "add",
"path": "/spec/configuration/additionalGuestMemoryOverheadRatio",
"value": "1.5"
}]'
check:
$ oc get kv -n openshift-cnv -o yaml
...
additionalGuestMemoryOverheadRatio: "1.5"
move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.13.3 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:4664 |
-------------------------------------------------- Description of problem: -------------------------------------------------- as a part of the new KubeVirt rate limiting testing, I tried to increase the number of virt-api pods using this which had no issues before, I suspect the new auto virt-api auto-scaler overwrites the manual patch. ================================================================================================ apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: annotations: deployOVS: "false" hco.kubevirt.io/tuningPolicy: '{"qps":200,"burst":400}' kubevirt.kubevirt.io/jsonpatch: '[{"op": "add", "path": "/spec/customizeComponents/patches", "value": [{"resourceType": "Deployment", "resourceName": "virt-api", "type": "json", "patch": "[{\"op\": \"replace\", \"path\": \"/spec/replicas\", \"value\": 6}]"}]}]' ================================================================================================= oc get deployment virt-api -o yaml readyReplicas: 2 replicas: 2 updatedReplicas: 2 ================================================================================================= oc -n openshift-cnv get kubevirt kubevirt -o yaml customizeComponents: patches: - patch: '[{"op": "replace", "path": "/spec/replicas", "value": 6}]' resourceName: virt-api resourceType: Deployment type: json ================================================================================================= Boaz Ben Shabat, 10:29 AM [root@e29-h19-740xd storms_tool]# oc get pod -n openshift-cnv|grep virt-api virt-api-8795754c5-npd5m 1/1 Running 1 30h virt-api-8795754c5-v8tn5 1/1 Running 1 30h ================================================================================================= -------------------------------------------------- Version-Release number of selected component (if applicable): -------------------------------------------------- openshift-cnv 4.13.0-1689 openshift-local-storage 4.12.0-202302061702 openshift-storage 4.12.0 -------------------------------------------------- How reproducible: -------------------------------------------------- every time. -------------------------------------------------- Steps to Reproduce: -------------------------------------------------- apply the following patch: oc patch hco -n openshift-cnv kubevirt-hyperconverged --type=merge -p '{"metadata":{"annotations":{"kubevirt.kubevirt.io/jsonpatch":"[{\"op\": \"add\", \"path\": \"/spec/customizeComponents/patches\", \"value\": [{\"resourceType\": \"Deployment\", \"resourceName\": \"virt-api\", \"type\": \"json\", \"patch\": \"[{\\\"op\\\": \\\"replace\\\", \\\"path\\\": \\\"/spec/replicas\\\", \\\"value\\\": 6}]\"}]}]"}}}' -------------------------------------------------- Expected results: -------------------------------------------------- the number of virt-api pods should increase to 6