Bug 2119309 - readinessProbe in VM stays on failed
Summary: readinessProbe in VM stays on failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Infrastructure
Version: 4.11.0
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
: 4.12.0
Assignee: Javier Cano Cano
QA Contact: Roni Kishner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-18 09:38 UTC by Roni Kishner
Modified: 2023-01-24 13:41 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-24 13:39:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CNV-20548 0 None None None 2022-08-24 08:13:24 UTC
Red Hat Product Errata RHSA-2023:0408 0 None None None 2023-01-24 13:41:11 UTC

Description Roni Kishner 2022-08-18 09:38:53 UTC
Description of problem:
When creating the file needed for the readinessProbe the readiness probe keeps staying on failed.
This should work the same way as described in the docs - https://github.com/kubevirt/kubevirt/blob/b52ebdc081868bf961d70683b2ec8d071fad6232/docs/probes.md#example

Version-Release number of selected component (if applicable):
4.11

How reproducible:
100%

Steps to Reproduce:
1. Create a VM using readinessProbe param:
spec:
  template:
    spec:
      readinessProbe:
        exec:
          command:
          - cat /tmp/healthy.txt
2. Watch the VM events using "oc get events -n <namespace>"
3. Start the VM (notice the readiness probe fails)
2. Console into the VM and create the file /tmp/healthy.txt
3. Validate the file was created using "cat /tmp/healthy.txt" command
4. Watch the VM events

Actual results:
The readiness probe keeps failing (VMI stays on "not ready")

Expected results:
The VMI should change to "ready"

Additional info:
The probe failure msg is:
Readiness probe failed: {"component":"virt-probe","level":"fatal","msg":"Failed executing the command","pos":"virt-probe.go:70","reason":"rpc error: code = Unknown desc = virError(Code=1, Domain=10, Message='internal error: unable to execute QEMU agent command 'guest-exec': Guest agent command failed, error was 'Failed to execute child process “cat /tmp/healthy.txt” (No such file or directory)'')","timestamp":"2022-08-18T07:54:53.312885Z"}...

I also tried using:
  command:
    - cat
    - /tmp/healthy.txt

The issue still persist with this, the only difference is the probe failure msg

Comment 3 Dominik Holler 2022-08-18 14:51:55 UTC
To keep the required permissions minimal, I would recommend to use 'test' instead of 'cat', e.g.:
"""
      readinessProbe:
        exec:
          command: ["test", "-f", "/tmp/healthy.txt"]
"""

In addition, the probe will only succeed, if the file /tmp/healthy.txt is created inside the VM, for testing porposes this could be done in cloud init, e.g.:
"""

      - cloudInitNoCloud:
          userData: |-
            #cloud-config
            chpasswd:
              expire: false
            password: xxxx
            user: fedora

            ssh_authorized_keys:
             [ssh-rsa AAAAB3Nxxx]
            runcmd: [ ..., 'touch /tmp/healthy.txt']
"""

@rkishner Do you agree that this bug might be used to track improving the documentation, instead of improving the software?

Comment 5 Roni Kishner 2022-08-18 16:55:11 UTC
Yes, only needed to change the docs if the solution you suggested works.

On a side note, I do wonder if in the past this did work in the current written method.

Comment 9 errata-xmlrpc 2023-01-24 13:39:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.12.0 Images security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0408


Note You need to log in before you can comment on or make changes to this bug.