RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2028408 - Podman healthcheck fails if the command contains unicode characters.
Summary: Podman healthcheck fails if the command contains unicode characters.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.5
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-02 10:42 UTC by Ashish Reddy
Modified: 2022-11-08 09:29 UTC (History)
11 users (show)

Fixed In Version: podman-4.0.2-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-08 09:14:55 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104535 0 None None None 2021-12-02 10:43:56 UTC
Red Hat Product Errata RHSA-2022:7457 0 None None None 2022-11-08 09:15:35 UTC

Description Ashish Reddy 2021-12-02 10:42:52 UTC
Description of problem:

Podman healthcheck fails if the command contains unicode characters. 

Version-Release number of selected component (if applicable):

podman-3.3.1


How reproducible:
Easy

Steps to Reproduce:
1. Launch the container with healthcheck command that contains ">" or "&" or "<" 
~~~
$ podman run -dit --rm --health-interval 5m --health-timeout 3s --health-cmd "mysqladmin --port 6446 --protocol TCP ping 2<&1 | grep Access || exit 1" ubi8/ubi-minimal
~~~

2. Check the healthcheck command with podman inspect


Actual results:
Inspect output contains unicode characters which results in failing of the command.

~~~
 "CreateCommand": [
                "podman",
                "run",
                "-dit",
                "--rm",
                "--health-interval",
                "5m",
                "--health-timeout",
                "3s",
                "--health-cmd",
                "mysqladmin --port 6446 --protocol TCP ping 2\u003c\u00261 | grep Access || exit 1",
                "ubi8/ubi-minimal"
            ],


~~~

Expected results:
The healthcheck command should have the correct characters as given while launching the container


Additional info:
While building with or using docker, there are no issues.


Cu is also able to reproduce this error with Dockerfile using the same healthcheck

~~~
FROM ubi8/ubi-minimal

HEALTHCHECK --interval=5m --timeout=3s \
  CMD mysqladmin --port 6446 --protocol TCP ping 2>&1 | grep Access || exit 1

~~~

The issue has been reproduced for ">" "&" "<" characters. I have tried with some other characters("," ";" ":" "^" "$" "%" "*" "/" "\") so far these characters does not have any issues.

Comment 1 Daniel Walsh 2021-12-02 15:01:06 UTC
Giuseppe could you take a look at this?

Comment 2 Giuseppe Scrivano 2021-12-03 16:16:20 UTC
the healthcheck command seems to be executed correctly:

$ podman run --name foo -d --health-interval 5m --health-timeout 3s --health-cmd "echo hi 2>&1" ubi8/ubi-minimal sleep 100000
$ (sleep 10; podman healthcheck run foo) &
# sudo /usr/share/bcc/tools/execsnoop  | grep echo

sh               342878 342874   0 /bin/sh -c echo hi 2>&1

So is the issue you are seeing just in the "podman inspect" output?  Even in this case, I am not sure it is an issue, since the command is correctly encoded:

$ echo -e "mysqladmin --port 6446 --protocol TCP ping 2\u003c\u00261 | grep Access || exit 1"
mysqladmin --port 6446 --protocol TCP ping 2<&1 | grep Access || exit 1


What is the '2<&1' for?  Did you mean '2>&1'?

How have you checked that the checkpoint command is launched incorrectly?

Comment 3 Ashish Reddy 2021-12-06 13:21:19 UTC
The exact command customer using for healthcheck is as below:

~~~
"mysqladmin --port 6446 --protocol TCP ping 2>&1 | grep Access || exit 1"
~~~

With docker, the container is getting the whole command as seen from the inspect output of the container. But with podman it gives Unicode values for ">" and "&" characters. So it is failing the healthchecks for them. They have verified this by the inspect output from the container.

I have also tested this from the inspect output.

Comment 4 Giuseppe Scrivano 2021-12-07 09:55:10 UTC
that doesn't answer my question.

The "inspect" output is just a formatting difference, that doesn't mean the Podman will use exactly that string to run the healthcheck.

The test I suggest to do is to create the container, then from another terminal run "/usr/share/bcc/tools/execsnoop" to see what processes are launched.  e.g.:


$ podman run -dit --rm --health-interval 5m --health-timeout 3s --health-cmd "echo Access 2>&1 | grep Access || exit 1" ubi8/ubi-minimal && sleep 20 && podman healthcheck run fa2880186f2bd596832aaf5705349e0d10d0b74155519ea904a7c6109d743609

(from another terminal)
# /usr/share/bcc/tools/execsnoop | grep echo
sh               702697 702693   0 /bin/sh -c echo Access 2>&1 | grep Access || exit 1

What command do you see launched?

Comment 5 Ashish Reddy 2021-12-14 15:01:21 UTC
Hello,

Yes, I can see the exact command in the execsnoop outputs.

But the cu came back with the below query

~~~

When I run podman healthcheck run on my image built with Docker built or the original mysql/enterprise-router:8.0.26 I get:


podman healthcheck run my_container
Error: healthcheck command exceeded timeout of 0s


The clue is 'exceeded timeout of 0s'. After a little further investigation a believe that the problem isn't the encoding, as you stated, but instead it's about default values for the healthcheck parameters.
If no timeout is set in the image Podman defaults it to 0s and as a consequence the healthcheck command fails to complete within that interval (of course)  and thus fails.
To compare, Docker has a default 30s which is plenty of time for this simple healthcheck.
Interval has a similar problem. Docker has a default of 30s but Podman defaults to nil. At least it appears so because the healthcheck is never run unless specified explicitly.
I haven't checked the retries parameter but I wouldn't be surprised if it defaults to infinity if not set. Dockers default i s 3

~~~

I could reproduce this with below dockerfile

~~~
[test@vm251-225 ~]$ cat Dockerfile 
FROM ubi8/ubi-minimal

HEALTHCHECK CMD mysqladmin --port 6446 --protocol TCP ping 2>&1 | grep Access || exit 1
[test@vm251-225 ~]$ 

~~~

Now when I run helathcheck on a container launched with this image, I get the below error.

~~~
[test@vm251-225 ~]$ podman healthcheck run 6916529718cf 
Error: healthcheck command exceeded timeout of 0s
[test@vm251-225 ~]$ 
~~~

From podman docs, I see that the default value is correct and is "30s"

- https://docs.podman.io/en/latest/markdown/podman-run.1.html#health-interval-interval


Also, is there anyway we could improve the output on the "podman inspect", so that it would be less confusing

Comment 6 Giuseppe Scrivano 2021-12-14 17:15:17 UTC
the issue seems to be that when there is no timeout specified in the image, we default to 0s.

I am not very familiar with health checks and what we are supposed to do in this case.

Does timeout=0s mean infinite?  If it is set to 0 (we don't seem to make a difference between 0 and not set), should it be 30s?

Brent, do you have any hints?

Comment 8 Brent Baude 2021-12-15 20:11:33 UTC
upstream fixed proposed -> https://github.com/containers/podman/pull/12614

Comment 10 Giuseppe Scrivano 2022-03-07 08:20:00 UTC
the fix was merged and it is part of podman 4

Comment 15 Alex Jia 2022-03-22 08:04:26 UTC
This bug has been verified on podman-4.0.2-1.module+el8.7.0+14421+e3b24aca.

[root@sweetpig-18 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 Beta (Ootpa)

[root@sweetpig-18 ~]# rpm -q podman runc systemd kernel
podman-4.0.2-1.module+el8.7.0+14421+e3b24aca.x86_64
runc-1.0.3-3.module+el8.7.0+14440+ed5f482d.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.2.1.el8.x86_64

[root@sweetpig-18 ~]# podman build --format docker -t foobar .
STEP 1/2: FROM quay.io/libpod/alpine
Trying to pull quay.io/libpod/alpine:latest...
Getting image source signatures
Copying blob 9d16cba9fb96 done  
Copying config 9617696764 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: HEALTHCHECK CMD ls -l / 2>&1
COMMIT foobar
--> f55ad067db6
Successfully tagged localhost/foobar:latest
f55ad067db6d4aae7aa483dcd53425a58e1f80c2b36159fd88ad5c3ec2ea5dca

[root@sweetpig-18 ~]# podman run -td --name hctest foobar ls
1eccb36e11f9e51be9a718f462c90b8429ba101498039d8cd4f0c6586575b018

[root@sweetpig-18 ~]# podman ps -a
CONTAINER ID  IMAGE                    COMMAND     CREATED        STATUS                               PORTS       NAMES
1eccb36e11f9  localhost/foobar:latest  ls          7 seconds ago  Exited (0) 5 seconds ago (starting)              hctest

[root@sweetpig-18 ~]# podman inspect hctest | grep -A2 CMD-SHELL
                         "CMD-SHELL",
                         "ls -l / 2>&1"
                    ],

Comment 20 errata-xmlrpc 2022-11-08 09:14:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7457


Note You need to log in before you can comment on or make changes to this bug.