Bug 1386569 - Better error message on container start with failed sysctls
Summary: Better error message on container start with failed sysctls
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.0
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Daniel Walsh
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1372658
TreeView+ depends on / blocked
 
Reported: 2016-10-19 09:06 UTC by Stefan Schimanski
Modified: 2019-03-06 01:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-30 15:25:16 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1372658 0 medium CLOSED [infrastructure_public_178]Using invalid sysctls value will cause pod keep in ContainerCreating 2021-02-22 00:41:40 UTC

Description Stefan Schimanski 2016-10-19 09:06:29 UTC
Description of problem:

Running a container with sysctls which is rejected by the kernel gives a "System error: could not synchronise with container process." error message from the /container/<id>/start API call (https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/start-a-container):

docker run -it --sysctl=kernel.shm_rmid_forced=hello ubuntu /bin/bash -c "sysctl kernel.shm_rmid_forced"                             sttts-sni
write /proc/sys/kernel/shm_rmid_forced: invalid argument
docker: Error response from daemon: Cannot start container 2424aea6711a7335b5ea1c171f544a39862fa2886240685b2b73a695a655487c: [9] System error: could not synchronise with container process.

The actual sysctl error shown above is from the container logs, not part of the error message of the API call. The later is too generic. Parsing logs in order to react properly on the error is not an option.

Compare this with e.g. the --user flags:

docker run -it -u foo ubuntu /bin/bash -c "sysctl kernel.shm_rmid_forced"                                                            sttts-sni
unable to find user foo: no matching entries in passwd file
docker: Error response from daemon: Cannot start container 7d63df9760ff964bd4a99e512c8d53777e8b59577959f47de8ad9285482fbe21: [9] System error: unable to find user foo: no matching entries in passwd file.

Here the error message is explicit.

Why is this relevant?

The Kubernetes kubelet must be able to react on sysctl errors properly, i.e. without restarting the container again and again. Instead the error should be reported to the user as an pod event. For that a proper, well known and stable error message is required. Parsing the logs for arbitrary plain text message is not a viable option here as such a message might change any time with future Docker versions or kernels.

Version-Release number of selected component (if applicable):

v1.12

Expected results:

docker run -it --sysctl=kernel.shm_rmid_forced=hello ubuntu /bin/bash -c "sysctl kernel.shm_rmid_forced"                             sttts-sni
write /proc/sys/kernel/shm_rmid_forced: invalid argument
docker: Error response from daemon: Cannot start container 2424aea6711a7335b5ea1c171f544a39862fa2886240685b2b73a695a655487c: [9] System error: cannot write /proc/sys/kernel/shm_rmid_forced: invalid argument

Comment 2 Stefan Schimanski 2016-10-31 13:16:52 UTC
Any chance to backport this to our 1.10.x?

Comment 3 Dusty Mabe 2017-06-07 14:10:22 UTC
re-assigning to containers team, this may already be fixed


Note You need to log in before you can comment on or make changes to this bug.