Bug 1748218 - kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Summary: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kubernetes
Version: 30
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jan Chaloupka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-03 07:56 UTC by alexander.kondrachuk
Modified: 2020-05-26 14:34 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 14:34:13 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf 0 None None None 2020-07-14 04:38:13 UTC

Description alexander.kondrachuk 2019-09-03 07:56:51 UTC
Description of problem:

Kubernetes control-plane node initialization (kubeadm init) from fresh install won't work due to systemd kubelet service failing.
The second way to experience the same issue is to upgrade kubernetes-node-12.5-2.fc30 to kubernetes-node-1.15.2-1.fc30

Version-Release number of selected component (if applicable):
cat /etc/fedora-release
Fedora release 30 (Thirty)

uname -a
Linux qubit 5.2.9-200.fc30.x86_64 #1 SMP Fri Aug 16 21:37:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

dnf list --installed | grep kube
kubernetes.x86_64                                 1.15.2-1.fc30                                                               
kubernetes-client.x86_64                          1.15.2-1.fc30                                                                
kubernetes-kubeadm.x86_64                         1.15.2-1.fc30                                                                
kubernetes-master.x86_64                          1.15.2-1.fc30                                                                
kubernetes-node.x86_64                            1.15.2-1.fc30 



How reproducible:


Steps to Reproduce:
1. dnf install kubernetes-master kubernetes-node kubernetes-kubeadm kubernetes-client docker
2. systemctl enable kubelet  
3. kubeadm init --pod-network-cidr=10.244.0.0/16

Note: docker and kubelet should have same cgroup driver systemd
docker info | grep Cgroup
Cgroup Driver: systemd


Actual results:

kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.1. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [qubit kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.43.88]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [qubit localhost] and IPs [192.168.43.88 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [qubit localhost] and IPs [192.168.43.88 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

Log output for kubelet:
journalctl -u kubelet.service -f
Sep 03 14:15:28 qubit systemd[1]: Started Kubernetes Kubelet Server.
Sep 03 14:15:28 qubit kubelet[8101]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 03 14:15:28 qubit kubelet[8101]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 03 14:15:28 qubit kubelet[8101]: Usage:
Sep 03 14:15:28 qubit kubelet[8101]:   kubelet [flags]
Sep 03 14:15:28 qubit kubelet[8101]: Flags:
Sep 03 14:15:28 qubit kubelet[8101]:       --address 0.0.0.0                                                                                           The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --allowed-unsafe-sysctls strings                                                                            Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. Sysctls feature gate is enabled by default.
Sep 03 14:15:28 qubit kubelet[8101]:       --alsologtostderr                                                                                           log to standard error as well as files
Sep 03 14:15:28 qubit kubelet[8101]:       --anonymous-auth                                                                                            Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --application-metrics-count-limit int                                                                       Max number of application metrics to store (per container) (default 100) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --authentication-token-webhook                                                                              Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --authentication-token-webhook-cache-ttl duration                                                           The duration to cache responses from the webhook token authenticator. (default 2m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --authorization-mode string                                                                                 Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (default "AlwaysAllow") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --authorization-webhook-cache-authorized-ttl duration                                                       The duration to cache 'authorized' responses from the webhook authorizer. (default 5m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --authorization-webhook-cache-unauthorized-ttl duration                                                     The duration to cache 'unauthorized' responses from the webhook authorizer. (default 30s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --azure-container-registry-config string                                                                    Path to the file containing Azure container registry configuration information.
Sep 03 14:15:28 qubit kubelet[8101]:       --boot-id-file string                                                                                       Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --bootstrap-checkpoint-path string                                                                          <Warning: Alpha feature> Path to the directory where the checkpoints are stored
Sep 03 14:15:28 qubit kubelet[8101]:       --bootstrap-kubeconfig string                                                                               Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
Sep 03 14:15:28 qubit kubelet[8101]:       --cert-dir string                                                                                           The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "/var/lib/kubelet/pki")
Sep 03 14:15:28 qubit kubelet[8101]:       --cgroup-driver string                                                                                      Driver that the kubelet uses to manipulate cgroups on the host.  Possible values: 'cgroupfs', 'systemd' (default "cgroupfs") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cgroup-root string                                                                                        Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cgroups-per-qos                                                                                           Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --chaos-chance float                                                                                        If > 0.0, introduce random client errors and latency. Intended for testing.
Sep 03 14:15:28 qubit kubelet[8101]:       --client-ca-file string                                                                                     If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cloud-config string                                                                                       The path to the cloud provider configuration file.  Empty string for no configuration file.
Sep 03 14:15:28 qubit kubelet[8101]:       --cloud-provider string                                                                                     The provider for cloud services. Specify empty string for running with no cloud provider. If set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used).
Sep 03 14:15:28 qubit kubelet[8101]:       --cluster-dns strings                                                                                       Comma-separated list of DNS server IP address.  This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst". Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cluster-domain string                                                                                     Domain for this cluster.  If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cni-bin-dir string                                                                                        <Warning: Alpha feature> A comma-separated list of full paths of directories in which to search for CNI plugin binaries. This docker-specific flag only works when container-runtime is set to docker. (default "/opt/cni/bin")
Sep 03 14:15:28 qubit kubelet[8101]:       --cni-conf-dir string                                                                                       <Warning: Alpha feature> The full path of the directory in which to search for CNI config files. This docker-specific flag only works when container-runtime is set to docker. (default "/etc/cni/net.d")
Sep 03 14:15:28 qubit kubelet[8101]:       --config string                                                                                             The Kubelet will load its initial configuration from this file. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this flag to use the built-in default configuration values. Command-line flags override configuration from this file.
Sep 03 14:15:28 qubit kubelet[8101]:       --container-hints string                                                                                    location of the container hints file (default "/etc/cadvisor/container_hints.json") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --container-log-max-files int32                                                                             <Warning: Beta feature> Set the maximum number of container log files that can be present for a container. The number must be >= 2. This flag can only be used with --container-runtime=remote. (default 5) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --container-log-max-size string                                                                             <Warning: Beta feature> Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with --container-runtime=remote. (default "10Mi") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --container-runtime string                                                                                  The container runtime to use. Possible values: 'docker', 'remote', 'rkt (deprecated)'. (default "docker")
Sep 03 14:15:28 qubit kubelet[8101]:       --container-runtime-endpoint string                                                                         [Experimental] The endpoint of remote runtime service. Currently unix socket endpoint is supported on Linux, while npipe and tcp endpoints are supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim' (default "unix:///var/run/dockershim.sock")
Sep 03 14:15:28 qubit kubelet[8101]:       --containerd string                                                                                         containerd endpoint (default "/run/containerd/containerd.sock") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --containerized                                                                                             Running kubelet in a container. (DEPRECATED: This feature will be removed in a later release.)
Sep 03 14:15:28 qubit kubelet[8101]:       --contention-profiling                                                                                      Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cpu-cfs-quota                                                                                             Enable CPU CFS quota enforcement for containers that specify CPU limits (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cpu-cfs-quota-period duration                                                                             Sets CPU CFS quota period value, cpu.cfs_period_us, defaults to Linux Kernel default (default 100ms) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cpu-manager-policy string                                                                                 CPU Manager policy to use. Possible values: 'none', 'static'. Default: 'none' (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --cpu-manager-reconcile-period NodeStatusUpdateFrequency                                                    <Warning: Alpha feature> CPU Manager reconciliation period. Examples: '10s', or '1m'. If not supplied, defaults to NodeStatusUpdateFrequency (default 10s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker string                                                                                             docker endpoint (default "unix:///var/run/docker.sock") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-endpoint string                                                                                    Use this for the docker endpoint to communicate with. This docker-specific flag only works when container-runtime is set to docker. (default "unix:///var/run/docker.sock")
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-env-metadata-whitelist string                                                                      a comma-separated list of environment variable keys that needs to be collected for docker containers (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-only                                                                                               Only report docker containers in addition to root stats (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-root string                                                                                        DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker) (default "/var/lib/docker")
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-tls                                                                                                use TLS to connect to docker (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-tls-ca string                                                                                      path to trusted CA (default "ca.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-tls-cert string                                                                                    path to client certificate (default "cert.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --docker-tls-key string                                                                                     path to private key (default "key.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --dynamic-config-dir string                                                                                 The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. The DynamicKubeletConfig feature gate must be enabled to pass this flag; this gate currently defaults to true because the feature is beta.
Sep 03 14:15:28 qubit kubelet[8101]:       --enable-cadvisor-json-endpoints                                                                            Enable cAdvisor json /spec and /stats/* endpoints. (default true)
Sep 03 14:15:28 qubit kubelet[8101]:       --enable-controller-attach-detach                                                                           Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --enable-debugging-handlers                                                                                 Enables server endpoints for log collection and local running of containers and commands (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --enable-load-reader                                                                                        Whether to enable cpu load reader (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --enable-server                                                                                             Enable the Kubelet's server (default true)
Sep 03 14:15:28 qubit kubelet[8101]:       --enforce-node-allocatable strings                                                                          A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are 'none', 'pods', 'system-reserved', and 'kube-reserved'. If the latter two options are specified, '--system-reserved-cgroup' and '--kube-reserved-cgroup' must also be set, respectively. If 'none' is specified, no additional options should be set. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (default [pods]) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --event-burst int32                                                                                         Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --event-qps int32                                                                                           If > 0, limit event creations per second to this value. If 0, unlimited. (default 5) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --event-storage-age-limit string                                                                            Max length of time for which to store events (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is a duration. Default is applied to all non-specified event types (default "default=0") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --event-storage-event-limit string                                                                          Max number of events to store (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is an integer. Default is applied to all non-specified event types (default "default=0") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-hard mapStringString                                                                             A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. (default imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-max-pod-grace-period int32                                                                       Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met.  If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-minimum-reclaim mapStringString                                                                  A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-pressure-transition-period duration                                                              Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (default 5m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-soft mapStringString                                                                             A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --eviction-soft-grace-period mapStringString                                                                A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --exit-on-lock-contention                                                                                   Whether kubelet should exit upon lock-file contention.
Sep 03 14:15:28 qubit kubelet[8101]:       --experimental-allocatable-ignore-eviction                                                                  When set to 'true', Hard Eviction Thresholds will be ignored while calculating Node Allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. [default=false]
Sep 03 14:15:28 qubit kubelet[8101]:       --experimental-bootstrap-kubeconfig string                                                                   (DEPRECATED: Use --bootstrap-kubeconfig)
Sep 03 14:15:28 qubit kubelet[8101]:       --experimental-check-node-capabilities-before-mount                                                         [Experimental] if set true, the kubelet will check the underlying node for required components (binaries, etc.) before performing the mount
Sep 03 14:15:28 qubit kubelet[8101]:       --experimental-kernel-memcg-notification                                                                    If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling.
Sep 03 14:15:28 qubit kubelet[8101]:       --experimental-mounter-path string                                                                          [Experimental] Path of mounter binary. Leave empty to use the default mount.
Sep 03 14:15:28 qubit kubelet[8101]:       --fail-swap-on                                                                                              Makes the Kubelet fail to start if swap is enabled on the node.  (default true) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --feature-gates mapStringBool                                                                               A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
Sep 03 14:15:28 qubit kubelet[8101]:                 APIListChunking=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 APIResponseCompression=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 AllAlpha=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 AppArmor=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 AttachVolumeLimit=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 BlockVolume=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CPUManager=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CRIContainerLogRotation=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIBlockVolume=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIDriverRegistry=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIInlineVolume=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigration=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigrationAWS=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigrationAzureDisk=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigrationAzureFile=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigrationGCE=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSIMigrationOpenStack=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CSINodeInfo=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomResourceDefaulting=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomResourcePublishOpenAPI=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomResourceSubresources=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomResourceValidation=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 CustomResourceWebhookConversion=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 DebugContainers=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 DevicePlugins=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 DryRun=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 DynamicAuditing=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 DynamicKubeletConfig=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ExpandCSIVolumes=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ExpandInUsePersistentVolumes=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ExpandPersistentVolumes=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 HyperVContainer=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 KubeletPodResources=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 LocalStorageCapacityIsolation=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 MountContainers=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 NodeLease=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 NonPreemptingPriority=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 PodShareProcessNamespace=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ProcMountType=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 QOSReserved=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 RemainingItemCount=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 RequestManagement=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ResourceQuotaScopeSelectors=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 RotateKubeletClientCertificate=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 RotateKubeletServerCertificate=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 RunAsGroup=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 RuntimeClass=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 SCTPSupport=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ScheduleDaemonSetPods=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ServerSideApply=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ServiceLoadBalancerFinalizer=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 ServiceNodeExclusion=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 StorageVersionHash=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 StreamingProxyRedirects=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 SupportNodePidsLimit=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 SupportPodPidsLimit=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 Sysctls=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 TTLAfterFinished=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 TaintBasedEvictions=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 TaintNodesByCondition=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 TokenRequest=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 TokenRequestProjection=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 ValidateProxyRedirects=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 VolumePVCDataSource=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 VolumeSnapshotDataSource=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 VolumeSubpathEnvExpansion=true|false (BETA - default=true)
Sep 03 14:15:28 qubit kubelet[8101]:                 WatchBookmark=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 WinDSR=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 WinOverlay=true|false (ALPHA - default=false)
Sep 03 14:15:28 qubit kubelet[8101]:                 WindowsGMSA=true|false (ALPHA - default=false) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --file-check-frequency duration                                                                             Duration between checking config files for new data (default 20s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --global-housekeeping-interval duration                                                                     Interval between global housekeepings (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Sep 03 14:15:28 qubit kubelet[8101]:       --hairpin-mode string                                                                                       How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none". (default "promiscuous-bridge") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit kubelet[8101]:       --healthz-bind-address 0.0.0.0                                                                              The IP address for the healthz server to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 127.0.0.1) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 03 14:15:28 qubit systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION



Expected results:
kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.1. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [qubit localhost] and IPs [192.168.43.88 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [qubit localhost] and IPs [192.168.43.88 127.0.0.1 ::1]
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [qubit kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.43.88]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 39.002197 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node qubit as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node qubit as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[kubelet-check] Initial timeout of 40s passed.
[bootstrap-token] Using token: s4797h.3veascl0nj36tgj8
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.43.88:6443 --token s4797h.3veascl0nj36tgj8 \
    --discovery-token-ca-cert-hash sha256:88a617913b447ebb788e4fd52f651838c13d9b30a18db24cddeff7f7a9d902e6 




Additional info:
The cause of issue is that in kubernetes 15.2 version, parameters should be set via the config file specified by the Kubelet's --config flag.

From official k8s repository the systemd service unit has next configuration:
https://github.com/kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Where it has Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" linked to yaml generating by kubadm during initialization.

Comment 1 Jan Pazdziora 2019-09-12 18:17:42 UTC
As packaged in kubernetes-node-1.15.2-1.fc30.x86_64, the kubelet.service is misconfigured.

Just plain

# dnf install -y /usr/lib/systemd/system/kubelet.service
# systemctl start kubelet.service
# systemctl status kubelet.service

yields

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2019-09-12 20:09:08 CEST; 4min 2s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 10018 ExecStart=/usr/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_ARGS (code=exited, status=255/EXCEPTION)
 Main PID: 10018 (code=exited, status=255/EXCEPTION)
      CPU: 125ms

Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Consumed 125ms CPU time.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Service RestartSec=100ms expired, scheduling restart.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 5.
Sep 12 20:09:08 machine.example.test systemd[1]: Stopped Kubernetes Kubelet Server.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Consumed 125ms CPU time.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Start request repeated too quickly.
Sep 12 20:09:08 machine.example.test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 12 20:09:08 machine.example.test systemd[1]: Failed to start Kubernetes Kubelet Server.

In the journal, there's

Sep 12 20:09:08 machine.example.test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/syste
md/systemd" hostname=? addr=? terminal=? res=success'
Sep 12 20:09:08 machine.example.test kubelet[10018]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https
://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 20:09:08 machine.example.test kubelet[10018]: Usage:
Sep 12 20:09:08 machine.example.test kubelet[10018]:   kubelet [flags]
Sep 12 20:09:08 machine.example.test kubelet[10018]: Flags:
Sep 12 20:09:08 machine.example.test kubelet[10018]:       --address 0.0.0.0                                                                                           The IP address for the
 Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by t
he Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Sep 12 20:09:08 machine.example.test kubelet[10018]:       --allowed-unsafe-sysctls strings                                                                            Comma-separated whitel
[...]
Sep 12 18:23:14 kvm-02-guest18.rhts.eng.brq.redhat.com kubelet[14673]:       --vmodule moduleSpec                                                                               Sep 12 18:23:14 kvm-02-guest18.rhts.eng.brq.redhat.com kubelet[14673]:       --volume-plugin-dir string                                                                         Sep 12 18:23:14 kvm-02-guest18.rhts.eng.brq.redhat.com kubelet[14673]:       --volume-stats-agg-period duration                                                                 Sep 12 18:23:14 kvm-02-guest18.rhts.eng.brq.redhat.com kubelet[14673]: F0912 18:23:14.356051   14673 server.go:156] unknown flag: --allow-privileged

That --allow-privileged comes from

# grep -r allow-privileged /etc/kubernetes
/etc/kubernetes/config:KUBE_ALLOW_PRIV="--allow-privileged=false"

When I comment out that KUBE_ALLOW_PRIV line in /etc/kubernetes/config, the kubelet.service with its out-of-box configuration at least starts and does not die immediatelly.

So the /etc/kubernetes/config is likely the first thing that needs to change to accomodate Kubernetes 1.15.

Comment 2 alexander.kondrachuk 2019-09-25 03:51:10 UTC
Hello Jan Pazdzioram,

(In reply to Jan Pazdziora from comment #1)
> As packaged in kubernetes-node-1.15.2-1.fc30.x86_64, the kubelet.service is
> misconfigured.
> 

> 
> So the /etc/kubernetes/config is likely the first thing that needs to change
> to accomodate Kubernetes 1.15.


What would be the approach to fix it? 
I might help to build stable rpm. 
Shall it be remained as kubernetes v1.15 or it can be switched to latest v1.16 ?

Sincerely
Alexander

Comment 3 Jan Pazdziora 2019-09-25 07:05:07 UTC
Given that even rawhide only has 1.15 now, and given that different Kubernetes versions bring small incompatibilities, I'd say rebase to 1.16 in rawhide and possibly in Fedora 30, and fix the config for the 1.15 bulid in Fedora 30.

Comment 4 Ben Cotton 2020-04-30 21:04:40 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 5 Ben Cotton 2020-05-26 14:34:13 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.