Attachment 1818463 Details for Bug 1998673 – machine-config-daemon logs

machine-config-daemon logs

mcd.log (text/plain), 14.49 KB, created by Sai Sindhur Malleni on 2021-08-27 22:42:56 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Sai Sindhur Malleni

Created: 2021-08-27 22:42:56 UTC

Size: 14.49 KB

patch

obsolete

>I0825 00:00:58.941337    6634 start.go:108] Version: v4.7.0-202105111858.p0-dirty (e3863b02b7403342cdf0f981889e8c3cfc2d86bb)
>I0825 00:00:58.943788    6634 start.go:121] Calling chroot("/rootfs")
>I0825 00:00:58.943932    6634 rpm-ostree.go:261] Running captured: rpm-ostree status --json
>I0825 00:00:59.090036    6634 daemon.go:218] Booted osImageURL: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e850d731de8eb871c5ec632e3e750e7e82e03e61c82b056f5c2b8af200e15a4d (47.83.202105111846-0)
>I0825 00:00:59.132942    6634 start.go:97] Copied self to /run/bin/machine-config-daemon on host
>I0825 00:00:59.134330    6634 metrics.go:105] Registering Prometheus metrics
>I0825 00:00:59.134378    6634 metrics.go:110] Starting metrics listener on 127.0.0.1:8797
>I0825 00:00:59.136239    6634 update.go:1904] Starting to manage node: worker037-fc640
>I0825 00:00:59.139676    6634 rpm-ostree.go:261] Running captured: rpm-ostree status
>I0825 00:00:59.166786    6634 daemon.go:849] State: idle
>Deployments:
>* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e850d731de8eb871c5ec632e3e750e7e82e03e61c82b056f5c2b8af200e15a4d
>              CustomOrigin: Managed by machine-config-operator
>                   Version: 47.83.202105111846-0 (2021-05-11T18:49:55Z)
>
>  ostree://3fdd1488024f054e39b1be508781d535d1ac7ed423bb3b4b656c2f345934220d
>                   Version: 47.83.202103251640-0 (2021-03-25T16:44:03Z)
>I0825 00:00:59.166800    6634 rpm-ostree.go:261] Running captured: journalctl --list-boots
>I0825 00:00:59.170731    6634 daemon.go:856] journalctl --list-boots:
>-1 0f91f69d564b4bcb9d443a7c24486b9e Tue 2021-08-24 23:51:28 UTCâTue 2021-08-24 23:55:40 UTC
> 0 c61c7f1f7ae54d09a4650c749c235a6d Tue 2021-08-24 23:58:48 UTCâWed 2021-08-25 00:00:59 UTC
>I0825 00:00:59.170742    6634 rpm-ostree.go:261] Running captured: systemctl list-units --state=failed --no-legend
>I0825 00:00:59.176131    6634 daemon.go:869] systemctl --failed:
>NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online
>I0825 00:00:59.176140    6634 daemon.go:603] Starting MachineConfigDaemon
>I0825 00:00:59.176218    6634 daemon.go:610] Enabling Kubelet Healthz Monitor
>I0825 00:01:29.136967    6634 trace.go:205] Trace[646203300]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (25-Aug-2021 00:00:59.136) (total time: 30000ms):
>Trace[646203300]: [30.000541733s] [30.000541733s] END
>E0825 00:01:29.136996    6634 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://172.30.0.1:443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 172.30.0.1:443: i/o timeout
>I0825 00:01:29.136974    6634 trace.go:205] Trace[1106410694]: "Reflector ListAndWatch" name:github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101 (25-Aug-2021 00:00:59.136) (total time: 30000ms):
>Trace[1106410694]: [30.000554922s] [30.000554922s] END
>E0825 00:01:29.137010    6634 reflector.go:138] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to watch *v1.MachineConfig: failed to list *v1.MachineConfig: Get "https://172.30.0.1:443/apis/machineconfiguration.openshift.io/v1/machineconfigs?limit=500&resourceVersion=0": dial tcp 172.30.0.1:443: i/o timeout
>I0825 00:01:31.630013    6634 daemon.go:387] Node worker037-fc640 is not labeled node-role.kubernetes.io/master
>I0825 00:01:31.630134    6634 node.go:24] No machineconfiguration.openshift.io/currentConfig annotation on node worker037-fc640: map[k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac:0a:58:0a:83:1a:03 k8s.ovn.org/l3-gateway-config:{"default":{"mode":"shared","interface-id":"br-ex_worker037-fc640","mac-address":"40:a6:b7:00:5f:b1","ip-addresses":["192.168.216.50/21"],"ip-address":"192.168.216.50/21","next-hops":["192.168.216.1"],"next-hop":"192.168.216.1","node-port-enable":"true","vlan-id":"0"}} k8s.ovn.org/node-chassis-id:0173f2f6-51dc-4963-be35-57d0a01d6a3f k8s.ovn.org/node-local-nat-ip:{"default":["169.254.4.255"]} k8s.ovn.org/node-mgmt-port-mac-address:fa:d3:79:e9:0b:84 k8s.ovn.org/node-primary-ifaddr:{"ipv4":"192.168.216.50/21"} k8s.ovn.org/node-subnets:{"default":"10.131.26.0/23"} machine.openshift.io/machine:openshift-machine-api/test922-bklht-worker-0-l4qbd volumes.kubernetes.io/controller-managed-attach-detach:true], in cluster bootstrap, loading initial node annotation from /etc/machine-config-daemon/node-annotations.json
>I0825 00:01:31.630463    6634 node.go:45] Setting initial node config: rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0825 00:01:31.643410    6634 daemon.go:767] In bootstrap mode
>I0825 00:01:31.643425    6634 daemon.go:795] Current+desired config: rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0825 00:01:31.647915    6634 daemon.go:1047] No bootstrap pivot required; unlinking bootstrap node annotations
>I0825 00:01:31.647963    6634 daemon.go:1085] Validating against pending config rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0825 00:01:31.660838    6634 daemon.go:1096] Validated on-disk state
>I0825 00:01:31.672014    6634 daemon.go:1151] Completing pending config rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0825 00:01:31.672029    6634 update.go:1904] completed update for config rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0825 00:01:31.674438    6634 daemon.go:1167] In desired config rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:42.361957    6634 update.go:596] Checking Reconcilable for config rendered-worker-be7070fffc9b1bb28637063800e3cfef to rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:42.394420    6634 update.go:1904] Starting update from rendered-worker-be7070fffc9b1bb28637063800e3cfef to rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef: &{osUpdate:false kargs:false fips:false passwd:false files:false units:false kernelType:false extensions:false}
>I0827 18:11:42.424809    6634 update.go:627] Changes do not require drain, skipping.
>I0827 18:11:42.424838    6634 update.go:1219] Updating files
>I0827 18:11:42.436979    6634 update.go:1616] Writing file "/etc/NetworkManager/conf.d/99-keyfiles.conf"
>I0827 18:11:42.438810    6634 update.go:1616] Writing file "/etc/NetworkManager/dispatcher.d/40-mdns-hostname"
>I0827 18:11:42.440481    6634 update.go:1616] Writing file "/etc/NetworkManager/conf.d/99-kni.conf"
>I0827 18:11:42.441872    6634 update.go:1616] Writing file "/etc/NetworkManager/dispatcher.d/30-resolv-prepender"
>I0827 18:11:42.443190    6634 update.go:1616] Writing file "/etc/NetworkManager/dispatcher.d/30-static-dhcp"
>I0827 18:11:42.444326    6634 update.go:1616] Writing file "/etc/NetworkManager/dispatcher.d/30-static-dhcpv6"
>I0827 18:11:42.445723    6634 update.go:1616] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
>I0827 18:11:42.447014    6634 update.go:1616] Writing file "/etc/keepalived/monitor.conf"
>I0827 18:11:42.448367    6634 update.go:1616] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
>I0827 18:11:42.449630    6634 update.go:1616] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem"
>I0827 18:11:42.450820    6634 update.go:1616] Writing file "/usr/local/bin/configure-ovs.sh"
>I0827 18:11:42.452844    6634 update.go:1616] Writing file "/etc/containers/storage.conf"
>I0827 18:11:42.454888    6634 update.go:1616] Writing file "/etc/kubernetes/static-pod-resources/coredns/Corefile.tmpl"
>I0827 18:11:42.456041    6634 update.go:1616] Writing file "/etc/kubernetes/manifests/coredns.yaml"
>I0827 18:11:42.457142    6634 update.go:1616] Writing file "/etc/mco/proxy.env"
>I0827 18:11:42.458209    6634 update.go:1616] Writing file "/etc/systemd/system/crio.service.d/20-stream-address.conf"
>I0827 18:11:42.459443    6634 update.go:1616] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
>I0827 18:11:42.460494    6634 update.go:1616] Writing file "/etc/modules-load.d/iptables.conf"
>I0827 18:11:42.462471    6634 update.go:1616] Writing file "/etc/kubernetes/static-pod-resources/keepalived/keepalived.conf.tmpl"
>I0827 18:11:42.463531    6634 update.go:1616] Writing file "/etc/kubernetes/manifests/keepalived.yaml"
>I0827 18:11:42.464621    6634 update.go:1616] Writing file "/etc/kubernetes/kubelet-ca.crt"
>I0827 18:11:42.465709    6634 update.go:1616] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
>I0827 18:11:42.466658    6634 update.go:1616] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
>I0827 18:11:42.467636    6634 update.go:1616] Writing file "/etc/kubernetes/static-pod-resources/mdns/config.hcl.tmpl"
>I0827 18:11:42.468617    6634 update.go:1616] Writing file "/etc/kubernetes/manifests/mdns-publisher.yaml"
>I0827 18:11:42.469746    6634 update.go:1616] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
>I0827 18:11:42.470787    6634 update.go:1616] Writing file "/etc/tmpfiles.d/nm.conf"
>I0827 18:11:42.471810    6634 update.go:1616] Writing file "/var/lib/kubelet/config.json"
>I0827 18:11:42.472993    6634 update.go:1616] Writing file "/etc/kubernetes/ca.crt"
>I0827 18:11:42.474010    6634 update.go:1616] Writing file "/etc/ssh/sshd_config.d/10-disable-ssh-key-dir.conf"
>I0827 18:11:42.475068    6634 update.go:1616] Writing file "/etc/sysctl.d/forward.conf"
>I0827 18:11:42.476101    6634 update.go:1616] Writing file "/etc/sysctl.d/inotify.conf"
>I0827 18:11:42.477020    6634 update.go:1616] Writing file "/usr/local/sbin/set-valid-hostname.sh"
>I0827 18:11:42.478131    6634 update.go:1616] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy"
>I0827 18:11:42.479781    6634 update.go:1616] Writing file "/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
>I0827 18:11:42.481108    6634 update.go:1616] Writing file "/etc/containers/registries.conf"
>I0827 18:11:42.482541    6634 update.go:1616] Writing file "/etc/crio/crio.conf.d/00-default"
>I0827 18:11:42.483857    6634 update.go:1616] Writing file "/etc/containers/policy.json"
>I0827 18:11:42.485332    6634 update.go:1616] Writing file "/etc/kubernetes/cloud.conf"
>I0827 18:11:42.486243    6634 update.go:1616] Writing file "/etc/kubernetes/kubelet.conf"
>I0827 18:11:42.487607    6634 update.go:1616] Writing file "/etc/sysconfig/network-scripts/ifcfg-eno1"
>I0827 18:11:42.488963    6634 update.go:1616] Writing file "/etc/sysconfig/network-scripts/ifcfg-eno1np0"
>I0827 18:11:42.489933    6634 update.go:1616] Writing file "/etc/chrony.conf"
>I0827 18:11:42.491018    6634 update.go:1509] Dropin for 10-mco-default-env.conf has no content, skipping write
>I0827 18:11:42.491033    6634 update.go:1516] Removing "/etc/systemd/system/crio.service.d/10-mco-default-env.conf", updated file has zero length
>I0827 18:11:42.491060    6634 update.go:1523] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
>I0827 18:11:42.491766    6634 update.go:1523] Writing systemd unit dropin "10-mco-default-madv.conf"
>I0827 18:11:42.746118    6634 update.go:1496] Preset systemd unit crio.service
>I0827 18:11:42.746154    6634 update.go:1523] Writing systemd unit dropin "mco-disabled.conf"
>I0827 18:11:42.752783    6634 update.go:1594] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
>)
>I0827 18:11:42.752797    6634 update.go:1509] Dropin for 10-mco-default-env.conf has no content, skipping write
>I0827 18:11:42.752806    6634 update.go:1516] Removing "/etc/systemd/system/kubelet.service.d/10-mco-default-env.conf", updated file has zero length
>I0827 18:11:42.752835    6634 update.go:1523] Writing systemd unit dropin "10-mco-default-madv.conf"
>I0827 18:11:42.753057    6634 update.go:1557] Writing systemd unit "kubelet.service"
>I0827 18:11:42.753287    6634 update.go:1557] Writing systemd unit "machine-config-daemon-firstboot.service"
>I0827 18:11:42.753476    6634 update.go:1557] Writing systemd unit "machine-config-daemon-pull.service"
>I0827 18:11:42.753677    6634 update.go:1557] Writing systemd unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
>I0827 18:11:42.753858    6634 update.go:1557] Writing systemd unit "node-valid-hostname.service"
>I0827 18:11:42.754035    6634 update.go:1557] Writing systemd unit "nodeip-configuration.service"
>I0827 18:11:42.754208    6634 update.go:1557] Writing systemd unit "ovs-configuration.service"
>I0827 18:11:42.754377    6634 update.go:1523] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
>I0827 18:11:42.967988    6634 update.go:1496] Preset systemd unit ovs-vswitchd.service
>I0827 18:11:42.968015    6634 update.go:1523] Writing systemd unit dropin "10-ovsdb-restart.conf"
>I0827 18:11:42.968469    6634 update.go:1509] Dropin for 10-mco-default-env.conf has no content, skipping write
>I0827 18:11:42.968480    6634 update.go:1516] Removing "/etc/systemd/system/pivot.service.d/10-mco-default-env.conf", updated file has zero length
>I0827 18:11:42.973934    6634 update.go:1594] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
>)
>I0827 18:11:42.973957    6634 update.go:1523] Writing systemd unit dropin "mco-disabled.conf"
>I0827 18:11:42.979162    6634 update.go:1594] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
>)
>I0827 18:11:43.194896    6634 update.go:1474] Enabled systemd units: [kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service etc-NetworkManager-system\x2dconnections\x2dmerged.mount node-valid-hostname.service nodeip-configuration.service openvswitch.service ovs-configuration.service ovsdb-server.service]
>I0827 18:11:43.194952    6634 update.go:1292] Deleting stale data
>I0827 18:11:43.208661    6634 update.go:1735] Writing SSHKeys at "/home/core/.ssh/authorized_keys"
>I0827 18:11:43.227189    6634 update.go:1904] Node has Desired Config rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef, skipping reboot
>I0827 18:11:43.229449    6634 daemon.go:802] Current config: rendered-worker-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:43.229464    6634 daemon.go:803] Desired config: rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:43.239230    6634 daemon.go:1151] Completing pending config rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:43.239246    6634 update.go:1904] completed update for config rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef
>I0827 18:11:43.241044    6634 daemon.go:1167] In desired config rendered-worker-group-3-be7070fffc9b1bb28637063800e3cfef
>I0827 22:14:21.978047    6634 daemon.go:586] Got SIGTERM, but actively updating

Actions: View

Attachments on bug 1998673: 1818463