Bug 1721546
| Summary: | Openshift-on-OpenStack Add protection from a docker/K8s bug | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jon Uriarte <juriarte> |
| Component: | Installer | Assignee: | Michał Dulko <mdulko> |
| Installer sub component: | openshift-ansible | QA Contact: | Jon Uriarte <juriarte> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | itbrown, ltomasbo |
| Version: | 3.11.0 | Keywords: | Triaged |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-03 15:56:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Adding correct logs when the issue is reproduced: $ oc -n kuryr logs -f kuryr-cni-ds-s65sh -c kuryr-cni container_linux.go:247: starting container process caused "process_linux.go:364: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/origin/openshift.local.volumes/pods/0b1d0cd7-8ec1-11e9-a55c-fa163ee3d6f8/volume-subpaths/config-volume/kuryr-cni/2\\\" to rootfs \\\"/var/lib/docker/overlay2/400ab2b1ad7563404e51d04959ce6dfb2ce51df0a3d4c168eb42822ef52ff862/merged\\\" at \\\"/var/lib/docker/overlay2/400ab2b1ad7563404e51d04959ce6dfb2ce51df0a3d4c168eb42822ef52ff862/merged/etc/kuryr/kuryr.conf\\\" caused \\\"no such file or directory\\\"\"" I switched this to medium as this only happens when Kuryr's ConfigMap gets edited. This is now fixed, so I guess ON_QA is the correct status. Verified with: openshift v3.11.136 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2580 |
Description of problem: kury-cni pod crashes if the kuryr-config ConfigMap is manually edited due to a bug in docker/K8s [1]. A protection can be added in the playbooks. Version-Release number of the following components: $ rpm -q openshift-ansible openshift-ansible-3.11.117-1.git.0.add13ff.el7.noarch $ rpm -q ansible ansible-2.5.15-1.el7ae.noarch $ ansible --version ansible 2.5.15 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: always Steps to Reproduce: 1. Install OSP 13 or 14 with Octavia (in a hybrid environment) 2. Deploy the ansible-host and DNS server on the overcloud 3. Enable kuryr (inventory/group_vars/all.yml) 4. Run installation playbooks 5. Edit kuryr-config ConfigMap (i.e debug = true): $ oc -n kuryr edit cm kuryr-config 6. Restart kuryr-cni container - sudo docker restart <kuryr-cni> Actual results: [openshift@master-0 ~]$ ock logs -f kuryr-cni-ds-4zplv -c kuryr-cni + cleanup + rm -f /etc/cni/net.d/10-kuryr.conf + rm -f /opt/cni/bin/kuryr-cni + deploy + POD_NAMESPACE=kuryr + cat + cp /kuryr-cni /opt/cni/bin/kuryr-cni + chmod +x /opt/cni/bin/kuryr-cni + cp /etc/kuryr-cni/10-kuryr.conf /etc/cni/net.d + exec kuryr-daemon --config-file /etc/kuryr/kuryr.conf Traceback (most recent call last): File "/usr/bin/kuryr-daemon", line 10, in <module> sys.exit(start()) File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/cni/daemon/service.py", line 299, in start config.init(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/config.py", line 299, in init CONF(args=args, project='kuryr-k8s', version=version_k8s, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2577, in __call__ raise ConfigFilesNotFoundError(self._namespace._files_not_found) oslo_config.cfg.ConfigFilesNotFoundError: Failed to find some config files: /etc/kuryr/kuryr.conf Expected results: no crash [1] https://github.com/kubernetes/kubernetes/issues/68211