Bug 1854008
| Summary: | The symlink logs file in /var/log/containers are not created after finishing the OCP3.11 installation | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jie Wu <jiewu> |
| Component: | Installer | Assignee: | Russell Teague <rteague> |
| Installer sub component: | openshift-ansible | QA Contact: | Gaoyun Pei <gpei> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | bleanhar, jiewu, sfu, xtian |
| Version: | 3.11.0 | Keywords: | UpcomingSprint |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-18 14:09:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The OpenShift installation '--log-driver json-file' is defined in Ansible Inventory file # docker options openshift_docker_options='--selinux-enabled --insecure-registry=172.30.0.0/16 --signature-verification=false --log-opt max-size=512m --log-opt max-file=3 --log-driver=json-file' This is not a 3.11 release blocker. This may have already been fixed in https://github.com/openshift/openshift-ansible/pull/12244 Please verify. This should be caused by the same problem as https://bugzilla.redhat.com/show_bug.cgi?id=1882572, "restart container runtime" handler task was skipped during configuring the container runtime on ansible-2.9. Setup a 3.11.317 cluster with openshift-ansible-3.11.317-1.git.0.de92996.el7.noarch.rpm. TASK [container_runtime : Fix SELinux Permissions on /var/lib/containers] ****** Tuesday 10 November 2020 13:53:22 +0800 (0:00:00.464) 0:01:57.102 ****** ok: [ci-vm-10-0-150-5.hosted.upshift.rdu2.redhat.com] => {"changed": false, "cmd": ["restorecon", "-R", "/var/lib/containers/"], "delta": "0:00:00.009963", "end": "2020-11-10 00:53:21.111723", "rc": 0, "start": "2020-11-10 00:53:21.101760", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ... RUNNING HANDLER [container_runtime : restart container runtime] **************** Tuesday 10 November 2020 13:53:22 +0800 (0:00:00.412) 0:01:57.514 ****** changed: [ci-vm-10-0-150-5.hosted.upshift.rdu2.redhat.com] => {"attempts": 1, "changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2020-11-10 00:51:14 EST", ... After fresh install, check /var/log/containers on master and node, symlink to /var/log/pods/ were created. [root@sunilc-311master-etcd-1 ~]# ls -al /var/log/containers/ total 8 drwxr-xr-x. 2 root root 4096 Nov 10 01:13 . drwxr-xr-x. 15 root root 4096 Nov 10 01:02 .. lrwxrwxrwx. 1 root root 58 Nov 10 01:13 apiserver-hs46x_openshift-template-service-broker_c-9f08796281b3ea6232bcf385fed20668f2de811b8e5c1c38cf18915ca160bf75.log -> /var/log/pods/d3170a81-231b-11eb-ace5-fa163e5972f8/c/0.log lrwxrwxrwx. 1 root root 66 Nov 10 01:11 apiserver-v5rgk_kube-service-catalog_apiserver-79e976fc721e8b4223f443e8346d1e64481617146e5f68691638e611f6467b0b.log -> /var/log/pods/84f170c1-231b-11eb-ace5-fa163e5972f8/apiserver/0.log lrwxrwxrwx. 1 root root 64 Nov 10 01:10 console-58d75f67-tmxjr_openshift-console_console-25e36da2c0d9a03b88aeddd433d63770ad5a1cd8706de44e02e59026f828d85b.log -> /var/log/pods/6181f991-231b-11eb-ace5-fa163e5972f8/console/0.log ... [root@sunilc-311node-registry-router-1 ~]# ls -al /var/log/containers total 8 drwxr-xr-x. 2 root root 4096 Nov 10 01:13 . drwxr-xr-x. 15 root root 4096 Nov 10 01:07 .. lrwxrwxrwx. 1 root root 69 Nov 10 01:10 alertmanager-main-1_openshift-monitoring_alertmanager-66a655a6e5a4f136c38b85eb42b5586372b812d2200f524a09ff510c5f8dee0b.log -> /var/log/pods/779d5845-231b-11eb-ace5-fa163e5972f8/alertmanager/0.log lrwxrwxrwx. 1 root root 75 Nov 10 01:10 alertmanager-main-1_openshift-monitoring_alertmanager-proxy-de36a1748aae0639170758830f09c86e75da2b9ba053f46be624873e67e67c63.log -> /var/log/pods/779d5845-231b-11eb-ace5-fa163e5972f8/alertmanager-proxy/0.log lrwxrwxrwx. 1 root root 72 Nov 10 01:10 alertmanager-main-1_openshift-monitoring_config-reloader-1e02149bb426097eafef4313ae2186396334a5feb5e5460f2486c8e959e12042.log -> /var/log/pods/779d5845-231b-11eb-ace5-fa163e5972f8/config-reloader/0.log lrwxrwxrwx. 1 root root 60 Nov 10 01:13 asb-1-6tc9l_openshift-ansible-service-broker_asb-921202b28ca092035544579c29959902f0817f04d4dd3112c147afd383c4b335.log -> /var/log/pods/cd8d4c46-231b-11eb-ace5-fa163e5972f8/asb/0.log lrwxrwxrwx. 1 root root 65 Nov 10 01:08 docker-registry-1-966h2_default_registry-88770bea5fc80808ff0cf43645b95d6718076749849ec51418dba8c2bd3bb707.log -> /var/log/pods/2bbcbc24-231b-11eb-ace5-fa163e5972f8/registry/0.log ... Thanks for fixing this bugs! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.318 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5107 |
Description of problem: The symlink logs file in /var/log/containers are not created, and need to restart the docker service after finishing the OCP installation. Version-Release number of the following components: openshift-ansible 3.11.188 ansible 2.8 above How reproducible: always Steps to Reproduce: 1. Ansible version 2.8 above or 2.9 2. Pre-installed docker and keep starting the docker service yum install docker -y systemctl enable docker --now 3. Run the ansible-openshift playbook to install the OpenShift Cluster ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml 4. Confirm the logging driver is 'journald' and the symlink logs file in /var/log/containers are NOT created. Actual results: The symlink logs file in /var/log/containers are NOT created. Handler Task'restart container runtime' is 'skipped'. /usr/share/ansible/openshift-ansible/roles/container_runtime/handlers/main.yml Expected results: The symlink logs file in /var/log/containers are created. Handler Task'restart container runtime' is 'changed'. /usr/share/ansible/openshift-ansible/roles/container_runtime/handlers/main.yml Additional info: I found the reason that the handler tasks for 'restart container runtime' is skipped, so the symlink logs file in /var/log/containers will not be created. The problem is happened only ansible 2.8 or 2.9 above, 'restart container runtime' will NOT skip in ansible version 2.6 or 2.7. About Docker restart handlers tasks. #cat /usr/share/ansible/openshift-ansible/roles/container_runtime/handlers/main.yml ---------------------------------------------------------------------------------------- - name: restart container runtime systemd: name: "{{ openshift_docker_service_name }}" state: restarted daemon_reload: yes register: r_docker_restart_docker_result until: not (r_docker_restart_docker_result is failed) retries: 3 delay: 30 when: - not docker_service_status_changed | default(false) | bool - not openshift_use_crio_only ---------------------------------------------------------------------------------------- The symlink logs files are created in /var/log/containers when /etc/sysconfig/docker is --log-driver json-file ---------------------------------------------------------------------------------------- # docker info Logging Driver: json-file ---------------------------------------------------------------------------------------- The symlink logs files are NOT created in /var/log/containers when /etc/sysconfig/docker is --log-driver journald ---------------------------------------------------------------------------------------- # docker info Logging Driver: journald ----------------------------------------------------------------------------------------