Bug 1572419 - 3.9.27: logging-curator pod crash looping - run.sh not found
Summary: 3.9.27: logging-curator pod crash looping - run.sh not found
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.9.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1564350
TreeView+ depends on / blocked
 
Reported: 2018-04-27 01:18 UTC by Mike Fiedler
Modified: 2018-05-17 06:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Curator pod is crash-looping because it was unable to find its entrypoint script due to a bad merge from origin into downstream dist-git Consequence: The pod was not functional and cycled crash-looping Fix: The code was synced with upstream. Result:
Clone Of:
Environment:
Last Closed: 2018-05-17 06:43:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1566 None None None 2018-05-17 06:44:00 UTC

Description Mike Fiedler 2018-04-27 01:18:59 UTC
Description of problem:

logging-curator pod can't start on 3.9.27.  Pod log:

# oc logs logging-curator-1-p9xx4              
sh: run.sh: No such file or directory            


Version-Release number of selected component (if applicable): logging 3.9.27

registry.reg-aws.openshift.com:443/openshift3/logging-curator             v3.9                fc2ed628d296        6 hours ago         228 MB

How reproducible: Always


Steps to Reproduce:
1. Install logging with inventory below
2. oc get pods -n logging
3.

Actual results:
 
logging-curator is crash looping

Expected results:

logging-curator starts

Additional info:

[OSEv3:children]                                                      
masters                                                               
etcd                                                                  
                                                                

[masters]                                                             
ip-172-31-48-218

[etcd]                                                                
ip-172-31-48-218

                      

[OSEv3:vars]                                                          
deployment_type=openshift-enterprise                                  

openshift_deployment_type=openshift-enterprise                        
openshift_release=v3.9                                                
openshift_docker_additional_registries=registry.reg-aws.openshift.com 


openshift_logging_install_logging=true                                
openshift_logging_master_url=https://ec2-54-187-232-229.us-west-2.compute.amazonaws.com:8443
openshift_logging_master_public_url=https://ec2-54-187-232-229.us-west-2.compute.amazonaws.com:8443
openshift_logging_kibana_hostname=kibana.apps.0426-361.qe.rhcloud.com                                        
openshift_logging_image_prefix=registry.reg-aws.openshift.com:443/openshift3/                                                                
openshift_logging_image_version=v3.9                               
openshift_logging_es_cluster_size=1                                   
openshift_logging_es_pvc_dynamic=true                                 
openshift_logging_es_pvc_size=20Gi                                    
openshift_logging_es_pvc_storage_class_name=gp2                       
openshift_logging_fluentd_read_from_head=false                                                           
openshift_logging_curator_nodeselector={"region": "infra"}            
openshift_logging_kibana_nodeselector={"region": "infra"}             
openshift_logging_es_nodeselector={"region": "infra"}

Comment 5 Junqi Zhao 2018-05-02 02:33:44 UTC
Tested with logging-curator/images/v3.9.27-1,issue is not fixed

Comment 8 Junqi Zhao 2018-05-03 01:58:17 UTC
Tested with logging-curator/images/v3.9.27-2, there is not run.sh in curator pod.
The image is not right

# oc get po | grep curator
logging-curator-1-btvfl                   1/1       Running   4          20m

# oc rsh logging-curator-1-btvfl
sh-4.2$ ls -al
total 44
drwxrwxr-x. 2 root       root  176 May  3 01:54 .
drwxr-xr-x. 3 root       root   17 May  2 13:59 ..
-rw-------. 1 1000070000 root   47 May  3 01:56 .bash_history
-rwxrwxr--. 1 root       root    0 May  2 13:58 __init__.py
-rwxrwxr-x. 1 root       root 8093 May  2 13:58 curator_cmd.py
-rw-r--r--. 1 1000070000 root 6753 May  3 01:38 curator_cmd.pyc
-rwxrwxr-x. 1 root       root 4235 May  2 13:58 parser.py
-rw-r--r--. 1 1000070000 root 4095 May  3 01:38 parser.pyc
-rwxrwxr-x. 1 root       root 3427 May  2 13:58 run_cron.py
-rwxrwxr-x. 1 root       root  221 May  2 13:58 util.py
-rw-r--r--. 1 1000070000 root  488 May  3 01:38 util.pyc

Comment 9 Jeff Cantrill 2018-05-03 02:36:35 UTC
There is not supposed to be a run.sh.  The entrypoint was changed:
http://pkgs.devel.redhat.com/cgit/rpms/logging-curator-docker/tree/Dockerfile?h=rhaos-3.9-rhel-7#n44

Per your output in c#8... not the pod is running

Comment 10 Junqi Zhao 2018-05-03 04:08:15 UTC
changed to VERIFIED, tested with logging-curator/images/v3.9.27-2, the entrypoint was changed

and pod could be started up

Comment 11 Junqi Zhao 2018-05-14 08:52:29 UTC
Fix is back ported to logging-curator-v3.9.27-1, curator pod could be started up
# oc get po -n logging | grep curator
logging-curator-1-9h7g8                   1/1       Running   0          12m

Comment 14 errata-xmlrpc 2018-05-17 06:43:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566


Note You need to log in before you can comment on or make changes to this bug.