Bug 2081562

Summary: lifecycle.posStart hook does not have network connectivity.
Product: OpenShift Container Platform Reporter: Seunghwan Jung <jseunghw>
Component: NetworkingAssignee: Nadia Pinaeva <npinaeva>
Networking sub component: openshift-sdn QA Contact: zhaozhanqi <zzhao>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, ddelcian, ffernand, hyoskim, mapandey, npinaeva, swasthan, wking, zzhao
Version: 4.10   
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
no doc update
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 11:09:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2112353    

Description Seunghwan Jung 2022-05-04 01:58:46 UTC
Description of problem:

lifecycle.posStart does not have network connectivity on OpenShiftSDN CNI. (OVNKubernetes does not have the issue)


Version-Release number of selected component (if applicable):
4.10

How reproducible:
always

Steps to Reproduce:
1. create statefulset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ oc create -f statefulset.yaml
$ cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: httpd
spec:
  serviceName: "httpd"
  replicas: 1
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
        - containerPort: 80
          name: web
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - curl -k https://<IP:PORT> > /tmp/urltest.txt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Actual results:

PostStartHook fails 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
36s         Normal    Killing               pod/httpd-0         FailedPostStartHook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Expected results:

PostStartHook should not fail.



Additional info:

by adding a dummy initContainers, you can workaround the issue. 
something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    spec:
      initContainers:
      - name: init-myservice
        image: busybox:1.28
        command: ['sh', '-c', 'sleep 2']
      containers:
      - name: httpd
        image: registry.redhat.io/rhel8/httpd-24:1-191
        ports:
        - containerPort: 80
          name: web
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - curl -k <IP:PORT> > /tmp/urltest.txt
        ....
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment 1 Ryan Phillips 2022-05-11 19:48:10 UTC
crio's contract with networking is to have networking up when the container starts. Moving to the openshift-sdn team to help triage what is going on.

Comment 7 zhaozhanqi 2022-07-27 06:40:30 UTC
this fixed PR is merged to build 4.12.0-0.nightly-2022-07-24-180529
So I update the target version to 4.12 version.

Comment 8 zhaozhanqi 2022-07-27 06:48:33 UTC
still failed on build 4.12.0-0.nightly-2022-07-26-131732

Creating above statefulset and pod still cannot be worked with same error 

27s         Warning   FailedPostStartHook                                pod/httpd-0                                       Exec lifecycle hook ([/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt]) for Container "httpd" in Pod "httpd-0_default(7e519841-7092-4513-928b-03c7783ddc7d)" failed - error: command '/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt' exited with 1: /bin/sh: -c: line 0: syntax error near unexpected token `>'...
85s         Normal    Killing                                            pod/httpd-0                                       FailedPostStartHook

Comment 14 errata-xmlrpc 2022-08-10 11:09:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069