Bug 1876912

Summary:	dns-node-resolver fails: svc_ips[*]: unbound variable
Product:	OpenShift Container Platform	Reporter:	Andy McCrae <amccrae>
Component:	Networking	Assignee:	Miciah Dashiel Butler Masters <mmasters>
Networking sub component:	DNS	QA Contact:	Hongan Li <hongli>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, jcallen, john.wong, kholtz, lshilin, rdossant, sgreene, sreichar, yprokule
Version:	4.6
Target Milestone:	---
Target Release:	4.6.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 16:38:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andy McCrae 2020-09-08 13:49:56 UTC

Description of problem:

Since https://github.com/openshift/cluster-dns-operator/commit/6b4360e49537e2a1667165b730746db642b32751 the dns-node-resolver fails to start in 4.6 with the following error:

/bin/bash: line 38: svc_ips[*]: unbound variable

Version-Release number of selected component (if applicable):
4.6

How reproducible:
Consistent

Steps to Reproduce:
1. Deploy Openshift 
2. Wait for dns-default pod to CrashLoopBackOff
3. Check pod definition

Actual results:

dns-node-resolver doesn't start with the above error.

Expected results:

dns-node-resolver starts normally.

Additional info:

This is related to https://bugzilla.redhat.com/show_bug.cgi?id=1872080 in that the change made to the svc_ips approach is compatible with newer bash versions included in Rhel8+, but not compatible with the older bash in Rhel 7.

For e.g. on Rhel 8.2 /w bash 4.4.19(1):

# bash -c 'set -u; declare -a ary; echo "${#ary[@]}"'
bash: ary: unbound variable
# bash -c 'set -u; declare -A ary; echo "${ary[*]}"'
# echo $?
0

On Rhel 7.6 /w bash 4.2.46(2):
# bash -c 'set -u; declare -a ary; echo "${#ary[@]}"'
0
# bash -c 'set -u; declare -A ary; echo "${ary[*]}"'
bash: ary[*]: unbound variable

https://bugzilla.redhat.com/show_bug.cgi?id=1875566 is filed to deal with the initial case in bash itself.

Comment 1 Stephen Greene 2020-09-08 17:39:39 UTC

*** Bug 1876928 has been marked as a duplicate of this bug. ***

Comment 4 Hongan Li 2020-09-14 08:36:26 UTC

verified with 4.6.0-0.nightly-2020-09-12-230035 and dns pods work well. 

# oc -n openshift-dns get ds/dns-default -oyaml
<---snip--->
            # Update /etc/hosts only if we get valid service IPs
            # We will not update /etc/hosts when there is coredns service outage or api unavailability
            # Stale entries could exist in /etc/hosts if the service is deleted
            if [[ -n "${svc_ips[*]-}" ]]; then
              # Build a new hosts file from /etc/hosts with our custom entries filtered out

# oc -n openshift-dns get pod
NAME                READY   STATUS    RESTARTS   AGE
dns-default-4p4dl   3/3     Running   0          7h46m
dns-default-7wgpb   3/3     Running   0          7h46m
dns-default-9fccl   3/3     Running   0          7h55m
dns-default-gqjxc   3/3     Running   0          7h55m
dns-default-kk9h5   3/3     Running   0          7h46m
dns-default-l2nr8   3/3     Running   0          7h55m

Comment 6 errata-xmlrpc 2020-10-27 16:38:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196