Bug 2001479

Summary: IBM Cloud DNS unable to create/update records
Product: OpenShift Container Platform Reporter: Attila Szucs <Attila.Szucs7>
Component: NetworkingAssignee: Candace Holman <cholman>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, bmcelvee, hongli, mmasters, pamoedom
Version: 4.9   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:07:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Attila Szucs 2021-09-06 08:21:33 UTC
Description of problem:
On IBMCloud IPI clusters DNS operations fail due to a few bugs in IBM DNS: 
-uninitialized map
-incorrect CIS url
-wildcard TTL value (30) can't be set for CIS
-listing records filter doesn't trim dots from the end of DNSName

OpenShift release version:
4.9

Cluster Platform:
IBMCloud IPI

How reproducible:
100%

Steps to Reproduce (in detail):
1.Create an IPI cluster on IBMCloud using the installer. Ingress-operator will fail due to bugs in IBM Cloud DNS provider code.
2.
3.


Actual results:
Ingress operator fails to create DNS records in CIS for the wildcard record

Expected results:
Ingress operator successfully creates DNS records in CIS for the wildcard record

Impact of the problem:
Blocks IPI deployment 

Additional info:
IBM is developing a fix to solve these issues.


** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Miciah Dashiel Butler Masters 2021-09-07 16:09:20 UTC
The target release needs to be set to 4.10.0 to get the fix merged into the master branch.  We can backport to the release-4.9 branch if needed.

Comment 6 Arvind iyengar 2021-10-27 09:56:21 UTC
Verified in "4.10.0-0.nightly-2021-10-25-190146" release version. The installation in IBM cloud happens successfully. There are no more issue noted with the ingress operator. 
-----------
oc get co                                                                                                 
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.nightly-2021-10-25-190146   True        False         False      6m19s   
baremetal                                  4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
cloud-controller-manager                   4.10.0-0.nightly-2021-10-25-190146   True        False         False      90m     
cloud-credential                           4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
cluster-autoscaler                         4.10.0-0.nightly-2021-10-25-190146   True        False         False      76m     
config-operator                            4.10.0-0.nightly-2021-10-25-190146   True        False         False      78m     
console                                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      5m18s   
csi-snapshot-controller                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      77m     
dns                                        4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
etcd                                       4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
image-registry                             4.10.0-0.nightly-2021-10-25-190146   True        False         False      69m     
ingress                                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      62m     
insights                                   4.10.0-0.nightly-2021-10-25-190146   True        False         False      71m     
kube-apiserver                             4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
kube-controller-manager                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
kube-scheduler                             4.10.0-0.nightly-2021-10-25-190146   True        False         False      75m     
kube-storage-version-migrator              4.10.0-0.nightly-2021-10-25-190146   True        False         False      78m     
machine-api                                4.10.0-0.nightly-2021-10-25-190146   True        False         False      72m     
machine-approver                           4.10.0-0.nightly-2021-10-25-190146   True        False         False      76m     
machine-config                             4.10.0-0.nightly-2021-10-25-190146   True        False         False      69m     
marketplace                                4.10.0-0.nightly-2021-10-25-190146   True        False         False      76m     
monitoring                                 4.10.0-0.nightly-2021-10-25-190146   True        False         False      63m     
network                                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      79m     
node-tuning                                4.10.0-0.nightly-2021-10-25-190146   True        False         False      76m     
openshift-apiserver                        4.10.0-0.nightly-2021-10-25-190146   True        False         False      69m     
openshift-controller-manager               4.10.0-0.nightly-2021-10-25-190146   True        False         False      68m     
openshift-samples                          4.10.0-0.nightly-2021-10-25-190146   True        False         False      74m     
operator-lifecycle-manager                 4.10.0-0.nightly-2021-10-25-190146   True        False         False      77m     
operator-lifecycle-manager-catalog         4.10.0-0.nightly-2021-10-25-190146   True        False         False      77m     
operator-lifecycle-manager-packageserver   4.10.0-0.nightly-2021-10-25-190146   True        False         False      73m     
service-ca                                 4.10.0-0.nightly-2021-10-25-190146   True        False         False      78m     
storage                                    4.10.0-0.nightly-2021-10-25-190146   True        False         False      78m  

oc get co ingress -o yaml                                                                                      ✔  admin ⎈ 
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
...

 status:
  conditions:
  - lastTransitionTime: "2021-10-27T08:41:12Z"
    message: The "default" ingress controller reports Available=True.
    reason: IngressAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-10-27T08:41:12Z"
    message: desired and current number of IngressControllers are equal
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: "2021-10-27T09:30:52Z"
    message: The "default" ingress controller reports Degraded=False.
    reason: IngressNotDegraded
    status: "False"
    type: Degraded  
-----------

Comment 9 Brandi Munilla 2022-02-10 20:36:04 UTC
Hi, if there is anything that customers should know about this bug or if there are any important workarounds that should be outlined in the bug fixes section OpenShift Container Platform 4.10 release notes, please update the Doc Type and Doc Text fields. If not, can you please mark it as "no doc update"? Thanks!

Comment 10 Miciah Dashiel Butler Masters 2022-02-23 22:43:10 UTC
IBM Cloud is a newly supported platform, so no doc text is needed for this fix.

Comment 12 errata-xmlrpc 2022-03-10 16:07:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056