Bug 1697262

Description of problem:

Version-Release number of the following components:
4.0.0-0.nightly-2019-04-05-165550

How reproducible:
Always

Steps to Reproduce:
1. Follow https://github.com/openshift/installer/blob/master/docs/user/aws/install_upi.md#configure-router-for-upi-dns to create UPI on aws.
2.
3.

Actual results:
After installation, cluster is not getting into some healthy state.
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-04-05-165550   False       True          3h54m   Unable to apply 4.0.0-0.nightly-2019-04-05-165550: the cluster operator console has not yet successfully rolled out

# oc get clusteroperator
NAME                                 VERSION                             AVAILABLE   PROGRESSING   FAILING   SINCE
authentication                                                           False       False         True      3h46m
console                              4.0.0-0.nightly-2019-04-05-165550   False       True          False     3h46m

Both operators are not getting into healthy state due to *.apps DNS issue.

After workaround BZ#1697236, found no DNS *.apps registration.
Check ingress router log:
# oc logs -f ingress-operator-7dc9b877-qzss2 -n openshift-ingress-operator
<--snip-->
2019-04-08T07:54:50.788Z	ERROR	operator.init.kubebuilder.controller	controller/controller.go:217	Reconciler error	{"controller": "operator-controller", "request": "openshift-ingress-operator/default", "error": "failed to ensure clusteringress: failed to ensure DNS for default: failed to ensure DNS record &{{ map[Name:jialiuuuu1-xbhm2-int kubernetes.io/cluster/jialiuuuu1-xbhm2:owned]} ALIAS *.apps.jialiuuuu1.qe1.devcluster.openshift.com -> a854be69459ad11e9a77f02661f80878-1072364770.us-east-2.elb.amazonaws.com} for openshift-ingress-operator/default: failed to find hosted zone for record &{{ map[Name:jialiuuuu1-xbhm2-int kubernetes.io/cluster/jialiuuuu1-xbhm2:owned]} ALIAS *.apps.jialiuuuu1.qe1.devcluster.openshift.com -> a854be69459ad11e9a77f02661f80878-1072364770.us-east-2.elb.amazonaws.com}: no matching hosted zone found", "errorCauses": [{"error": "failed to ensure clusteringress: failed to ensure DNS for default: failed to ensure DNS record &{{ map[Name:jialiuuuu1-xbhm2-int kubernetes.io/cluster/jialiuuuu1-xbhm2:owned]} ALIAS *.apps.jialiuuuu1.qe1.devcluster.openshift.com -> a854be69459ad11e9a77f02661f80878-1072364770.us-east-2.elb.amazonaws.com} for openshift-ingress-operator/default: failed to find hosted zone for record &{{ map[Name:jialiuuuu1-xbhm2-int kubernetes.io/cluster/jialiuuuu1-xbhm2:owned]} ALIAS *.apps.jialiuuuu1.qe1.devcluster.openshift.com -> a854be69459ad11e9a77f02661f80878-1072364770.us-east-2.elb.amazonaws.com}: no matching hosted zone found"}]}


Expected results:
*.apps dns should be created automatically.

Additional info:
Compare with a IPI env:
IPI:
# aws route53 list-tags-for-resource --resource-type hostedzone --resource-id  Z3AV2FDEBVHLZN
{
    "ResourceTagSet": {
        "ResourceType": "hostedzone", 
        "ResourceId": "Z3AV2FDEBVHLZN", 
        "Tags": [
            {
                "Value": "owned", 
                "Key": "kubernetes.io/cluster/qe-jialiu-jj626"
            }, 
            {
                "Value": "2019-04-08T03:15:02.479817+00:00", 
                "Key": "openshift_creationDate"
            }, 
            {
                "Value": "2019-04-10T03:15:02.479817+00:00", 
                "Key": "openshift_expirationDate"
            }, 
            {
                "Value": "qe-jialiu-jj626-int", 
                "Key": "Name"
            }
        ]
    }
}
UPI:
# aws route53 list-tags-for-resource --resource-type hostedzone --resource-id   Z295LDR9P4ZPLI
{
    "ResourceTagSet": {
        "ResourceType": "hostedzone", 
        "ResourceId": "Z295LDR9P4ZPLI", 
        "Tags": [
            {
                "Value": "2019-04-08T03:15:02.479817+00:00", 
                "Key": "openshift_creationDate"
            }, 
            {
                "Value": "2019-04-10T03:15:02.479817+00:00", 
                "Key": "openshift_expirationDate"
            }
        ]
    }
}

Workaround:
# aws route53 change-tags-for-resource --resource-type hostedzone --resource-id Z295LDR9P4ZPLI --add-tags Key=kubernetes.io/cluster/jialiuuuu1-xbhm2,Value=owned Key=Name,Value=jialiuuuu1-xbhm2-int

# aws route53 list-tags-for-resource --resource-type hostedzone --resource-id   Z295LDR9P4ZPLI{
    "ResourceTagSet": {
        "ResourceType": "hostedzone", 
        "ResourceId": "Z295LDR9P4ZPLI", 
        "Tags": [
            {
                "Value": "owned", 
                "Key": "kubernetes.io/cluster/jialiuuuu1-xbhm2"
            }, 
            {
                "Value": "2019-04-08T03:15:02.479817+00:00", 
                "Key": "openshift_creationDate"
            }, 
            {
                "Value": "owned", 
                "Key": "kubernetes.io/cluster/kubernetes.io/cluster/jialiuuuu1-xbhm2"
            }, 
            {
                "Value": "2019-04-10T03:15:02.479817+00:00", 
                "Key": "openshift_expirationDate"
            }, 
            {
                "Value": "jialiuuuu1-xbhm2-int", 
                "Key": "Name"
            }
        ]
    }
}

Wait for some mins, *.apps is created by ingress operator, cluster works.

NOTE:
hostzone tag is using 'infraID' string as tag name, that is different with BZ#1697236 which is using 'cluster name' string as tag name, ideally they should be keep insistent.