Document URL: https://docs.openshift.com/container-platform/4.1/installing/installing_aws_user_infra/installing-aws-user-infra.html Section Number and Name: Describe the issue: The *.apps DNS record is required for installation to complete as it is needed for the authentication and console operators. Suggestions for improvement: Additional information:
As far as I know, *.apps DNS would be provisioned by ingress router, so I do not think this is an issue.
The ingress router is not creating any *.apps records. So maybe this is a bug with the ingress router then instead of with the docs.
2019-05-31T17:29:03.729Z INFO operator.controller controller/controller.go:101 reconciling {"request": "openshift-ingress-operator/default"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a7cfb41276cf111e987000e616f5cccb", "dns name": "a7cfb41276cf111e987000e616f5cccb-1923067711.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "aa6d3339575a911e99a080a416d87bfe", "dns name": "aa6d3339575a911e99a080a416d87bfe-1287501337.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a4be8a3a2767811e99b9602fb71b1e13", "dns name": "a4be8a3a2767811e99b9602fb71b1e13-242697147.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "ad1febd93767e11e99a080a416d87bfe", "dns name": "ad1febd93767e11e99a080a416d87bfe-2084259110.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a98e7eafe78c411e9a1db02fb71b1e13", "dns name": "a98e7eafe78c411e9a1db02fb71b1e13-826423417.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a40b997b47b3911e9b4ce0eb27b1ef28", "dns name": "a40b997b47b3911e9b4ce0eb27b1ef28-1634745472.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a442d30927cd211e99062126c29a244b", "dns name": "a442d30927cd211e99062126c29a244b-1310351826.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.966Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a8a619a6f7d7311e9bd6212235e364a8", "dns name": "a8a619a6f7d7311e9bd6212235e364a8-277882918.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.967Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a50fdf6bd816511e9ad82022aedbe4da", "dns name": "a50fdf6bd816511e9ad82022aedbe4da-331156363.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.967Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a344e406d817711e9b0710e162b90e27", "dns name": "a344e406d817711e9b0710e162b90e27-306263612.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.967Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a59848b08821c11e99ffa0e877aca9fc", "dns name": "a59848b08821c11e99ffa0e877aca9fc-43722952.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.967Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "a09829fb5822111e99dbe02fb71b1e13", "dns name": "a09829fb5822111e99dbe02fb71b1e13-1299202005.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.967Z INFO operator.dns aws/dns.go:199 found load balancer {"name": "aff9afbe8830111e98d1a0e877aca9fc", "dns name": "internal-aff9afbe8830111e98d1a0e877aca9fc-1525245753.us-east-1.elb.amazonaws.com", "hosted zone ID": "Z35SXDOTRQ7X7K"} 2019-05-31T17:29:03.992Z ERROR operator.init.controller-runtime.controller controller/controller.go:217 Reconciler error {"controller": "operator-controller", "request": "openshift-ingress-operator/default", "error": "failed to ensure ingresscontroller: failed to ensure DNS for default: failed to ensure DNS record &{{ map[Name:upi-bqhv9-int kubernetes.io/cluster/upi-bqhv9:owned]} ALIAS *.apps.upi.aws.sysdeseng.com -> a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com} for openshift-ingress-operator/default: failed to get hosted zone for load balancer target \"a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com\": couldn't find hosted zone ID of ELB a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com", "errorCauses": [{"error": "failed to ensure ingresscontroller: failed to ensure DNS for default: failed to ensure DNS record &{{ map[Name:upi-bqhv9-int kubernetes.io/cluster/upi-bqhv9:owned]} ALIAS *.apps.upi.aws.sysdeseng.com -> a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com} for openshift-ingress-operator/default: failed to get hosted zone for load balancer target \"a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com\": couldn't find hosted zone ID of ELB a13d5fb0683c411e9a0d902722c57c88-420581299.us-east-2.elb.amazonaws.com"}]}
We need a doc which describes how a user would determine the name of the ELB created by the router service and then tell the user to create CNAME records for that ELB. In UPI we have no promises that the ingress controller will have any ability to program route53. If it works, that's almost by accident :) I'm ok if the cloudformation template does leave things in a state where the ingress operator is able to finish the job, but this is NOT how most UPI customers should work.
Now that it's clear what we need to do we can create that A/Alias in an aws cli command. No problem. I'll provide that cmd on Mon if someone else doesn't get to it first.
Sorry everyone I lost track of this bugzilla. I have a thorough command handy... (awscli, jq from epel and yq from pip are required!) SET ENV VARS export CLUSTER_NAME=`cat metadata.json | jq -r .clusterName` export HOSTED_ZONE_NAME=`cat install-config.yaml.bak | yq -r .baseDomain` export INFRA_NAME=`cat metadata.json | jq -r .infraID` export VPCID=`cat .stack-vpc.json | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "VpcId").OutputValue'` CREATE APPS SecurityGroup export sg_apps="`aws ec2 create-security-group \ --vpc-id ${VPCID} \ --group-name appsSg \ --description appsSg`" printf "${sg_apps}" > .sg_apps if [ "`aws ec2 describe-security-groups --query "SecurityGroups[? VpcId == '${VPCID}' && GroupName == 'appsSg'].IpPermissions" | jq '.[] | length'`" == 0 ]; then aws ec2 authorize-security-group-ingress \ --group-id `echo ${sg_apps} | jq -r '.GroupId'` \ --ip-permissions '[ {"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}, {"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]} ]' fi CREATE APPS ELB aws cloudformation describe-stacks --stack-name ${INFRA_NAME}-vpc > .stack-vpc.json IFS=',' read -r -a subnet <<< `cat .stack-vpc.json | jq -r '.Stacks[0].Outputs[] | select(.OutputKey | contains("PrivateSubnetIds")).OutputValue'` declare -p subnet echo "Creating ELB..." aws elb create-load-balancer \ --load-balancer-name apps-${INFRA_NAME} \ --listener \ Protocol=TCP,LoadBalancerPort=80,InstanceProtocol=TCP,InstancePort=80 \ Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \ --security-groups `cat .sg_apps | jq -r '.GroupId'` \ --scheme internet-facing \ --subnets ${subnet[0]} \ --tags Key=name,Value=apps-${INFRA_NAME} Key=kubernetes.io/service-name,Value=openshift-ingress/router-default Key=kubernetes.io/cluster/${CLUSTER_NAME},Value=owned aws elb modify-load-balancer-attributes \ --load-balancer-name apps-${INFRA_NAME} \ --load-balancer-attributes '{ "CrossZoneLoadBalancing":{"Enabled":false}, "ConnectionDraining":{"Enabled":false} }' aws elb configure-health-check \ --load-balancer-name apps-${INFRA_NAME} \ --health-check Target=TCP:443,HealthyThreshold=2,Interval=5,Timeout=2,UnhealthyThreshold=2 for i in `seq 0 $(( ${#subnet[@]} - 1))`; do if ((${#subnet[@]} == 1)); then aws elb attach-load-balancer-to-subnets --load-balancer-name apps-${INFRA_NAME} --subnets ${subnet[0]} elif ((${#subnet[@]} == 2)); then aws elb attach-load-balancer-to-subnets --load-balancer-name apps-${INFRA_NAME} --subnets ${subnet[$((${i} % 2))]} else aws elb attach-load-balancer-to-subnets --load-balancer-name apps-${INFRA_NAME} --subnets ${subnet[${i}]} fi done IFS=',' read -r -a instances <<< `aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" "Name=tag:aws:cloudformation:logical-id,Values=Worker0" --query "Reservations[? Instances[0].VpcId == '${VPCID}'] | [].Instances[0].InstanceId" --output text | sed -e "s#\s#,#g"` declare -p instances for i in `seq 0 $(( ${#instances[@]} - 1))`; do aws elb register-instances-with-load-balancer --load-balancer-name apps-${INFRA_NAME} --instances ${instances[\${i}]} done CREATE APPS ROUTE53 Resource Record Set echo "Discovering ELBs in VPC: ${VPCID}" for i in 1 2 3 4 5; do if `aws elb describe-load-balancers --load-balancer-names apps-${INFRA_NAME} > /dev/null 2>&1`; then echo "Found ELB: apps-${INFRA_NAME} ..." aws elb describe-load-balancers --load-balancer-names apps-${INFRA_NAME} > .elb-apps-${INFRA_NAME} echo "Setting route53_apps_dns_alias_record.json: ResourceRecordSet.AliasTarget.DNSName value now" cat route53_apps_dns_alias_record.json | \ jq ".Comment = \"${INFRA_NAME}\"" | \ jq ".Changes[0].ResourceRecordSet.Name = \"*.apps.${CLUSTER_NAME}.${HOSTED_ZONE_NAME}.\"" | \ jq ".Changes[0].ResourceRecordSet.AliasTarget.DNSName = \"dualstak.`cat .elb-apps-${INFRA_NAME} | jq -r '.LoadBalancerDescriptions[0].DNSName'`\"" | \ jq ".Changes[0].ResourceRecordSet.AliasTarget.HostedZoneId = \"`cat .elb-apps-${INFRA_NAME} | jq -r '.LoadBalancerDescriptions[0].CanonicalHostedZoneNameID'`\"" > \ .route53_apps_dns_alias_record.json && \ /bin/mv .route53_apps_dns_alias_record.json route53_apps_dns_alias_record.json break else printf "*.apps ELB has not been created yet ... sleeping for 30 sec\n" sleep 30 fi done aws route53 change-resource-record-sets --hosted-zone-id ${HOSTED_ZONE} --change-batch file://route53_apps_dns_alias_record.json
Also worker nodes must be deployed before wait-for installation-complete command is run. They are reqd to handle the ingress-router pods. I have probably 2000 UPI deployments under my belt in the last 3 weeks using a Jenkins pipeline. My workflow is as follows... 1) Verify inputs ... pull secrets, ssh keys, cidr's, etc 2) Prepare installer install-config, manifests, & ignition files 3) Prepare infrastructure vpc 4) Prepare infrastructure security group and roles 5) Prepare infrastructure network and load balancers 6) Prepare infrastructure S3 buckets 7) Prepare infrastructure bootstrap incl s3 put ign file 8) Prepare controlplane nodes 9) Prepare workers nodes 10) Prepare *.apps ELB and Route53 dns record 11) Run wait-for bootstrap-complete 12) Run wait-for installation-complete
I just kicked off a round of our AWS UPI CI testing [1]. The procedure it uses is [2], and it does not do anything explicit around creating the *.apps record. And here is the ingress controller creating it for us [3]: $ curl -s --compressed https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/2123/pull-ci-openshift-installer-master-e2e-aws-upi/4/artifacts/e2e-aws-upi/must-gather/namespaces/openshift-ingress-operator/pods/ingress-operator-58f474b79-flmwm/ingress-operator/ingress-operator/logs/current.log | grep -1 '\*\.apps' 2019-08-01T15:21:13.29444427Z 2019-08-01T15:21:13.294Z INFO operator.ingress_controller ingress/controller.go:113 reconciling {"request": "openshift-ingress-operator/default"} 2019-08-01T15:21:13.475294314Z 2019-08-01T15:21:13.475Z INFO operator.ingress_controller ingress/dns.go:35 created dnsrecord {"dnsrecord": {"metadata":{"name":"default-wildcard","namespace":"openshift-ingress-operator","selfLink":"/apis/ingress.operator.openshift.io/v1/namespaces/openshift-ingress-operator/dnsrecords/default-wildcard","uid":"ff97e09c-b46f-11e9-b19a-0e7c537a4586","resourceVersion":"8802","generation":1,"creationTimestamp":"2019-08-01T15:21:13Z","labels":{"ingresscontroller.operator.openshift.io/owning-ingresscontroller":"default"},"ownerReferences":[{"apiVersion":"operator.openshift.io/v1","kind":"IngressController","name":"default","uid":"fcd2fe92-b46f-11e9-b19a-0e7c537a4586","controller":true,"blockOwnerDeletion":true}],"finalizers":["operator.openshift.io/ingress-dns"]},"spec":{"dnsName":"*.apps.ci-op-690h820f-9d51b.origin-ci-int-aws.dev.rhcloud.com.","targets":["afd5a8c49b46f11e9b19a0e7c537a458-111304965.us-east-1.elb.amazonaws.com"],"recordType":"CNAME"},"status":{}}} 2019-08-01T15:21:13.618808325Z 2019-08-01T15:21:13.618Z INFO operator.dns aws/dns.go:183 found hosted zone using tags {"zone id": "Z2A83K5ILP3F12", "tags": {"Name":"ci-op-690h820f-9d51b-k9nr7-int","kubernetes.io/cluster/ci-op-690h820f-9d51b-k9nr7":"owned"}} 2019-08-01T15:21:14.127951541Z 2019-08-01T15:21:14.127Z INFO operator.dns aws/dns.go:322 updated DNS record {"zone id": "Z2A83K5ILP3F12", "domain": "*.apps.ci-op-690h820f-9d51b.origin-ci-int-aws.dev.rhcloud.com.", "target": "afd5a8c49b46f11e9b19a0e7c537a458-111304965.us-east-1.elb.amazonaws.com", "response": "{\n ChangeInfo: {\n Id: \"/change/C3JBSPNG0D1ED1\",\n Status: \"PENDING\",\n SubmittedAt: 2019-08-01 15:21:14.342 +0000 UTC\n }\n}"} 2019-08-01T15:21:14.128113748Z 2019-08-01T15:21:14.128Z INFO operator.dns aws/dns.go:281 upserted DNS record {"record": {"metadata":{"name":"default-wildcard","namespace":"openshift-ingress-operator","selfLink":"/apis/ingress.operator.openshift.io/v1/namespaces/openshift-ingress-operator/dnsrecords/default-wildcard","uid":"ff97e09c-b46f-11e9-b19a-0e7c537a4586","resourceVersion":"8802","generation":1,"creationTimestamp":"2019-08-01T15:21:13Z","labels":{"ingresscontroller.operator.openshift.io/owning-ingresscontroller":"default"},"ownerReferences":[{"apiVersion":"operator.openshift.io/v1","kind":"IngressController","name":"default","uid":"fcd2fe92-b46f-11e9-b19a-0e7c537a4586","controller":true,"blockOwnerDeletion":true}],"finalizers":["operator.openshift.io/ingress-dns"]},"spec":{"dnsName":"*.apps.ci-op-690h820f-9d51b.origin-ci-int-aws.dev.rhcloud.com.","targets":["afd5a8c49b46f11e9b19a0e7c537a458-111304965.us-east-1.elb.amazonaws.com"],"recordType":"CNAME"},"status":{}}} 2019-08-01T15:21:14.163673376Z 2019-08-01T15:21:14.163Z INFO operator.dns aws/dns.go:322 updated DNS record {"zone id": "Z2GYOLTZHS5VK", "domain": "*.apps.ci-op-690h820f-9d51b.origin-ci-int-aws.dev.rhcloud.com.", "target": "afd5a8c49b46f11e9b19a0e7c537a458-111304965.us-east-1.elb.amazonaws.com", "response": "{\n ChangeInfo: {\n Id: \"/change/C1MDSLUVY30OYI\",\n Status: \"PENDING\",\n SubmittedAt: 2019-08-01 15:21:14.377 +0000 UTC\n }\n}"} 2019-08-01T15:21:14.163705559Z 2019-08-01T15:21:14.163Z INFO operator.dns aws/dns.go:281 upserted DNS record {"record": {"metadata":{"name":"default-wildcard","namespace":"openshift-ingress-operator","selfLink":"/apis/ingress.operator.openshift.io/v1/namespaces/openshift-ingress-operator/dnsrecords/default-wildcard","uid":"ff97e09c-b46f-11e9-b19a-0e7c537a4586","resourceVersion":"8802","generation":1,"creationTimestamp":"2019-08-01T15:21:13Z","labels":{"ingresscontroller.operator.openshift.io/owning-ingresscontroller":"default"},"ownerReferences":[{"apiVersion":"operator.openshift.io/v1","kind":"IngressController","name":"default","uid":"fcd2fe92-b46f-11e9-b19a-0e7c537a4586","controller":true,"blockOwnerDeletion":true}],"finalizers":["operator.openshift.io/ingress-dns"]},"spec":{"dnsName":"*.apps.ci-op-690h820f-9d51b.origin-ci-int-aws.dev.rhcloud.com.","targets":["afd5a8c49b46f11e9b19a0e7c537a458-111304965.us-east-1.elb.amazonaws.com"],"recordType":"CNAME"},"status":{}}} 2019-08-01T15:21:14.181285483Z 2019-08-01T15:21:14.181Z DEBUG operator.init.controller-runtime.controller controller/controller.go:236 Successfully Reconciled {"controller": "dns_controller", "request": "openshift-ingress-operator/default-wildcard"} *.apps records are being created in both the public zone is Z2GYOLTZHS5VK [4] and the private zone Z2A83K5ILP3F12. Of course, if you don't give the ingress operator sufficient credentials to create those records, you'll need to create them yourselves. I'm fine mentioning that in the UPI docs and/or shifting our CI so that it tests a UPI flow where the cluster creds lack record-creation auth. Do we have a set of permissions that we expect UPI clusters to have? If we remove *all* resource-creation permissions, then we lose the ability to create LoadBalancer services [5], etc., etc. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2123/pull-ci-openshift-installer-master-e2e-aws-upi/4 [2]: https://github.com/openshift/release/blob/92ea08af3806b213e1d6ae3f0ab58df943d5147f/ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml#L298-L680 [3]: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/2123/pull-ci-openshift-installer-master-e2e-aws-upi/4/artifacts/e2e-aws-upi/must-gather/namespaces/openshift-ingress-operator/pods/ingress-operator-58f474b79-flmwm/ingress-operator/ingress-operator/logs/current.log [4]: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/2123/pull-ci-openshift-installer-master-e2e-aws-upi/4/artifacts/e2e-aws-upi/must-gather/cluster-scoped-resources/config.openshift.io/dnses.yaml [5]: https://kubernetes.io/docs/concepts/services-networking/#loadbalancer
Hello Chris and Team, Thanks for all your inputs. I have checked with the customer and he confirmed that the annotation has been added in the router service only. ----- kind: Service apiVersion: v1 metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 name: router-default namespace: openshift-ingress ----- Regarding AWS ec2 api endpoint, they have not configured it yet but the ask is whether it is essential or not? If yes, is it documented somewhere?
Talking this over among the installer team, we're thinking about moving the installer's docs to configure the *.apps records externally, and to clear the zone properties [1] in the installer-generated DNS config [2] so the ingress operator knows it is not supposed to create records. We are not planning on adjusting our docs to limit LoadBalancer Service creation, registry storage creation, or other cluster-created resources at this time, although we can revisit those on a case-by-case basis in follow-up bugs. [1]: https://github.com/openshift/api/blob/b76189cc788c46038952dba8fbccd76b3eba866f/config/v1/types_dns.go#L33-L50 [2]: https://github.com/openshift/installer/blob/b50a68e4b891800ed4823dbf81b21b501ab3c213/pkg/asset/manifests/dns.go#L56
I see Scott changed the Target Release to 4.2.0. I've changed Version to match, and we can clone a new bug if we decide we want to backport or officially WONTFIX the 4.1.z installer docs. Not sure if openshift-docs would need its own bug for this? They might decide to port changes to their 4.1.z branch even if the installer does not.
Per https://github.com/openshift/installer/pull/2221/files, seem like this become a document, but no downstream offical doc PR for my review, so change state to "ASSIGNED".
> ... but no downstream offical doc PR for my review... No official docs from the installer repo, so I'm changing the component to Documentation (which is where the official docs come from) if we need official docs to close this bug.
No update till now, set target release to 4.2.z.
Docs are in flight with the to openshift-docs PRs I'm linking.
Gaoyun Pei, will you PTAL? https://github.com/openshift/openshift-docs/pull/17190/
Comment in the PR
Jianlin, thank you for updating the bug, and my apologies for missing Gaoyun Pei's feedback! I've incorporated some changes to the PR. Will you please take a look?
LGTM.
Thanks! I've merged the change, and it's live now. docs.openshift.com: https://docs.openshift.com/container-platform/4.2/installing/installing_aws_user_infra/installing-aws-user-infra.html#installation-create-ingress-dns-records_installing-aws-user-infra the portal: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.2/html-single/installing/index#installation-create-ingress-dns-records_installing-aws-user-infra