Bug 1663097 - Long time wasted in `DEBUG listing AWS hosted zones (page ...)` during `openshift-install create`
Summary: Long time wasted in `DEBUG listing AWS hosted zones (page ...)` during `opens...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.1.0
Assignee: Alex Crawford
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-03 06:53 UTC by Xingxing Xia
Modified: 2019-03-12 14:27 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-09 19:21:19 UTC
Target Upstream Version:


Attachments (Terms of Use)
Hosted zones not many in console (8.24 KB, image/png)
2019-01-07 03:26 UTC, Xingxing Xia
no flags Details

Description Xingxing Xia 2019-01-03 06:53:35 UTC
Description of problem:
During `openshift-install/create`, there are many repeated `DEBUG listing AWS hosted zones (page ...)` messages, leading to time waste.
If this is expected due to AWS issue, we should provide interactive approach (e.g. CTRL+whateverkey) and prompt to switch to manual input for Base Domain.

Version-Release number of the following components:
$ bin/openshift-install version
bin/openshift-install v0.8.0-master-8-g713289e20bd6afccb06f2e4ff7ed89d2483fac9a

How reproducible:
Always

Steps to Reproduce:
1. $ bin/openshift-install create install-config --dir=initial-xxia --log-level=debug
...
? SSH Public Key /home/xxia/.ssh/libra.pub
DEBUG   Fetching "Base Domain"...                  
DEBUG     Fetching "Platform"...                   
DEBUG     Generating "Platform"...                 
? Platform aws
? Region us-west-2
DEBUG   Generating "Base Domain"...                
DEBUG listing AWS hosted zones (page 0)
...
DEBUG listing AWS hosted zones (page 20000)
...

Actual results:
1. There are many repeated `DEBUG listing AWS hosted zones (page ...)` messages, leading to time waste.

The repetition depends on region I select.
If I select Region us-east-1, the message disappeared after trying page 200+ and manual input prompt then appeared as below.
...
DEBUG listing AWS hosted zones (page 244)
ERROR list hosted zones: Throttling: Rate exceeded
        status code: 400, request id: 5f8902f0-0f22-11e9-b8e1-6927c728939c
? Base Domain [? for help]

If I select us-east-2, manual input prompt appeared after page 600+.
But if I select us-west-2, manual input prompt never appeared, it kept trying up to page 20000+

Expected results:
1. If this is expected due to AWS issue, we should provide interactive approach (e.g. CTRL+whateverkey) and prompt to switch to manual input for Base Domain.

Additional info:

Comment 1 W. Trevor King 2019-01-05 06:05:12 UTC
Your account has 20000+ pages of hosted zones!  Can you confirm that (e.g. via [1], which should show the total number of zones)?

[1]: https://console.aws.amazon.com/route53/home?region=us-west-2#Dashboard:

Comment 2 Xingxing Xia 2019-01-07 03:26:06 UTC
Created attachment 1518880 [details]
Hosted zones not many in console

No, I checked web console, it doesn't have that numerous pages, see attachment.

For me to skip the long time and to work, I checked https://github.com/openshift/installer/blob/master/pkg/asset/installconfig/aws/basedomain.go#L34
34         for i := 0; true; i++ {
35                 logrus.Debugf("listing AWS hosted zones (page %d)", i)
I change it to:
34         for i := 0; i < 1; i++ {
And rebuild, then I can quickly install the cluster without the message repetition:
DEBUG       Generating "Platform"...
? Platform aws
? Region us-west-2
DEBUG     Generating "Base Domain"...
DEBUG listing AWS hosted zones (page 0)   # Only shown once due to above code change
? Base Domain devcluster.openshift.com    # Can automatically show domain options for selection
DEBUG     Fetching "Cluster Name"...
DEBUG     Generating "Cluster Name"...
? Cluster Name xxia

So there seems must be some bug.

Comment 3 W. Trevor King 2019-01-08 00:58:16 UTC
Can you run with:

$ git diff
diff --git a/pkg/asset/installconfig/aws/basedomain.go b/pkg/asset/installconfig/aws/basedomain.go
index 58da961..024d1f2 100644
--- a/pkg/asset/installconfig/aws/basedomain.go
+++ b/pkg/asset/installconfig/aws/basedomain.go
@@ -49,6 +49,7 @@ func GetBaseDomain() (string, error) {
                }
 
                input.Marker = response.Marker
+               logrus.Infof("marker %v", input.Marker)
        }
 
        publicZones := make([]string, 0, len(publicZoneMap))


applied locally?  I don't understand why pagination isn't terminating for you.

Comment 4 Xingxing Xia 2019-01-08 07:17:45 UTC
OK, I pulled latest repo, added logrus.Debugf("marker %v", input.Marker) in the line where you suggested, and saw the messages became much less today (not sure why much less today):
$ hack/build.sh
$ bin/openshift-install create cluster --dir=install-xxia --log-level=debug
? Platform aws
? Region us-east-2
DEBUG     Generating "Base Domain"...
DEBUG listing AWS hosted zones (page 0)
DEBUG marker <nil>
...
DEBUG listing AWS hosted zones (page 13)
DEBUG marker <nil>
DEBUG listing AWS hosted zones (page 14)
ERROR list hosted zones: Throttling: Rate exceeded
        status code: 400, request id: 664b86a6-1314-11e9-bd6d-75fd7b08b429
? Base Domain [? for help]  # Just prompt for input, *instead* of automatically show domain options for selection as my comment 2

Comment 5 W. Trevor King 2019-01-08 19:34:32 UTC
> DEBUG marker <nil>

Ah, that's not good ;).  Fix in flight [1].

[1]: https://github.com/openshift/installer/pull/1014

Comment 6 Xingxing Xia 2019-01-09 08:10:16 UTC
Pulled latest repo and rebuild, now after listing at most "page 3", the options come out for selection. I'd move to VERIFIED directly (skipping ON_QA). Thx


Note You need to log in before you can comment on or make changes to this bug.