Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1752409

Summary: [ci][vsphere-upi] failed to create aws route53 record for ARRDATAIllegalIPv4Address
Product: OpenShift Container Platform Reporter: sheng.lao <shlao>
Component: InstallerAssignee: Joseph Callen <jcallen>
Installer sub component: openshift-installer QA Contact: Johnny Liu <jialiu>
Status: CLOSED DEFERRED Docs Contact:
Severity: high    
Priority: high CC: adahiya, jlebon
Version: 4.3.0Keywords: Reopened
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-30 21:34:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description sheng.lao 2019-09-16 10:09:22 UTC
Description of problem:
https://prow.k8s.io/view/gcs/origin-ci-test/logs/canary-openshift-ocp-installer-e2e-vsphere-upi-4.2/102


Container setup exited with code 1, reason Error
---
043a18
* module.dns.aws_route53_record.api-internal: 1 error(s) occurred:

* aws_route53_record.api-internal: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: a3e064e1-ed01-4811-9dfe-cb1939508ffd
* module.dns.aws_route53_record.control_plane_nodes[2]: 1 error(s) occurred:

* aws_route53_record.control_plane_nodes.2: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: d8e78ad9-3f82-4ae3-8f93-6a5e0a7b7108
* module.dns.aws_route53_record.api-external: 1 error(s) occurred:

* aws_route53_record.api-external: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: b287d547-b80c-4b6c-98b7-219514a22e26
* module.dns.aws_route53_record.compute_nodes[1]: 1 error(s) occurred:

* aws_route53_record.compute_nodes.1: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: 3408b53c-b931-4aaf-b26f-c9bc9970733c
* module.dns.aws_route53_record.etcd_a_nodes[2]: 1 error(s) occurred:

* aws_route53_record.etcd_a_nodes.2: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: 08d042c4-952f-46a5-b6c3-d4cd09e3ecb7


Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Joseph Callen 2019-09-16 16:23:59 UTC
I noticed this late last week monitoring the vsphere prow jobs.  It seems to occur randomly and I was unable to reproduce with the cluster bot.
Maybe its an issue with IPAM.  I will investigate further.

Comment 3 Scott Dodson 2019-09-17 17:17:30 UTC
This is specific to our test environment setup, moving to 4.3 but leaving as high priority.

Comment 4 Joseph Callen 2019-09-25 13:45:50 UTC
Since investigating the current vSphere CI issues I have not seen these errors.  Certainly we should keep this BZ - just an update.

Comment 5 Scott Dodson 2019-09-30 17:13:37 UTC
How often is this occurring?

Comment 6 sheng.lao 2019-10-06 06:58:47 UTC
It occurs hardly rarely during my duty of monitoring build-cop jobs

Comment 7 Joseph Callen 2019-10-09 18:04:14 UTC
I haven't see this error for quite a while.  Going to mark as notabug and will reopen if I see this again.

Comment 8 Jonathan Lebon 2019-12-04 15:24:42 UTC
Just saw this again now:
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-vsphere-upi-serial-4.3/438

```
Error: Error applying plan:

12 error(s) occurred:

* module.dns.aws_route53_record.control_plane_nodes[2]: 1 error(s) occurred:

* aws_route53_record.control_plane_nodes.2: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: 8b376c38-db2b-40b7-9be0-747c3ebcd21c
* module.dns.aws_route53_record.control_plane_nodes[0]: 1 error(s) occurred:

* aws_route53_record.control_plane_nodes.0: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: e345d3ba-0abb-4407-a799-c87e5dda4db4
* module.dns.aws_route53_record.etcd_a_nodes[1]: 1 error(s) occurred:

* aws_route53_record.etcd_a_nodes.1: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: 06989f72-5581-4d85-9c15-a6c3eae8018f
* module.dns.aws_route53_record.api-internal: 1 error(s) occurred:

* aws_route53_record.api-internal: [ERR]: Error building changeset: InvalidChangeBatch: [Invalid Resource Record: FATAL problem: ARRDATAIllegalIPv4Address (Value is not a valid IPv4 address) encountered with '']
	status code: 400, request id: d48a38a0-ee51-4ca0-89e6-c42d9a0407d1
* module.dns.aws_route53_record.etcd_a_nodes[0]: 1 error(s) occurred:

...
```

Tentatively re-opening.

Comment 9 Joseph Callen 2019-12-04 15:52:28 UTC
This always happens if a job runs at 7:xx am est.  I have no explanation for it.

Comment 10 Abhinav Dahiya 2020-01-30 21:34:28 UTC
https://ci-search-ci-search-next.svc.ci.openshift.org/?search=ARRDATAIllegalIPv4Address&maxAge=336h&context=2&type=all

there doesn't seem to be any failures in last 2 weeks so closing..