Bug 2065265

Summary: hw event proxy is not binding on ipv6 local address
Product: OpenShift Container Platform Reporter: Jack Ding <jacding>
Component: Cloud Native EventsAssignee: Jack Ding <jacding>
Cloud Native Events sub component: Hardware Event Proxy QA Contact: nwaizer
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: aos-bugs, aputtur, grajaiya, ijolliff, nparekh, nwaizer
Version: 4.10Keywords: Reopened
Target Milestone: ---   
Target Release: 4.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2062849 Environment:
Last Closed: 2022-04-08 05:04:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2062849    
Bug Blocks:    

Description Jack Ding 2022-03-17 14:36:04 UTC 
+++ This bug was initially created as a clone of Bug #2062849 +++

Description of problem:
Hw-event-proxy is unreachable in a single stack ipv6 environment 

Version-Release number of selected component (if applicable):
hw-event-proxy 4.10

How reproducible:


Steps to Reproduce:
1. Deploy according to https://github.com/jzding/notes/tree/main/hw-event-proxy-operator
2. get app url using : oc get route
|-> hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com

3. from a host in the ipv6 issue this request:
curl -k -X POST -d '' https://hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com/webook




Actual results:
503 error

Expected results:
20x OK

Additional info:
When connecting to the app and issuing curl [::1]:9087 I get an error response.
curl 127.0.0.1:9087 gives ok

And from inside the cluster:
[kni ~]$ oc -n hw-event-proxy-operator-system exec -it hw-event-proxy-9bdb586d8-bxbqm -- curl http://127.0.0.1:9087 -I
Defaulted container "hw-event-proxy" out of: hw-event-proxy, cloud-event-proxy, kube-rbac-proxy
HTTP/1.1 404 Not Found
Server: fasthttp
Date: Thu, 10 Mar 2022 16:56:15 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 19

[kni ~]$ oc -n hw-event-proxy-operator-system exec -it hw-event-proxy-9bdb586d8-bxbqm -- curl http://[::1]:9087 -I
Defaulted container "hw-event-proxy" out of: hw-event-proxy, cloud-event-proxy, kube-rbac-proxy
curl: (7) Failed to connect to ::1 port 9087: Connection refused
command terminated with exit code 7

--- Additional comment from  on 2022-03-10 17:32:26 UTC ---

yes, only ipv4 is avail now

--- Additional comment from Jack Ding on 2022-03-11 14:04:40 UTC ---

I confirm this is an issue with included fasthttp library in hw-event-proxy.
fasthttp server only listens to tcp4 sockets. 
https://github.com/valyala/fasthttp/blob/master/server.go#L1592

Will work on a fix ASAP.

--- Additional comment from Jack Ding on 2022-03-11 18:06:03 UTC ---

Fix available to test:

PROXY_IMG=quay.io/jacding/hw-event-proxy:latest
CONSUMER_IMG=quay.io/redhat-cne/cloud-event-consumer:release-4.10

--- Additional comment from  on 2022-03-14 17:22:08 UTC ---

After deploying new code I see this on the app container:
time="2022-03-14T16:51:32Z" level=info msg="checking for rest service health\n"
time="2022-03-14T16:51:32Z" level=info msg="health check http://localhost:9085/api/cloudNotifications/v1/health "
time="2022-03-14T16:51:33Z" level=info msg="rest service returned healthy status"
time="2022-03-14T16:51:35Z" level=info msg="Created publisher {a789d06c-0ea7-41a6-afc4-b6fe8197d29c http://localhost:9085/api/cloudNotifications/v1/dummy http://localhost:9085/api/cloudNotifications/v1/publishers/a789d06c-0ea7-41a6-afc4-b6fe8197d29c /cluster/node/sno.kni-qe-2.lab.eng.rdu2.redhat.com/redfish/event}"
time="2022-03-14T16:51:35Z" level=info msg="waiting for events"
2022-03-14 16:51:47,913 - DEBUG - Starting new HTTPS connection (1): 2620:52:0:11d:b67a:f1ff:feae:98d9:443
2022-03-14 16:51:48,730 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/ HTTP/1.1" 200 None
2022-03-14 16:51:48,731 - INFO - Redfish version: 1.6.0
2022-03-14 16:51:48,731 - INFO - Preloading Redfish Registries...
2022-03-14 16:51:49,114 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:49,645 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/ HTTP/1.1" 200 None
2022-03-14 16:51:49,647 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:49,994 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/Base HTTP/1.1" 200 None
2022-03-14 16:51:50,025 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:50,525 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeCommon HTTP/1.1" 200 None
2022-03-14 16:51:50,549 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:50,902 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeDcpmmDiags HTTP/1.1" 200 None
2022-03-14 16:51:50,933 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:51,303 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLO HTTP/1.1" 200 None
2022-03-14 16:51:51,333 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:51,741 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLOeRS HTTP/1.1" 200 None
2022-03-14 16:51:51,770 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:52,127 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLOEvents HTTP/1.1" 200 None
2022-03-14 16:51:52,153 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:52,544 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/TaskEvent HTTP/1.1" 200 None
2022-03-14 16:51:52,573 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:53,001 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/%23SUTReg.v2_8_0.SUTReg HTTP/1.1" 200 None
2022-03-14 16:51:53,028 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:53,402 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/BiosAttributeRegistryH08.v1_1_88 HTTP/1.1" 200 None
2022-03-14 16:51:53,423 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:53,806 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeBiosMessageRegistry.v1_0_0 HTTP/1.1" 200 None
2022-03-14 16:51:53,830 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:54,187 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/Base.json HTTP/1.1" 200 None
2022-03-14 16:51:54,217 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:54,570 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/Base.json HTTP/1.1" 200 None
2022-03-14 16:51:54,614 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:55,043 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeCommon.json HTTP/1.1" 200 None
2022-03-14 16:51:55,079 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:55,459 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeCommon.json HTTP/1.1" 200 None
2022-03-14 16:51:55,511 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:55,861 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeDcpmmDiags.json HTTP/1.1" 200 None
2022-03-14 16:51:55,900 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:56,267 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeDcpmmDiags.json HTTP/1.1" 200 None
2022-03-14 16:51:56,295 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:56,663 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLO.json HTTP/1.1" 200 None
2022-03-14 16:51:56,693 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:57,061 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLO.json HTTP/1.1" 200 None
2022-03-14 16:51:57,116 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:57,645 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOeRS.json HTTP/1.1" 200 None
2022-03-14 16:51:57,712 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:58,108 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOeRS.json HTTP/1.1" 200 None
2022-03-14 16:51:58,113 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:58,582 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOEvents.json HTTP/1.1" 200 None
2022-03-14 16:51:58,613 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:59,088 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOEvents.json HTTP/1.1" 200 None
2022-03-14 16:51:59,119 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:51:59,883 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/TaskEvent.json HTTP/1.1" 200 None
2022-03-14 16:51:59,885 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:00,318 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/TaskEvent.json HTTP/1.1" 200 None
2022-03-14 16:52:00,320 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:00,735 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/sutreg.v2_8_0.sutreg HTTP/1.1" 200 None
2022-03-14 16:52:00,738 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:01,122 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/sutreg.v2_8_0.sutreg HTTP/1.1" 200 None
2022-03-14 16:52:01,127 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:01,632 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None
2022-03-14 16:52:01,717 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:02,416 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None
2022-03-14 16:52:02,516 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:03,143 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None
2022-03-14 16:52:03,232 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:04,670 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/hpebiosmessageregistry.v1_0_0 HTTP/1.1" 200 None
2022-03-14 16:52:04,672 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9
2022-03-14 16:52:05,168 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/hpebiosmessageregistry.v1_0_0 HTTP/1.1" 200 None
2022-03-14 16:52:05,169 - INFO - Preloading Redfish Registries DONE
2022-03-14 16:52:05,172 - INFO - server ready on port '9097'



Yet when submitting test event, no event appears in the app's log

--- Additional comment from  on 2022-03-14 17:33:05 UTC ---

when issuing this from a host in the ipv6 network 
$ curl -v -k -X POST -d '' https://hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com/webhook

I get 503 error as before

--- Additional comment from  on 2022-03-14 17:39:25 UTC ---

when ssh to the app container I see it still does not answer local ipv6. It does answer ipv4:
Unsupported path %s(app-root) bash-4.4$ curl -v http://127.0.0.1:9087/webhook
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9087 (#0)
> GET /webhook HTTP/1.1
> Host: 127.0.0.1:9087
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: fasthttp
< Date: Mon, 14 Mar 2022 17:35:42 GMT
< Content-Length: 0
< 
* Connection #0 to host 127.0.0.1 left intact


(app-root) bash-4.4$ curl -v http://[::1]:9087/webhook
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 9087 failed: Connection refused
* Failed to connect to ::1 port 9087: Connection refused
* Closing connection 0
curl: (7) Failed to connect to ::1 port 9087: Connection refused

--- Additional comment from Jack Ding on 2022-03-14 18:35:48 UTC ---

The image you used does not contain the fix since the fix was merged in mainstream, not in 4.10. I need it QE verified before cherry-picking it to 4.10.

     "image": "registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/openshift/origin-baremetal-hardware-event-proxy:4.10",

Please use image quay.io/openshift/origin-baremetal-hardware-event-proxy:latest

--- Additional comment from Jack Ding on 2022-03-14 22:50:17 UTC ---

You should be able to see this if the image is correct:

$ oc exec -it hw-event-proxy-5c4cf9b7c8-p25x9 -c hw-event-proxy -- curl -X POST [::1]:9087/webhook -I
HTTP/1.1 200 OK
Date: Mon, 14 Mar 2022 21:45:04 GMT
Content-Length: 0

--- Additional comment from  on 2022-03-16 07:34:58 UTC ---

New image installed on the disconnected IPv6 node, and indeed the command above works.
I also see the events, that I generate using SubmitTestEvent arriving at the app and sent to the message queue.
How ever the consumer installed according to https://github.com/jzding/notes/blob/main/hw-event-proxy-operator/consumer.yaml
Does not receive the events.
I notice the consumer is using IPV4 to connect to the broker, and asked Aneesh Puttur if I can use IPv6 for the AMQ.
He claims that IPv6 was never tested.
Will the user use IPV4 in the cluster for the message queue , and this is what I should test?
Or, should the AMQ be configured to use IPv6 and I should test that?

--- Additional comment from Jack Ding on 2022-03-16 13:09:50 UTC ---

(In reply to nwaizer from comment #9)
> New image installed on the disconnected IPv6 node, and indeed the command
> above works.
> I also see the events, that I generate using SubmitTestEvent arriving at the
> app and sent to the message queue.
> How ever the consumer installed according to
> https://github.com/jzding/notes/blob/main/hw-event-proxy-operator/consumer.
> yaml
> Does not receive the events.
> I notice the consumer is using IPV4 to connect to the broker, and asked
> Aneesh Puttur if I can use IPv6 for the AMQ.
> He claims that IPv6 was never tested.
> Will the user use IPV4 in the cluster for the message queue , and this is
> what I should test?
> Or, should the AMQ be configured to use IPv6 and I should test that?
Could you please fix the AMQ image and see if it works. I hope ipv4 works with localhosts in this environment.

--- Additional comment from Jack Ding on 2022-03-16 20:27:20 UTC ---

After fixing AMQ image and changing listener host from `0.0.0.0` to `::` in amq router config, events are received from consumer.

--- a/manifests/amq-installer/config.yaml
+++ b/manifests/amq-installer/config.yaml
@@ -14,7 +14,7 @@ data:
     }
 
     listener {
-        host: 0.0.0.0
+        host: ::
         port: 5672
         role: normal
     }

--- Additional comment from  on 2022-03-17 07:13:02 UTC ---

tested fine on HPE
Comment 3 nwaizer 2022-03-30 17:55:53 UTC 
verified on the same setup the issue was reported.
Comment 6 errata-xmlrpc 2022-04-08 05:04:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.10.8 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1162