+++ This bug was initially created as a clone of Bug #2062849 +++ Description of problem: Hw-event-proxy is unreachable in a single stack ipv6 environment Version-Release number of selected component (if applicable): hw-event-proxy 4.10 How reproducible: Steps to Reproduce: 1. Deploy according to https://github.com/jzding/notes/tree/main/hw-event-proxy-operator 2. get app url using : oc get route |-> hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com 3. from a host in the ipv6 issue this request: curl -k -X POST -d '' https://hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com/webook Actual results: 503 error Expected results: 20x OK Additional info: When connecting to the app and issuing curl [::1]:9087 I get an error response. curl 127.0.0.1:9087 gives ok And from inside the cluster: [kni ~]$ oc -n hw-event-proxy-operator-system exec -it hw-event-proxy-9bdb586d8-bxbqm -- curl http://127.0.0.1:9087 -I Defaulted container "hw-event-proxy" out of: hw-event-proxy, cloud-event-proxy, kube-rbac-proxy HTTP/1.1 404 Not Found Server: fasthttp Date: Thu, 10 Mar 2022 16:56:15 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 19 [kni ~]$ oc -n hw-event-proxy-operator-system exec -it hw-event-proxy-9bdb586d8-bxbqm -- curl http://[::1]:9087 -I Defaulted container "hw-event-proxy" out of: hw-event-proxy, cloud-event-proxy, kube-rbac-proxy curl: (7) Failed to connect to ::1 port 9087: Connection refused command terminated with exit code 7 --- Additional comment from on 2022-03-10 17:32:26 UTC --- yes, only ipv4 is avail now --- Additional comment from Jack Ding on 2022-03-11 14:04:40 UTC --- I confirm this is an issue with included fasthttp library in hw-event-proxy. fasthttp server only listens to tcp4 sockets. https://github.com/valyala/fasthttp/blob/master/server.go#L1592 Will work on a fix ASAP. --- Additional comment from Jack Ding on 2022-03-11 18:06:03 UTC --- Fix available to test: PROXY_IMG=quay.io/jacding/hw-event-proxy:latest CONSUMER_IMG=quay.io/redhat-cne/cloud-event-consumer:release-4.10 --- Additional comment from on 2022-03-14 17:22:08 UTC --- After deploying new code I see this on the app container: time="2022-03-14T16:51:32Z" level=info msg="checking for rest service health\n" time="2022-03-14T16:51:32Z" level=info msg="health check http://localhost:9085/api/cloudNotifications/v1/health " time="2022-03-14T16:51:33Z" level=info msg="rest service returned healthy status" time="2022-03-14T16:51:35Z" level=info msg="Created publisher {a789d06c-0ea7-41a6-afc4-b6fe8197d29c http://localhost:9085/api/cloudNotifications/v1/dummy http://localhost:9085/api/cloudNotifications/v1/publishers/a789d06c-0ea7-41a6-afc4-b6fe8197d29c /cluster/node/sno.kni-qe-2.lab.eng.rdu2.redhat.com/redfish/event}" time="2022-03-14T16:51:35Z" level=info msg="waiting for events" 2022-03-14 16:51:47,913 - DEBUG - Starting new HTTPS connection (1): 2620:52:0:11d:b67a:f1ff:feae:98d9:443 2022-03-14 16:51:48,730 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/ HTTP/1.1" 200 None 2022-03-14 16:51:48,731 - INFO - Redfish version: 1.6.0 2022-03-14 16:51:48,731 - INFO - Preloading Redfish Registries... 2022-03-14 16:51:49,114 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:49,645 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/ HTTP/1.1" 200 None 2022-03-14 16:51:49,647 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:49,994 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/Base HTTP/1.1" 200 None 2022-03-14 16:51:50,025 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:50,525 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeCommon HTTP/1.1" 200 None 2022-03-14 16:51:50,549 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:50,902 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeDcpmmDiags HTTP/1.1" 200 None 2022-03-14 16:51:50,933 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:51,303 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLO HTTP/1.1" 200 None 2022-03-14 16:51:51,333 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:51,741 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLOeRS HTTP/1.1" 200 None 2022-03-14 16:51:51,770 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:52,127 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/iLOEvents HTTP/1.1" 200 None 2022-03-14 16:51:52,153 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:52,544 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/TaskEvent HTTP/1.1" 200 None 2022-03-14 16:51:52,573 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:53,001 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/%23SUTReg.v2_8_0.SUTReg HTTP/1.1" 200 None 2022-03-14 16:51:53,028 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:53,402 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/BiosAttributeRegistryH08.v1_1_88 HTTP/1.1" 200 None 2022-03-14 16:51:53,423 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:53,806 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/Registries/HpeBiosMessageRegistry.v1_0_0 HTTP/1.1" 200 None 2022-03-14 16:51:53,830 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:54,187 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/Base.json HTTP/1.1" 200 None 2022-03-14 16:51:54,217 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:54,570 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/Base.json HTTP/1.1" 200 None 2022-03-14 16:51:54,614 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:55,043 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeCommon.json HTTP/1.1" 200 None 2022-03-14 16:51:55,079 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:55,459 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeCommon.json HTTP/1.1" 200 None 2022-03-14 16:51:55,511 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:55,861 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeDcpmmDiags.json HTTP/1.1" 200 None 2022-03-14 16:51:55,900 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:56,267 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/HpeDcpmmDiags.json HTTP/1.1" 200 None 2022-03-14 16:51:56,295 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:56,663 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLO.json HTTP/1.1" 200 None 2022-03-14 16:51:56,693 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:57,061 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLO.json HTTP/1.1" 200 None 2022-03-14 16:51:57,116 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:57,645 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOeRS.json HTTP/1.1" 200 None 2022-03-14 16:51:57,712 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:58,108 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOeRS.json HTTP/1.1" 200 None 2022-03-14 16:51:58,113 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:58,582 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOEvents.json HTTP/1.1" 200 None 2022-03-14 16:51:58,613 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:59,088 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/iLOEvents.json HTTP/1.1" 200 None 2022-03-14 16:51:59,119 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:51:59,883 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/TaskEvent.json HTTP/1.1" 200 None 2022-03-14 16:51:59,885 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:00,318 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/RegistryStore/registries/en/TaskEvent.json HTTP/1.1" 200 None 2022-03-14 16:52:00,320 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:00,735 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/sutreg.v2_8_0.sutreg HTTP/1.1" 200 None 2022-03-14 16:52:00,738 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:01,122 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/sutreg.v2_8_0.sutreg HTTP/1.1" 200 None 2022-03-14 16:52:01,127 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:01,632 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None 2022-03-14 16:52:01,717 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:02,416 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None 2022-03-14 16:52:02,516 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:03,143 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/biosattributeregistryh08.v1_1_88 HTTP/1.1" 200 None 2022-03-14 16:52:03,232 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:04,670 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/hpebiosmessageregistry.v1_0_0 HTTP/1.1" 200 None 2022-03-14 16:52:04,672 - DEBUG - Resetting dropped connection: 2620:52:0:11d:b67a:f1ff:feae:98d9 2022-03-14 16:52:05,168 - DEBUG - https://2620:52:0:11d:b67a:f1ff:feae:98d9:443 "GET /redfish/v1/registrystore/registries/en/hpebiosmessageregistry.v1_0_0 HTTP/1.1" 200 None 2022-03-14 16:52:05,169 - INFO - Preloading Redfish Registries DONE 2022-03-14 16:52:05,172 - INFO - server ready on port '9097' Yet when submitting test event, no event appears in the app's log --- Additional comment from on 2022-03-14 17:33:05 UTC --- when issuing this from a host in the ipv6 network $ curl -v -k -X POST -d '' https://hw-event-proxy-hw-event-proxy-operator-system.apps.kni-qe-2.lab.eng.rdu2.redhat.com/webhook I get 503 error as before --- Additional comment from on 2022-03-14 17:39:25 UTC --- when ssh to the app container I see it still does not answer local ipv6. It does answer ipv4: Unsupported path %s(app-root) bash-4.4$ curl -v http://127.0.0.1:9087/webhook * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 9087 (#0) > GET /webhook HTTP/1.1 > Host: 127.0.0.1:9087 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Server: fasthttp < Date: Mon, 14 Mar 2022 17:35:42 GMT < Content-Length: 0 < * Connection #0 to host 127.0.0.1 left intact (app-root) bash-4.4$ curl -v http://[::1]:9087/webhook * Trying ::1... * TCP_NODELAY set * connect to ::1 port 9087 failed: Connection refused * Failed to connect to ::1 port 9087: Connection refused * Closing connection 0 curl: (7) Failed to connect to ::1 port 9087: Connection refused --- Additional comment from Jack Ding on 2022-03-14 18:35:48 UTC --- The image you used does not contain the fix since the fix was merged in mainstream, not in 4.10. I need it QE verified before cherry-picking it to 4.10. "image": "registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/openshift/origin-baremetal-hardware-event-proxy:4.10", Please use image quay.io/openshift/origin-baremetal-hardware-event-proxy:latest --- Additional comment from Jack Ding on 2022-03-14 22:50:17 UTC --- You should be able to see this if the image is correct: $ oc exec -it hw-event-proxy-5c4cf9b7c8-p25x9 -c hw-event-proxy -- curl -X POST [::1]:9087/webhook -I HTTP/1.1 200 OK Date: Mon, 14 Mar 2022 21:45:04 GMT Content-Length: 0 --- Additional comment from on 2022-03-16 07:34:58 UTC --- New image installed on the disconnected IPv6 node, and indeed the command above works. I also see the events, that I generate using SubmitTestEvent arriving at the app and sent to the message queue. How ever the consumer installed according to https://github.com/jzding/notes/blob/main/hw-event-proxy-operator/consumer.yaml Does not receive the events. I notice the consumer is using IPV4 to connect to the broker, and asked Aneesh Puttur if I can use IPv6 for the AMQ. He claims that IPv6 was never tested. Will the user use IPV4 in the cluster for the message queue , and this is what I should test? Or, should the AMQ be configured to use IPv6 and I should test that? --- Additional comment from Jack Ding on 2022-03-16 13:09:50 UTC --- (In reply to nwaizer from comment #9) > New image installed on the disconnected IPv6 node, and indeed the command > above works. > I also see the events, that I generate using SubmitTestEvent arriving at the > app and sent to the message queue. > How ever the consumer installed according to > https://github.com/jzding/notes/blob/main/hw-event-proxy-operator/consumer. > yaml > Does not receive the events. > I notice the consumer is using IPV4 to connect to the broker, and asked > Aneesh Puttur if I can use IPv6 for the AMQ. > He claims that IPv6 was never tested. > Will the user use IPV4 in the cluster for the message queue , and this is > what I should test? > Or, should the AMQ be configured to use IPv6 and I should test that? Could you please fix the AMQ image and see if it works. I hope ipv4 works with localhosts in this environment. --- Additional comment from Jack Ding on 2022-03-16 20:27:20 UTC --- After fixing AMQ image and changing listener host from `0.0.0.0` to `::` in amq router config, events are received from consumer. --- a/manifests/amq-installer/config.yaml +++ b/manifests/amq-installer/config.yaml @@ -14,7 +14,7 @@ data: } listener { - host: 0.0.0.0 + host: :: port: 5672 role: normal } --- Additional comment from on 2022-03-17 07:13:02 UTC --- tested fine on HPE
verified on the same setup the issue was reported.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.10.8 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1162