Description of problem (please be detailed as possible and provide log snippets): Tier1 test "tests/e2e/workloads/app/amq/test_rgw_kafka_notifications.py::TestRGWAndKafkaNotifications::test_rgw_kafka_notifications" fails with below error. > raise Exception( "Error: Messages are not recieved from Kafka side." "RGW bucket notification is not working as expected." ) E Exception: Error: Messages are not recieved from Kafka side.RGW bucket notification is not working as expected. Version of all relevant components (if applicable): OCP 4.13.0-rc.6 ODF 4.13.0-203 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? yes Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Install OCP and ODF 4.13 2. Execute tier1 test "tests/e2e/workloads/app/amq/test_rgw_kafka_notifications.py::TestRGWAndKafkaNotifications::test_rgw_kafka_notifications" 3. Actual results: Expected results: Additional info: Logs available in google drive : https://drive.google.com/file/d/180nzo7xi8cWU_wy3aVb1MteOPYI1AiNp/view?usp=sharing
Jiffin, PTAL
Reran the test again after setting debug level as mentioned. sh-5.1$ ceph config set client.rgw.ocs.storagecluster.cephobjectstore.a debug_rgw 20 sh-5.1$ ceph config get client.rgw.ocs.storagecluster.cephobjectstore.a debug_rgw 20/20 sh-5.1$ please find the logs here : https://drive.google.com/file/d/1em1jGjkYh3Gax6-JMP_4dGpMwbcNdu_n/view?usp=sharing
Marking as failed QA since "tests/e2e/workloads/app/amq/test_rgw_kafka_notifications.py::TestRGWAndKafkaNotifications::test_rgw_kafka_notifications" is still failing in our runs. Versions tested on: OCP: 4.14.0-0.nightly-2023-09-15-055234 ODF: 4.14.0-134 ceph version 17.2.6-138.el9cp (b488c8dad42b2ecffcd96f3d76eeeecce48b8590) quincy (stable) Logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/mashetty-svm15/mashetty-svm15_20230915T122502/logs/ocs-ci-logs-1695113189/by_outcome/failed/tests/e2e/workloads/app/amq/test_rgw_kafka_notifications.py/ Versions tested on: OCP: 4.14.0-0.nightly-2023-08-11-055332 ODF: 4.14.0-115 Logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-001vii1cs30-t1/j-001vii1cs30-t1_20230824T043853/logs/ocs-ci-logs-1692855470/by_outcome/failed/tests/e2e/workloads/app/amq/test_rgw_kafka_notifications.py/
Moving the non-blocker out of 4.14
(In reply to Mudit Agarwal from comment #22) > Moving the non-blocker out of 4.14 The test that found this failure was added based on a customer BZ(https://bugzilla.redhat.com/show_bug.cgi?id=1937100) earlier where notifications were not working. Requesting this bz to be fixed in 4.14 as it breaks the earlier working functionality.
(In reply to Harish NV Rao from comment #24) > (In reply to Mudit Agarwal from comment #22) > > Moving the non-blocker out of 4.14 > > The test that found this failure was added based on a customer > BZ(https://bugzilla.redhat.com/show_bug.cgi?id=1937100) earlier where > notifications were not working. > Requesting this bz to be fixed in 4.14 as it breaks the earlier working > functionality. requesting this to be fixed in 4.14.
Ceph fix is pushed to 7.0, I don't see how we can fix this in 4.14 Keeping the needinfo on me to create a ceph tracker
Hi Matt, >> Does this issue only affect s390x? Harish, can you please reply to this question >> Also, you state that the fix is "pushed to 7.0"--but it's actually fixed on current rhcs-6.1. My bad, I read the comments wrong, thanks for checking This is actually a failed qa so we should debug https://bugzilla.redhat.com/show_bug.cgi?id=2209616#c21 Can someone please take a look? Thanks Mudit I am still keeping it at 4.15 given that the fix will be in Ceph and we don't have a release vehicle.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:1383