Description of problem:
Backport and introduce as tech preview the currently S3 notification functionality as tech preview.
This feature would be fantastic. We (the AI Center of Excellence) have been pushing Ceph as a critical piece of a reference architecture for Big Data, AI, and ML. We have a need to be able to receive notifications of new data as it lands in Ceph so that ETL jobs can grab that data and process it. This can be a big part of doing batch processing of streaming data as well, and would certainly work its way into our reference architecture. One such use case is when a user uploads a new training data set, being able to fire off the training job immediately instead of having to poll constantly for changing data. Kafka (AMQ Streams) is currently the message bus of choice for the reference architecture.
Currently AMQP0.9.1 (the version used by the rabbitmq message broker) is implemented in the pubsub feature.
There are two options for supporting Kafka:
(1) native, by embedding a C/C++ kafka client library into our code (e.g. librdkafka seems well maintained)
(2) by using a translation broker like ActiveMQ, which translate AMQP1.0 to Kafka. This will still require dev effort of adding native AMQP1.0 client (e.g. Qpid Proton which is maintained by Redhat)
Both options would be similar effort (since, sadly, AMQP1.0 would require a different client than AMQP0.9.1). If the end goal is Kafka, I would recommend (1).
We may add AMQP1.0 later to support ActiveMQ regardless of Kafka/AMQ-Streams.
Approach #1 sounds the most attractive to me. Kafka support would allow the triggering of Knative functions using KafkaSource:
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.