Bug 2341711

Summary: rgw: bucket logging fixes and enhancements
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Yuval Lifshitz <ylifshit>
Component: RGWAssignee: Yuval Lifshitz <ylifshit>
Status: VERIFIED --- QA Contact: Hemanth Sai <hmaheswa>
Severity: medium Docs Contact: Rivka Pollack <rpollack>
Priority: unspecified    
Version: 8.1CC: ceph-eng-bugs, cephqe-warriors, hmaheswa, rpollack, tserlin
Target Milestone: ---   
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.2.1-3.el9cp Doc Type: Technology Preview
Doc Text:
.Bucket logging support for Ceph Object Gateway with bug fixes and enhancements Bucket logging was introduced in Red Hat Ceph Storage 8.0. Bucket logging provides a mechanism for logging all access to a bucket. The log data can be used to monitor bucket activity, detect unauthorized access, get insights into the bucket usage and use the logs as a journal for bucket changes. The log records are stored in objects in a separate bucket and can be analyzed later. Logging configuration is done at the bucket level and can be enabled or disabled at any time. The log bucket can accumulate logs from multiple buckets. The configured `prefix` may be used to distinguish between logs from different buckets. For performance reasons, even though the log records are written to persistent storage, the log object appears in the log bucket only after a configurable amount of time or when reaching the maximum object size of 128 MB. Adding a log object to the log bucket is done in such a way that if no more records are written to the object, it might remain outside of the log bucket even after the configured time has passed. There are two logging types: `standard` and `journal`. The default logging type is `standard`. When set to `standard` the log records are written to the log bucket after the bucket operation is completed. As a result the logging operation can fail with no indication to the client. When set to `journal` the records are written to the log bucket before the bucket operation is complete. As a result, the operation does not run if the logging action fails and an error is returned to the client. You can complete the following bucket logging actions: enable, disable, and get. Red Hat Ceph Storage 8.1 enhancements introduce several improvements to bucket logging, including support for source and destination buckets across different tenants, suffix/prefix-based key filtering, and standardized AWS operation names in log records. A new REST-based flush (POST) API has been added, along with the `bucket logging info admin` command for retrieving logging configurations. Fixes address concurrency issues causing multiple temporary objects, missing object size in certain cases, and retry attributes in race conditions. Additional safeguards now ensure that source and log buckets are distinct and that log buckets do not have encryption. Cleanup mechanisms have been improved to remove pending objects when source buckets are deleted, logging is disabled or reconfigured, or when target buckets are removed. Logging records now include missing fields related to authentication and transport layer information, ensuring more comprehensive logging capabilities.
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2351689    

Description Yuval Lifshitz 2025-01-23 08:53:20 UTC
Description of problem:
in 8.0 we provided partial bucket logging support (see: https://bugzilla.redhat.com/show_bug.cgi?id=2308169), it was mising the following fixes and enhancements:
* add REST (POST) based flush API
* doc and examples fixes
* object size fix
* fix concurrency issue causing multiple temporary objects
* add suffix/prefix based key filtering
* using standard (AWS) operation names in logs records
* support source and destination buckets on different tenants
* verify source and log bucket must be different (original bz: https://bugzilla.redhat.com/show_bug.cgi?id=2321568)
* log bucket must not have encryption
* retry attribuite set in case of race
* clean pending objects
  * when source bucket is deleted
  * when logging is disabled
  * when logging conf changes
  * when target bucket is deleted
* add "bucket logging info" admin command
  * returning logging conf for source bucket
  * list of source buckets for log bucket
* add missing fields in lof records regarding authentication and transport layer info

Comment 1 Storage PM bot 2025-01-23 08:53:29 UTC
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.