Bug 2281421

+++ This bug was initially created as a clone of Bug #2166576 +++

Description of problem:
Add support to generate event notification for object transition via lifecycle rule. 
ref BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1857813

Version-Release number of selected component (if applicable):
ceph version 17.2.5-63.el9cp 

How reproducible:
always

Steps to Reproduce:
1. ceph cluster deployed on rhcs6.0

2. create a storage class 'hot.test' as mentioned in the downstream guide - https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/object_gateway_guide/administration#transitioning-an-object-from-one-storage-class-to-another_rgw 

3. create a bucket and set lifecycle rule to transit object to storage class 'hot.test'
lc rule:
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>rule1</ID>
		<Filter>
			<Prefix>zg1</Prefix>
		</Filter>
		<Status>Enabled</Status>
		<Transition>
			<Days>2</Days>
			<StorageClass>hot.test</StorageClass>
		</Transition>
	</Rule>
</LifecycleConfiguration>
[root@extensa001 ~]# 

conf changed for testing - rgw_lc_debug_interval=300

4. create 2 objects with prefix 'zg1'
[root@extensa001 ~]# s3cmd ls s3://testbucket-hot1
2023-01-31 23:11          349  s3://testbucket-hot1/zg1-1
2023-01-31 23:11          349  s3://testbucket-hot1/zg1-2


5. add notification including the Events for "s3:ObjectLifecycle:*" 

6. upon lc rule trigger, the storage class of the object, reflects 'hot.test' 
snippet:
before lc: 
    {
        "name": "zg1-1",
            "storage_class": "STANDARD",
...
after lc:
    {
        "name": "zg1-1",
            "storage_class": "hot.test",
... 

7. However, there is no notification seen on transition.


Actual results:
no event notification seen on object transition

Expected results:
event is generated for object transition

Additional info:

console logs and rgw logs are available at http://magna002.ceph.redhat.com/ceph-qe-logs/madhavi/bz2166576/

--- Additional comment from Ken Dreyer (Red Hat) on 2023-04-03 19:40:50 UTC ---

What commits fix this BZ?

--- Additional comment from Matt Benjamin (redhat) on 2023-04-05 16:32:52 UTC ---

This has not yet merged upstream and is not urgent, moving to 7.0.

Matt

--- Additional comment from errata-xmlrpc on 2023-11-16 09:23:13 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2023:118213-01
https://errata.devel.redhat.com/advisory/118213

--- Additional comment from errata-xmlrpc on 2023-11-16 09:23:20 UTC ---

This bug has been added to advisory RHBA-2023:118213 by Thomas Serlin (tserlin)

--- Additional comment from Madhavi Kasturi on 2023-11-23 11:41:37 UTC ---

on 7.0 build ( ceph version 18.2.0-127.el9cp) with the steps mentioned in the description, upon lifecycle transition to storage class 'cold' , there is no event notification seen on kafka. 

1. steps followed as mentioned in the description with storage class 'cold'
2. upon trigger of lc, the object transitioned to cold storage class
snippet:
before lc: 
    {
        "name": "obj1",
            "storage_class": "STANDARD",
...
after lc:
    {
        "name": "obj1",
            "storage_class": "cold",

3. No event notification seen on kafka.

PFA, rgw logs , lc conf & notification.py at node 
details:
10.0.205.157 ; root/passwd
path: /root/BZ2166576

Moving this BZ to assigned.

--- Additional comment from Scott Ostapovicz on 2023-11-24 09:11:29 UTC ---

Based on the level of urgency as described in comment 2, and on the fact that the final RC build for the 7.0 release is on Monday (11/28), this is being retargeted to 7.0 z1.  If you feel I was wrong in retargeting this, please advise.

--- Additional comment from errata-xmlrpc on 2023-11-29 16:48:30 UTC ---

This bug has been dropped from advisory RHBA-2023:118213 by Vivek Das (vdas)

--- Additional comment from Matt Benjamin (redhat) on 2023-11-29 17:17:38 UTC ---

(In reply to Scott Ostapovicz from comment #6)
> Based on the level of urgency as described in comment 2, and on the fact
> that the final RC build for the 7.0 release is on Monday (11/28), this is
> being retargeted to 7.0 z1.  If you feel I was wrong in retargeting this,
> please advise.

I do not, but would appreciate being tagged in slack.

thanks,

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-02-15 19:11:01 UTC ---

moving to 7.0z2, as backport is late for 7.0z1 release

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-04-04 17:02:13 UTC ---

fixes are in place, ready for test

Matt

--- Additional comment from  on 2024-04-04 17:10:22 UTC ---

Using latest Fixed In Version.

Thomas

--- Additional comment from errata-xmlrpc on 2024-04-04 17:11:00 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:129008-01
https://errata.engineering.redhat.com/advisory/129008

--- Additional comment from errata-xmlrpc on 2024-04-04 17:11:10 UTC ---

This bug has been added to advisory RHBA-2024:129008 by Thomas Serlin (tserlin)

--- Additional comment from Madhavi Kasturi on 2024-04-14 16:06:41 UTC ---

On ceph version 18.2.0-190.el9cp, with steps mentioned in description , notification is not seen upon transition of the object.

Steps followed and rgw logs are at http://magna002.ceph.redhat.com/ceph-qe-logs/madhavi/BZ_verify/BZ2166576/ 

Moving this BZ to assigned.

--- Additional comment from Matt Benjamin (redhat) on 2024-04-17 12:52:38 UTC ---

For schedule reasons, moving to 7.1 as MODIFIED for restest

Matt

--- Additional comment from  on 2024-04-17 14:39:05 UTC ---

Dropping from 7.0 z2 errata advisory and re-attaching to the 7.1 errata advisory.

Using latest "Fixed in Version" for 7.1.

Thomas

--- Additional comment from errata-xmlrpc on 2024-04-17 14:39:24 UTC ---

This bug has been dropped from advisory RHBA-2024:129008 by Thomas Serlin (tserlin)

--- Additional comment from errata-xmlrpc on 2024-04-17 15:18:23 UTC ---

This bug has been added to advisory RHBA-2024:126567 by Thomas Serlin (tserlin)

--- Additional comment from errata-xmlrpc on 2024-04-17 15:18:37 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:126567-01
https://errata.engineering.redhat.com/advisory/126567

--- Additional comment from Madhavi Kasturi on 2024-05-02 16:57:19 UTC ---

On ceph version 18.2.1-155.el9cp, with steps mentioned in description , notification is not seen upon transition of the object.

RGW logs are added here http://magna002.ceph.redhat.com/ceph-qe-logs/madhavi/BZ_verify/BZ2166576/7.1/

Moving the Bz to assigned.

--- Additional comment from Matt Benjamin (redhat) on 2024-05-09 12:07:36 UTC ---

hi madhavi,

the workflow in https://bugzilla.redhat.com/show_bug.cgi?id=2279339 shows that notifications were being fired for transition, but the rca of that bz entailed invalid sharing of part of the object metadata
with notification, so I think this bz needs to be retested with those fixes (now present on 7.1).

thanks,

Matt

--- Additional comment from Madhavi Kasturi on 2024-05-17 08:45:09 UTC ---

Hi Matt,

Ack on retest based on fix provided for BZ https://bugzilla.redhat.com/show_bug.cgi?id=2279339.

On ceph version 18.2.1-179.el9cp performed retest with steps mentioned description, however notifications on transition are not seen. 

PFA, the rgw.logs at http://magna002.ceph.redhat.com/ceph-qe-logs/madhavi/BZ_verify/BZ2166576/build_179/ceph-client.rgw.rgw.1.ceph-rgw-7-1-x5xrri-node5.iynfks.log 


Moving this BZ to assigned.

--- Additional comment from Matt Benjamin (redhat) on 2024-05-18 18:22:01 UTC ---

Hi Madhavi,

Yes, notify on transition *to a non-cloud target* was not being sent (but from other results, I think we know that notifications are being sent in that case).

Great work catching this!

Matt