Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1945772

Summary: [RBD] ceph-immutable-object-cache status Active: failed (thawing)
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Harish Munjulur <hmunjulu>
Component: RBDAssignee: Ilya Dryomov <idryomov>
Status: CLOSED ERRATA QA Contact: Harish Munjulur <hmunjulu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0CC: ceph-eng-bugs, idryomov, tserlin
Target Milestone: ---   
Target Release: 5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rhel-8.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-30 08:29:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Harish Munjulur 2021-04-01 21:40:34 UTC
Description of problem: ceph-immutable-object-cache daemon status 
Active: failed (thawing) (Result: exit-code) since Wed 2021-03-31 18:46:08 UTC; 1 day 2h ago


Version-Release number of selected component (if applicable):


Steps to Reproduce:
1. Have the ceph-immutable-object-cache daemon active and running. 

Actual results:
[root@magna031 tmp]# systemctl status ceph-immutable-object-cache
● ceph-immutable-object-cache.service - Ceph immutable object cache daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-immutable-object-cache@.service; enabled; vendor preset: disabled)
   Active: failed (thawing) (Result: exit-code) since Wed 2021-03-31 18:46:08 UTC; 1 day 2h ago
 Main PID: 822964 (code=exited, status=1/FAILURE)



Expected results: Active: active (running)


Additional info:
[root@magna031 tmp]# journalctl -u ceph-immutable-object-cache.service
-- Logs begin at Mon 2021-03-29 16:26:42 UTC, end at Thu 2021-04-01 21:35:20 UTC. --
Mar 31 18:31:39 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:31:39 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:31:39 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:32:00 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:32:00 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:32:00 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:32:01 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:32:01 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:32:01 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:34:42 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:34:42 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:34:42 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:38:11 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:38:11 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:38:11 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:38:12 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:38:12 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:38:12 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:38:13 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:38:13 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:38:13 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Mar 31 18:46:08 magna031 systemd[1]: Stopping Ceph immutable object cache daemon...
Mar 31 18:46:08 magna031 ceph-immutable-object-cache[822964]: 2021-03-31T18:46:08.829+0000 7f737e38c700 -1 received  signal: Terminated from /usr/lib/systemd/systemd --switched-root --system --deserialize 17  (>
Mar 31 18:46:08 magna031 systemd[1]: ceph-immutable-object-cache.service: Main process exited, code=exited, status=1/FAILURE
Mar 31 18:46:08 magna031 systemd[1]: ceph-immutable-object-cache.service: Failed with result 'exit-code'.
Mar 31 18:46:08 magna031 systemd[1]: Stopped Ceph immutable object cache daemon.
Mar 31 18:46:38 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:17: Unknown lvalue 'ProtectClock' in section 'Service'
Mar 31 18:46:38 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:20: Unknown lvalue 'ProtectHostname' in section 'Service'
Mar 31 18:46:38 magna031 systemd[1]: /usr/lib/systemd/system/ceph-immutable-object-cache@.service:21: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Apr 01 21:35:13 magna031 systemd[1]: Started Ceph immutable object cache daemon.

Comment 1 Harish Munjulur 2021-04-01 21:43:07 UTC
Tried starting the daemon. 

[root@magna031 tmp]# systemctl start ceph-immutable-object-cache
[root@magna031 tmp]# systemctl status ceph-immutable-object-cache
● ceph-immutable-object-cache.service - Ceph immutable object cache daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-immutable-object-cache@.service; enabled; vendor preset: disabled)
   Active: active (running) (thawing) since Thu 2021-04-01 21:35:13 UTC; 3s ago
 Main PID: 2515163 (ceph-immutable-)
    Tasks: 15 (limit: 204080)
   Memory: 7.4M
   CGroup: /system.slice/system-ceph\x2dimmutable\x2dobject\x2dcache.slice/ceph-immutable-object-cache.service
           └─2515163 /usr/bin/ceph-immutable-object-cache -f --cluster ceph --id ceph-immutable-object-cache.readonlycache --setuser ceph --setgroup ceph

Apr 01 21:35:13 magna031 systemd[1]: Started Ceph immutable object cache daemon.


1. Is it ok for the daemon to stop by systemd? Is it expected?
2. Active: active (running) (thawing). What does thawing indicate?

Comment 3 Harish Munjulur 2021-04-19 09:00:51 UTC
1. Thanks I will update this BZ if it reproduces again 

2. Are you testing on RHEL 8.3?  What version of systemd is in use? 

Yes I am testing on Rhel 8.3. 

[root@magna031 ~]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"

[root@magna031 ~]# systemctl --version
systemd 239 (239-40.el8)

Even if I stop and start the ceph-immutable-object-cache daemon I see this  

Active: active (running) (thawing) since Fri 2021-04-09 20:01:31 UTC; 1 weeks 2 days ago

Comment 5 Harish Munjulur 2021-04-21 04:36:56 UTC
Thanks Ilya. I will close this after verifying the issue with the corresponding systemd version

Comment 7 Harish Munjulur 2021-05-24 09:58:02 UTC
Working on this. Looks like RHEL 8.3 doesn't have the systemd version required will test on RHEL 8.4 and update.

Comment 9 Harish Munjulur 2021-06-06 16:00:17 UTC
QA verified on RHEL 8.4

[root@ceph-hary-1622989934418-node1-mon-mgr-installer-node-exporter-a ceph]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"


[root@ceph-hary-1622989934418-node1-mon-mgr-installer-node-exporter-a ceph]# systemctl --version
systemd 239 (239-45.el8)



1. systemctl status ceph-immutable-object-cache

make sure the status is active

2. stop the service and start 

3. status should be active (thawing is not seen)


[root@ceph-hary-1622989934418-node1-mon-mgr-installer-node-exporter-a ceph]# systemctl status ceph-immutable-object-cache
● ceph-immutable-object-cache.service - Ceph immutable object cache daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-immutable-object-cache@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-06-06 11:56:00 EDT; 3s ago
 Main PID: 62802 (ceph-immutable-)
    Tasks: 15 (limit: 23484)
   Memory: 5.9M
   CGroup: /system.slice/system-ceph\x2dimmutable\x2dobject\x2dcache.slice/ceph-immutable-object-cache@ceph-immutable->
           └─62802 /usr/bin/ceph-immutable-object-cache -f --cluster ceph --id ceph-immutable-object-cache.readonlycac>

Comment 12 errata-xmlrpc 2021-08-30 08:29:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294