Bug 1671484

Summary: OSD Bluestore debug not reaching the docker logs.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: daniel parkes <dparkes>
Component: ContainerAssignee: Sébastien Han <shan>
Status: CLOSED DUPLICATE QA Contact: Vasishta <vashastr>
Severity: medium Docs Contact:
Priority: high    
Version: 3.2CC: anharris, ceph-eng-bugs, dparkes, dsavinea, evelu, gabrioux, mamccoma, vumrao
Target Milestone: rc   
Target Release: 4.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-30 18:36:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description daniel parkes 2019-01-31 16:59:10 UTC 
Description of problem:

On a containerised 3.2 deployment, when we enable for example bluestore debug, the actual bluestore logs are not reaching the host journal.

Using the default logging options in the conf file

# cat /etc/ceph/ceph.conf | grep log
log file = /dev/null
mon cluster log file = /dev/null

Enable Bluestore debug

[root@servera ~]#  ceph daemon osd.11 config get debug_bluestore
{
    "debug_bluestore": "5/5"
}

We have no logs in the host journal

[root@servera ~]# journalctl -b | grep bluestore.Mempool

If we configure the osd to dump it0s log to a file we get the bluestore logs like expected:
[root@servera ~]#  ceph daemon osd.11 config get debug_bluestore
{
    "debug_bluestore": "5/5"
}
[root@servera ~]# ceph daemon osd.11 config set log_file /tmp/osd11.log
{
    "success": "log_file = '/tmp/osd11.log' "
}
[root@servera ~]# ceph daemon osd.11 log reopen
{}
[root@servera ~]# docker exec -it ceph-osd-11 cat /tmp/osd11.log | tail -2
2019-01-31 11:56:16.012799 7f1f5a3bb700  5 bluestore.MempoolThread(0x55645df471d0) _trim_shards cache_size: 2845415832 kv_alloc: 1218696969 kv_used: 1837362 meta_alloc: 1084479241 meta_used: 71604 data_alloc: 542239620 data_used: 0
2019-01-31 11:56:16.866433 7f1f5a3bb700  5 bluestore.MempoolThread(0x55645df471d0) _tune_cache_size target: 4294967296 heap: 75497472 unmapped: 22568960 mapped: 52928512 old cache_size: 2845415832 new cache size: 2845415832





Version-Release number of selected component (if applicable):

3.2

How reproducible:

Install Containerised 3.2 deployment 
Enable Bluestore Debug

Steps to Reproduce:
1.Install Containerised 3.2 deployment
2.Enable Bluestore Debug
3.

Actual results:

No Bluestore debug logs are present

Expected results:

The OSD debug logs should be present on the host journal

Additional info:
Comment 2 mamccoma 2019-04-12 20:04:11 UTC 
Hello,

- Support was able to re-produce this issue in filestore OSD as well.

# docker exec ca8005010f73 ceph daemon osd.7 config set debug_filestore 20
{
"success": ""
}

- Checking journalctl I don't seem to get any debug logs for osd.7

# journalctl -b | tail -f
Apr 12 15:03:38 vm251-254.gsslab.pnq2.redhat.com dockerd-current[3851]: 2019-04-12 15:03:38.097227 7f1f8afd7700  1 mgr send_beacon active
Apr 12 15:03:38 vm251-254.gsslab.pnq2.redhat.com docker[4109]: 2019-04-12 15:03:38.097227 7f1f8afd7700  1 mgr send_beacon active


- I created the log with: 

# docker exec ca8005010f73 ceph daemon osd.7 config set log_file /tmp/osd7.log
{
    "success": "log_file = '/tmp/osd7.log' "
}

- Checking this log file I can see the debug level 20 log entries

# docker exec -it ca8005010f73 cat /tmp/osd7.log | tail -2
2019-04-12 14:53:05.047242 7f486c9e1700 20 filestore(/var/lib/ceph/osd/ceph-7) sync_entry(3989): woke after 5.000143
2019-04-12 14:53:05.047296 7f486c9e1700 20 filestore(/var/lib/ceph/osd/ceph-7) sync_entry(3973):  waiting for max_interval 5.000000



- In an additional note, we could not duplicate this for RGW container as changing rgw logging to debug produced debug logs in journalctl without issue.
Comment 3 Dimitri Savineau 2019-05-30 18:36:12 UTC 

*** This bug has been marked as a duplicate of bug 1710548 ***