Bug 1671484 - OSD Bluestore debug not reaching the docker logs.
Summary: OSD Bluestore debug not reaching the docker logs.
Keywords:
Status: CLOSED DUPLICATE of bug 1710548
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Container
Version: 3.2
Hardware: All
OS: All
high
medium
Target Milestone: rc
: 4.0
Assignee: Sébastien Han
QA Contact: Vasishta
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-31 16:59 UTC by daniel parkes
Modified: 2019-05-30 18:36 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-30 18:36:12 UTC
Embargoed:


Attachments (Terms of Use)

Description daniel parkes 2019-01-31 16:59:10 UTC
Description of problem:

On a containerised 3.2 deployment, when we enable for example bluestore debug, the actual bluestore logs are not reaching the host journal.

Using the default logging options in the conf file

# cat /etc/ceph/ceph.conf | grep log
log file = /dev/null
mon cluster log file = /dev/null

Enable Bluestore debug

[root@servera ~]#  ceph daemon osd.11 config get debug_bluestore
{
    "debug_bluestore": "5/5"
}

We have no logs in the host journal

[root@servera ~]# journalctl -b | grep bluestore.Mempool

If we configure the osd to dump it0s log to a file we get the bluestore logs like expected:
[root@servera ~]#  ceph daemon osd.11 config get debug_bluestore
{
    "debug_bluestore": "5/5"
}
[root@servera ~]# ceph daemon osd.11 config set log_file /tmp/osd11.log
{
    "success": "log_file = '/tmp/osd11.log' "
}
[root@servera ~]# ceph daemon osd.11 log reopen
{}
[root@servera ~]# docker exec -it ceph-osd-11 cat /tmp/osd11.log | tail -2
2019-01-31 11:56:16.012799 7f1f5a3bb700  5 bluestore.MempoolThread(0x55645df471d0) _trim_shards cache_size: 2845415832 kv_alloc: 1218696969 kv_used: 1837362 meta_alloc: 1084479241 meta_used: 71604 data_alloc: 542239620 data_used: 0
2019-01-31 11:56:16.866433 7f1f5a3bb700  5 bluestore.MempoolThread(0x55645df471d0) _tune_cache_size target: 4294967296 heap: 75497472 unmapped: 22568960 mapped: 52928512 old cache_size: 2845415832 new cache size: 2845415832





Version-Release number of selected component (if applicable):

3.2

How reproducible:

Install Containerised 3.2 deployment 
Enable Bluestore Debug

Steps to Reproduce:
1.Install Containerised 3.2 deployment
2.Enable Bluestore Debug
3.

Actual results:

No Bluestore debug logs are present

Expected results:

The OSD debug logs should be present on the host journal

Additional info:

Comment 2 mamccoma 2019-04-12 20:04:11 UTC
Hello,

- Support was able to re-produce this issue in filestore OSD as well.

# docker exec ca8005010f73 ceph daemon osd.7 config set debug_filestore 20
{
"success": ""
}

- Checking journalctl I don't seem to get any debug logs for osd.7

# journalctl -b | tail -f
Apr 12 15:03:38 vm251-254.gsslab.pnq2.redhat.com dockerd-current[3851]: 2019-04-12 15:03:38.097227 7f1f8afd7700  1 mgr send_beacon active
Apr 12 15:03:38 vm251-254.gsslab.pnq2.redhat.com docker[4109]: 2019-04-12 15:03:38.097227 7f1f8afd7700  1 mgr send_beacon active


- I created the log with: 

# docker exec ca8005010f73 ceph daemon osd.7 config set log_file /tmp/osd7.log
{
    "success": "log_file = '/tmp/osd7.log' "
}

- Checking this log file I can see the debug level 20 log entries

# docker exec -it ca8005010f73 cat /tmp/osd7.log | tail -2
2019-04-12 14:53:05.047242 7f486c9e1700 20 filestore(/var/lib/ceph/osd/ceph-7) sync_entry(3989): woke after 5.000143
2019-04-12 14:53:05.047296 7f486c9e1700 20 filestore(/var/lib/ceph/osd/ceph-7) sync_entry(3973):  waiting for max_interval 5.000000



- In an additional note, we could not duplicate this for RGW container as changing rgw logging to debug produced debug logs in journalctl without issue.

Comment 3 Dimitri Savineau 2019-05-30 18:36:12 UTC

*** This bug has been marked as a duplicate of bug 1710548 ***


Note You need to log in before you can comment on or make changes to this bug.