DescriptionMichael J. Kidd
2016-09-06 21:38:03 UTC
Created attachment 1198425[details]
OSD logs & configs for 134, 135 and 224
Description of problem:
PG scrub code requires a chunk it's scrubbing to begin and end on a hash boundary.
It seems in the case where it does not find a hash boundary, it keeps expanding the list (possibly by up to osd_scrub_chunk_max *PAST* the nearest hash boundary).
Version-Release number of selected component (if applicable):
Ceph 0.94.3
How reproducible:
Sporadic, data captured during deep scrub with 'osd_scrub_sleep' set to 2
Steps to Reproduce:
1. Set 'osd_scrub_sleep' to non-zero value
2. Set 'osd_scrub_chunk_max' to a low value ( 5 in this case )
3. Force a deep scrub on a PG with a few thousand objects in it
4. Monitor for extended periods between 'sleep' periods in the log output
5. Analyze object count between extended 'sleep' periods
Actual results:
PG scrub proceeded correctly for several iterations ( 5 objects read, 2 second sleep, repeat ), until 2016-09-06 09:59:51.772691, at which point 435 objects are read in, and the scrub chunk takes > 40 seconds which causes slow requests to stack up and impacts client IO.
Expected results:
'osd_scrub_chunk_max' objects to be read at a time, with 'osd_scrub_sleep' seconds between chunks until all objects have been read/scrubbed
Additional info:
* config dumps attached
* logs attached from the OSDs which were responsible for PG 3.f09 which was under deep scrub.
* OSD 134 was primary and logged the sleeep / slept statements
* debug_filestore 20, debug_osd 20, debug_ms 1 set during PG scrub
Created attachment 1198425 [details] OSD logs & configs for 134, 135 and 224 Description of problem: PG scrub code requires a chunk it's scrubbing to begin and end on a hash boundary. It seems in the case where it does not find a hash boundary, it keeps expanding the list (possibly by up to osd_scrub_chunk_max *PAST* the nearest hash boundary). Version-Release number of selected component (if applicable): Ceph 0.94.3 How reproducible: Sporadic, data captured during deep scrub with 'osd_scrub_sleep' set to 2 Steps to Reproduce: 1. Set 'osd_scrub_sleep' to non-zero value 2. Set 'osd_scrub_chunk_max' to a low value ( 5 in this case ) 3. Force a deep scrub on a PG with a few thousand objects in it 4. Monitor for extended periods between 'sleep' periods in the log output 5. Analyze object count between extended 'sleep' periods Actual results: PG scrub proceeded correctly for several iterations ( 5 objects read, 2 second sleep, repeat ), until 2016-09-06 09:59:51.772691, at which point 435 objects are read in, and the scrub chunk takes > 40 seconds which causes slow requests to stack up and impacts client IO. Expected results: 'osd_scrub_chunk_max' objects to be read at a time, with 'osd_scrub_sleep' seconds between chunks until all objects have been read/scrubbed Additional info: * config dumps attached * logs attached from the OSDs which were responsible for PG 3.f09 which was under deep scrub. * OSD 134 was primary and logged the sleeep / slept statements * debug_filestore 20, debug_osd 20, debug_ms 1 set during PG scrub