Bug 1369555

Summary: kworker/0:0 writes to hard disk storage once a second every second like a metronome; md / bcache / lvm ?
Product: [Fedora] Fedora Reporter: Robert <redhat-bz>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 24CC: gansalmon, ichavero, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab, redhat-bz
Target Milestone: ---Flags: jforbes: needinfo?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-28 17:24:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
mount flags showing noatime in all the right places none

Description Robert 2016-08-23 18:16:33 UTC
Created attachment 1193407 [details]
mount flags showing noatime in all the right places

Description of problem:

I recently replaced by root partition with a sweet layered cake... three magnetic hard disks in a md (raid5) configuration below a bcache acceleration layer (with an SSD on the side for some speed) below LVM (sparse; to dice up the large-and-fast storage into various units) below the root "/".

Now... even with (effectively) no processes running, "kworker/0:0" causes a write to the disk every second (maybe even on the second) like a metronome. I can even watch my wall clock and see it beating in time with my hard disks.

I pretty sure the problem is... "in the kernel"... but I don't know how to further isolate it into one of these components (md / bcache / lvm) or how to tell if it is just an interesting feedback-reaction of using them all together.


Version-Release number of selected component (if applicable):
kernel-4.6.6-300.fc24.x86_64 (updates)
kernel-4.6.7-300.fc24.x86_64 (updates-testing)

How reproducible:
100% (on my machine)
???% (at large)

Steps to Reproduce:
1. create an md/bcache/lvm bootable layered root storage device
2. reboot
3. kill all the services & processes you can get your hands on
4. watch hard drive activity light or "iotop"

Actual results:

Even though "nothing" is writing to the disk, "kworker" shows up as writing to the disk; and the write/head noise can be heard literally every second.

Expected results:

When idle, the system should be... idle... and quiet.

Additional info:

/boot is formed from a raid-1 mirror...

I've already ruled out the 'noatime' effect (at the file system level, at least).

This is how narrow I got the "pstree" and was still not able to drop this funky beat from my computer:

systemd-+-agetty
        |-dbus-daemon
        |-dnsmasq---dnsmasq
        |-irqbalance
        |-login---bash---pstree
        |-rpcbind
        |-systemd---(sd-pam)
        |-systemd-journal
        |-systemd-logind
        `-systemd-udevd

'journal' would show up in the iotop every now and then, but not nearly so regular as kworker/0:0 ... which was always there at 0.13%.

Comment 1 Laura Abbott 2016-09-23 19:19:57 UTC
*********** MASS BUG UPDATE **************
 
We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 24 kernel bugs.
 
Fedora 24 has now been rebased to 4.7.4-200.fc24.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.
 
If you have moved on to Fedora 25, and are still experiencing this issue, please change the version to Fedora 25.
 
If you experience different issues, please open a new bug report for those.

Comment 2 Robert 2016-09-23 20:48:17 UTC
Still present in 4.7.4-200.fc24.x86_64

Comment 3 Justin M. Forbes 2017-04-11 14:48:55 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 24 kernel bugs.

Fedora 25 has now been rebased to 4.10.9-100.fc24.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26.

If you experience different issues, please open a new bug report for those.

Comment 4 Justin M. Forbes 2017-04-28 17:24:54 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the 
relevant data from the latest kernel you are running and any data that might have been requested previously.

Comment 5 Robert 2017-06-08 17:46:29 UTC
I can confirm the issue remains as far as 4.10.12-200.fc25

However, I tested a 4.12-rc and did not notice it... so I'm 90% sure this will go away "on it's own".