Description of problem: Running F15 i get these: Apr 14 18:01:21 xxxxx kernel: [ 16.798256] systemd-readahead-collect[470]: Failed to read event: Value too large for defined data type Apr 14 18:01:21 xxxxx systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 # systemctl status systemd-readahead-collect.service systemd-readahead-collect.service - Collect Read-Ahead Data Loaded: loaded (/lib/systemd/system/systemd-readahead-collect.service) Active: active (exited) since Thu, 14 Apr 2011 15:01:09 -0300; 4h 2min ago Process: 470 ExecStart=/lib/systemd/systemd-readahead-collect (code=exited, status=1/FAILURE) Status: "Collecting readahead data" CGroup: name=systemd:/system/systemd-readahead-collect.service Version-Release number of selected component (if applicable): i think it is readahead-1.5.7-3.fc15.x86_64 How reproducible: Steps to Reproduce: 1. Actual results: Expected results: Additional info:
reassigning to systemd
Now running beta, it get the same errors on a new install. # cat /var/log/messages | grep -i ahead Apr 25 12:59:00 ulquiorra systemd-readahead-collect[449]: Failed to read event: Permission denied Apr 25 12:59:00 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 08:00:37 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 09:44:57 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 13:08:56 ulquiorra systemd-readahead-collect[501]: Failed to read event: Permission denied Apr 26 13:08:56 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 16:32:26 ulquiorra systemd-readahead-collect[469]: Failed to read event: Permission denied Apr 26 16:32:26 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 16:50:14 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 # systemctl status systemd-readahead-collect.service systemd-readahead-collect.service - Collect Read-Ahead Data Loaded: loaded (/lib/systemd/system/systemd-readahead-collect.service) Active: active (exited) since Tue, 26 Apr 2011 13:50:04 -0300; 3h 20min ago Process: 486 ExecStart=/lib/systemd/systemd-readahead-collect (code=exited, status=1/FAILURE) Status: "Collecting readahead data" CGroup: name=systemd:/system/systemd-readahead-collect.service # systemctl stop systemd-readahead-collect.service # systemctl start systemd-readahead-collect.service messages: Apr 26 17:12:58 ulquiorra systemd[1]: Unit systemd-readahead-collect.service entered failed state. audit.log: type=SERVICE_STOP msg=audit(1303848778.945:90): user pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=': comm="systemd-readahead-collect" exe="/bin/systemd" hostname=? addr=? terminal=? res=failed' type=SERVICE_START msg=audit(1303848798.650:91): user pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=': comm="systemd-readahead-collect" exe="/bin/systemd" hostname=? addr=? terminal=? res=success' SELinux is now enforcing beacuse i reinstalled. # cat ./systemd-readahead-collect.service # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=Collect Read-Ahead Data DefaultDependencies=no Wants=systemd-readahead-done.timer Conflicts=shutdown.target Before=sysinit.target shutdown.target [Service] Type=notify ExecStart=/lib/systemd/systemd-readahead-collect RemainAfterExit=yes [Install] WantedBy=default.target # ls -l /lib/systemd/systemd-readahead-collect -rwxr-xr-x. 1 root root 58384 Apr 20 22:51 /lib/systemd/systemd-readahead-collect # ls -lZ /lib/systemd/systemd-readahead-collect -rwxr-xr-x. root root system_u:object_r:readahead_exec_t:s0 /lib/systemd/systemd-readahead-collect # /lib/systemd/systemd-readahead-collect --help systemd-readahead-collect [OPTIONS...] [DIRECTORY] Collect read-ahead data on early boot. -h --help Show this help --max-files=INT Maximum number of files to read ahead --max-file-size=BYTES Maximum size of files to read ahead --timeout=USEC Maximum time to spend collecting data So i cam execute it manually. So the problem is "Failed to read event: Permission denied" # ./systemd/system/default.target.wants/systemd-readahead-replay.service bash: ./systemd/system/default.target.wants/systemd-readahead-replay.service: Permission denied # ./systemd/system/default.target.wants/systemd-readahead-replay.service bash: ./systemd/system/default.target.wants/systemd-readahead-replay.service: Permission denied # ls -l ./systemd/system/default.target.wants/systemd-readahead-replay.service lrwxrwxrwx. 1 root root 52 Apr 8 18:33 ./systemd/system/default.target.wants/systemd-readahead-replay.service -> /lib/systemd/system/systemd-readahead-replay.service # ls -l /lib/systemd/system/systemd-readahead-replay.service -rw-r--r--. 1 root root 570 Apr 20 22:51 /lib/systemd/system/systemd-readahead-replay.service # ls -lZ /lib/systemd/system/systemd-readahead-replay.service -rw-r--r--. root root system_u:object_r:systemd_unit_file_t:s0 /lib/systemd/system/systemd-readahead-replay.service # cat /lib/systemd/system/systemd-readahead-replay.service # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=Replay Read-Ahead Data DefaultDependencies=no Conflicts=shutdown.target Before=sysinit.target shutdown.target ConditionPathExists=/.readahead [Service] Type=notify ExecStart=/lib/systemd/systemd-readahead-replay RemainAfterExit=yes [Install] WantedBy=default.target # ls -l /lib/systemd/systemd-readahead-replay -rwxr-xr-x. 1 root root 47688 Apr 20 22:51 /lib/systemd/systemd-readahead-replay # ls -lZ /lib/systemd/systemd-readahead-replay -rwxr-xr-x. root root system_u:object_r:readahead_exec_t:s0 /lib/systemd/systemd-readahead-replay # /lib/systemd/systemd-readahead-replay --help systemd-readahead-replay [OPTIONS...] [DIRECTORY] Replay collected read-ahead data on early boot. -h --help Show this help --max-file-size=BYTES Maximum size of files to read ahead # /lib/systemd/systemd-readahead-replay This caused: Apr 26 17:25:56 ulquiorra systemd-readahead-replay[2878]: Bumped block_nr parameter of /dev/sda to 16384. This is a temporary hack and should be removed one day. Example of the error: Apr 26 16:32:24 ulquiorra kernel: [ 51.576002] eth1: Link changed: 100Mbps, full duplex Apr 26 16:32:26 ulquiorra systemd-readahead-collect[469]: Failed to read event: Permission denied Apr 26 16:32:26 ulquiorra systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Apr 26 16:32:26 ulquiorra auditd[1213]: Started dispatcher: /sbin/audispd pid: 1215 I is slightly different, no "Failed to read event: Value too large for defined data type" message.
so you got E2BIG and EPERM on read() on an fanotify fd. I wonder what this means. If you disable selinux enforcing mode, does the EPERM go away? E2BIG still puzzles me... Eric, do you have an idea what E2BIG and EPERM when read()ing on an fanotify mean? The code for this is here: http://cgit.freedesktop.org/systemd/tree/src/readahead-collect.c#n381 Right now the buffer we read into is sizes 4K. Is it possible this is too small and you return E2BIG due to that?
Oops, it is actually EACCES and EOVERFLOW, not EPERM and E2BIG.
The EOVERFLOW issue seems to be a kernel bug, according to Eric. Reassigning to kernel. Reartes, Eric will try to fix the EOVERFLOW issue for you. If this issue is fixed but the EACCES issue remains, please file a new bug. In the meantime hope that this is the same issue in some way.
I am also seeing something similar in my messages on boot: Jun 2 22:01:25 localhost kernel: [ 30.272748] systemd-readahead-collect[374]: Failed to read event: Value too large for defined data type Jun 2 22:01:25 localhost systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1 Jun 2 22:01:25 localhost kernel: [ 30.640772] Ebtables v2.0 registered
Since i installed the released F15 (not an upgrade) i have not seen it on my logs anymore. I only used it for a couple of days, time will tell. SELinux is most of the time in enforcing and somethimes in permisive. # grep -i -readahead-collect /var/log/messages <nothing> # systemctl status systemd-readahead-collect.service systemd-readahead-collect.service - Collect Read-Ahead Data Loaded: loaded (/lib/systemd/system/systemd-readahead-collect.service) Active: active (exited) since Fri, 03 Jun 2011 03:53:51 -0300; 8h ago Process: 490 ExecStart=/lib/systemd/systemd-readahead-collect (code=exited, status=0/SUCCESS) Status: "Collecting readahead data" CGroup: name=systemd:/system/systemd-readahead-collect.service Versions: systemd-sysv-26-2.fc15.x86_64 systemd-units-26-2.fc15.x86_64 systemd-26-2.fc15.x86_64
Created attachment 915327 [details] Comment (This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Eric, did this get fixed?
Fedora 15 has reached it's end of life as of June 26, 2012. As a result, we will not be fixing any remaining bugs found in Fedora 15. In the event that you have upgraded to a newer release and the bug you reported is still present, please reopen the bug and set the version field to the newest release you have encountered the issue with. Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered. Thank you for taking the time to file a report. We hope newer versions of Fedora suit your needs.
(In reply to comment #9) > Eric, did this get fixed? No, it didn't. This bug still exists on F17: https://bugzilla.redhat.com/show_bug.cgi?id=834423 Reopening.
*** Bug 834423 has been marked as a duplicate of this bug. ***
https://bugzilla.redhat.com/show_bug.cgi?id=834423 has some more information about this problem on an F17 kernel.
Currently a problem with F18-Beta following a change in the configuration of SEL to Permissive. Bug 889549
*** Bug 889549 has been marked as a duplicate of this bug. ***
I've tracked the cause of this down. The actual error is EOVERFLOW, and the cause is: - systemd-readahead-collect uses fanotify to track files that are read - during boot, libvirtd starts - libvirt reads the VM images at start up (I don't know why it does that, but...) (see [1]) - I have a VM image > 2 GB - fanotify_init says: - O_LARGEFILE support files exceeding 2 GB. Failing to set this flag will result in an EOVERFLOW error when trying to open a large file which is monitored by a fanotify group. - Systemd does set O_LARGEFILE in the call to fanotify_init. - BUT, on x86_64, O_LARGEFILE is defined as 0, so that doesn't actually do anything. - see https://lkml.org/lkml/2013/7/11/553 Patching systemd to add "#define O_LARGEFILE 0100000" after the #include of <fcntl.h> "fixes" this. Not sure if this is a kernel bug or a glibc one? [1] If I enable debugs with the patched systemd-readahead-collect, I get: - Jul 22 23:41:05 pear.home lt-systemd-readahead[220]: Not preloading file /var/lib/libvirt/images/win.img with size out of bounds 14010049536
The kernel's implementation of the open() syscall in fs/open.c does: SYSCALL_DEFINE3(open, const char __user *, filename, int, flags, umode_t, mode) { if (force_o_largefile()) flags |= O_LARGEFILE; ... } So maybe fanotify_init needs to do the same? Eric?
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs. Fedora 19 has now been rebased to 3.11.1-200.fc19. Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
Still a problem on 3.11.1-200.fc19.x86_64: Sep 21 13:21:23 pear.home systemd-readahead[229]: Failed to read event: Value too large for defined data type No more comments in the lkml email thread either
Still a problem on 3.11.4-201.fc19.x86_64 [mar oct 15 11:02:24 2013] systemd-readahead[269]: Failed to read event: Value too large for defined data type
Still a problem on 3.11.4-201.fc19.x86_64 Oct 17 08:33:54 ivan kernel: 34.905025] systemd-readahead[316]: Failed to read event: Value too large for defined data type Oct 17 08:50:30 ivan systemd[1]: systemd-readahead-collect.service: main process exited, code=exited, status=1/FAILURE
Just throwing this one out there. Those experiencing this did you (fed)upgrade, if the answer is yes ( or no ) do you have data=writeback in your /etc/fstab in the / line if the answer is yes what happens if you remove it?
Yes. I have fedup to fedora 19. #grep " / " /etc/fstab /dev/mapper/VolGroup-lv_root / ext4 defaults 1 1 #mount |grep " / " /dev/mapper/VolGroup-lv_root on / type ext4 (rw,relatime,data=ordered)
Its nothing to do with fedup or writeback - see comment 16
well was not the patch for that here http://permalink.gmane.org/gmane.linux.kernel/1441629 ? In anycase I only was aware of the fedup issue and that systemd-readahead does not work if you have a swapfile instead of swap partition which you can simply confirm/deny by hashing that entry in the /etc/fstab
# grep swap /etc/fstab /dev/mapper/VolGroup-lv_swap swap swap defaults 0 0 # lvs |grep swap lv_swap VolGroup -wi-ao--- 3,94g # free |grep Swap Swap: 4128764 0 4128764 # swapon -s Filename Type Size Used Priority /dev/dm-1 partition 4128764 0 -1
Here [1] it sounds the nvidia driver is one of the culprit triggering this after upgrade ( as odd as that sounds ) "I had encountered this before, in an earlier Fedora and found it just meant that the nvidia driver was not installed properly. " 1.https://ask.fedoraproject.org/question/29686/nvidia-problem-after-fedup-fedora-17-to-fedora-19/ In anycase to rule out some fallout happening at upgrade ( which seems to be the most common dominator with the exception of what's pointed out in comment 16 ) someone needs to step up and do a fresh install and see if it's triggered after it.
Has that patch been applied? The swapfile is presumably the same issue if the swap file is > 2GB
and the nvidia thing is unrelated; its just the last message displayed before X starts (or in that case, doesn't start)
# rpm -qa |grep nvidia |sort akmod-nvidia-304xx-304.108-1.fc19.1.x86_64 kmod-nvidia-304xx-3.10.13-101.fc18.x86_64-304.88-2.fc18.6.x86_64 kmod-nvidia-304xx-3.11.3-201.fc19.x86_64-304.108-1.fc19.2.x86_64 kmod-nvidia-304xx-3.11.4-201.fc19.x86_64-304.108-1.fc19.3.x86_64 nvidia-settings-319.32-1.fc19.x86_64 nvidia-xconfig-319.32-1.fc19.x86_64 xorg-x11-drv-nvidia-304xx-304.108-1.fc19.x86_64 xorg-x11-drv-nvidia-304xx-libs-304.108-1.fc19.x86_64 # rpm -q kernel kernel-3.10.13-101.fc18.x86_64 kernel-3.11.3-201.fc19.x86_64 kernel-3.11.4-201.fc19.x86_64 # uname -a Linux ivan.oduv.so-ups.ru 3.11.4-201.fc19.x86_64 #1 SMP Thu Oct 10 14:11:18 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
No. I have not patched kernel.
My swap area located on device, not in file. [PATCH] fanotify: fix support of large files.
(In reply to Bradley from comment #16) > I've tracked the cause of this down. The actual error is EOVERFLOW, and the > cause is: > > - systemd-readahead-collect uses fanotify to track files that are read > - during boot, libvirtd starts > - libvirt reads the VM images at start up (I don't know why it does that, > but...) (see [1]) > - I have a VM image > 2 GB > - fanotify_init says: > - O_LARGEFILE > support files exceeding 2 GB. Failing to set this flag will result in an > EOVERFLOW error when trying to open a large file which is monitored by a > fanotify group. > - Systemd does set O_LARGEFILE in the call to fanotify_init. > - BUT, on x86_64, O_LARGEFILE is defined as 0, so that doesn't actually do > anything. > - see https://lkml.org/lkml/2013/7/11/553 > > Patching systemd to add "#define O_LARGEFILE 0100000" after the #include of > <fcntl.h> "fixes" this. Not sure if this is a kernel bug or a glibc one? > > [1] If I enable debugs with the patched systemd-readahead-collect, I get: > > - Jul 22 23:41:05 pear.home lt-systemd-readahead[220]: Not preloading file > /var/lib/libvirt/images/win.img with size out of bounds 14010049536 Same here with kernel 3.11.7-200.fc19.x86_64. I had to: systemctl disable libvirtd.service to have my GUI back working again. I think that this kind of problem on GUI unrelated services shouldn't block my graphical login manager to start.
> I think that this kind of problem on GUI unrelated services shouldn't block my graphical login manager to start. That's unrelated - because readahead runs until the boot process is finished, this line is the last one printed. So any failure to start the GUI will leave this as the last line.
*** Bug 1026265 has been marked as a duplicate of this bug. ***
Still an issue on F20, although the error is only in journalctl, not the syslog - that just logs that the service failed to start up
Also observed this with F20 (after fedup from F19, don't remember seeing this there too, but cannot confirm now). Re [comment 16]: I also have several > 2 GB VM images (standard location) and libvirtd service enabled. systemd-208-9.fc20.x86_64 3.12.6-200.fc19.x86_64 (as it was newer than what's there for F20 ATM?)
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs. Fedora 20 has now been rebased to 3.13.4-200.fc20. Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
On my up-to-date F20 systems, I am not experiencing this problem. Brgds, Eduard
Still an issue: Mar 01 20:42:44 pear.home systemd-readahead[316]: Failed to read event: Value too large for defined data type
Created attachment 869665 [details] !ConditionPathExists # /usr/lib/systemd/systemd --version systemd 210 +PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP -APPARMOR # rpm -qf /usr/lib/systemd/systemd systemd-210-2.fc21.x86_64
Comment on attachment 869665 [details] !ConditionPathExists This one is Big Mama's L! :)
Created attachment 869697 [details] And this one is Big Mama's R! as Rules! :)
We definitely need something like /etc/readahead.conf: # Exclude dirs RAC_EXCLUDE="/var/lib/libvirt/images/ ..." Easy peasy lemon squeezy. Don't drag this to infinity, s'il vous plaît!
Excluding the image directory is probably a good idea, but this is still a kernel bug.
Created attachment 889396 [details] fanotify-fix-EOVERFLOW-on-64-bit.patch Here's a kernel patch (against 3.14.1-200.fc20) that fixes the problem. I've tested it locally and readahead-collect no longer exits with an error. Yay! I also sent a version of this upstream: http://marc.info/?l=linux-kernel&m=139835974112096&w=2
Patches added in Fedora git. Thanks Will!
kernel-3.14.2-200.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.14.2-200.fc20
Package kernel-3.14.2-200.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.14.2-200.fc20' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-5808/kernel-3.14.2-200.fc20 then log in and leave karma (feedback).
kernel-3.14.2-200.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.