Description of problem: After the Kernel upgrade to 3.5.0-2.fc17.x86_64 from 3.4.6-2.fc17.x86_64 vsftpd-3.0.0-2.fc17.x86_64 started giving what appeared to be random Fatal error: 500 OOPS: priv_sock_ aborts. With no explanation. Further examination reveals the following: Under 3.5 vsftpd is unable to get directory information for a directory of 32 items or greater. Which means that neither directory listings nor file operations can be run. Reverting to an earlier Kernel restores normal vsftpd function. So although the problem is visible through vsftpd, I would suspect that it is a Kernel 3.5 X64 issue. Kernel 3.5.0-2.fc17.i686 with vsftpd-3.0.0-2.fc17.i686 does not seem to have the same problems. Version-Release number of selected component (if applicable): vsftpd-3.0.0-2.fc17.x86_64 kernel-3.5.0-2.fc17.x86_64 How reproducible: Easily. Boot with kernel-3.5.0-2.fc17.x86_64 and attempt file operations on that machine using vsftpd-3.0.0-2.fc17.x86_64 Steps to Reproduce: 1. Install and boot to kernel-3.5.0-2.fc17.x86_64 2. Install and system start vsftpd-3.0.0-2.fc17.x86_64 3. Attempt to either ftp or lftp to that machine from any other machine to a directory with 32 or more files in it. Actual results: Fatal error: 500 OOPS: priv_sock_get_cmd Expected results: Directory listing in this case. Additional info:
It could be both a kernel and vsftpd issue actually. Some change in the Kernel that vsftpd needs to take into account? But it might well have wider implications, so both kernel and vsftpd developers may need to investigate. The number of files being 32 is an interesting number...
Interesting, I ran into this and hadn't attributed it to a kernel update, but... I'm running kernel 3.5.1-1.fc17.x86_64 More info: -I have selinux completely disabled from boot -Setting the following allows it to work, but doesn't allow access to anything outside of a user's home dir: chroot_local_user=YES allow_writeable_chroot=YES Using the chroot list to attempt to allow access for specific users fails with the same error. Hope this gets fixed soon..It's rather crippling for FTP...
Continues to not work with kernel 3.5.2-1.fc17.x86_64...
confirmed here as well. vsftpd.log shows nothing beyond "150 here comes the directory connection" even though verbose logging is turned on. tried to use the hack above (chroot etc), does not work that way either. tried both passive and active ftp connections, the "500 OOPS: priv_sock_get_cmd" error message pops up in passive mode only, active mode only returns nothing and on the next command attempt the connection is lost (seems like the vsftpd child process dies without a whimper). please advise whether any work is commencing on fixing this bug - my clients are ready to tear me a new one. (i know, using sftp/scp could be an alternative, but there is a reason why ssh is disabled for most of the logins.) ps: actually, starting to regret my decision to upgrade to fedora 17. php is now running without any accelerators (both apc and xcache fail in various ways with php 5.4.5, actually to the extent that the phpMyAdmin installation from the official rpm doesn't work), and now this. what happened?!
I changed the Component to Kernel, because I'm pretty sure that's where the problem lies. There's something dodgy with the buffers. But I haven't had time to work out what. Incidentally, if you don't do a file list, you can still transfer files just fine... James :) Collins.
(In reply to comment #5) > Incidentally, if you don't do a file list, you can still transfer files just > fine... try telling that to clients using total commander. :/ (and actually, i'm using total commander myself as well sometimes. i now prefer winscp but due to security concerns i severely restrict who's allowed to ssh in, so that's not really a solution for "simple" users either.) btw, i suspected that something may be off in the kernel, so i added this to the config: use_sendfile=NO but the bug remained still, so it's not a sendfile problem.
any selinux denials in /var/log/audit/ ?
(In reply to comment #7) > any selinux denials in /var/log/audit/ ? using SELINUXTYPE=DISABLED... :) it's not an SElinux issue, but besides, no log entries like that.
It seems that feature is usually disabled on systems exhibiting this behaviour, so it's not outside the realm of possibilities that it is part of the problem. Perhaps disabling SElinux is causing the errors? However it doesn't answer the genereal flakiness of machines which are run on 3.5 vs 3.4
Still NOT working on kernel 3.5.3-1.fc17.x86_64, I'm going back to 3.4.6, it works there.
workaround: seccomp_sandbox=NO
(In reply to comment #12) > workaround: > > seccomp_sandbox=NO a resounding THANK YOU. finally seems to work. now if only apc would be fixed... but that's a different bugtrack number. :)
Excellent!! Thank you very much...
btw, vsftpd author (Chris Evans) is aware and promised to look which syscall is causing the trouble.
The problem is the seccomp filter doesn't include __NR_munmap in the default case, which glibc is using to back (FILE*) IO. (For the interested reader, qsort is being called by vsftpd, which results in glibc doing a sysconf(SC_PHYS_PAGES) to find out how much memory there is, which results in the mmap/munmap of /proc/meminfo. http://kyle.fedorapeople.org/vsftpd-3.0.0-seccomp-fix-munmap-sigsys.patch should fix the issue when seccomp_sandbox=YES. [There's a secondary issue when using text_userdb_names, I'll address that separately.] regards, Kyle
Thanks Kyle, this is fixed in the latest vsftpd-3.0.1 too. I plan to rebase rawhide as well as f17, f18. Jiri
Thanks for the workaround Kyle and for the fix Jiri. All of us in the "Trenches" really appreciate your hard work! James.
No problem, I'm not sure if Chris fixed the text_userdb_names issue as I've not had a chance to look at the 3.0.1 diff yet, but I'll do that today. --Kyle
And one more fix was needed, but it should be good in vsftpd-3.0.2 >> vsftpd-3.0.2 is released - the only noteworthy fixes are two seccomp sandbox policy tweaks which stops session crashes when listing large directories.
vsftpd-3.0.2-1.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/vsftpd-3.0.2-1.fc17
Package vsftpd-3.0.2-1.fc17: * should fix your issue, * was pushed to the Fedora 17 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing vsftpd-3.0.2-1.fc17' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-14450/vsftpd-3.0.2-1.fc17 then log in and leave karma (feedback).
With kernel-3.5.4-2.fc17.x86_64 and vsftpd-3.0.2-1.fc17.x86_64 I still seem to be hitting this issue: ftp> dir 500 OOPS: priv_sock_get_int Passive mode refused. ftp> seccomp_sandbox=NO fixes this. The interesting part of strace -ff (without seccomp_sandbox=NO) looks like: [...] "PASV\r\n", 4096, MSG_PEEK, NULL, NULL) = 6 [pid 4774] read(0, "PASV\r\n", 6) = 6 [pid 4774] write(5, "\n", 1 <unfinished ...> [pid 4772] <... read resumed> "\n", 1) = 1 [pid 4774] <... write resumed> ) = 1 [pid 4772] write(4, "\1", 1 <unfinished ...> [pid 4774] read(5, <unfinished ...> [pid 4772] <... write resumed> ) = 1 [pid 4774] <... read resumed> "\1", 1) = 1 [pid 4772] read(4, <unfinished ...> [pid 4774] write(5, "\f", 1) = 1 [pid 4772] <... read resumed> "\f", 1) = 1 [pid 4772] +++ killed by SIGSYS +++ [pid 4765] <... accept resumed> 0x7fff221031c0, [28]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) [pid 4765] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=4772, si_status=SIGSYS, si_utime=0, si_stime=0} --- [pid 4765] alarm(1) = 0 [pid 4765] rt_sigreturn() = -1 EINTR (Interrupted system call) [pid 4765] alarm(0) = 1 [pid 4765] wait4(-1, NULL, WNOHANG, NULL) = 4772 [pid 4765] wait4(-1, NULL, WNOHANG, NULL) = -1 ECHILD (No child processes) [pid 4765] accept(3, <unfinished ...> [pid 4774] read(5, "", 4) = 0 [pid 4774] fcntl(0, F_GETFL) = 0x2 (flags O_RDWR) [pid 4774] fcntl(0, F_SETFL, O_RDWR|O_NONBLOCK) = 0 [pid 4774] write(0, "500 OOPS: ", 10) = 10 [pid 4774] write(0, "priv_sock_get_int", 17) = 17 [pid 4774] write(0, "\r\n", 2) = 2 [pid 4774] exit_group(2) = ? [pid 4774] +++ exited with 2 +++ [...] If I rerun the same job with seccomp_sandbox=NO I'm getting: the following syscalls after where above the SIGSYS struck: [...] [pid 4864] gettimeofday({1349110483, 633323}, NULL) = 0 [pid 4864] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 [pid 4864] setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 [pid 4864] bind(5, {sa_family=AF_INET, sin_port=htons(59857), sin_addr=inet_addr("217.197.83.146")}, 16) = 0 I take it this was tested on bare metal on a modern machine where glibc avoids the gettimeofday(2) syscall but not on an old system without TSC or a VM where gettimeofday(2) needs to be called? Something like that.
Hrm. Yeah, that does sound likely.
Also seeing the same problem on Fedora 18 (just upgraded from Fedora 16) with v3.0.2-1 of VSFTPd. Can get directory listings, but attempts at file transfers abort the connection without any data received. Adding the line "seccomp_sandbox=NO" to vsftpd.conf resolves the problem in FC18 too.
I can confirm this bug on Fedora 18, kernel 3.8.2-206.fc18.x86_64 and vsftpd-3.0.2-1.fc18.x86_64. ftp> ls 500 OOPS: priv_sock_get_int Passive mode refused. seccomp_sandbox=NO resolves the problem.
i can also confirm the same thing after an update on Fedora 17, vsftpd version 3.0.0-2.fc17. the seccomp_sandbox=NO resolves the problem as before.
Looks like mmap is still being problematic, along with wait4 and alarm. I'll poke upstream about this.
I'm seeing this on a newly installed F17 system with all updates applied. Kernel kernel-3.8.12-100.fc17.x86_64, vsftpd vsftpd-3.0.0-2.fc17.x86_64. When I connect with a ftp client, I immediately get $ ftp xxx Connected to xxx. 500 OOPS: child died ftp> quit Adding seccomp_sandbox=NO fixes this.
This message is a reminder that Fedora 17 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 17. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '17'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 17's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 17 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior to Fedora 17's end of life. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Confirming that this is still an issue on Fedora 19.
I'll look at it soon unless Jiri beats me.
I don't see the issue on F19 with kernel 3.10.3-300.fc19.x86_64. Could someone confirm it? Unfortunately the issue persists on F18 (kernel 3.10.6-100.fc18.x86_64). I'll investigate what's behind.
I've changed temporarily default value of seccomp_sandbox to NO in f18 branch as a workaround together with number of other changes.
With respect to comments #33 and #34 inclusive the fact I didn't hear about the bug after commenting I'm going to close the bug.
If anyone stumbles across this, even though this bugzilla is closed, it's still a bug on F21 with vsftpd-3.0.2-12.fc21.x86_64. Putting seccomp_sandbox=NO into vstpd.conf still works around the problem.
This problem still there in Fedora 22, it gets solve with adding seccomp_sandbox=NO into /etc/vsftpd/vsftpd.conf as mentioned up.
Jiri, please see comment #37. I've also seen a similar (or same) issue with Fedora 23. kernel-4.4.2-301.fc23.x86_64 vsftpd-3.0.2-14.fc23.x86_64 echo 'seccomp_sandbox=NO' >> /etc/vsftpd/vsftpd.conf fixed it for me. Perhaps you need a new bug open for that ?