Description of problem: Firefox and/or qemu are killed during the night by the kernel because "out of memory" Version-Release number of selected component (if applicable): How reproducible: Most nights either or both are killed. I suspect that a rsync crontab was running at the time of out of memory condition Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Here are the out of memory messages: Jun 28 01:13:57 lx140e kernel: Out of memory: Killed process 2545 (firefox) total-vm:3122000kB, anon-rss:412084kB, file-rss:0kB, shmem-rss:2672kB Jun 28 15:20:58 lx140e kernel: Out of memory: Killed process 1136 (qemu-system-x86) total-vm:3530480kB, anon-rss:296716kB, file-rss:0kB, shmem-rss:0kB Jun 29 22:40:28 lx140e kernel: Out of memory: Killed process 10048 (qemu-system-x86) total-vm:3580940kB, anon-rss:633272kB, file-rss:0kB, shmem-rss:0kB Jun 30 01:53:35 lx140e kernel: Out of memory: Killed process 13357 (qemu-system-x86) total-vm:3536848kB, anon-rss:860128kB, file-rss:0kB, shmem-rss:0kB Jun 30 04:40:30 lx140e kernel: Out of memory: Killed process 2234 (firefox) total-vm:3474660kB, anon-rss:711064kB, file-rss:0kB, shmem-rss:3224kB Jul 1 23:32:06 lx140e kernel: Out of memory: Killed process 1126 (qemu-system-x86) total-vm:3539732kB, anon-rss:800372kB, file-rss:0kB, shmem-rss:0kB Jul 2 04:32:06 lx140e kernel: Out of memory: Killed process 2760 (firefox) total-vm:3984140kB, anon-rss:1011316kB, file-rss:0kB, shmem-rss:3508kB Here are two larger greps from /var/log/messages: Jul 1 23:32:06 lx140e kernel: [ 1126] 107 1126 884933 200093 3170304 86937 0 qemu-system-x86 Jul 1 23:32:06 lx140e kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d1\x2dfedora21.scope,task=qemu-system-x86,pid=1126,uid=107 Jul 1 23:32:06 lx140e kernel: Out of memory: Killed process 1126 (qemu-system-x86) total-vm:3539732kB, anon-rss:800372kB, file-rss:0kB, shmem-rss:0kB Jul 1 23:32:06 lx140e kernel: oom_reaper: reaped process 1126 (qemu-system-x86), now anon-rss:0kB, file-rss:12kB, shmem-rss:0kB Jul 1 23:32:06 lx140e journal[899]: internal error: End of file from qemu monitor Jul 1 23:32:06 lx140e systemd[1]: machine-qemu\x2d1\x2dfedora21.scope: Succeeded. Jul 1 23:32:06 lx140e systemd-machined[749]: Machine qemu-1-fedora21 terminated. Jul 2 04:32:06 lx140e kernel: [ 2760] 1000 2760 996035 253706 6094848 182556 0 firefox Jul 2 04:32:06 lx140e kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-2.scope,task=firefox,pid=2760,uid=1000 Jul 2 04:32:06 lx140e kernel: Out of memory: Killed process 2760 (firefox) total-vm:3984140kB, anon-rss:1011316kB, file-rss:0kB, shmem-rss:3508kB Jul 2 04:32:06 lx140e kernel: oom_reaper: reaped process 2760 (firefox), now anon-rss:0kB, file-rss:0kB, shmem-rss:3500kB Jul 2 04:32:06 lx140e kernel: Cameras IPC[11287]: segfault at 0 ip 000056067c4b774f sp 00007fe67d1798d0 error 6 in firefox[56067c4ae000+20000] Jul 2 04:32:06 lx140e audit[3118]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=3118 comm=43616D6572617320495043 exe="/usr/lib64/firefox/firefox" sig=11 res=1 Jul 2 04:32:06 lx140e audit[3164]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=3164 comm=43616D6572617320495043 exe="/usr/lib64/firefox/firefox" sig=11 res=1 Jul 2 04:32:18 lx140e systemd-coredump[14786]: Process 3164 (Web Content) of user 1000 dumped core.#012#012Stack trace of thread 11287:#012#0 0x000056067c4b774f MOZ_CrashPrintf (firefox)#012#1 0x00007fe69eabbfce n/a (libxul.so)#012#2 0x00007fe69eac71b5 n/a (libxul.so)#012#3 0x00007fe69ccf5446 n/a (libxul.so)#012#4 0x00007fe69e98b41e n/a (libxul.so)#012#5 0x00007fe69ded0ffa n/a (libxul.so)#012#6 0x00007fe69ed5abd1 n/a (libxul.so)#012#7 0x00007fe69ed5a06a n/a (libxul.so)#012#8 0x00007fe69ed0e559 n/a (libxul.so)#012#9 0x00007fe69ed59a97 n/a (libxul.so)#012#10 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#11 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#12 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3174:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x000056067c4c5ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fe69ed499dd n/a (libxul.so)#012#4 0x00007fe69ed496ea n/a (libxul.so)#012#5 0x00007fe69ed4967f n/a (libxul.so)#012#6 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#7 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3164:#012#0 0x00007fe6a40235c7 __poll (libc.so.6)#012#1 0x00007fe69f1ab7bf n/a (libxul.so)#012#2 0x00007fe6a2a831de n/a (libglib-2.0.so.0)#012#3 0x00007fe6a2a83313 g_main_context_iteration (libglib-2.0.so.0)#012#4 0x00007fe69f1ab4ff n/a (libxul.so)#012#5 0x00007fe69ed5a4e6 n/a (libxul.so)#012#6 0x00007fe69ed5a2f0 n/a (libxul.so)#012#7 0x00007fe69f30b84b n/a (libxul.so)#012#8 0x00007fe69eab985b n/a (libxul.so)#012#9 0x00007fe69ed0e559 n/a (libxul.so)#012#10 0x00007fe69f30b31d n/a (libxul.so)#012#11 0x00007fe69eab9806 n/a (libxul.so)#012#12 0x00007fe69ed0e559 n/a (libxul.so)#012#13 0x00007fe69d2c9e0e n/a (libxul.so)#012#14 0x000056067c4bc517 n/a (firefox)#012#15 0x00007fe6a3f56f33 __libc_start_main (libc.so.6)#012#16 0x000056067c4b8c2e _start (firefox)#012#012Stack trace of thread 3171:#012#0 0x00007fe6a446888a pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007fe6a3f09593 n/a (libnspr4.so)#012#2 0x00007fe6a3f09a50 PR_WaitCondVar (libnspr4.so)#012#3 0x00007fe69ed4727a n/a (libxul.so)#012#4 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#5 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#6 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3173:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x000056067c4c5ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fe69ed499dd n/a (libxul.so)#012#4 0x00007fe69ed496ea n/a (libxul.so)#012#5 0x00007fe69ed4967f n/a (libxul.so)#012#6 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#7 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3175:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x000056067c4c5ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fe69ed499dd n/a (libxul.so)#012#4 0x00007fe69ed496ea n/a (libxul.so)#012#5 0x00007fe69ed4967f n/a (libxul.so)#012#6 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#7 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3179:#012#0 0x00007fe6a446883c pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5de2 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#2 0x00007fe69ed5b741 n/a (libxul.so)#012#3 0x00007fe69ed5abd1 n/a (libxul.so)#012#4 0x00007fe69ed5a06a n/a (libxul.so)#012#5 0x00007fe69ed0e559 n/a (libxul.so)#012#6 0x00007fe69ed59a97 n/a (libxul.so)#012#7 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#8 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#9 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 11282:#012#0 0x00007fe6a4025e3b __select (libc.so.6)#012#1 0x00007fe69eac1217 n/a (libxul.so)#012#2 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#3 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3324:#012#0 0x00007fe6a4028fad syscall (libc.so.6)#012#1 0x00007fe69ed0f19b epoll_wait (libxul.so)#012#2 0x00007fe69fb37ef9 n/a (libxul.so)#012#3 0x00007fe69fb37b7c n/a (libxul.so)#012#4 0x00007fe6a015067e n/a (libxul.so)#012#5 0x00007fe6a01523a8 n/a (libxul.so)#012#6 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#7 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3323:#012#0 0x00007fe6a3ffc820 __sysconf (libc.so.6)#012#1 0x00007fe69fb3fbe8 n/a (libxul.so)#012#2 0x00007fe69fb0dab0 n/a (libxul.so)#012#3 0x00007fe69fb0946a n/a (libxul.so)#012#4 0x00007fe69fb2cb2c n/a (libxul.so)#012#5 0x00007fe69fb2b487 n/a (libxul.so)#012#6 0x00007fe69fb141a6 n/a (libxul.so)#012#7 0x00007fe69fb12c36 n/a (libxul.so)#012#8 0x00007fe69fb11eec n/a (libxul.so)#012#9 0x00007fe6a015067e n/a (libxul.so)#012#10 0x00007fe6a01523a8 n/a (libxul.so)#012#11 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#12 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3176:#012#0 0x00007fe6a40235c7 __poll (libc.so.6)#012#1 0x00007fe6a3f0b51e n/a (libnspr4.so)#012#2 0x00007fe69ed5e044 n/a (libxul.so)#012#3 0x00007fe69ed5abd1 n/a (libxul.so)#012#4 0x00007fe69ed5a2f0 n/a (libxul.so)#012#5 0x00007fe69ed5a0e1 n/a (libxul.so)#012#6 0x00007fe69ed0e559 n/a (libxul.so)#012#7 0x00007fe69ed59a97 n/a (libxul.so)#012#8 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#9 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#10 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3193:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fe69fa3799d n/a (libxul.so)#012#3 0x00007fe69ed59d27 n/a (libxul.so)#012#4 0x00007fe69ed5a50b n/a (libxul.so)#012#5 0x00007fe69ed5a2f0 n/a (libxul.so)#012#6 0x00007fe69ed5a0e1 n/a (libxul.so)#012#7 0x00007fe69ed0e559 n/a (libxul.so)#012#8 0x00007fe69ed59a97 n/a (libxul.so)#012#9 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#10 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#11 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3168:#012#0 0x00007fe69eb08b02 n/a (libxul.so)#012#1 0x00007fe69ed4f00d n/a (libxul.so)#012#2 0x00007fe69ed0efcb n/a (libxul.so)#012#3 0x00007fe69ed0e623 n/a (libxul.so)#012#4 0x00007fe69ed0e559 n/a (libxul.so)#012#5 0x00007fe69ed0d46e n/a (libxul.so)#012#6 0x00007fe69ed0d36b n/a (libxul.so)#012#7 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#8 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 3186:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fe69fa3799d n/a (libxul.so)#012#3 0x00007fe69ed59d27 n/a (libxul.so)#012#4 0x00007fe69ed5a50b n/a (libxul.so)#012#5 0x00007fe69ed5a2f0 n/a (libxul.so)#012#6 0x00007fe69ed5a0e1 n/a (libxul.so)#012#7 0x00007fe69ed0e559 n/a (libxul.so)#012#8 0x00007fe69ed59a97 n/a (libxul.so)#012#9 0x00007fe6a3f0f8a9 n/a (libnspr4.so)#012#10 0x00007fe6a44625a2 start_thread (libpthread.so.0)#012#11 0x00007fe6a402e303 __clone (libc.so.6)#012#012Stack trace of thread 11264:#012#0 0x00007fe6a44684d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056067c4c5e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fe69fa3799d n/a (libxul.so)#012#3 0x00007fe69ed59d27 n/a (libxul. Jul 2 04:32:21 lx140e systemd-coredump[14785]: Process 3118 (Web Content) of user 1000 dumped core.#012#012Stack trace of thread 11327:#012#0 0x00005633b6d0674f MOZ_CrashPrintf (firefox)#012#1 0x00007fb3cfafcfce n/a (libxul.so)#012#2 0x00007fb3cfb081b5 n/a (libxul.so)#012#3 0x00007fb3cdd36446 n/a (libxul.so)#012#4 0x00007fb3cf9cc41e n/a (libxul.so)#012#5 0x00007fb3cef11ffa n/a (libxul.so)#012#6 0x00007fb3cfd9bbd1 n/a (libxul.so)#012#7 0x00007fb3cfd9b06a n/a (libxul.so)#012#8 0x00007fb3cfd4f559 n/a (libxul.so)#012#9 0x00007fb3cfd9aa97 n/a (libxul.so)#012#10 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#11 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#12 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3125:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00005633b6d14ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fb3cfd8a9dd n/a (libxul.so)#012#4 0x00007fb3cfd8a6ea n/a (libxul.so)#012#5 0x00007fb3cfd8a67f n/a (libxul.so)#012#6 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#7 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3123:#012#0 0x00007fb3d54ab88a pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007fb3d4f4c593 n/a (libnspr4.so)#012#2 0x00007fb3d4f4ca50 PR_WaitCondVar (libnspr4.so)#012#3 0x00007fb3cfd8827a n/a (libxul.so)#012#4 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#5 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#6 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3124:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00005633b6d14ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fb3cfd8a9dd n/a (libxul.so)#012#4 0x00007fb3cfd8a6ea n/a (libxul.so)#012#5 0x00007fb3cfd8a67f n/a (libxul.so)#012#6 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#7 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3126:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00005633b6d14ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fb3cfd8a9dd n/a (libxul.so)#012#4 0x00007fb3cfd8a6ea n/a (libxul.so)#012#5 0x00007fb3cfd8a67f n/a (libxul.so)#012#6 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#7 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3127:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00005633b6d14ce8 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#3 0x00007fb3cfd8a9dd n/a (libxul.so)#012#4 0x00007fb3cfd8a6ea n/a (libxul.so)#012#5 0x00007fb3cfd8a67f n/a (libxul.so)#012#6 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#7 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 5514:#012#0 0x00007fb3d506bfad syscall (libc.so.6)#012#1 0x00007fb3cfd5019b epoll_wait (libxul.so)#012#2 0x00007fb3d0b78ef9 n/a (libxul.so)#012#3 0x00007fb3d0b78b7c n/a (libxul.so)#012#4 0x00007fb3d119167e n/a (libxul.so)#012#5 0x00007fb3d11933a8 n/a (libxul.so)#012#6 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#7 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3135:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fb3d0a7899d n/a (libxul.so)#012#3 0x00007fb3cfd9ad27 n/a (libxul.so)#012#4 0x00007fb3cfd9b50b n/a (libxul.so)#012#5 0x00007fb3cfd9b2f0 n/a (libxul.so)#012#6 0x00007fb3cfd9b0e1 n/a (libxul.so)#012#7 0x00007fb3cfd4f559 n/a (libxul.so)#012#8 0x00007fb3cfd9aa97 n/a (libxul.so)#012#9 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#10 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#11 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3121:#012#0 0x00007fb3cfb4910b n/a (libxul.so)#012#1 0x00007fb3cfb49b19 n/a (libxul.so)#012#2 0x00007fb3cfd9000d n/a (libxul.so)#012#3 0x00007fb3cfd4ffcb n/a (libxul.so)#012#4 0x00007fb3cfd4f623 n/a (libxul.so)#012#5 0x00007fb3cfd4f559 n/a (libxul.so)#012#6 0x00007fb3cfd4e46e n/a (libxul.so)#012#7 0x00007fb3cfd4e36b n/a (libxul.so)#012#8 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#9 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3128:#012#0 0x00007fb3d50665c7 __poll (libc.so.6)#012#1 0x00007fb3d4f4e51e n/a (libnspr4.so)#012#2 0x00007fb3cfd9f044 n/a (libxul.so)#012#3 0x00007fb3cfd9bbd1 n/a (libxul.so)#012#4 0x00007fb3cfd9b06a n/a (libxul.so)#012#5 0x00007fb3cfd4f559 n/a (libxul.so)#012#6 0x00007fb3cfd9aa97 n/a (libxul.so)#012#7 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#8 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#9 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3160:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fb3d0a7899d n/a (libxul.so)#012#3 0x00007fb3cfd9ad27 n/a (libxul.so)#012#4 0x00007fb3cfd9b50b n/a (libxul.so)#012#5 0x00007fb3cfd9b2f0 n/a (libxul.so)#012#6 0x00007fb3cfd9b0e1 n/a (libxul.so)#012#7 0x00007fb3cfd4f559 n/a (libxul.so)#012#8 0x00007fb3cfd9aa97 n/a (libxul.so)#012#9 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#10 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#11 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3129:#012#0 0x00007fb3d54ab83c pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14de2 _ZN7mozilla6detail21ConditionVariableImpl8wait_forERNS0_9MutexImplERKNS_16BaseTimeDurationINS_27TimeDurationValueCalculatorEEE (firefox)#012#2 0x00007fb3cfd9c741 n/a (libxul.so)#012#3 0x00007fb3cfd9bbd1 n/a (libxul.so)#012#4 0x00007fb3cfd9b2f0 n/a (libxul.so)#012#5 0x00007fb3cfd9b0e1 n/a (libxul.so)#012#6 0x00007fb3cfd4f559 n/a (libxul.so)#012#7 0x00007fb3cfd9aa97 n/a (libxul.so)#012#8 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#9 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#10 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3161:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fb3d0a7899d n/a (libxul.so)#012#3 0x00007fb3cfd9ad27 n/a (libxul.so)#012#4 0x00007fb3cfd9b50b n/a (libxul.so)#012#5 0x00007fb3cfd9b2f0 n/a (libxul.so)#012#6 0x00007fb3cfd9b0e1 n/a (libxul.so)#012#7 0x00007fb3cfd4f559 n/a (libxul.so)#012#8 0x00007fb3cfd9aa97 n/a (libxul.so)#012#9 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#10 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#11 0x00007fb3d5071303 __clone (libc.so.6)#012#012Stack trace of thread 3285:#012#0 0x00007fb3d54ab4d5 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00005633b6d14e4a _ZN7mozilla6detail21ConditionVariableImpl4waitERNS0_9MutexImplE (firefox)#012#2 0x00007fb3d0a7899d n/a (libxul.so)#012#3 0x00007fb3cfd9ad27 n/a (libxul.so)#012#4 0x00007fb3cfd9b50b n/a (libxul.so)#012#5 0x00007fb3cfd9b2f0 n/a (libxul.so)#012#6 0x00007fb3cfd9b0e1 n/a (libxul.so)#012#7 0x00007fb3cfd4f559 n/a (libxul.so)#012#8 0x00007fb3cfd9aa97 n/a (libxul.so)#012#9 0x00007fb3d4f528a9 n/a (libnspr4.so)#012#10 0x00007fb3d54a55a2 start_thread (libpthread.so.0)#012#11 0x00007fb3d5071303 Jul 2 04:32:26 lx140e abrt-notification[14841]: Process 3164 (firefox) crashed in MOZ_CrashPrintf() There are two crontab rsync scripts which work with a large number of files. Note that I have been using these for years on F21, F22, F24, and F28: 1 3 * * * rsync -tvz ftp.rfc-editor.org::rfcs/*.txt /home/common/ietf/rfcs 1 4 * * * rsync -tvz rsync.tools.ietf.org::tools.id/*.txt /home/common/ietf/drafts But the rsync may be a red heering.
This is a continual issue. Some of these do occur when the rsync may still be running but the last two nights were not during an rsyc: # grep "of memory" messages Jun 28 01:13:57 lx140e kernel: Out of memory: Killed process 2545 (firefox) total-vm:3122000kB, anon-rss:412084kB, file-rss:0kB, shmem-rss:2672kB Jun 28 15:20:58 lx140e kernel: Out of memory: Killed process 1136 (qemu-system-x86) total-vm:3530480kB, anon-rss:296716kB, file-rss:0kB, shmem-rss:0kB Jun 29 22:40:28 lx140e kernel: Out of memory: Killed process 10048 (qemu-system-x86) total-vm:3580940kB, anon-rss:633272kB, file-rss:0kB, shmem-rss:0kB Jun 30 01:53:35 lx140e kernel: Out of memory: Killed process 13357 (qemu-system-x86) total-vm:3536848kB, anon-rss:860128kB, file-rss:0kB, shmem-rss:0kB Jun 30 04:40:30 lx140e kernel: Out of memory: Killed process 2234 (firefox) total-vm:3474660kB, anon-rss:711064kB, file-rss:0kB, shmem-rss:3224kB Jul 1 23:32:06 lx140e kernel: Out of memory: Killed process 1126 (qemu-system-x86) total-vm:3539732kB, anon-rss:800372kB, file-rss:0kB, shmem-rss:0kB Jul 2 04:32:06 lx140e kernel: Out of memory: Killed process 2760 (firefox) total-vm:3984140kB, anon-rss:1011316kB, file-rss:0kB, shmem-rss:3508kB Jul 3 04:27:38 lx140e kernel: Out of memory: Killed process 14996 (firefox) total-vm:3403492kB, anon-rss:592604kB, file-rss:0kB, shmem-rss:2512kB Jul 4 02:45:51 lx140e kernel: Out of memory: Killed process 1142 (qemu-system-x86) total-vm:3552068kB, anon-rss:596580kB, file-rss:0kB, shmem-rss:0kB Jul 5 01:45:49 lx140e kernel: Out of memory: Killed process 15128 (qemu-system-x86) total-vm:3536848kB, anon-rss:764304kB, file-rss:0kB, shmem-rss:0kB
Just had an out of memory kill during working hours. Here is what I am seeing in messages: Jul 5 12:45:51 lx140e kernel: firefox invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 5 12:45:51 lx140e kernel: CPU: 3 PID: 2268 Comm: firefox Tainted: P OE 5.1.15-300.fc30.x86_64 #1 Jul 5 12:45:51 lx140e kernel: Hardware name: LENOVO 20BLS00400/20BLS00400, BIOS GSET58WW (2.03 ) 02/24/2014 Jul 5 12:45:51 lx140e kernel: Call Trace: Jul 5 12:45:51 lx140e kernel: dump_stack+0x5c/0x80 Jul 5 12:45:51 lx140e kernel: dump_header+0x51/0x2ff Jul 5 12:45:51 lx140e kernel: ? sched_clock+0x5/0x10 Jul 5 12:45:51 lx140e kernel: ? oom_unkillable_task+0x9a/0xd0 Jul 5 12:45:51 lx140e kernel: oom_kill_process.cold+0xb/0x10 Jul 5 12:45:51 lx140e kernel: out_of_memory+0x1cd/0x470 Jul 5 12:45:51 lx140e kernel: __alloc_pages_slowpath+0xccb/0xdf0 Jul 5 12:45:51 lx140e kernel: __alloc_pages_nodemask+0x2f2/0x340 Jul 5 12:45:51 lx140e kernel: pagecache_get_page+0xb4/0x370 Jul 5 12:45:51 lx140e kernel: filemap_fault+0x702/0xad0 Jul 5 12:45:51 lx140e kernel: ? page_add_file_rmap+0xfa/0x2d0 Jul 5 12:45:51 lx140e kernel: ? alloc_set_pte+0x2d9/0x6b0 Jul 5 12:45:51 lx140e kernel: ? xas_load+0x5/0x70 Jul 5 12:45:51 lx140e kernel: ? xas_find+0x159/0x1a0 Jul 5 12:45:51 lx140e kernel: ? filemap_map_pages+0x1da/0x3a0 Jul 5 12:45:51 lx140e kernel: ext4_filemap_fault+0x2d/0x3f Jul 5 12:45:51 lx140e kernel: __do_fault+0x36/0x180 Jul 5 12:45:51 lx140e kernel: __handle_mm_fault+0x1091/0x1ac0 Jul 5 12:45:51 lx140e kernel: ? __switch_to_asm+0x41/0x70 Jul 5 12:45:51 lx140e kernel: handle_mm_fault+0xdc/0x210 Jul 5 12:45:51 lx140e kernel: do_user_addr_fault+0x1f6/0x450 Jul 5 12:45:51 lx140e kernel: do_page_fault+0x33/0x120 Jul 5 12:45:51 lx140e kernel: ? page_fault+0x8/0x30 Jul 5 12:45:51 lx140e kernel: page_fault+0x1e/0x30 Jul 5 12:45:51 lx140e kernel: RIP: 0033:0x7fbed482e4ee Jul 5 12:45:51 lx140e kernel: Code: Bad RIP value. Jul 5 12:45:51 lx140e kernel: RSP: 002b:00007ffecf50a800 EFLAGS: 00010246 Jul 5 12:45:51 lx140e kernel: RAX: 0000000000000001 RBX: fffffffffffff000 RCX: 0000000000100000 Jul 5 12:45:51 lx140e kernel: RDX: 0000000000100000 RSI: 00007ffecf50a800 RDI: 00007ffecf50ac48 Jul 5 12:45:51 lx140e kernel: RBP: 00007ffecf50ac48 R08: 00007fbe79a20060 R09: 002e006f0063002e Jul 5 12:45:51 lx140e kernel: R10: 007e007c006b0075 R11: 0063006900720070 R12: 0000000000000001 Jul 5 12:45:51 lx140e kernel: R13: 0000000000000001 R14: 00007fbe73098000 R15: 00007fbed757ece0 Jul 5 12:45:51 lx140e kernel: Mem-Info: Jul 5 12:45:51 lx140e kernel: active_anon:526121 inactive_anon:195842 isolated_anon:0#012 active_file:112 inactive_file:40 isolated_file:0#012 unevictable:8 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:12791 slab_unreclaimable:30195#012 mapped:4949 shmem:8638 pagetables:9255 bounce:0#012 free:20475 free_pcp:0 free_cma:0 Jul 5 12:45:51 lx140e kernel: Node 0 active_anon:2104484kB inactive_anon:783368kB active_file:448kB inactive_file:160kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:19796kB dirty:0kB writeback:0kB shmem:34552kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 167936kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Jul 5 12:45:51 lx140e kernel: Node 0 DMA free:12944kB min:328kB low:408kB high:488kB active_anon:908kB inactive_anon:1916kB active_file:4kB inactive_file:20kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:8kB pagetables:52kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jul 5 12:45:51 lx140e kernel: lowmem_reserve[]: 0 2717 3158 3158 3158 Jul 5 12:45:51 lx140e kernel: Node 0 DMA32 free:59608kB min:57872kB low:72340kB high:86808kB active_anon:1902520kB inactive_anon:708000kB active_file:104kB inactive_file:360kB unevictable:0kB writepending:0kB present:2910336kB managed:2820340kB mlocked:0kB kernel_stack:11200kB pagetables:30040kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jul 5 12:45:51 lx140e kernel: lowmem_reserve[]: 0 0 440 440 440 Jul 5 12:45:51 lx140e kernel: Node 0 Normal free:9348kB min:9380kB low:11724kB high:14068kB active_anon:201056kB inactive_anon:73452kB active_file:24kB inactive_file:624kB unevictable:32kB writepending:0kB present:507904kB managed:451088kB mlocked:32kB kernel_stack:3752kB pagetables:6928kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jul 5 12:45:51 lx140e kernel: lowmem_reserve[]: 0 0 0 0 0 Jul 5 12:45:51 lx140e kernel: Node 0 DMA: 8*4kB (UME) 1*8kB (M) 4*16kB (UME) 2*32kB (UE) 2*64kB (ME) 3*128kB (UME) 2*256kB (UE) 1*512kB (E) 3*1024kB (UME) 2*2048kB (ME) 1*4096kB (M) = 12968kB Jul 5 12:45:51 lx140e kernel: Node 0 DMA32: 885*4kB (UME) 586*8kB (UME) 262*16kB (UME) 223*32kB (UE) 162*64kB (UE) 70*128kB (UE) 36*256kB (UME) 20*512kB (E) 2*1024kB (UM) 0*2048kB 0*4096kB = 60388kB Jul 5 12:45:51 lx140e kernel: Node 0 Normal: 500*4kB (UME) 196*8kB (UME) 127*16kB (UE) 62*32kB (UME) 16*64kB (UE) 1*128kB (E) 0*256kB 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 9760kB Jul 5 12:45:51 lx140e kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 5 12:45:51 lx140e kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 5 12:45:51 lx140e kernel: 27688 total pagecache pages Jul 5 12:45:51 lx140e kernel: 18874 pages in swap cache Jul 5 12:45:51 lx140e kernel: Swap cache stats: add 25418496, delete 25399486, find 9314718/16014139 Jul 5 12:45:51 lx140e kernel: Free swap = 0kB Jul 5 12:45:51 lx140e kernel: Total swap = 3419132kB Jul 5 12:45:51 lx140e kernel: 858557 pages RAM Jul 5 12:45:51 lx140e kernel: 0 pages HighMem/MovableOnly Jul 5 12:45:51 lx140e kernel: 36724 pages reserved Jul 5 12:45:51 lx140e kernel: 0 pages cma reserved Jul 5 12:45:51 lx140e kernel: 0 pages hwpoisoned Jul 5 12:45:51 lx140e kernel: Tasks state (memory values in pages): Jul 5 12:45:51 lx140e kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jul 5 12:45:51 lx140e kernel: [ 538] 0 538 17411 21 143360 429 0 systemd-journal Jul 5 12:45:51 lx140e kernel: [ 563] 0 563 11928 0 98304 787 -1000 systemd-udevd Jul 5 12:45:51 lx140e kernel: [ 711] 0 711 26265 0 77824 231 -1000 auditd Jul 5 12:45:51 lx140e kernel: [ 713] 0 713 1589 0 53248 98 0 sedispatch Jul 5 12:45:51 lx140e kernel: [ 733] 81 733 67649 0 65536 467 -900 dbus-broker-lau Jul 5 12:45:51 lx140e kernel: [ 734] 81 734 2040 1 57344 838 -900 dbus-broker Jul 5 12:45:51 lx140e kernel: [ 735] 0 735 78310 0 106496 404 0 ModemManager Jul 5 12:45:51 lx140e kernel: [ 736] 70 736 7786 5 65536 104 0 avahi-daemon Jul 5 12:45:51 lx140e kernel: [ 737] 0 737 78032 0 110592 303 0 rngd Jul 5 12:45:51 lx140e kernel: [ 739] 172 739 38231 0 69632 113 0 rtkit-daemon Jul 5 12:45:51 lx140e kernel: [ 741] 0 741 100266 0 131072 734 0 udisksd Jul 5 12:45:51 lx140e kernel: [ 742] 0 742 10901 0 102400 393 0 sssd Jul 5 12:45:51 lx140e kernel: [ 743] 0 743 30574 1 139264 4988 0 firewalld Jul 5 12:45:51 lx140e kernel: [ 748] 0 748 13457 0 81920 179 0 gssproxy Jul 5 12:45:51 lx140e kernel: [ 751] 0 751 1242 0 49152 67 0 alsactl Jul 5 12:45:51 lx140e kernel: [ 758] 991 758 21758 0 57344 124 0 chronyd Jul 5 12:45:51 lx140e kernel: [ 770] 0 770 2655 4 57344 332 0 smartd Jul 5 12:45:51 lx140e kernel: [ 784] 0 784 4528 0 77824 312 0 systemd-machine Jul 5 12:45:51 lx140e kernel: [ 786] 0 786 19830 11 57344 49 0 irqbalance Jul 5 12:45:51 lx140e kernel: [ 789] 0 789 2069 0 53248 137 0 bluetoothd Jul 5 12:45:51 lx140e kernel: [ 791] 0 791 65207 86 188416 184 0 rsyslogd Jul 5 12:45:51 lx140e kernel: [ 799] 70 799 7753 0 57344 83 0 avahi-daemon Jul 5 12:45:51 lx140e kernel: [ 800] 993 800 421866 0 225280 1894 0 polkitd Jul 5 12:45:51 lx140e kernel: [ 801] 0 801 11105 7 98304 623 0 sssd_be Jul 5 12:45:51 lx140e kernel: [ 810] 0 810 14279 28 139264 302 0 sssd_nss Jul 5 12:45:51 lx140e kernel: [ 848] 0 848 62494 0 98304 270 0 accounts-daemon Jul 5 12:45:51 lx140e kernel: [ 849] 0 849 7037 0 86016 568 0 systemd-logind Jul 5 12:45:51 lx140e kernel: [ 855] 0 855 84206 154 139264 639 0 NetworkManager Jul 5 12:45:51 lx140e kernel: [ 870] 0 870 6605 0 69632 216 -1000 sshd Jul 5 12:45:51 lx140e kernel: [ 871] 0 871 10221 1 98304 448 0 cupsd Jul 5 12:45:51 lx140e kernel: [ 881] 0 881 3932 0 53248 236 0 crond Jul 5 12:45:51 lx140e kernel: [ 883] 0 883 4540 0 53248 63 0 atd Jul 5 12:45:51 lx140e kernel: [ 884] 0 884 78501 0 102400 265 0 lightdm Jul 5 12:45:51 lx140e kernel: [ 891] 986 891 62194 0 114688 632 0 colord Jul 5 12:45:51 lx140e kernel: [ 896] 0 896 389459 430 409600 8424 0 libvirtd Jul 5 12:45:51 lx140e kernel: [ 909] 0 909 271680 3652 847872 12590 0 Xorg Jul 5 12:45:51 lx140e kernel: [ 942] 0 942 3083 41 57344 208 0 wpa_supplicant Jul 5 12:45:51 lx140e kernel: [ 1085] 977 1085 5700 30 57344 65 0 dnsmasq Jul 5 12:45:51 lx140e kernel: [ 1086] 0 1086 5693 0 57344 84 0 dnsmasq Jul 5 12:45:51 lx140e kernel: [ 1136] 0 1136 10348 0 102400 331 -900 virtlogd Jul 5 12:45:51 lx140e kernel: [ 1257] 0 1257 47775 1 110592 382 0 lightdm Jul 5 12:45:51 lx140e kernel: [ 1286] 1000 1286 5032 0 86016 466 0 systemd Jul 5 12:45:51 lx140e kernel: [ 1288] 1000 1288 12978 0 98304 1298 0 (sd-pam) Jul 5 12:45:51 lx140e kernel: [ 1293] 1000 1293 320856 0 131072 899 0 pulseaudio Jul 5 12:45:51 lx140e kernel: [ 1301] 1000 1301 68617 18 151552 1350 0 xfce4-session Jul 5 12:45:51 lx140e kernel: [ 1306] 1000 1306 69174 0 57344 113 0 dbus-broker-lau Jul 5 12:45:51 lx140e kernel: [ 1313] 1000 1313 1420 105 53248 116 0 dbus-broker Jul 5 12:45:51 lx140e kernel: [ 1339] 1000 1339 58165 0 90112 259 0 imsettings-daem Jul 5 12:45:51 lx140e kernel: [ 1342] 1000 1342 59220 0 90112 320 0 gvfsd Jul 5 12:45:51 lx140e kernel: [ 1416] 1000 1416 57558 0 77824 340 0 xfconfd Jul 5 12:45:51 lx140e kernel: [ 1480] 1000 1480 1777 9 53248 127 0 ssh-agent Jul 5 12:45:51 lx140e kernel: [ 1506] 1000 1506 76252 0 86016 240 0 at-spi-bus-laun Jul 5 12:45:51 lx140e kernel: [ 1511] 1000 1511 67078 0 61440 81 0 dbus-broker-lau Jul 5 12:45:51 lx140e kernel: [ 1512] 1000 1512 1294 0 45056 124 0 dbus-broker Jul 5 12:45:51 lx140e kernel: [ 1516] 1000 1516 40156 0 77824 213 0 at-spi2-registr Jul 5 12:45:51 lx140e kernel: [ 1521] 1000 1521 38681 48 65536 34 0 gpg-agent Jul 5 12:45:51 lx140e kernel: [ 1522] 1000 1522 226968 818 462848 5711 0 xfwm4 Jul 5 12:45:51 lx140e kernel: [ 1534] 1000 1534 49144 49 143360 1557 0 xfsettingsd Jul 5 12:45:51 lx140e kernel: [ 1535] 1000 1535 77632 1457 229376 5520 0 xfce4-panel Jul 5 12:45:51 lx140e kernel: [ 1547] 1000 1547 46789 535 131072 745 0 panel-16-xfce4- Jul 5 12:45:51 lx140e kernel: [ 1549] 1000 1549 113418 308 151552 1270 0 panel-17-pulsea Jul 5 12:45:51 lx140e kernel: [ 1550] 1000 1550 46379 401 126976 879 0 panel-6-systray Jul 5 12:45:51 lx140e kernel: [ 1551] 1000 1551 48942 157 139264 1161 0 panel-2-actions Jul 5 12:45:51 lx140e kernel: [ 1563] 1000 1563 68162 0 167936 1651 0 xfce4-notifyd Jul 5 12:45:51 lx140e kernel: [ 1578] 1000 1578 78976 194 200704 3370 0 xfdesktop Jul 5 12:45:52 lx140e kernel: [ 1586] 1000 1586 99763 0 126976 939 0 gvfs-udisks2-vo Jul 5 12:45:52 lx140e kernel: [ 1590] 1000 1590 58385 0 86016 262 0 gvfs-mtp-volume Jul 5 12:45:52 lx140e kernel: [ 1599] 1000 1599 77748 0 102400 424 0 gvfsd-trash Jul 5 12:45:52 lx140e kernel: [ 1606] 1000 1606 39905 0 77824 249 0 gvfsd-metadata Jul 5 12:45:52 lx140e kernel: [ 1657] 1000 1657 8315 54 81920 351 0 xscreensaver Jul 5 12:45:52 lx140e kernel: [ 1662] 1000 1662 68963 0 147456 1758 0 xfce-polkit Jul 5 12:45:52 lx140e kernel: [ 1664] 1000 1664 104042 0 159744 924 0 abrt-applet Jul 5 12:45:52 lx140e kernel: [ 1666] 1000 1666 86575 640 176128 1147 0 nm-applet Jul 5 12:45:52 lx140e kernel: [ 1667] 1000 1667 53660 0 192512 4169 0 seapplet Jul 5 12:45:52 lx140e kernel: [ 1671] 1000 1671 58559 0 81920 160 0 agent Jul 5 12:45:52 lx140e kernel: [ 1685] 1000 1685 49277 718 147456 879 0 xfce4-power-man Jul 5 12:45:52 lx140e kernel: [ 1704] 0 1704 62354 4 110592 386 0 upowerd Jul 5 12:45:52 lx140e kernel: [ 1903] 1000 1903 489695 3024 864256 52912 0 virt-manager Jul 5 12:45:52 lx140e kernel: [ 1908] 1000 1908 39027 0 65536 209 0 dconf-service Jul 5 12:45:52 lx140e kernel: [ 1959] 1000 1959 94257 22 204800 3116 0 xfce4-terminal Jul 5 12:45:52 lx140e kernel: [ 1963] 1000 1963 4119 0 65536 504 0 bash Jul 5 12:45:52 lx140e kernel: [ 1999] 1000 1999 4131 0 65536 502 0 bash Jul 5 12:45:52 lx140e kernel: [ 2037] 1000 2037 641307 63924 2568192 73099 0 thunderbird Jul 5 12:45:52 lx140e kernel: [ 2100] 0 2100 152336 0 131072 573 0 pcscd Jul 5 12:45:52 lx140e kernel: [ 2160] 1000 2160 492603 13801 1298432 33669 0 thunderbird Jul 5 12:45:52 lx140e kernel: [ 2263] 1000 2263 141019 0 229376 3849 0 xfce4-appfinder Jul 5 12:45:52 lx140e kernel: [ 2268] 1000 2268 716001 125198 3739648 69282 0 firefox Jul 5 12:45:52 lx140e kernel: [ 2345] 1000 2345 468832 4737 1761280 17157 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2439] 1000 2439 580753 199493 3702784 33467 0 WebExtensions Jul 5 12:45:52 lx140e kernel: [ 2506] 1000 2506 502544 6479 2297856 26140 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2522] 1000 2522 487258 5958 2162688 32939 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2540] 1000 2540 391518 6258 1187840 16572 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2568] 1000 2568 374001 5480 999424 13446 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2588] 1000 2588 334799 1800 606208 9237 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2609] 1000 2609 394834 18653 1421312 8845 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2637] 1000 2637 477246 2833 2002944 28990 0 Web Content Jul 5 12:45:52 lx140e kernel: [ 2668] 1000 2668 324232 264 458752 4621 0 file:// Content Jul 5 12:45:52 lx140e kernel: [ 2950] 1000 2950 96219 0 106496 420 0 gvfsd-network Jul 5 12:45:52 lx140e kernel: [ 2956] 1000 2956 78059 0 98304 304 0 gvfsd-dnssd Jul 5 12:45:52 lx140e kernel: [ 3534] 1000 3534 542628 44649 1708032 54557 0 thunderbird Jul 5 12:45:52 lx140e kernel: [ 3620] 1000 3620 501449 12217 1355776 47513 0 thunderbird Jul 5 12:45:52 lx140e kernel: [ 5263] 1000 5263 4099 0 57344 477 0 bash Jul 5 12:45:52 lx140e kernel: [ 10691] 1000 10691 41149 0 77824 217 0 oosplash Jul 5 12:45:52 lx140e kernel: [ 10702] 1000 10702 230787 1419 868352 34489 0 soffice.bin Jul 5 12:45:52 lx140e kernel: [ 10791] 1000 10791 59329 0 90112 306 0 gnome-keyring-d Jul 5 12:45:52 lx140e kernel: [ 15171] 1000 15171 4099 1 65536 477 0 bash Jul 5 12:45:52 lx140e kernel: [ 15234] 1000 15234 11522 1 98304 389 0 su Jul 5 12:45:52 lx140e kernel: [ 15245] 0 15245 4165 0 61440 503 0 bash Jul 5 12:45:52 lx140e kernel: [ 18286] 1000 18286 108834 9 196608 3169 0 mousepad Jul 5 12:45:52 lx140e kernel: [ 27283] 1000 27283 327349 1582 303104 5542 0 Thunar Jul 5 12:45:52 lx140e kernel: [ 29902] 0 29902 3803 1 69632 479 0 dhclient Jul 5 12:45:52 lx140e kernel: [ 30566] 107 30566 882677 167476 3055616 119342 0 qemu-system-x86 Jul 5 12:45:52 lx140e kernel: [ 31714] 0 31714 65818 0 122880 503 0 abrtd Jul 5 12:45:52 lx140e kernel: [ 31718] 0 31718 25357 4 163840 461 0 abrt-dump-journ Jul 5 12:45:52 lx140e kernel: [ 31720] 0 31720 33548 0 221184 496 0 abrt-dump-journ Jul 5 12:45:52 lx140e kernel: [ 31736] 0 31736 25359 0 155648 489 0 abrt-dump-journ Jul 5 12:45:52 lx140e kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d3\x2dfedora21.scope,task=qemu-system-x86,pid=30566,uid=107 Jul 5 12:45:52 lx140e kernel: Out of memory: Killed process 30566 (qemu-system-x86) total-vm:3530708kB, anon-rss:669904kB, file-rss:0kB, shmem-rss:0kB Jul 5 12:45:52 lx140e kernel: oom_reaper: reaped process 30566 (qemu-system-x86), now anon-rss:0kB, file-rss:12kB, shmem-rss:0kB Jul 5 12:45:52 lx140e kernel: virbr0: port 2(vnet0) entered disabled state Jul 5 12:45:52 lx140e kernel: device vnet0 left promiscuous mode Jul 5 12:45:52 lx140e kernel: virbr0: port 2(vnet0) entered disabled state Jul 5 12:45:52 lx140e audit: ANOM_PROMISCUOUS dev=vnet0 prom=0 old_prom=256 auid=4294967295 uid=107 gid=107 ses=4294967295 Jul 5 12:45:52 lx140e journal[896]: internal error: End of file from agent monitor Jul 5 12:45:52 lx140e avahi-daemon[736]: Interface vnet0.IPv6 no longer relevant for mDNS. Jul 5 12:45:52 lx140e journal[896]: internal error: End of file from qemu monitor Jul 5 12:45:52 lx140e avahi-daemon[736]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fef6:b09b. Jul 5 12:45:52 lx140e systemd[1]: machine-qemu\x2d3\x2dfedora21.scope: Succeeded. Jul 5 12:45:52 lx140e avahi-daemon[736]: Withdrawing address record for fe80::fc54:ff:fef6:b09b on vnet0. Jul 5 12:45:52 lx140e NetworkManager[855]: <info> [1562345152.0966] device (vnet0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jul 5 12:45:52 lx140e NetworkManager[855]: <info> [1562345152.1127] device (virbr0): bridge port vnet0 was detached Jul 5 12:45:52 lx140e NetworkManager[855]: <info> [1562345152.1138] device (vnet0): released from master device virbr0 Jul 5 12:45:52 lx140e systemd[1]: Starting Network Manager Script Dispatcher Service... Jul 5 12:45:52 lx140e systemd-machined[784]: Machine qemu-3-fedora21 terminated. Jul 5 12:45:52 lx140e audit[896]: VIRT_CONTROL pid=896 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=stop reason=failed vm="fedora21" uuid=6b5e6ecd-b1cb-41f1-9d60-5f3060b88d0a vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' Jul 5 12:45:52 lx140e systemd[1]: Started Network Manager Script Dispatcher Service. Jul 5 12:45:52 lx140e audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 5 12:45:52 lx140e nm-dispatcher[5065]: req:1 'down' [vnet0]: new request (3 scripts) Jul 5 12:45:52 lx140e nm-dispatcher[5065]: req:1 'down' [vnet0]: start running ordered scripts... Jul 5 12:46:02 lx140e systemd[1]: NetworkManager-dispatcher.service: Succeeded. Jul 5 12:46:02 lx140e audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Qemu image was killed again. I went to restart it and get: Error starting domain: internal error: process exited while connecting to monitor: 2019-07-10T00:49:57.794355Z qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 66, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1420, in startup self._backend.create() File "/usr/lib64/python3.7/site-packages/libvirt.py", line 1089, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirt.libvirtError: internal error: process exited while connecting to monitor: 2019-07-10T00:49:57.794355Z qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory There was that big qemu update today that I installed, but tracer reported that all I had to do was restart the virt manager. That was hours prior to this event. So reboot time.
So far 3 crashes today. Note that all were before the hour. Jul 11 05:02:21 lx140e kernel: Out of memory: Killed process 2246 (firefox) total-vm:3670148kB, anon-rss:714756kB, file-rss:0kB, shmem-rss:2444kB Jul 12 03:20:16 lx140e kernel: Out of memory: Killed process 2188 (qemu-system-x86) total-vm:3561520kB, anon-rss:849436kB, file-rss:0kB, shmem-rss:0kB Jul 15 02:59:22 lx140e kernel: Out of memory: Killed process 2032 (qemu-system-x86) total-vm:3558696kB, anon-rss:682320kB, file-rss:0kB, shmem-rss:0kB Jul 16 01:59:15 lx140e kernel: Out of memory: Killed process 1124 (qemu-system-x86) total-vm:3568716kB, anon-rss:773260kB, file-rss:0kB, shmem-rss:0kB Jul 17 03:57:53 lx140e kernel: Out of memory: Killed process 3100 (qemu-system-x86) total-vm:3553296kB, anon-rss:621552kB, file-rss:0kB, shmem-rss:0kB Jul 17 07:52:40 lx140e kernel: Out of memory: Killed process 20080 (qemu-system-x86) total-vm:3533764kB, anon-rss:555388kB, file-rss:0kB, shmem-rss:0kB Jul 17 17:50:37 lx140e kernel: Out of memory: Killed process 21504 (qemu-system-x86) total-vm:3537876kB, anon-rss:726788kB, file-rss:0kB, shmem-rss:0kB Here is everything from messages at that 17:50:37 time: Jul 17 17:50:37 lx140e kernel: in:imjournal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 17 17:50:37 lx140e kernel: CPU: 1 PID: 21702 Comm: in:imjournal Tainted: P OE 5.1.17-300.fc30.x86_64 #1 Jul 17 17:50:37 lx140e kernel: Hardware name: LENOVO 20BLS00400/20BLS00400, BIOS GSET58WW (2.03 ) 02/24/2014 Jul 17 17:50:37 lx140e kernel: Call Trace: Jul 17 17:50:37 lx140e kernel: dump_stack+0x5c/0x80 Jul 17 17:50:37 lx140e kernel: dump_header+0x51/0x2ff Jul 17 17:50:37 lx140e kernel: ? oom_unkillable_task+0x4a/0xd0 Jul 17 17:50:37 lx140e kernel: ? oom_unkillable_task+0x9a/0xd0 Jul 17 17:50:37 lx140e kernel: oom_kill_process.cold+0xb/0x10 Jul 17 17:50:37 lx140e kernel: out_of_memory+0x1cd/0x470 Jul 17 17:50:37 lx140e kernel: __alloc_pages_slowpath+0xccb/0xdf0 Jul 17 17:50:37 lx140e kernel: __alloc_pages_nodemask+0x2f2/0x340 Jul 17 17:50:37 lx140e kernel: pagecache_get_page+0xb4/0x370 Jul 17 17:50:37 lx140e kernel: filemap_fault+0x702/0xad0 Jul 17 17:50:37 lx140e kernel: ? page_add_file_rmap+0xfa/0x2d0 Jul 17 17:50:37 lx140e kernel: ? alloc_set_pte+0x2d9/0x6b0 Jul 17 17:50:37 lx140e kernel: ? xas_load+0x5/0x70 Jul 17 17:50:37 lx140e kernel: ? xas_find+0x159/0x1a0 Jul 17 17:50:37 lx140e kernel: ? filemap_map_pages+0x1da/0x3a0 Jul 17 17:50:37 lx140e kernel: ext4_filemap_fault+0x2d/0x3f Jul 17 17:50:37 lx140e kernel: __do_fault+0x36/0x180 Jul 17 17:50:37 lx140e kernel: __handle_mm_fault+0x1091/0x1ac0 Jul 17 17:50:37 lx140e kernel: ? remove_wait_queue+0x20/0x60 Jul 17 17:50:37 lx140e kernel: handle_mm_fault+0xdc/0x210 Jul 17 17:50:37 lx140e kernel: do_user_addr_fault+0x1f6/0x450 Jul 17 17:50:37 lx140e kernel: do_page_fault+0x33/0x120 Jul 17 17:50:37 lx140e kernel: ? page_fault+0x8/0x30 Jul 17 17:50:37 lx140e kernel: page_fault+0x1e/0x30 Jul 17 17:50:37 lx140e kernel: RIP: 0033:0x7f20dab0cd9d Jul 17 17:50:37 lx140e kernel: Code: Bad RIP value. Jul 17 17:50:37 lx140e kernel: RSP: 002b:00007f20d0a99bd0 EFLAGS: 00010206 Jul 17 17:50:37 lx140e kernel: RAX: 00007f20d3f60000 RBX: 000055a75574aa20 RCX: 0000000000000001 Jul 17 17:50:37 lx140e kernel: RDX: 000000000000083c RSI: 000055a75574abd8 RDI: 000055a75574abc8 Jul 17 17:50:37 lx140e kernel: RBP: 0000000000000001 R08: 0000000000000013 R09: 00007ffc4e9970b0 Jul 17 17:50:37 lx140e kernel: R10: 00007ffc4e997080 R11: 0000000000000000 R12: 000055a755766460 Jul 17 17:50:37 lx140e kernel: R13: 0000000000000000 R14: 00007f20c4001f40 R15: 00007f20d0a99c00 Jul 17 17:50:37 lx140e kernel: Mem-Info: Jul 17 17:50:37 lx140e kernel: active_anon:477617 inactive_anon:225022 isolated_anon:0#012 active_file:50 inactive_file:39 isolated_file:0#012 unevictable:8 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:20615 slab_unreclaimable:30761#012 mapped:7832 shmem:7752 pagetables:9564 bounce:0#012 free:20412 free_pcp:24 free_cma:0 Jul 17 17:50:37 lx140e kernel: Node 0 active_anon:1910468kB inactive_anon:900088kB active_file:200kB inactive_file:156kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:31328kB dirty:0kB writeback:0kB shmem:31008kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 444416kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes Jul 17 17:50:37 lx140e kernel: Node 0 DMA free:12952kB min:328kB low:408kB high:488kB active_anon:2452kB inactive_anon:172kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:80kB pagetables:92kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jul 17 17:50:37 lx140e kernel: lowmem_reserve[]: 0 2747 3158 3158 3158 Jul 17 17:50:37 lx140e kernel: Node 0 DMA32 free:60052kB min:58508kB low:73132kB high:87756kB active_anon:1735352kB inactive_anon:819392kB active_file:268kB inactive_file:0kB unevictable:0kB writepending:0kB present:2910336kB managed:2844800kB mlocked:0kB kernel_stack:12256kB pagetables:33052kB bounce:0kB free_pcp:96kB local_pcp:48kB free_cma:0kB Jul 17 17:50:37 lx140e kernel: lowmem_reserve[]: 0 0 410 410 410 Jul 17 17:50:37 lx140e kernel: Node 0 Normal free:8644kB min:8740kB low:10924kB high:13108kB active_anon:172664kB inactive_anon:80524kB active_file:0kB inactive_file:0kB unevictable:32kB writepending:0kB present:507904kB managed:426628kB mlocked:32kB kernel_stack:2848kB pagetables:5112kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jul 17 17:50:37 lx140e kernel: lowmem_reserve[]: 0 0 0 0 0 Jul 17 17:50:37 lx140e kernel: Node 0 DMA: 2*4kB (U) 2*8kB (M) 4*16kB (UME) 4*32kB (UM) 5*64kB (UME) 1*128kB (E) 2*256kB (ME) 1*512kB (E) 3*1024kB (UME) 2*2048kB (ME) 1*4096kB (M) = 12952kB Jul 17 17:50:37 lx140e kernel: Node 0 DMA32: 662*4kB (UME) 179*8kB (UME) 81*16kB (UME) 233*32kB (UME) 99*64kB (UM) 57*128kB (UME) 45*256kB (UME) 12*512kB (UME) 15*1024kB (UM) 1*2048kB (M) 0*4096kB = 61536kB Jul 17 17:50:37 lx140e kernel: Node 0 Normal: 410*4kB (UME) 181*8kB (UME) 219*16kB (UME) 70*32kB (UME) 6*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9216kB Jul 17 17:50:37 lx140e kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jul 17 17:50:37 lx140e kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 17 17:50:37 lx140e kernel: 34445 total pagecache pages Jul 17 17:50:37 lx140e kernel: 26500 pages in swap cache Jul 17 17:50:37 lx140e kernel: Swap cache stats: add 14681407, delete 14654832, find 4730396/8843282 Jul 17 17:50:37 lx140e kernel: Free swap = 0kB Jul 17 17:50:37 lx140e kernel: Total swap = 3419132kB Jul 17 17:50:37 lx140e kernel: 858557 pages RAM Jul 17 17:50:37 lx140e kernel: 0 pages HighMem/MovableOnly Jul 17 17:50:37 lx140e kernel: 36724 pages reserved Jul 17 17:50:37 lx140e kernel: 0 pages cma reserved Jul 17 17:50:37 lx140e kernel: 0 pages hwpoisoned Jul 17 17:50:37 lx140e kernel: Tasks state (memory values in pages): Jul 17 17:50:37 lx140e kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jul 17 17:50:37 lx140e kernel: [ 540] 0 540 17370 112 143360 294 0 systemd-journal Jul 17 17:50:37 lx140e kernel: [ 565] 0 565 11928 3 98304 772 -1000 systemd-udevd Jul 17 17:50:37 lx140e kernel: [ 706] 0 706 24684 3 69632 192 -1000 auditd Jul 17 17:50:37 lx140e kernel: [ 708] 0 708 1589 19 49152 75 0 sedispatch Jul 17 17:50:37 lx140e kernel: [ 728] 81 728 67183 0 57344 213 -900 dbus-broker-lau Jul 17 17:50:37 lx140e kernel: [ 729] 81 729 1667 124 53248 346 -900 dbus-broker Jul 17 17:50:37 lx140e kernel: [ 730] 172 730 38231 0 65536 123 0 rtkit-daemon Jul 17 17:50:37 lx140e kernel: [ 732] 70 732 7785 35 65536 73 0 avahi-daemon Jul 17 17:50:37 lx140e kernel: [ 736] 0 736 10901 14 98304 380 0 sssd Jul 17 17:50:37 lx140e kernel: [ 739] 0 739 30574 1 147456 4985 0 firewalld Jul 17 17:50:37 lx140e kernel: [ 741] 0 741 4592 22 77824 363 0 systemd-machine Jul 17 17:50:37 lx140e kernel: [ 749] 0 749 1242 0 49152 74 0 alsactl Jul 17 17:50:37 lx140e kernel: [ 754] 0 754 78327 0 106496 420 0 ModemManager Jul 17 17:50:37 lx140e kernel: [ 755] 0 755 13457 1 81920 174 0 gssproxy Jul 17 17:50:37 lx140e kernel: [ 758] 0 758 78032 0 114688 334 0 rngd Jul 17 17:50:37 lx140e kernel: [ 761] 0 761 100301 243 126976 498 0 udisksd Jul 17 17:50:37 lx140e kernel: [ 768] 991 768 21758 13 57344 95 0 chronyd Jul 17 17:50:37 lx140e kernel: [ 771] 0 771 2655 31 57344 305 0 smartd Jul 17 17:50:37 lx140e kernel: [ 775] 0 775 2069 2 53248 108 0 bluetoothd Jul 17 17:50:37 lx140e kernel: [ 776] 0 776 19830 28 61440 39 0 irqbalance Jul 17 17:50:37 lx140e kernel: [ 787] 0 787 65819 0 126976 500 0 abrtd Jul 17 17:50:37 lx140e kernel: [ 795] 70 795 7753 0 61440 83 0 avahi-daemon Jul 17 17:50:37 lx140e kernel: [ 806] 0 806 11089 18 102400 597 0 sssd_be Jul 17 17:50:37 lx140e kernel: [ 809] 993 809 421618 90 212992 1725 0 polkitd Jul 17 17:50:37 lx140e kernel: [ 825] 0 825 62222 28 348160 451 0 abrt-dump-journ Jul 17 17:50:37 lx140e kernel: [ 834] 0 834 47890 33 253952 433 0 abrt-dump-journ Jul 17 17:50:37 lx140e kernel: [ 835] 0 835 47888 29 286720 436 0 abrt-dump-journ Jul 17 17:50:37 lx140e kernel: [ 836] 0 836 14279 96 135168 232 0 sssd_nss Jul 17 17:50:37 lx140e kernel: [ 886] 0 886 62496 0 110592 303 0 accounts-daemon Jul 17 17:50:37 lx140e kernel: [ 887] 0 887 7039 44 86016 521 0 systemd-logind Jul 17 17:50:37 lx140e kernel: [ 966] 0 966 84207 271 147456 506 0 NetworkManager Jul 17 17:50:37 lx140e kernel: [ 992] 0 992 10219 0 94208 462 0 cupsd Jul 17 17:50:37 lx140e kernel: [ 997] 0 997 6605 0 69632 216 -1000 sshd Jul 17 17:50:37 lx140e kernel: [ 1002] 0 1002 3932 14 61440 209 0 crond Jul 17 17:50:37 lx140e kernel: [ 1005] 0 1005 4540 12 53248 39 0 atd Jul 17 17:50:37 lx140e kernel: [ 1014] 986 1014 62200 1 114688 593 0 colord Jul 17 17:50:37 lx140e kernel: [ 1015] 0 1015 389726 440 413696 8801 0 libvirtd Jul 17 17:50:37 lx140e kernel: [ 1608] 977 1608 5700 30 57344 65 0 dnsmasq Jul 17 17:50:37 lx140e kernel: [ 1609] 0 1609 5693 18 57344 66 0 dnsmasq Jul 17 17:50:37 lx140e kernel: [ 2118] 0 2118 3053 55 61440 179 0 wpa_supplicant Jul 17 17:50:37 lx140e kernel: [ 2135] 0 2135 60068 0 94208 266 0 lightdm Jul 17 17:50:37 lx140e kernel: [ 2142] 0 2142 268352 10092 835584 10729 0 Xorg Jul 17 17:50:37 lx140e kernel: [ 2258] 0 2258 47774 1 102400 384 0 lightdm Jul 17 17:50:37 lx140e kernel: [ 2268] 1000 2268 5031 0 81920 465 0 systemd Jul 17 17:50:37 lx140e kernel: [ 2270] 1000 2270 33467 0 106496 1303 0 (sd-pam) Jul 17 17:50:37 lx140e kernel: [ 2275] 1000 2275 320824 0 135168 857 0 pulseaudio Jul 17 17:50:37 lx140e kernel: [ 2283] 1000 2283 68587 164 159744 1134 0 xfce4-session Jul 17 17:50:37 lx140e kernel: [ 2288] 1000 2288 69143 0 65536 119 0 dbus-broker-lau Jul 17 17:50:37 lx140e kernel: [ 2295] 1000 2295 1417 116 45056 101 0 dbus-broker Jul 17 17:50:37 lx140e kernel: [ 2321] 1000 2321 58165 0 90112 258 0 imsettings-daem Jul 17 17:50:37 lx140e kernel: [ 2324] 1000 2324 77653 0 102400 279 0 gvfsd Jul 17 17:50:37 lx140e kernel: [ 2398] 1000 2398 57496 117 77824 110 0 xfconfd Jul 17 17:50:37 lx140e kernel: [ 2462] 1000 2462 1777 9 53248 127 0 ssh-agent Jul 17 17:50:37 lx140e kernel: [ 2488] 1000 2488 76260 0 90112 234 0 at-spi-bus-laun Jul 17 17:50:37 lx140e kernel: [ 2493] 1000 2493 67078 0 57344 81 0 dbus-broker-lau Jul 17 17:50:37 lx140e kernel: [ 2494] 1000 2494 1321 0 49152 134 0 dbus-broker Jul 17 17:50:37 lx140e kernel: [ 2498] 1000 2498 40156 0 73728 212 0 at-spi2-registr Jul 17 17:50:37 lx140e kernel: [ 2503] 1000 2503 38681 48 65536 34 0 gpg-agent Jul 17 17:50:37 lx140e kernel: [ 2504] 1000 2504 225423 1381 442368 4014 0 xfwm4 Jul 17 17:50:37 lx140e kernel: [ 2516] 1000 2516 49173 179 143360 1202 0 xfsettingsd Jul 17 17:50:37 lx140e kernel: [ 2517] 1000 2517 74877 2247 208896 3279 0 xfce4-panel Jul 17 17:50:37 lx140e kernel: [ 2523] 1000 2523 229667 1809 282624 3364 0 Thunar Jul 17 17:50:37 lx140e kernel: [ 2528] 1000 2528 46792 559 139264 731 0 panel-16-xfce4- Jul 17 17:50:37 lx140e kernel: [ 2529] 1000 2529 113429 306 147456 1234 0 panel-17-pulsea Jul 17 17:50:37 lx140e kernel: [ 2530] 1000 2530 46382 166 139264 1086 0 panel-6-systray Jul 17 17:50:37 lx140e kernel: [ 2531] 1000 2531 48941 165 143360 1122 0 panel-2-actions Jul 17 17:50:37 lx140e kernel: [ 2546] 1000 2546 68484 0 176128 1743 0 xfce4-notifyd Jul 17 17:50:37 lx140e kernel: [ 2557] 1000 2557 78812 1550 188416 1886 0 xfdesktop Jul 17 17:50:37 lx140e kernel: [ 2563] 1000 2563 99685 258 122880 553 0 gvfs-udisks2-vo Jul 17 17:50:37 lx140e kernel: [ 2567] 1000 2567 58233 0 86016 232 0 gvfs-mtp-volume Jul 17 17:50:37 lx140e kernel: [ 2574] 1000 2574 77748 0 98304 353 0 gvfsd-trash Jul 17 17:50:37 lx140e kernel: [ 2579] 1000 2579 39895 0 77824 231 0 gvfsd-metadata Jul 17 17:50:37 lx140e kernel: [ 2587] 1000 2587 93926 300 192512 2485 0 xfce4-terminal Jul 17 17:50:37 lx140e kernel: [ 2594] 1000 2594 8283 246 90112 142 0 xscreensaver Jul 17 17:50:37 lx140e kernel: [ 2599] 1000 2599 68855 30 155648 1569 0 xfce-polkit Jul 17 17:50:37 lx140e kernel: [ 2601] 1000 2601 67185 0 151552 898 0 abrt-applet Jul 17 17:50:37 lx140e kernel: [ 2602] 1000 2602 86585 403 172032 1368 0 nm-applet Jul 17 17:50:37 lx140e kernel: [ 2603] 1000 2603 53668 31 196608 4101 0 seapplet Jul 17 17:50:37 lx140e kernel: [ 2607] 1000 2607 58559 0 86016 159 0 agent Jul 17 17:50:37 lx140e kernel: [ 2628] 1000 2628 49290 448 139264 1100 0 xfce4-power-man Jul 17 17:50:37 lx140e kernel: [ 2683] 1000 2683 4131 1 65536 490 0 bash Jul 17 17:50:37 lx140e kernel: [ 2687] 0 2687 62349 179 102400 210 0 upowerd Jul 17 17:50:37 lx140e kernel: [ 2707] 0 2707 65971 101 135168 437 0 abrt-dbus Jul 17 17:50:37 lx140e kernel: [ 2808] 1000 2808 4132 1 61440 503 0 bash Jul 17 17:50:37 lx140e kernel: [ 2869] 0 2869 3803 1 69632 478 0 dhclient Jul 17 17:50:37 lx140e kernel: [ 2982] 0 2982 152336 0 122880 573 0 pcscd Jul 17 17:50:37 lx140e kernel: [ 3004] 1000 3004 4099 1 49152 479 0 bash Jul 17 17:50:37 lx140e kernel: [ 3060] 1000 3060 488599 5354 860160 51787 0 virt-manager Jul 17 17:50:37 lx140e kernel: [ 3065] 1000 3065 39027 0 69632 201 0 dconf-service Jul 17 17:50:37 lx140e kernel: [ 3091] 0 3091 10348 0 102400 343 -900 virtlogd Jul 17 17:50:37 lx140e kernel: [ 3130] 1000 3130 140746 55 225280 3406 0 xfce4-appfinder Jul 17 17:50:37 lx140e kernel: [ 3142] 1000 3142 976823 106483 3801088 65873 0 firefox Jul 17 17:50:37 lx140e kernel: [ 3214] 1000 3214 630971 2477 860160 14138 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3277] 1000 3277 714235 108997 2310144 12518 0 WebExtensions Jul 17 17:50:37 lx140e kernel: [ 3342] 1000 3342 769483 4717 2211840 25081 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3358] 1000 3358 770807 3899 2260992 25555 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3404] 1000 3404 783135 5244 2613248 40900 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3424] 1000 3424 596874 1349 610304 10500 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3449] 1000 3449 647036 1364 966656 20334 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3473] 1000 3473 755611 1613 2158592 25732 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 3506] 1000 3506 589583 264 499712 5239 0 file:// Content Jul 17 17:50:37 lx140e kernel: [ 4944] 1000 4944 516887 23455 1437696 31019 0 thunderbird Jul 17 17:50:37 lx140e kernel: [ 6365] 1000 6365 782463 56929 3072000 38895 0 Web Content Jul 17 17:50:37 lx140e kernel: [ 6431] 1000 6431 24765 0 69632 216 0 oosplash Jul 17 17:50:37 lx140e kernel: [ 6442] 1000 6442 317808 7173 1294336 65190 0 soffice.bin Jul 17 17:50:37 lx140e kernel: [ 6499] 1000 6499 96226 0 110592 389 0 gvfsd-network Jul 17 17:50:37 lx140e kernel: [ 6505] 1000 6505 78059 15 98304 265 0 gvfsd-dnssd Jul 17 17:50:37 lx140e kernel: [ 6556] 1000 6556 104066 0 172032 2416 0 mousepad Jul 17 17:50:37 lx140e kernel: [ 9815] 1000 9815 59328 0 94208 289 0 gnome-keyring-d Jul 17 17:50:37 lx140e kernel: [ 11507] 1000 11507 77691 0 106496 275 0 gvfsd-burn Jul 17 17:50:37 lx140e kernel: [ 11548] 1000 11548 17494 0 118784 524 0 kdeinit5 Jul 17 17:50:37 lx140e kernel: [ 11549] 1000 11549 101377 55 241664 1889 0 klauncher Jul 17 17:50:37 lx140e kernel: [ 11705] 1000 11705 559797 71289 1908736 25881 0 thunderbird Jul 17 17:50:37 lx140e kernel: [ 12065] 1000 12065 569846 36678 1822720 78623 0 thunderbird Jul 17 17:50:37 lx140e kernel: [ 12270] 1000 12270 513805 33935 1466368 43543 0 thunderbird Jul 17 17:50:37 lx140e kernel: [ 21504] 107 21504 884469 181697 3055616 105928 0 qemu-system-x86 Jul 17 17:50:37 lx140e kernel: [ 21544] 1000 21544 4099 1 61440 476 0 bash Jul 17 17:50:37 lx140e kernel: [ 21574] 1000 21574 11522 1 94208 389 0 su Jul 17 17:50:37 lx140e kernel: [ 21609] 0 21609 4166 0 53248 504 0 bash Jul 17 17:50:37 lx140e kernel: [ 21700] 0 21700 91832 83 368640 91 0 rsyslogd Jul 17 17:50:37 lx140e kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d3\x2dfedora21.scope,task=qemu-system-x86,pid=21504,uid=107 Jul 17 17:50:37 lx140e kernel: Out of memory: Killed process 21504 (qemu-system-x86) total-vm:3537876kB, anon-rss:726788kB, file-rss:0kB, shmem-rss:0kB Jul 17 17:50:37 lx140e journal[1015]: internal error: End of file from agent monitor Jul 17 17:50:37 lx140e kernel: oom_reaper: reaped process 21504 (qemu-system-x86), now anon-rss:0kB, file-rss:12kB, shmem-rss:0kB Jul 17 17:50:37 lx140e journal[1015]: internal error: End of file from qemu monitor Jul 17 17:50:37 lx140e kernel: virbr0: port 2(vnet0) entered disabled state Jul 17 17:50:37 lx140e kernel: device vnet0 left promiscuous mode Jul 17 17:50:37 lx140e kernel: virbr0: port 2(vnet0) entered disabled state Jul 17 17:50:37 lx140e audit: ANOM_PROMISCUOUS dev=vnet0 prom=0 old_prom=256 auid=4294967295 uid=107 gid=107 ses=4294967295 Jul 17 17:50:37 lx140e avahi-daemon[732]: Interface vnet0.IPv6 no longer relevant for mDNS. Jul 17 17:50:37 lx140e avahi-daemon[732]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fef6:b09b. Jul 17 17:50:37 lx140e avahi-daemon[732]: Withdrawing address record for fe80::fc54:ff:fef6:b09b on vnet0. Jul 17 17:50:37 lx140e systemd[1]: machine-qemu\x2d3\x2dfedora21.scope: Succeeded. Jul 17 17:50:37 lx140e NetworkManager[966]: <info> [1563400237.4093] device (vnet0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jul 17 17:50:37 lx140e NetworkManager[966]: <info> [1563400237.4222] device (virbr0): bridge port vnet0 was detached Jul 17 17:50:37 lx140e NetworkManager[966]: <info> [1563400237.4223] device (vnet0): released from master device virbr0 Jul 17 17:50:37 lx140e systemd-machined[741]: Machine qemu-3-fedora21 terminated. Jul 17 17:50:37 lx140e systemd[1]: Starting Network Manager Script Dispatcher Service... Jul 17 17:50:37 lx140e systemd[1]: Started Network Manager Script Dispatcher Service. Jul 17 17:50:37 lx140e audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 17 17:50:37 lx140e nm-dispatcher[28682]: req:1 'down' [vnet0]: new request (3 scripts) Jul 17 17:50:37 lx140e nm-dispatcher[28682]: req:1 'down' [vnet0]: start running ordered scripts... Jul 17 17:50:37 lx140e audit[1015]: VIRT_CONTROL pid=1015 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=stop reason=failed vm="fedora21" uuid=6b5e6ecd-b1cb-41f1-9d60-5f3060b88d0a vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' J
The messages prior to this time were: Jul 17 17:40:33 lx140e audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jul 17 17:40:33 lx140e audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [root@lx140e log]# Nothing between 17:40 and 17:50
Just had a crash trying to restart my qemu image. The system was non-responsive for a few minutes: # grep "Jul 17 18:0" messages |head Jul 17 18:06:45 lx140e dnsmasq-dhcp[1608]: DHCPREQUEST(virbr0) 192.168.122.138 52:54:00:f6:b0:9b Jul 17 18:06:46 lx140e dnsmasq-dhcp[1608]: DHCPACK(virbr0) 192.168.122.138 52:54:00:f6:b0:9b Jul 17 18:07:10 lx140e systemd[2268]: dbus-:1.2-org.freedesktop.thumbnails.Cache1: Succeeded. # grep "Jul 17 18:1" messages |head Jul 17 18:10:29 lx140e journal[1015]: internal error: connection closed due to keepalive timeout Jul 17 18:11:34 lx140e kernel: qemu-system-x86 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 17 18:11:34 lx140e kernel: CPU: 0 PID: 28783 Comm: qemu-system-x86 Tainted: P OE 5.1.17-300.fc30.x86_64 #1 Jul 17 18:11:34 lx140e kernel: Hardware name: LENOVO 20BLS00400/20BLS00400, BIOS GSET58WW (2.03 ) 02/24/2014 Jul 17 18:11:34 lx140e kernel: Call Trace: Jul 17 18:11:34 lx140e kernel: dump_stack+0x5c/0x80 Jul 17 18:11:34 lx140e kernel: dump_header+0x51/0x2ff Jul 17 18:11:34 lx140e kernel: ? sched_clock+0x5/0x10 Jul 17 18:11:34 lx140e kernel: ? oom_unkillable_task+0x9a/0xd0 Jul 17 18:11:34 lx140e kernel: oom_kill_process.cold+0xb/0x10
Further, the crashes only seem to occur when I am away from my system. Keyboard idle and screen locked: Jul 17 03:57:53 lx140e kernel: Out of memory: Killed process 3100 (qemu-system-x86) total-vm:3553296kB, anon-rss:621552kB, file-rss:0kB, shmem-rss:0kB Jul 17 07:52:40 lx140e kernel: Out of memory: Killed process 20080 (qemu-system-x86) total-vm:3533764kB, anon-rss:555388kB, file-rss:0kB, shmem-rss:0kB Jul 17 17:50:37 lx140e kernel: Out of memory: Killed process 21504 (qemu-system-x86) total-vm:3537876kB, anon-rss:726788kB, file-rss:0kB, shmem-rss:0kB Jul 17 18:11:34 lx140e kernel: Out of memory: Killed process 28783 (qemu-system-x86) total-vm:3575052kB, anon-rss:1061096kB, file-rss:0kB, shmem-rss:0kB Jul 17 18:20:00 lx140e kernel: Out of memory: Killed process 29164 (qemu-system-x86) total-vm:3508064kB, anon-rss:1112360kB, file-rss:0kB, shmem-rss:0kB Jul 19 01:32:11 lx140e kernel: Out of memory: Killed process 1830 (qemu-system-x86) total-vm:3558440kB, anon-rss:715824kB, file-rss:0kB, shmem-rss:0kB Jul 19 07:32:03 lx140e kernel: Out of memory: Killed process 26847 (qemu-system-x86) total-vm:3542248kB, anon-rss:864756kB, file-rss:0kB, shmem-rss:0kB Jul 19 10:32:08 lx140e kernel: Out of memory: Killed process 28586 (qemu-system-x86) total-vm:3528624kB, anon-rss:778920kB, file-rss:0kB, shmem-rss:0kB Jul 19 13:32:04 lx140e kernel: Out of memory: Killed process 1997 (firefox) total-vm:4898936kB, anon-rss:968432kB, file-rss:0kB, shmem-rss:168kB All the above I was elsewhere for some time prior. The last 2 occurred while I was away for an extended time (dr appt.)
I have added 4GB of swap space via a swap file and the problem has stop. I just did not have enough space with 4GB real memory and 4GB swap partition. Unfortunately the messages were not clear and something (probably FIrefox) would grow, take up too much memory and the system nicely killed something (Firefox or QEMU) to get breathing room. So consider this closed and the conclusion is: You really do not have enough memory. Get more.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 30 kernel bugs. Fedora 30 has now been rebased to 5.2.9-200.fc30. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 31, and are still experiencing this issue, please change the version to Fedora 31. If you experience different issues, please open a new bug report for those.
Firefox on F30 is a memory hog, period. Not sure if this has more to do with the latest versions of firefox, fedora, gnome, or some combination(s) thereof. I'm seeing twice as much memory usage for firefox on F30 than on previous fedora versions. I noticed this primarily because I started looking into why memory usage under F30 was constantly hovering around 90%. On this particular system, I have 8GB of ram, and firefox is using anywhere between 1/5 to 1/4 of that at any given time, when I add up the process memory usage of firefox and all the "WebExtensions" and "Web Process" sub-processes / threads. Admittedly, I tend to keep a lot of tabs open, but I haven't changed my browsing habits in years, and I've never seen memory utilization this bad from firefox before. Incidentally, memory usage overall on F30 does not seem as efficient as under previous versions. Firefox is accounting for a lot of that, but not all. I also had a few epiphany incognito windows open, and that freed up about 3GB of ram when I closed those(!).
I am much better off with 8GB of actual memory and 4GB swap. Firefox still eats up memory and eventually crashes: $ free total used free shared buff/cache available Mem: 7403260 5967640 488288 315748 947332 859792 Swap: 3419132 2151988 1267144 Normally I get to a reboot after an update before it gets too bad. I just live with it. Sigh.
This message is a reminder that Fedora 30 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '30'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 30 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days