1262130 – system cannot be controlled at all

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1262130 - system cannot be controlled at all

Summary: system cannot be controlled at all

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	systemd
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	7.2
Assignee:	systemd-maint
QA Contact:	qe-baseos-daemons
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-09-10 21:42 UTC by Vladimir Benes
Modified:	2015-10-19 19:44 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-10-08 11:27:53 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Vladimir Benes 2015-09-10 21:42:47 UTC

Description of problem:
Systemd is somehow stuck with these errors in journal.

Sep 10 17:34:31 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:31 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com sshd[7564]: Accepted publickey for root from 10.36.6.63 port 40490 ssh2: RSA 7d:1c:b6:b3:91:3d:dd:e3:76:a9:d2:b8:e3:35:b0:24
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Requested transaction contradicts existing jobs: Transaction is destructive.
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com sshd[7564]: pam_systemd(sshd:session): Failed to create session: Resource deadlock avoided
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com sshd[7564]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd-logind[592]: Failed to start session scope session-19.scope: Transaction is destructive. org.freedesktop.systemd1.TransactionIsDestructive
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com dbus[593]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Sep 10 17:34:32 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com dbus[593]: [system] Successfully activated service 'org.freedesktop.problems'
Sep 10 17:34:33 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:34 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:36 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:38 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:39 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:41 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:43 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:44 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:46 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:47 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:47 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:47 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/pppoe-server.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:48 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:49 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:49 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:49 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:50 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:51 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/pppoe-server.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:51 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/pppoe-server.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:51 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:53 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:55 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:57 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:58 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:34:59 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:59 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:34:59 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Configuration file /usr/lib/systemd/system/pppoe-server.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 10 17:35:00 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:02 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:03 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:05 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:07 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:08 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com polkitd[627]: Registered Authentication Agent for unix-process:7619:4963079 (system bus name :1.423 [/usr/bin/pkttyagent --notify-fd 6 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Sep 10 17:35:08 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Requested transaction contradicts existing jobs: Transaction is destructive.
Sep 10 17:35:08 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com polkitd[627]: Unregistered Authentication Agent for unix-process:7619:4963079 (system bus name :1.423, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Sep 10 17:35:08 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:10 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:12 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:13 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:15 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:17 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:18 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:20 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:22 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:23 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
Sep 10 17:35:25 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.

I cannot restart NM service, I cannot reboot. Nothing. Thi s happened after a set of NM tests was executed. 


system is doing nothing
[root@qe-dell-ovs5-vm-16 ~]# systemctl list-jobs
  JOB UNIT                                                         TYPE  STATE  
22275 NetworkManager-wait-online.service                           stop  waiting
22569 brandbot.path                                                stop  waiting
22603 dev-disk-by\x2duuid...1\x2d4d9a\x2da9b2\x2dc04d3a43aa8f.swap stop  waiting
22592 systemd-hwdb-update.service                                  stop  waiting
22491 systemd-update-done.service                                  stop  waiting
22594 remote-fs.target                                             stop  waiting
22576 session-5.scope                                              stop  waiting
22508 systemd-vconsole-setup.service                               stop  waiting
21601 ntpd.service                                                 nop   waiting
22604 dev-mapper-rhel_qe\...dovs5\x2d\x2dvm\x2d\x2d16\x2dswap.swap stop  waiting
22562 tuned.service                                                stop  waiting
22510 systemd-sysctl.service                                       stop  waiting
22601 dev-disk-by\x2did-d...dovs5\x2d\x2dvm\x2d\x2d16\x2dswap.swap stop  waiting
22463 getty                                           stop  waiting
22614 plymouth-reboot.service                                      start waiting
22558 rsyslog.service                                              stop  waiting
22512 system-getty.slice                                           stop  waiting
22509 slices.target                                                stop  waiting
22493 beah-fwd-backend.service                                     stop  waiting
22574 session-12.scope                                             stop  waiting
22528 wpa_supplicant.service                                       stop  waiting
22516 session-14.scope                                             stop  waiting
22502 rhsmcertd.service                                            stop  waiting
22489 system-lvm2\x2dpvscan.slice                                  stop  waiting
22505 systemd-remount-fs.service                                   stop  waiting
22560 beah-srv.service                                             stop  waiting
22542 systemd-logind.service                                       stop  waiting
22543 multi-user.target                                            stop  waiting
22511 sockets.target                                               stop  waiting
22474 ldconfig.service                                             stop  waiting
22483 polkit.service                                               stop  waiting
22531 paths.target                                                 stop  waiting
22583 lvm2-lvmetad.service                                         stop  waiting
22612 boot.mount                                                   stop  waiting
22606 dev-dm\x2d1.swap                                             stop  waiting
22608 var-lib-nfs-rpc_pipefs.mount                                 stop  waiting
22454 reboot.target                                                start waiting
22595 network.target                                               stop  waiting
22506 systemd-tmpfiles-setup-dev.service                           stop  waiting
22605 dev-disk-by\x2did-d...ZJtyMwVcP4gwxOW2PPQ9BNnWdSZgbsjMt.swap stop  waiting
22581 getty.target                                                 stop  waiting
22529 nfs-client.target                                            stop  waiting
22577 session-16.scope                                             stop  waiting
22590 user-0.slice                                                 stop  waiting
22538 rhel-readonly.service                                        stop  waiting
22593 systemd-tmpfiles-setup.service                               stop  waiting
22551 dbus.socket                                                  stop  waiting
22532 lvm2-pvscan@252:2.service                                    stop  waiting
22579 systemd-ask-password-wall.path                               stop  waiting
22533 postfix.service                                              stop  waiting
22575 local-fs.target                                              stop  waiting
22570 remote-fs-pre.target                                         stop  waiting
22539 session-9.scope                                              stop  waiting
22540 session-13.scope                                             stop  waiting
22611 run-user-0.mount                                             stop  waiting
22487 sshd-keygen.service                                          stop  waiting
22552 basic.target                                                 stop  waiting
22546 swap.target                                                  stop  waiting
22541 cryptsetup.target                                            stop  waiting
22580 session-6.scope                                              stop  waiting
22514 time-sync.target                                             stop  waiting
22610 run-netns.mount                                              stop  waiting
22453 abrtd.service                                                nop   waiting
22523 session-11.scope                                             stop  waiting
22607 run-netns-vethsetup.mount                                    stop  waiting
22522 user.slice                                                   stop  waiting
22548 sysinit.target                                               stop  waiting
22527 systemd-journal-catalog-update.service                       stop  waiting
22503 session-8.scope                                              stop  waiting
22602 dev-rhel_qe\x2ddell\x2dovs5\x2dvm\x2d16-swap.swap            stop  waiting
22500 rhel-autorelabel-mark.service                                stop  waiting
22456 shutdown.target                                              start waiting
22486 sshd.service                                                 stop  waiting
22507 sound.target                                                 stop  waiting
22613 final.target                                                 start waiting
22557 gssproxy.service                                             stop  waiting
22488 system-serial\x2dgetty.slice                                 stop  waiting
22526 timers.target                                                stop  waiting
22600 umount.target                                                start waiting
22599 lvm2-monitor.service                                         stop  waiting
22537 systemd-ask-password-plymouth.path                           stop  waiting
22524 systemd-random-seed.service                                  stop  waiting
22565 atd.service                                                  stop  waiting
22457 dracut-shutdown.service                                      start waiting
22494 NetworkManager.service                                       stop  waiting
22534 dbus.service                                                 stop  waiting
22559 systemd-user-sessions.service                                stop  waiting
22568 crond.service                                                stop  waiting
22693 systemd-update-utmp.service                                  stop  waiting
22536 systemd-tmpfiles-clean.timer                                 stop  waiting
22530 serial-getty                                   stop  waiting
22561 NetworkManager-dispatcher.service                            stop  waiting
22586 session-7.scope                                              stop  waiting
22501 session-10.scope                                             stop  waiting
22455 systemd-reboot.service                                       start waiting
22598 session-15.scope                                             stop  waiting
22525 rhnsd.service                                                stop  waiting
22515 rhel-import-state.service                                    stop  waiting
22477 rpcbind.socket                                               stop  waiting
22513 rhel-dmesg.service                                           stop  waiting
22589 local-fs-pre.target                                          stop  waiting
22464 anamon.service                                               stop  waiting
22520 network-online.target                                        stop  waiting
22471 auditd.service                                               stop  waiting

104 jobs listed.
 
Version-Release number of selected component (if applicable):
systemd-219-13.el7.x86_64
kernel-3.10.0-315.el7.x86_64
dracut-033-339.el7.x86_64

x86_64 in a VM

How reproducible:
not sure

Comment 3 Vladimir Benes 2015-09-11 21:40:55 UTC

This may be connected. There is a test that connects two veth networks that are connected via bridge with stp on into another bridge without stp enabled.

console blocked with following messages and a crash:

[25286.337741] bridge0: received packet on eth2 with own address as source address
[25286.338629] bridge0: received packet on eth1 with own address as source address
[25286.339499] bridge0: received packet on eth2 with own address as source address
[25288.280824] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.281009] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.281009] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.281009]  0000000000200020 00000000d46be278 ffff88007fc03768 ffffffff81633971
[25288.281009]  ffff88007fc037f8 ffffffff8116de80 0000000000000010 0000000000000000
[25288.281009]  ffff88007ffd6800 00000000ffffffff 0000000000000001 00000000d46be278
[25288.281009] Call Trace:
[25288.281009]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.281009]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.281009]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.281009]  [<ffffffff810c523d>] ? trigger_load_balance+0x5d/0x200
[25288.281009]  [<ffffffff811237dd>] ? call_rcu_sched+0x1d/0x20
[25288.281009]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.281009]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.281009]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.281009]  [<ffffffff815314af>] ? dst_alloc+0x4f/0x170
[25288.281009]  [<ffffffff81526965>] ? netif_rx+0x45/0x120
[25288.281009]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.281009]  [<ffffffff815314af>] dst_alloc+0x4f/0x170
[25288.281009]  [<ffffffff8155f33c>] rt_dst_alloc+0x4c/0x50
[25288.281009]  [<ffffffff81561e7d>] ip_route_input_noref+0x10d/0xb90
[25288.281009]  [<ffffffff8103b283>] ? mce_chrdev_open+0x43/0x90
[25288.281009]  [<ffffffff8155aeb0>] ? nf_iterate+0x70/0xb0
[25288.281009]  [<ffffffff8156424f>] ip_rcv_finish+0x29f/0x350
[25288.281009]  [<ffffffff815649c6>] ip_rcv+0x2b6/0x410
[25288.281009]  [<ffffffff81563fb0>] ? inet_del_offload+0x40/0x40
[25288.281009]  [<ffffffff81528fe2>] __netif_receive_skb_core+0x582/0x7d0
[25288.281009]  [<ffffffffa043aa80>] ? br_multicast_flood+0x1b0/0x1b0 [bridge]
[25288.281009]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.281009]  [<ffffffff815292d0>] netif_receive_skb+0x40/0xc0
[25288.281009]  [<ffffffff8155af98>] ? nf_hook_slow+0xa8/0x110
[25288.281009]  [<ffffffff81529361>] netif_receive_skb_sk+0x11/0x20
[25288.281009]  [<ffffffffa043c035>] br_handle_frame_finish+0x205/0x420 [bridge]
[25288.281009]  [<ffffffff81529350>] ? netif_receive_skb+0xc0/0xc0
[25288.281009]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.281009]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.281009]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.281009]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.281009]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.281009]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.281009]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.281009]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.281009]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.281009]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.281009]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.281009]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.281009]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.281009]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.281009]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.281009]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.281009]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.281009]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.281009] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.281009]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.281009]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.322826] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.323611] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.323779] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.323779]  0000000000200020 00000000d46be278 ffff88007fc03918 ffffffff81633971
[25288.323779]  ffff88007fc039a8 ffffffff8116de80 ffffffff81644cdd ffff88007fc03938
[25288.323779]  ffff88007ffd6800 ffff88007fc03950 33203a736a626f20 00000000d46be278
[25288.323779] Call Trace:
[25288.323779]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.323779]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.323779]  [<ffffffff81644cdd>] ? apic_timer_interrupt+0x6d/0x80
[25288.323779]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.323779]  [<ffffffff81057b8f>] ? kvm_clock_get_cycles+0x1f/0x30
[25288.323779]  [<ffffffff810d776a>] ? __getnstimeofday64+0x3a/0xd0
[25288.323779]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.323779]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.323779]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.323779]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.323779]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.323779]  [<ffffffff81519539>] skb_clone+0x49/0xb0
[25288.323779]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.323779]  [<ffffffffa043a7e5>] deliver_clone+0x25/0x50 [bridge]
[25288.323779]  [<ffffffffa043aeb3>] br_flood+0xf3/0x150 [bridge]
[25288.323779]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.323779]  [<ffffffffa043b059>] br_flood_forward+0x19/0x20 [bridge]
[25288.323779]  [<ffffffffa043c0b7>] br_handle_frame_finish+0x287/0x420 [bridge]
[25288.323779]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.323779]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.323779]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.323779]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.323779]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.323779]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.323779]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.323779]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.323779]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.323779]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.323779]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.323779]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.323779]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.323779]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.323779]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.323779]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.323779]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.323779]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.323779] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.323779]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.323779]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.357413] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.358196] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.358394] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.358394]  0000000000200020 00000000d46be278 ffff88007fc03768 ffffffff81633971
[25288.358394]  ffff88007fc037f8 ffffffff8116de80 ffffffff810182df ffff88007c04bff8
[25288.358394]  ffff88007ffd6800 ffff88007c77fd90 ffffffff81888e1c 00000000d46be278
[25288.358394] Call Trace:
[25288.358394]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.358394]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.358394]  [<ffffffff810182df>] ? print_context_stack+0x8f/0x100
[25288.358394]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.358394]  [<ffffffff810adfe8>] ? __wake_up_common+0x58/0x90
[25288.358394]  [<ffffffff810a7661>] ? run_posix_cpu_timers+0x51/0x840
[25288.358394]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.358394]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.358394]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.358394]  [<ffffffff815314af>] ? dst_alloc+0x4f/0x170
[25288.358394]  [<ffffffff810dd42b>] ? clockevents_program_event+0x6b/0xf0
[25288.358394]  [<ffffffff810def84>] ? tick_program_event+0x24/0x30
[25288.358394]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.358394]  [<ffffffff815314af>] dst_alloc+0x4f/0x170
[25288.358394]  [<ffffffff8155f33c>] rt_dst_alloc+0x4c/0x50
[25288.358394]  [<ffffffff81561e7d>] ip_route_input_noref+0x10d/0xb90
[25288.358394]  [<ffffffff81630986>] ? __slab_alloc+0x3ba/0x48f
[25288.358394]  [<ffffffff8155aeb0>] ? nf_iterate+0x70/0xb0
[25288.358394]  [<ffffffff8156424f>] ip_rcv_finish+0x29f/0x350
[25288.358394]  [<ffffffff815649c6>] ip_rcv+0x2b6/0x410
[25288.358394]  [<ffffffff81563fb0>] ? inet_del_offload+0x40/0x40
[25288.358394]  [<ffffffff81528fe2>] __netif_receive_skb_core+0x582/0x7d0
[25288.358394]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.358394]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.358394]  [<ffffffff815292d0>] netif_receive_skb+0x40/0xc0
[25288.358394]  [<ffffffff8155af98>] ? nf_hook_slow+0xa8/0x110
[25288.358394]  [<ffffffff81529361>] netif_receive_skb_sk+0x11/0x20
[25288.358394]  [<ffffffffa043c035>] br_handle_frame_finish+0x205/0x420 [bridge]
[25288.358394]  [<ffffffff81529350>] ? netif_receive_skb+0xc0/0xc0
[25288.358394]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.358394]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.358394]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.358394]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.358394]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.358394]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.358394]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.358394]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.358394]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.358394]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.358394]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.358394]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.358394]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.358394]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.358394]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.358394]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.358394]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.358394]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.358394] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.358394]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.358394]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.399187] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.399938] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.400162] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.400162]  0000000000200020 00000000d46be278 ffff88007fc03918 ffffffff81633971
[25288.400162]  ffff88007fc039a8 ffffffff8116de80 ffffffff81644cdd ffff88007fc03938
[25288.400162]  ffff88007ffd6800 ffff88007fc03950 33203a736a626f20 00000000d46be278
[25288.400162] Call Trace:
[25288.400162]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.400162]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.400162]  [<ffffffff81644cdd>] ? apic_timer_interrupt+0x6d/0x80
[25288.400162]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.400162]  [<ffffffff81057b8f>] ? kvm_clock_get_cycles+0x1f/0x30
[25288.400162]  [<ffffffff810d776a>] ? __getnstimeofday64+0x3a/0xd0
[25288.400162]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.400162]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.400162]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.400162]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.400162]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.400162]  [<ffffffff81519539>] skb_clone+0x49/0xb0
[25288.400162]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.400162]  [<ffffffffa043a7e5>] deliver_clone+0x25/0x50 [bridge]
[25288.400162]  [<ffffffffa043aeb3>] br_flood+0xf3/0x150 [bridge]
[25288.400162]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.400162]  [<ffffffffa043b059>] br_flood_forward+0x19/0x20 [bridge]
[25288.400162]  [<ffffffffa043c0b7>] br_handle_frame_finish+0x287/0x420 [bridge]
[25288.400162]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.400162]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.400162]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.400162]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.400162]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.400162]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.400162]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.400162]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.400162]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.400162]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.400162]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.400162]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.400162]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.400162]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.400162]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.400162]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.400162]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.400162]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.400162] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.400162]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.400162]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.433136] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.433909] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.434121] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.434121]  0000000000200020 00000000d46be278 ffff88007fc03768 ffffffff81633971
[25288.434121]  ffff88007fc037f8 ffffffff8116de80 ffffffff810182df ffff88007c04bff8
[25288.434121]  ffff88007ffd6800 ffff88007c77fd90 ffffffff81888e1c 00000000d46be278
[25288.434121] Call Trace:
[25288.434121]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.434121]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.434121]  [<ffffffff810182df>] ? print_context_stack+0x8f/0x100
[25288.434121]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.434121]  [<ffffffff810adfe8>] ? __wake_up_common+0x58/0x90
[25288.434121]  [<ffffffff810a7661>] ? run_posix_cpu_timers+0x51/0x840
[25288.434121]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.434121]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.434121]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.434121]  [<ffffffff810d755c>] ? ktime_get+0x4c/0xd0
[25288.434121]  [<ffffffff815314af>] ? dst_alloc+0x4f/0x170
[25288.434121]  [<ffffffff81048a5d>] ? lapic_next_event+0x1d/0x30
[25288.434121]  [<ffffffff810dd42b>] ? clockevents_program_event+0x6b/0xf0
[25288.434121]  [<ffffffff810def84>] ? tick_program_event+0x24/0x30
[25288.434121]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.434121]  [<ffffffff815314af>] dst_alloc+0x4f/0x170
[25288.434121]  [<ffffffff8155f33c>] rt_dst_alloc+0x4c/0x50
[25288.434121]  [<ffffffff81561e7d>] ip_route_input_noref+0x10d/0xb90
[25288.434121]  [<ffffffff81630986>] ? __slab_alloc+0x3ba/0x48f
[25288.434121]  [<ffffffff8155aeb0>] ? nf_iterate+0x70/0xb0
[25288.434121]  [<ffffffff8156424f>] ip_rcv_finish+0x29f/0x350
[25288.434121]  [<ffffffff815649c6>] ip_rcv+0x2b6/0x410
[25288.434121]  [<ffffffff81563fb0>] ? inet_del_offload+0x40/0x40
[25288.434121]  [<ffffffff81528fe2>] __netif_receive_skb_core+0x582/0x7d0
[25288.434121]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.434121]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.434121]  [<ffffffff815292d0>] netif_receive_skb+0x40/0xc0
[25288.434121]  [<ffffffff8155af98>] ? nf_hook_slow+0xa8/0x110
[25288.434121]  [<ffffffff81529361>] netif_receive_skb_sk+0x11/0x20
[25288.434121]  [<ffffffffa043c035>] br_handle_frame_finish+0x205/0x420 [bridge]
[25288.434121]  [<ffffffff81529350>] ? netif_receive_skb+0xc0/0xc0
[25288.434121]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.434121]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.434121]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.434121]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.434121]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.434121]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.434121]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.434121]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.434121]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.434121]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.434121]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.434121]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.434121]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.434121]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.434121]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.434121]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.434121]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.434121]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.434121] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.434121]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.434121]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.477644] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.478418] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.478501] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.478501]  0000000000200020 00000000d46be278 ffff88007fc03918 ffffffff81633971
[25288.478501]  ffff88007fc039a8 ffffffff8116de80 ffff8800001a4f00 ffffffff815958c0
[25288.478501]  ffff88007ffd6800 ffffffff815164a7 ffff88007fc03970 00000000d46be278
[25288.478501] Call Trace:
[25288.478501]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.478501]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.478501]  [<ffffffff815958c0>] ? __udp4_lib_mcast_deliver+0x290/0x3c0
[25288.478501]  [<ffffffff815164a7>] ? kfree_skbmem+0x37/0x90
[25288.478501]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.478501]  [<ffffffff81057b8f>] ? kvm_clock_get_cycles+0x1f/0x30
[25288.478501]  [<ffffffff810d776a>] ? __getnstimeofday64+0x3a/0xd0
[25288.478501]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.478501]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.478501]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.478501]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.478501]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.478501]  [<ffffffff81519539>] skb_clone+0x49/0xb0
[25288.478501]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.478501]  [<ffffffffa043a7e5>] deliver_clone+0x25/0x50 [bridge]
[25288.478501]  [<ffffffffa043aeb3>] br_flood+0xf3/0x150 [bridge]
[25288.478501]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.478501]  [<ffffffffa043b059>] br_flood_forward+0x19/0x20 [bridge]
[25288.478501]  [<ffffffffa043c0b7>] br_handle_frame_finish+0x287/0x420 [bridge]
[25288.478501]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.478501]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.478501]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.478501]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.478501]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.478501]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.478501]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.478501]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.478501]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.478501]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.478501]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.478501]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.478501]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.478501]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.478501]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.478501]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.478501]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.478501] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.478501]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.478501]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.511411] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.512158] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.512389] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.512389]  0000000000200020 00000000d46be278 ffff88007fc03918 ffffffff81633971
[25288.512389]  ffff88007fc039a8 ffffffff8116de80 0000000000000000 ffff88007fc03930
[25288.512389]  ffff88007ffd6800 000000fb20ea0940 ffff88007c7af300 00000000d46be278
[25288.512389] Call Trace:
[25288.512389]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.512389]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.512389]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.512389]  [<ffffffff81057b8f>] ? kvm_clock_get_cycles+0x1f/0x30
[25288.512389]  [<ffffffff810d776a>] ? __getnstimeofday64+0x3a/0xd0
[25288.512389]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.512389]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.512389]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.512389]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.512389]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.512389]  [<ffffffff81519539>] skb_clone+0x49/0xb0
[25288.512389]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.512389]  [<ffffffffa043a7e5>] deliver_clone+0x25/0x50 [bridge]
[25288.512389]  [<ffffffffa043aeb3>] br_flood+0xf3/0x150 [bridge]
[25288.512389]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.512389]  [<ffffffffa043b059>] br_flood_forward+0x19/0x20 [bridge]
[25288.512389]  [<ffffffffa043c0b7>] br_handle_frame_finish+0x287/0x420 [bridge]
[25288.512389]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.512389]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.512389]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.512389]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.512389]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.512389]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.512389]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.512389]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.512389]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.512389]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.512389]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.512389]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.512389]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.512389]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.512389]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.512389]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.512389]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.512389] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.512389]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.512389]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.543021] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.543762] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.544008] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.544008]  0000000000200020 00000000d46be278 ffff88007fc03768 ffffffff81633971
[25288.544008]  ffff88007fc037f8 ffffffff8116de80 ffffffff810182df ffff88007c04bff8
[25288.544008]  ffff88007ffd6800 ffff88007c77fd90 ffffffff81888e1c 00000000d46be278
[25288.544008] Call Trace:
[25288.544008]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.544008]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.544008]  [<ffffffff810182df>] ? print_context_stack+0x8f/0x100
[25288.544008]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.544008]  [<ffffffff810adfe8>] ? __wake_up_common+0x58/0x90
[25288.544008]  [<ffffffff810a7661>] ? run_posix_cpu_timers+0x51/0x840
[25288.544008]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.544008]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.544008]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.544008]  [<ffffffff815314af>] ? dst_alloc+0x4f/0x170
[25288.544008]  [<ffffffff810dd42b>] ? clockevents_program_event+0x6b/0xf0
[25288.544008]  [<ffffffff810def84>] ? tick_program_event+0x24/0x30
[25288.544008]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.544008]  [<ffffffff815314af>] dst_alloc+0x4f/0x170
[25288.544008]  [<ffffffff810c523d>] ? trigger_load_balance+0x5d/0x200
[25288.544008]  [<ffffffff8155f33c>] rt_dst_alloc+0x4c/0x50
[25288.544008]  [<ffffffff81561e7d>] ip_route_input_noref+0x10d/0xb90
[25288.544008]  [<ffffffff810df5c0>] ? tick_sched_handle.isra.14+0x60/0x60
[25288.544008]  [<ffffffff812fa760>] ? timerqueue_add+0x60/0xb0
[25288.544008]  [<ffffffff8155aeb0>] ? nf_iterate+0x70/0xb0
[25288.544008]  [<ffffffff8156424f>] ip_rcv_finish+0x29f/0x350
[25288.544008]  [<ffffffff815649c6>] ip_rcv+0x2b6/0x410
[25288.544008]  [<ffffffff81563fb0>] ? inet_del_offload+0x40/0x40
[25288.544008]  [<ffffffff81528fe2>] __netif_receive_skb_core+0x582/0x7d0
[25288.544008]  [<ffffffff81646615>] ? smp_apic_timer_interrupt+0x45/0x60
[25288.544008]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.544008]  [<ffffffff815292d0>] netif_receive_skb+0x40/0xc0
[25288.544008]  [<ffffffff8155af98>] ? nf_hook_slow+0xa8/0x110
[25288.544008]  [<ffffffff81529361>] netif_receive_skb_sk+0x11/0x20
[25288.544008]  [<ffffffffa043c035>] br_handle_frame_finish+0x205/0x420 [bridge]
[25288.544008]  [<ffffffff81529350>] ? netif_receive_skb+0xc0/0xc0
[25288.544008]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.544008]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.544008]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.544008]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.544008]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.544008]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.544008]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.544008]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.544008]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.544008]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.544008]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.544008]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.544008]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.544008]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.544008]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.544008]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.544008]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.544008] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.544008]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.544008]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.585661] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.586517] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.586637] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.586637]  0000000000200020 00000000d46be278 ffff88007fc03918 ffffffff81633971
[25288.586637]  ffff88007fc039a8 ffffffff8116de80 ffffffff81644cdd ffff88007fc03938
[25288.586637]  ffff88007ffd6800 ffff88007fc03950 33203a736a626f20 00000000d46be278
[25288.586637] Call Trace:
[25288.586637]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.586637]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.586637]  [<ffffffff81644cdd>] ? apic_timer_interrupt+0x6d/0x80
[25288.586637]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.586637]  [<ffffffff81057b8f>] ? kvm_clock_get_cycles+0x1f/0x30
[25288.586637]  [<ffffffff810d776a>] ? __getnstimeofday64+0x3a/0xd0
[25288.586637]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.586637]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.586637]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.586637]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.586637]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.586637]  [<ffffffff81519539>] skb_clone+0x49/0xb0
[25288.586637]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.586637]  [<ffffffffa043a7e5>] deliver_clone+0x25/0x50 [bridge]
[25288.586637]  [<ffffffffa043aeb3>] br_flood+0xf3/0x150 [bridge]
[25288.586637]  [<ffffffffa043ab30>] ? br_forward_finish+0xb0/0xb0 [bridge]
[25288.586637]  [<ffffffffa043b059>] br_flood_forward+0x19/0x20 [bridge]
[25288.586637]  [<ffffffffa043c0b7>] br_handle_frame_finish+0x287/0x420 [bridge]
[25288.586637]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.586637]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.586637]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.586637]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.586637]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.586637]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.586637]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.586637]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.586637]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.586637]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.586637]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.586637]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.586637]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.586637]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.586637]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.586637]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.586637]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.586637] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.586637]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.586637]   node 0: slabs: 199031, objs: 3184496, free: 0
[25288.619431] rcuos/0: page allocation failure: order:0, mode:0x200020
[25288.620198] CPU: 0 PID: 11 Comm: rcuos/0 Not tainted 3.10.0-315.el7.x86_64 #1
[25288.620417] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[25288.620417]  0000000000200020 00000000d46be278 ffff88007fc03768 ffffffff81633971
[25288.620417]  ffff88007fc037f8 ffffffff8116de80 ffffffff810182df ffff88007c04bff8
[25288.620417]  ffff88007ffd6800 ffff88007c77fd90 ffffffff81888e1c 00000000d46be278
[25288.620417] Call Trace:
[25288.620417]  <IRQ>  [<ffffffff81633971>] dump_stack+0x19/0x1b
[25288.620417]  [<ffffffff8116de80>] warn_alloc_failed+0x110/0x180
[25288.620417]  [<ffffffff810182df>] ? print_context_stack+0x8f/0x100
[25288.620417]  [<ffffffff81172608>] __alloc_pages_nodemask+0x9a8/0xb90
[25288.620417]  [<ffffffff810adfe8>] ? __wake_up_common+0x58/0x90
[25288.620417]  [<ffffffff810a7661>] ? run_posix_cpu_timers+0x51/0x840
[25288.620417]  [<ffffffff811b3109>] alloc_pages_current+0xa9/0x170
[25288.620417]  [<ffffffff811bd6fc>] new_slab+0x2ec/0x300
[25288.620417]  [<ffffffff816308e1>] __slab_alloc+0x315/0x48f
[25288.620417]  [<ffffffff815314af>] ? dst_alloc+0x4f/0x170
[25288.620417]  [<ffffffff810dd42b>] ? clockevents_program_event+0x6b/0xf0
[25288.620417]  [<ffffffff810def84>] ? tick_program_event+0x24/0x30
[25288.620417]  [<ffffffff811bfcc3>] kmem_cache_alloc+0x193/0x1d0
[25288.620417]  [<ffffffff815314af>] dst_alloc+0x4f/0x170
[25288.620417]  [<ffffffff8155f33c>] rt_dst_alloc+0x4c/0x50
[25288.620417]  [<ffffffff81561e7d>] ip_route_input_noref+0x10d/0xb90
[25288.620417]  [<ffffffff81630986>] ? __slab_alloc+0x3ba/0x48f
[25288.620417]  [<ffffffff8155aeb0>] ? nf_iterate+0x70/0xb0
[25288.620417]  [<ffffffff8156424f>] ip_rcv_finish+0x29f/0x350
[25288.620417]  [<ffffffff815649c6>] ip_rcv+0x2b6/0x410
[25288.620417]  [<ffffffff81563fb0>] ? inet_del_offload+0x40/0x40
[25288.620417]  [<ffffffff81528fe2>] __netif_receive_skb_core+0x582/0x7d0
[25288.620417]  [<ffffffff81519539>] ? skb_clone+0x49/0xb0
[25288.620417]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.620417]  [<ffffffff815292d0>] netif_receive_skb+0x40/0xc0
[25288.620417]  [<ffffffff8155af98>] ? nf_hook_slow+0xa8/0x110
[25288.620417]  [<ffffffff81529361>] netif_receive_skb_sk+0x11/0x20
[25288.620417]  [<ffffffffa043c035>] br_handle_frame_finish+0x205/0x420 [bridge]
[25288.620417]  [<ffffffff81529350>] ? netif_receive_skb+0xc0/0xc0
[25288.620417]  [<ffffffffa043c3f5>] br_handle_frame+0x1a5/0x2d0 [bridge]
[25288.620417]  [<ffffffffa043be30>] ? br_handle_local_finish+0xa0/0xa0 [bridge]
[25288.620417]  [<ffffffff81528c42>] __netif_receive_skb_core+0x1e2/0x7d0
[25288.620417]  [<ffffffff81529248>] __netif_receive_skb+0x18/0x60
[25288.620417]  [<ffffffff81529f1e>] process_backlog+0xae/0x180
[25288.620417]  [<ffffffff81529702>] net_rx_action+0x152/0x240
[25288.620417]  [<ffffffff81083acf>] __do_softirq+0xef/0x280
[25288.620417]  [<ffffffff8164599c>] call_softirq+0x1c/0x30
[25288.620417]  <EOI>  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
[25288.620417]  [<ffffffff810833c4>] local_bh_enable+0x94/0xa0
[25288.620417]  [<ffffffff81122ef2>] rcu_nocb_kthread+0x232/0x370
[25288.620417]  [<ffffffff810a5aa0>] ? wake_up_atomic_t+0x30/0x30
[25288.620417]  [<ffffffff81122cc0>] ? rcu_start_gp+0x40/0x40
[25288.620417]  [<ffffffff810a4aaf>] kthread+0xcf/0xe0
[25288.620417]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.620417]  [<ffffffff81643fd8>] ret_from_fork+0x58/0x90
[25288.620417]  [<ffffffff810a49e0>] ? kthread_create_on_node+0x140/0x140
[25288.620417] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[25288.620417]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[25288.620417]   node 0: slabs: 199031, objs: 3184496, free: 0
[25291.333021] net_ratelimit: 1040685 callbacks suppressed
[25291.333668] bridge0: received packet on eth2 with own address as source address

Comment 4 Vladimir Benes 2015-09-11 22:19:42 UTC

I suppose system should be more robust to avoid crashes in bad configuration scenarios. Something like this happened.

 --------------------------------------------+-----------------------------------
                          network namespace  | normal system's area 
                                             |
DHCP -> veth pair -> bridge (STP On) -> veth | pair -> bridge (STP Off)
                                     -> veth | pair -> 
                                             |
                                             |

Comment 5 Michal Sekletar 2015-09-14 12:45:16 UTC

I tried to reproduce the scenario as described in comment #4 and I was able to create packet storm, but I was unable to crash the system. Kernel task ksoftirqd consumes a lot of CPU time and logs are full of following records, 

br0: received packet on veth2 with own address as source address
br0: received packet on veth4 with own address as source address.

However system is still operational.

[0 root@qeos-63 test]# uname -r
3.10.0-306.0.1.el7.x86_64

[0 root@qeos-63 test]# rpm -q systemd
systemd-219-11.el7.x86_64

Can you please provide more specific reproducer? Also removing blocker flag for now since it is not obvious to me that systemd is to blame here.

Comment 7 Michal Sekletar 2015-09-14 12:55:30 UTC

Arghh, immediately after posting previous comment I checked dmesg output again I can see stack traces from kernel. However system is still operational and systemd stays responsive.

Comment 8 Lukáš Nykrýn 2015-10-06 12:09:33 UTC

Can you try this with latest build of systemd? 219-18. 
The issue
Sep 10 17:34:34 qe-dell-ovs5-vm-16.dqe.lab.eng.bos.redhat.com systemd[1]: Looping too fast. Throttling execution a little.
has been fixed.

Comment 9 Lukáš Nykrýn 2015-10-08 11:27:53 UTC

I really believe that the issue was fixed in latest build of systemd. Closing as current release. Feel free to reopen the bug, if you are able to reproduce this with current version.

Comment 10 Vladimir Benes 2015-10-19 19:44:40 UTC

seems like not reproducible anymore

Note You need to log in before you can comment on or make changes to this bug.