Bug 662932

Summary: failed to allocate new chunk when set lots of vlan in RHEL6 32 bit guest
Product: Red Hat Enterprise Linux 5 Reporter: Joy Pu <ypu>
Component: kernelAssignee: Herbert Xu <herbert.xu>
Status: CLOSED CANTFIX QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.6CC: akong, mkenneth, riel, tburke, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-12-16 19:25:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580948    
Attachments:
Description Flags
Whole serial log of guest none

Description Joy Pu 2010-12-14 08:37:53 UTC 
Description:
If set 4094 vlans in the same guest. It will failed around 1560. The kernel will get this message(The same test in 64 bit guest of RHEL 6 is pass):
2010-12-10 11:53:25: 802.1Q VLAN Support v1.8 Ben Greear <greearb>
2010-12-10 11:53:25: All bugs added by David S. Miller <davem>
2010-12-10 12:04:09: PERCPU: allocation failed, size=2048 align=8, failed to allocate new chunk
2010-12-10 12:04:09: Pid: 5561, comm: vconfig Not tainted 2.6.32-71.7.1.el6.i686 #1
2010-12-10 12:04:09: Call Trace:
2010-12-10 12:04:09:  [<c05164db>] ? pcpu_alloc+0x85b/0x8d0
2010-12-10 12:04:09:  [<c04f148b>] ? kmemdup+0x1b/0x40
2010-12-10 12:04:09:  [<c080d808>] ? _write_lock_bh+0x8/0x20
2010-12-10 12:04:09:  [<c07bf5a6>] ? snmp_mib_init+0x16/0x60
2010-12-10 12:04:09:  [<f8b1307d>] ? ipv6_add_dev+0x13d/0x3a0 [ipv6]
2010-12-10 12:04:09:  [<f8b1393f>] ? addrconf_notify+0x5f/0x900 [ipv6]
2010-12-10 12:04:09:  [<c07c2a21>] ? ip_mc_init_dev+0x51/0x90
2010-12-10 12:04:09:  [<c07be305>] ? inetdev_init+0xb5/0x180
2010-12-10 12:04:09:  [<c07be817>] ? inetdev_event+0x3c7/0x490
2010-12-10 12:04:09:  [<c069d600>] ? get_device+0x10/0x20
2010-12-10 12:04:09:  [<c07f76f4>] ? klist_node_init+0x34/0x50
2010-12-10 12:04:09:  [<c05116c0>] ? kmem_cache_alloc_notrace+0xa0/0xb0
2010-12-10 12:04:09:  [<c077f1e9>] ? dropmon_net_event+0x79/0x170
2010-12-10 12:04:09:  [<c07e7f5e>] ? packet_notifier+0x1e/0x190
2010-12-10 12:04:09:  [<c080fe44>] ? notifier_call_chain+0x44/0x60
2010-12-10 12:04:09:  [<c0475ac7>] ? raw_notifier_call_chain+0x17/0x20
2010-12-10 12:04:09:  [<c0770232>] ? register_netdevice+0x292/0x350
2010-12-10 12:04:09:  [<c076f261>] ? alloc_netdev_mq+0xa1/0x1e0
2010-12-10 12:04:09:  [<f7ea59f2>] ? register_vlan_dev+0xa2/0x300 [8021q]
2010-12-10 12:04:09:  [<f7ea6270>] ? vlan_setup+0x0/0x60 [8021q]
2010-12-10 12:04:09:  [<f7ea5f93>] ? vlan_ioctl_handler+0x343/0x3f0 [8021q]
2010-12-10 12:04:09:  [<f7ea5c50>] ? vlan_ioctl_handler+0x0/0x3f0 [8021q]
2010-12-10 12:04:09:  [<c075c433>] ? sock_ioctl+0x203/0x260
2010-12-10 12:04:09:  [<c075c230>] ? sock_ioctl+0x0/0x260
2010-12-10 12:04:09:  [<c052dcbb>] ? vfs_ioctl+0x1b/0xa0
2010-12-10 12:04:09:  [<c052de9c>] ? do_vfs_ioctl+0x6c/0x5c0
2010-12-10 12:04:09:  [<c052e466>] ? sys_ioctl+0x76/0x90
2010-12-10 12:04:09:  [<c04099fb>] ? sysenter_do_call+0x12/0x28
2010-12-10 12:10:25: initctl: Event failed

Version-Release number of selected component (if applicable):
Host: 2.6.18-236.el5
Guest: 2.6.32-71.7.1.el6
Qemu:
rpm -qa |grep kvm
kmod-kvm-83-222.el5
etherboot-roms-kvm-5.4.4-13.el5
kmod-kvm-debug-83-222.el5
etherboot-zroms-kvm-5.4.4-13.el5
kvm-qemu-img-83-222.el5
kvm-tools-83-222.el5
kvm-debuginfo-83-222.el5
kvm-83-222.el5


How reproducible:
Always

Steps to Reproduce:
1.Boot up a rhel 6 32 bit guest
2.Set vlan in guest 
# /etc/init.d/iptables stop
# modprobe 8021q
# for i in `seq 4094`;do vconfig add eth0 $i; done



Actual results:
vlan add operate failed aroud 1560
Expected results:
Can set max number of vlan in guest

Additional info:
1.Host cpuinfo
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
stepping	: 4
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 5490.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management: [8]

2. cmdline
# /usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/qemu -name 'vm1' -monitor unix:'/tmp/monitor-humanmonitor1-20101210-113729-8lxP',server,nowait -serial unix:'/tmp/serial-20101210-113729-8lxP',server,nowait -drive file='/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/RHEL-Server-6.0-32-virtio.qcow2',index=0,if=ide,media=disk,cache=none,snapshot=on,format=qcow2 -net nic,vlan=0,model=virtio,macaddr='9a:16:10:0b:e0:0a' -net tap,vlan=0,ifname='t0-113729-8lxP',script='/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu qemu64,+sse2 -soundhw ac97 -vnc :0 -spice port=8000,disable-ticketing -qxl 1 -rtc-td-hack -M rhel5.5.0 -usbdevice tablet -no-kvm-pit-reinjection
Comment 1 Joy Pu 2010-12-14 09:02:59 UTC 
Created attachment 468561 [details]
Whole serial log of guest
Comment 3 Michael S. Tsirkin 2010-12-14 17:06:34 UTC 
Looks like a 6.0 guest bug in networking core so far.
Would appreciate it if someone looks at this from the
networking core perspective.
Comment 5 Herbert Xu 2010-12-15 15:03:07 UTC 
Looks like a problem with percpu allocations to me.  The network side requires a 2K allocation per IPv6 device, which does not seem to be too demanding to me.  As the guest has plenty of memory, I would say that this is a VM problem.
Comment 7 Rik van Riel 2010-12-16 14:06:42 UTC 
The amount of space for per cpu allocations is limited.  The reporter allocates 3120 kB of per-cpu data, by allocating 1560 vlans inside the guest.  After that, the per-cpu space is exhausted and further allocations fail.

Setting up that many vlans on a 32 bit guest is unreasonable, considering the limited amount of kernel address space and the constraints that places on the amount of address space available for per-cpu variables.

With a maximum 32 CPUs on 32 bit, 4MB maximum per-CPU memory completely exhausts the 128MB vmalloc memory.  Arguably that makes our per-cpu memory limit on 32 bits too high...
Comment 8 Bill Burns 2010-12-16 19:25:15 UTC 
Based on comment #7, closing this. If you disagree please reopen.