Bug 1078741

Summary:	[Rawhide] openvswitch-2.1.0 is available
Product:	[Fedora] Fedora	Reporter:	Upstream Release Monitoring <upstream-release-monitoring>
Component:	openvswitch	Assignee:	Flavio Leitner <fleitner>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	chrisw, fleitner, markmc, tgraf
Target Milestone:	---	Keywords:	FutureFeature, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch-2.1.0-1.fc21	Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:
Clones:	1080679 1080680 1080681 (view as bug list)		Environment:
Last Closed:	2014-04-16 18:59:59 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Upstream Release Monitoring 2014-03-20 09:26:14 UTC

Latest upstream release: 2.1.0
Current version/release in Fedora Rawhide: 2.0.1-1.fc21
URL: http://openvswitch.org/releases/

Please consult the package updates policy before you issue an update to a stable branch: https://fedoraproject.org/wiki/Updates_Policy

More information about the service that created this bug can be found at:
https://fedoraproject.org/wiki/Upstream_release_monitoring

Comment 1 Flavio Leitner 2014-03-26 16:20:35 UTC

I got this while testing OVS 2.1.0 on fedora rawhide
[root@i7 ~]# rpm -qi openvswitch
Name        : openvswitch
Version     : 2.1.0
Release     : 1.fc21
Architecture: x86_64
Install Date: Wed 26 Mar 2014 12:07:57 PM BRT
Group       : Unspecified
Size        : 7595484
License     : ASL 2.0 and LGPLv2+ and SISSL
Signature   : (none)
Source RPM  : openvswitch-2.1.0-1.fc21.src.rpm
Build Date  : Wed 26 Mar 2014 12:02:06 PM BRT
Build Host  : t520.home



[  746.284058] gre: GRE over IPv4 demultiplexor driver
[  746.315384] openvswitch: Open vSwitch switching datapath
[  746.434615] device ovs-system entered promiscuous mode
[  746.485194] device br0 entered promiscuous mode
[  746.490864] device p64p1 entered promiscuous mode

[  748.184923] =================================
[  748.189730] [ INFO: inconsistent lock state ]
[  748.194543] 3.14.0-0.rc7.git2.1.fc21.x86_64 #1 Tainted: G          I
[  748.201640] ---------------------------------
[  748.206455] inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
[  748.213080] swapper/1/0 [HC0[0]:SC1[5]:HE1:SE0] takes:
[  748.218748]  (&(&cpu_stats->lock)->rlock){+.?...}, at: [<ffffffffa065d83f>] ovs_flow_stats_update+0x4f/0xd0 [openvswitch]
[  748.230836] {SOFTIRQ-ON-W} state was registered at:
[  748.236197]   [<ffffffff810f97af>] __lock_acquire+0x68f/0x1c40
[  748.236199]   [<ffffffff810fb552>] lock_acquire+0xa2/0x1d0
[  748.236203]   [<ffffffff817dae7e>] _raw_spin_lock+0x3e/0x80
[  748.236207]   [<ffffffffa065d980>] ovs_flow_stats_get+0xc0/0x1b0 [openvswitch]
[  748.236211]   [<ffffffffa065a7f5>] ovs_flow_cmd_fill_info+0x185/0x360 [openvswitch]
[  748.236215]   [<ffffffffa065aea5>] ovs_flow_cmd_build_info.constprop.27+0x55/0x90 [openvswitch]
[  748.236218]   [<ffffffffa065b3bd>] ovs_flow_cmd_new_or_set+0x4dd/0x570 [openvswitch]
[  748.236221]   [<ffffffff816c44dd>] genl_family_rcv_msg+0x1cd/0x3f0
[  748.236223]   [<ffffffff816c478e>] genl_rcv_msg+0x8e/0xd0
[  748.236227]   [<ffffffff816c22b9>] netlink_rcv_skb+0xa9/0xc0
[  748.236232]   [<ffffffff816c2818>] genl_rcv+0x28/0x40
[  748.236235]   [<ffffffff816c18b0>] netlink_unicast+0x100/0x1e0
[  748.236238]   [<ffffffff816c1cd7>] netlink_sendmsg+0x347/0x770
[  748.236240]   [<ffffffff8166af4c>] sock_sendmsg+0x9c/0xe0
[  748.236242]   [<ffffffff8166b389>] ___sys_sendmsg+0x3a9/0x3c0
[  748.236245]   [<ffffffff8166c9c1>] __sys_sendmsg+0x51/0x90
[  748.236247]   [<ffffffff8166ca12>] SyS_sendmsg+0x12/0x20
[  748.236251]   [<ffffffff817e5de9>] system_call_fastpath+0x16/0x1b
[  748.236252] irq event stamp: 650346
[  748.236255] hardirqs last  enabled at (650346): [<ffffffff8175f640>] ip6_finish_output2+0x4f0/0x840
[  748.236257] hardirqs last disabled at (650345): [<ffffffff8175f5fb>] ip6_finish_output2+0x4ab/0x840
[  748.236262] softirqs last  enabled at (650286): [<ffffffff8109be82>] _local_bh_enable+0x22/0x50
[  748.236264] softirqs last disabled at (650287): [<ffffffff8109db75>] irq_exit+0xc5/0xd0
[  748.236265]
other info that might help us debug this:
[  748.236266]  Possible unsafe locking scenario:

[  748.236266]        CPU0
[  748.236266]        ----
[  748.236268]   lock(&(&cpu_stats->lock)->rlock);
[  748.236268]   <Interrupt>
[  748.236270]     lock(&(&cpu_stats->lock)->rlock);
[  748.236270]
 *** DEADLOCK ***

[  748.236271] 5 locks held by swapper/1/0:
[  748.236276]  #0:  (((&ifa->dad_timer))){+.-...}, at: [<ffffffff810a71c5>] call_timer_fn+0x5/0x320
[  748.236282]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff8178ab05>] mld_sendpack+0x5/0x4a0
[  748.236286]  #2:  (rcu_read_lock_bh){.+....}, at: [<ffffffff8175f1a9>] ip6_finish_output2+0x59/0x840
[  748.236291]  #3:  (rcu_read_lock_bh){.+....}, at: [<ffffffff8168db25>] __dev_queue_xmit+0x5/0x9b0
[  748.236297]  #4:  (rcu_read_lock){.+.+..}, at: [<ffffffffa0664125>] internal_dev_xmit+0x5/0x110 [openvswitch]
[  748.236298]
stack backtrace:
[  748.236300] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G          I  3.14.0-0.rc7.git2.1.fc21.x86_64 #1
[  748.236302] Hardware name:                  /DX58SO, BIOS SOX5810J.86A.5599.2012.0529.2218 05/29/2012
[  748.236305]  0000000000000000 a793fdefcb049f11 ffff88042d803828 ffffffff817d1f18
[  748.236308]  ffff8804275b8000 ffff88042d803878 ffffffff817cd7aa 0000000000000005
[  748.236310]  ffff880400000001 ffff880400000000 0000000000000006 ffff8804275b8000
[  748.236311] Call Trace:
[  748.236316]  <IRQ>  [<ffffffff817d1f18>] dump_stack+0x4d/0x66
[  748.236320]  [<ffffffff817cd7aa>] print_usage_bug+0x1f4/0x205
[  748.236323]  [<ffffffff810f7f80>] ? check_usage_backwards+0x180/0x180
[  748.236326]  [<ffffffff810f89d3>] mark_lock+0x223/0x2b0
[  748.236329]  [<ffffffff810f9743>] __lock_acquire+0x623/0x1c40
[  748.236331]  [<ffffffff810f5777>] ? __lock_is_held+0x57/0x80
[  748.236336]  [<ffffffffa0662636>] ? masked_flow_lookup+0x236/0x250 [openvswitch]
[  748.236339]  [<ffffffff810fb552>] lock_acquire+0xa2/0x1d0
[  748.236344]  [<ffffffffa065d83f>] ? ovs_flow_stats_update+0x4f/0xd0 [openvswitch]
[  748.236346]  [<ffffffff817dae7e>] _raw_spin_lock+0x3e/0x80
[  748.236350]  [<ffffffffa065d83f>] ? ovs_flow_stats_update+0x4f/0xd0 [openvswitch]
[  748.236354]  [<ffffffffa065d83f>] ovs_flow_stats_update+0x4f/0xd0 [openvswitch]
[  748.236358]  [<ffffffffa065cc04>] ovs_dp_process_received_packet+0x84/0x120 [openvswitch]
[  748.236361]  [<ffffffff810f9467>] ? __lock_acquire+0x347/0x1c40
[  748.236366]  [<ffffffffa0663b5a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[  748.236371]  [<ffffffffa0664188>] internal_dev_xmit+0x68/0x110 [openvswitch]
[  748.236375]  [<ffffffffa0664125>] ? internal_dev_xmit+0x5/0x110 [openvswitch]
[  748.236378]  [<ffffffff8168d556>] dev_hard_start_xmit+0x2e6/0x8b0
[  748.236381]  [<ffffffff8168df37>] __dev_queue_xmit+0x417/0x9b0
[  748.236384]  [<ffffffff8168db25>] ? __dev_queue_xmit+0x5/0x9b0
[  748.236386]  [<ffffffff8175f640>] ? ip6_finish_output2+0x4f0/0x840
[  748.236389]  [<ffffffff8168e4e0>] dev_queue_xmit+0x10/0x20
[  748.236391]  [<ffffffff8175f6a1>] ip6_finish_output2+0x551/0x840
[  748.236393]  [<ffffffff8176333a>] ? ip6_finish_output+0x9a/0x220
[  748.236395]  [<ffffffff8176333a>] ip6_finish_output+0x9a/0x220
[  748.236397]  [<ffffffff8176350f>] ip6_output+0x4f/0x1f0
[  748.236400]  [<ffffffff8178acd9>] mld_sendpack+0x1d9/0x4a0
[  748.236403]  [<ffffffff8178b668>] mld_send_initial_cr.part.32+0x88/0xa0
[  748.236406]  [<ffffffff8176b260>] ? addrconf_dad_completed+0x220/0x220
[  748.236409]  [<ffffffff817903b1>] ipv6_mc_dad_complete+0x31/0x50
[  748.236411]  [<ffffffff8176b187>] addrconf_dad_completed+0x147/0x220
[  748.236413]  [<ffffffff8176b260>] ? addrconf_dad_completed+0x220/0x220
[  748.236416]  [<ffffffff8176b3ff>] addrconf_dad_timer+0x19f/0x1c0
[  748.236418]  [<ffffffff810a7259>] call_timer_fn+0x99/0x320
[  748.236420]  [<ffffffff810a71c5>] ? call_timer_fn+0x5/0x320
[  748.236423]  [<ffffffff8176b260>] ? addrconf_dad_completed+0x220/0x220
[  748.236425]  [<ffffffff810a7734>] run_timer_softirq+0x254/0x3b0
[  748.236428]  [<ffffffff8109d4ed>] __do_softirq+0x12d/0x480
[  748.236431]  [<ffffffff8109db75>] irq_exit+0xc5/0xd0
[  748.236433]  [<ffffffff817e81b5>] smp_apic_timer_interrupt+0x45/0x60
[  748.236436]  [<ffffffff817e6af2>] apic_timer_interrupt+0x72/0x80
[  748.236441]  <EOI>  [<ffffffff8162d644>] ? cpuidle_enter_state+0x54/0xd0
[  748.236444]  [<ffffffff8162d640>] ? cpuidle_enter_state+0x50/0xd0
[  748.236447]  [<ffffffff8162d779>] cpuidle_idle_call+0xb9/0x380
[  748.236451]  [<ffffffff8102637e>] arch_cpu_idle+0xe/0x40
[  748.236455]  [<ffffffff8110d725>] cpu_startup_entry+0xf5/0x420
[  748.236458]  [<ffffffff8104dce0>] start_secondary+0x240/0x300

Comment 2 Flavio Leitner 2014-03-26 17:24:08 UTC

Now with upstream net tree
[   50.778281] =================================
[   50.783040] [ INFO: inconsistent lock state ]
[   50.787796] 3.14.0-rc8-00007-g632b06a #1 Tainted: G          I
[   50.794264] ---------------------------------
[   50.799014] inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
[   50.805576] swapper/0/0 [HC0[0]:SC1[5]:HE1:SE0] takes:
[   50.811186]  (&(&cpu_stats->lock)->rlock){+.?...}, at: [<ffffffffa05dd8a1>] ovs_flow_stats_update+0x51/0xd0 [openvswitch]
[   50.823212] {SOFTIRQ-ON-W} state was registered at:
[   50.828541]   [<ffffffff810f973f>] __lock_acquire+0x68f/0x1c40
[   50.834931]   [<ffffffff810fb4e2>] lock_acquire+0xa2/0x1d0
[   50.840942]   [<ffffffff817d8d9e>] _raw_spin_lock+0x3e/0x80
[   50.847039]   [<ffffffffa05dd9e4>] ovs_flow_stats_get+0xc4/0x1e0 [openvswitch]
[   50.854948]   [<ffffffffa05da855>] ovs_flow_cmd_fill_info+0x185/0x360 [openvswitch]
[   50.863328]   [<ffffffffa05daf05>] ovs_flow_cmd_build_info.constprop.27+0x55/0x90 [openvswitch]
[   50.872845]   [<ffffffffa05db41d>] ovs_flow_cmd_new_or_set+0x4dd/0x570 [openvswitch]
[   50.881319]   [<ffffffff816c245d>] genl_family_rcv_msg+0x1cd/0x3f0
[   50.888085]   [<ffffffff816c270e>] genl_rcv_msg+0x8e/0xd0
[   50.893994]   [<ffffffff816c0239>] netlink_rcv_skb+0xa9/0xc0
[   50.900189]   [<ffffffff816c0798>] genl_rcv+0x28/0x40
[   50.905723]   [<ffffffff816bf830>] netlink_unicast+0x100/0x1e0
[   50.912107]   [<ffffffff816bfc57>] netlink_sendmsg+0x347/0x770
[   50.918491]   [<ffffffff81668e9c>] sock_sendmsg+0x9c/0xe0
[   50.924395]   [<ffffffff816692d9>] ___sys_sendmsg+0x3a9/0x3c0
[   50.930682]   [<ffffffff8166a911>] __sys_sendmsg+0x51/0x90
[   50.936694]   [<ffffffff8166a962>] SyS_sendmsg+0x12/0x20
[   50.942507]   [<ffffffff817e3ce9>] system_call_fastpath+0x16/0x1b
[   50.949178] irq event stamp: 1740726
[   50.953085] hardirqs last  enabled at (1740726): [<ffffffff8175d5e0>] ip6_finish_output2+0x4f0/0x840

Comment 3 Flavio Leitner 2014-03-26 17:25:32 UTC

[   50.963078] hardirqs last disabled at (1740725): [<ffffffff8175d59b>] ip6_finish_output2+0x4ab/0x840
[   50.973071] softirqs last  enabled at (1740674): [<ffffffff8109be12>] _local_bh_enable+0x22/0x50
[   50.982684] softirqs last disabled at (1740675): [<ffffffff8109db05>] irq_exit+0xc5/0xd0
[   50.993529]
other info that might help us debug this:
[   51.004683]  Possible unsafe locking scenario:

[   51.015152]        CPU0
[   51.019783]        ----
[   51.024390]   lock(&(&cpu_stats->lock)->rlock);
[   51.031280]   <Interrupt>
[   51.036075]     lock(&(&cpu_stats->lock)->rlock);
[   51.043149]
 *** DEADLOCK ***

[   51.055241] 5 locks held by swapper/0/0:
[   51.061376]  #0:  (((&ifa->dad_timer))){+.-...}, at: [<ffffffff810a7155>] call_timer_fn+0x5/0x320
[   51.072972]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff81788a55>] mld_sendpack+0x5/0x4a0
[   51.083917]  #2:  (rcu_read_lock_bh){.+....}, at: [<ffffffff8175d149>] ip6_finish_output2+0x59/0x840
[   51.095825]  #3:  (rcu_read_lock_bh){.+....}, at: [<ffffffff8168ba75>] __dev_queue_xmit+0x5/0x9b0
[   51.107451]  #4:  (rcu_read_lock){.+.+..}, at: [<ffffffffa05e41b5>] internal_dev_xmit+0x5/0x110 [openvswitch]
[   51.120225]
stack backtrace:
[   51.128785] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G          I  3.14.0-rc8-00007-g632b06a #1
[   51.140004] Hardware name:                  /DX58SO, BIOS SOX5810J.86A.5599.2012.0529.2218 05/29/2012
[   51.151985]  0000000000000000 0fcf20709903df0c ffff88042d603808 ffffffff817cfe3c
[   51.162033]  ffffffff81c134c0 ffff88042d603858 ffffffff817cb6da 0000000000000005
[   51.172103]  ffffffff00000001 ffff880400000000 0000000000000006 ffffffff81c134c0
[   51.182176] Call Trace:
[   51.186801]  <IRQ>  [<ffffffff817cfe3c>] dump_stack+0x4d/0x66
[   51.195041]  [<ffffffff817cb6da>] print_usage_bug+0x1f4/0x205
[   51.203231]  [<ffffffff810f7f10>] ? check_usage_backwards+0x180/0x180
[   51.212170]  [<ffffffff810f8963>] mark_lock+0x223/0x2b0
[   51.219775]  [<ffffffff810f96d3>] __lock_acquire+0x623/0x1c40
[   51.227951]  [<ffffffff810f5707>] ? __lock_is_held+0x57/0x80
[   51.236024]  [<ffffffffa05e26c6>] ? masked_flow_lookup+0x236/0x250 [openvswitch]
[   51.246000]  [<ffffffff810fb4e2>] lock_acquire+0xa2/0x1d0
[   51.253803]  [<ffffffffa05dd8a1>] ? ovs_flow_stats_update+0x51/0xd0 [openvswitch]
[   51.263896]  [<ffffffff817d8d9e>] _raw_spin_lock+0x3e/0x80
[   51.271817]  [<ffffffffa05dd8a1>] ? ovs_flow_stats_update+0x51/0xd0 [openvswitch]
[   51.281923]  [<ffffffffa05dd8a1>] ovs_flow_stats_update+0x51/0xd0 [openvswitch]
[   51.291832]  [<ffffffffa05dcc64>] ovs_dp_process_received_packet+0x84/0x120 [openvswitch]
[   51.302691]  [<ffffffff810f93f7>] ? __lock_acquire+0x347/0x1c40
[   51.311071]  [<ffffffffa05e3bea>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[   51.320559]  [<ffffffffa05e4218>] internal_dev_xmit+0x68/0x110 [openvswitch]
[   51.330137]  [<ffffffffa05e41b5>] ? internal_dev_xmit+0x5/0x110 [openvswitch]
[   51.339823]  [<ffffffff8168b4a6>] dev_hard_start_xmit+0x2e6/0x8b0
[   51.348378]  [<ffffffff8168be87>] __dev_queue_xmit+0x417/0x9b0
[   51.356629]  [<ffffffff8168ba75>] ? __dev_queue_xmit+0x5/0x9b0
[   51.364879]  [<ffffffff8175d5e0>] ? ip6_finish_output2+0x4f0/0x840
[   51.373493]  [<ffffffff8168c430>] dev_queue_xmit+0x10/0x20
[   51.381349]  [<ffffffff8175d641>] ip6_finish_output2+0x551/0x840
[   51.389761]  [<ffffffff8176128a>] ? ip6_finish_output+0x9a/0x220
[   51.398162]  [<ffffffff8176128a>] ip6_finish_output+0x9a/0x220
[   51.406374]  [<ffffffff8176145f>] ip6_output+0x4f/0x1f0
[   51.413917]  [<ffffffff81788c29>] mld_sendpack+0x1d9/0x4a0
[   51.421741]  [<ffffffff817895b8>] mld_send_initial_cr.part.32+0x88/0xa0
[   51.430808]  [<ffffffff817691b0>] ? addrconf_dad_completed+0x220/0x220
[   51.439785]  [<ffffffff8178e301>] ipv6_mc_dad_complete+0x31/0x50
[   51.448192]  [<ffffffff817690d7>] addrconf_dad_completed+0x147/0x220
[   51.456987]  [<ffffffff817691b0>] ? addrconf_dad_completed+0x220/0x220
[   51.465977]  [<ffffffff8176934f>] addrconf_dad_timer+0x19f/0x1c0
[   51.474397]  [<ffffffff810a71e9>] call_timer_fn+0x99/0x320
[   51.482237]  [<ffffffff810a7155>] ? call_timer_fn+0x5/0x320
[   51.490144]  [<ffffffff817691b0>] ? addrconf_dad_completed+0x220/0x220
[   51.499097]  [<ffffffff810a76c4>] run_timer_softirq+0x254/0x3b0
[   51.507370]  [<ffffffff8109d47d>] __do_softirq+0x12d/0x480

Comment 4 Flavio Leitner 2014-03-27 14:15:00 UTC

Patch posted:
http://patchwork.openvswitch.org/patch/3452/