The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.
Bug 1947913 - [OVN][RFE] Add protection mechanism against gARPs / flapping ports
Summary: [OVN][RFE] Add protection mechanism against gARPs / flapping ports
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: ovn2.13
Version: FDP 21.B
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: lorenzo bianconi
QA Contact: ying xu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-09 14:02 UTC by Daniel Alvarez Sanchez
Modified: 2023-09-18 00:25 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-30 17:59:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-1223 0 None None None 2021-12-24 05:46:24 UTC
Red Hat Product Errata RHBA-2022:5446 0 None None None 2022-06-30 18:00:24 UTC

Description Daniel Alvarez Sanchez 2021-04-09 14:02:29 UTC
In an scenario with VIPs that are balanced at application level (by for example, keepalived) there could be issues at such level which leads to the VIP flapping very frequently.

This will make the app to send the gARPs announcing the location of the new IP address which will trigger updates of MAC_Binding entries very frequently if the ports flap at high rate. The result is that ovn-northd will stay at 100% CPU and ovsdb-server for SB database will be very loaded.

Just like normal switches have, it would be great if OVN could implement some protection mechanism that when such condition is detected, the MAC address is banned from the switch in one or more ports to prevent this situation from happening.

Comment 2 Dan Williams 2021-05-04 14:19:15 UTC
COPP upstream series posted: http://patchwork.ozlabs.org/project/ovn/list/?series=241400&state=%2A&archive=both

Comment 6 Dan Williams 2022-05-24 17:07:12 UTC
IIRC this support landed in OVN 21.09 mid 2021.

Comment 9 ying xu 2022-06-14 12:39:25 UTC

 as comment8 said, it is included in https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021,
I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64
the command for copp is not the same as that in ovn22.03.
I want to make sure I should verify this bug in ovn22.03 or ovn-2021?

Comment 10 lorenzo bianconi 2022-06-14 13:22:16 UTC
(In reply to ying xu from comment #9)
> 
>  as comment8 said, it is included in
> https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021,
> I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64
> the command for copp is not the same as that in ovn22.03.
> I want to make sure I should verify this bug in ovn22.03 or ovn-2021?

I would suggest to test both of them, it is just a tiny change I guess.

Comment 11 ying xu 2022-06-14 13:44:27 UTC
(In reply to lorenzo bianconi from comment #10)
> (In reply to ying xu from comment #9)
> > 
> >  as comment8 said, it is included in
> > https://errata.devel.redhat.com/advisory/96200, it's for ovn-2021,
> > I tried on version ovn-2021-21.12.0-73.el8fdp.x86_64
> > the command for copp is not the same as that in ovn22.03.
> > I want to make sure I should verify this bug in ovn22.03 or ovn-2021?
> 
> I would suggest to test both of them, it is just a tiny change I guess.

but why the commands are not the same for these two versions?
I think it is not friendly and not compatiable.

Comment 12 ying xu 2022-06-14 15:24:02 UTC
verified on version:
ovn-2021-21.12.0-73.el8fdp.x86_64
ovn22.03-22.03.0-52.el8fdp.x86_64

although there are some differences between the two versions, the test procedure is the same.
:: [ 06:17:13 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w reject1.pcap& '
:: [ 06:17:13 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w reject1.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:17:15 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py'
WARNING: Interface lo: no address assigned
:: [ 06:17:16 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0)
28 packets captured
29 packets received by filter
0 packets dropped by kernel
reading from file reject.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:17:20 ] :: [   PASS   ] :: reject packets count:1 should be 1 (Assert: '1' should equal '1')
SYNC_NC: sync_set client ovn_copp
SYNC_NC: sent "ovn_copp" to dell-per740-54.rhts.eng.pek2.redhat.com
SYNC_NC: sync_wait client ovn_copp
SYNC_NC: waiting "dell-per740-54.rhts.eng.pek2.redhat.com"
SYNC_NC: got "ovn_copp" from dell-per740-54.rhts.eng.pek2.redhat.com
:: [ 06:17:31 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w reject10.pcap& '
:: [ 06:17:31 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w reject10.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:17:33 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py'
WARNING: Interface lo: no address assigned
:: [ 06:17:34 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0)
32 packets captured
32 packets received by filter
0 packets dropped by kernel
reading from file reject10.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:17:38 ] :: [   PASS   ] :: reject packets count:10 should be 10 (Assert: '10' should equal '10')
SYNC_NC: sync_set client ovn_copp
SYNC_NC: sent "ovn_copp" to dell-per740-54.rhts.eng.pek2.redhat.com
SYNC_NC: sync_wait client ovn_copp
SYNC_NC: waiting "dell-per740-54.rhts.eng.pek2.redhat.com"
SYNC_NC: got "ovn_copp" from dell-per740-54.rhts.eng.pek2.redhat.com
:: [ 06:17:49 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv copp-del copp0 reject'
:: [ 06:17:49 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv copp-del copp0 reject' (Expected 0, got 0)
:: [ 06:17:49 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w rejectno.pcap& '
:: [ 06:17:49 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w rejectno.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:17:51 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py'
WARNING: Interface lo: no address assigned
:: [ 06:17:52 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendudp.py' (Expected 0, got 0)
42 packets captured
42 packets received by filter
0 packets dropped by kernel
reading from file rejectno.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:17:56 ] :: [   PASS   ] :: reject packets count:20 should be 20 (Assert: '20' should equal '20')

:: [ 06:18:22 ] :: [  BEGIN   ] :: Running 'ovn-nbctl meter-add icmp-meter drop 1 pktps 0'
:: [ 06:18:22 ] :: [   PASS   ] :: Command 'ovn-nbctl meter-add icmp-meter drop 1 pktps 0' (Expected 0, got 0)
:: [ 06:18:22 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv copp-add copp2 icmp4-error icmp-meter'
:: [ 06:18:22 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv copp-add copp2 icmp4-error icmp-meter' (Expected 0, got 0)
:: [ 06:18:22 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv lr-copp-add copp2 lr1'
:: [ 06:18:22 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv lr-copp-add copp2 lr1' (Expected 0, got 0)
:: [ 06:18:22 ] :: [  BEGIN   ] :: Running 'ovn-nbctl copp-list copp2 |grep icmp4-error'
icmp4-error: icmp-meter
:: [ 06:18:22 ] :: [   PASS   ] :: Command 'ovn-nbctl copp-list copp2 |grep icmp4-error' (Expected 0, got 0)
:: [ 06:18:22 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w icmperror.pcap& '
:: [ 06:18:22 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w icmperror.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:18:24 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py'
WARNING: Interface lo: no address assigned
:: [ 06:18:25 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' (Expected 0, got 0)
23 packets captured
23 packets received by filter
0 packets dropped by kernel
reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:18:29 ] :: [   PASS   ] :: ICMP packets count:1 should be 1 (Assert: '1' should equal '1')
:: [ 06:18:29 ] :: [  BEGIN   ] :: Running 'tcpdump -r icmperror.pcap -nnle'
reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
06:18:25.309472 Out 00:00:01:01:01:01 ethertype ARP (0x0806), length 44: Request who-has 192.168.1.254 tell 192.168.1.2, length 28
06:18:25.309619  In 00:00:01:0f:01:02 ethertype ARP (0x0806), length 44: Reply 192.168.1.254 is-at 00:00:01:0f:01:02, length 28
06:18:25.324666 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.327465 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.330200 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.331406  In 00:00:01:0f:01:02 ethertype IPv4 (0x0800), length 148: 192.168.1.254 > 192.168.1.2: ICMP time exceeded in-transit, length 112
06:18:25.332928 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.335530 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.336832 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.338115 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.339397 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.340687 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.341983 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.343278 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.344337 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.345497 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.346488 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.347482 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.348478 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.349473 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.350462 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.351455 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
06:18:25.352461 Out 00:00:01:01:01:01 ethertype IPv4 (0x0800), length 120: 192.168.1.2.20 > 172.16.1.1.8080: Flags [S], seq 0:64, win 8192, length 64: HTTP
:: [ 06:18:29 ] :: [   PASS   ] :: Command 'tcpdump -r icmperror.pcap -nnle' (Expected 0, got 0)
:: [ 06:18:29 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv copp-del copp2 icmp4-error'
:: [ 06:18:29 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv copp-del copp2 icmp4-error' (Expected 0, got 0)
:: [ 06:18:29 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w icmperror.pcap& '
:: [ 06:18:29 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w icmperror.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:18:31 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py'
WARNING: Interface lo: no address assigned
:: [ 06:18:32 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendtcp_ttl1.py' (Expected 0, got 0)
42 packets captured
42 packets received by filter
0 packets dropped by kernel
reading from file icmperror.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:18:36 ] :: [   PASS   ] :: ICMP packets count:20 should be 20 (Assert: '20' should equal '20')
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl meter-add sctp-meter drop 1 pktps 0'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl meter-add sctp-meter drop 1 pktps 0' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv copp-add copp3 reject sctp-meter'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv copp-add copp3 reject sctp-meter' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv ls-copp-add copp3 ls1'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv ls-copp-add copp3 ls1' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl  lb-add lb 30.0.0.1:8000 ""'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl  lb-add lb 30.0.0.1:8000 ""' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d protocol=sctp'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d protocol=sctp' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set logical_switch ls1 load_balancer=3c69661c-ad6c-4c19-9226-cc863445996d'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl set logical_switch ls1 load_balancer=3c69661c-ad6c-4c19-9226-cc863445996d' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d options:reject=true'
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ovn-nbctl set load_balancer 3c69661c-ad6c-4c19-9226-cc863445996d options:reject=true' (Expected 0, got 0)
:: [ 06:18:36 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w sctp.pcap& '
:: [ 06:18:36 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w sctp.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:18:38 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py'
WARNING: Interface lo: no address assigned
:: [ 06:18:39 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' (Expected 0, got 0)
23 packets captured
23 packets received by filter
0 packets dropped by kernel
reading from file sctp.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:18:43 ] :: [   PASS   ] :: SCTP ABORT packets count:1 should be 1 (Assert: '1' should equal '1')
:: [ 06:18:43 ] :: [  BEGIN   ] :: Running 'ovn-nbctl --wait=hv copp-del copp3 reject'
:: [ 06:18:43 ] :: [   PASS   ] :: Command 'ovn-nbctl --wait=hv copp-del copp3 reject' (Expected 0, got 0)
:: [ 06:18:43 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 tcpdump -U -i any  -w sctp.pcap& '
:: [ 06:18:43 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 tcpdump -U -i any  -w sctp.pcap& ' (Expected 0, got 0)
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
:: [ 06:18:45 ] :: [  BEGIN   ] :: Running 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py'
WARNING: Interface lo: no address assigned
:: [ 06:18:46 ] :: [   PASS   ] :: Command 'ip netns exec ls1p1 /usr/libexec/platform-python /mnt/tests/kernel/networking/openvswitch/ovn/ls-lr/sendsctp.py' (Expected 0, got 0)
42 packets captured
42 packets received by filter
0 packets dropped by kernel
reading from file sctp.pcap, link-type LINUX_SLL (Linux cooked v1)
dropped privs to tcpdump
:: [ 06:18:50 ] :: [   PASS   ] :: SCTP ABORT packets count:20 should be 20 (Assert: '20' should equal '20')

Comment 14 errata-xmlrpc 2022-06-30 17:59:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5446

Comment 15 Red Hat Bugzilla 2023-09-18 00:25:42 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.