Bug 2109622 - 20220708 linux-firmware release breaks internet for Realtek RTL 8822ce card
Summary: 20220708 linux-firmware release breaks internet for Realtek RTL 8822ce card
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: linux-firmware
Version: 36
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: David Woodhouse
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2117866 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-21 15:59 UTC by Michael Kronenberg
Modified: 2022-11-10 22:19 UTC (History)
11 users (show)

Fixed In Version: linux-firmware-20221012-141.fc36 linux-firmware-20221012-141.fc35 linux-firmware-20221012-142.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-10-16 22:27:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
These iwl firmware updates broke my internet connection (130.05 KB, image/png)
2022-07-21 15:59 UTC, Michael Kronenberg
no flags Details
Description here mentions my RTL 8822ce wifi card (164.87 KB, image/png)
2022-07-21 16:00 UTC, Michael Kronenberg
no flags Details
dmesg of working version (84.86 KB, text/plain)
2022-07-29 12:00 UTC, Michael Schlechtinger
no flags Details
dmesg of broken version (84.78 KB, text/plain)
2022-07-29 12:00 UTC, Michael Schlechtinger
no flags Details

Description Michael Kronenberg 2022-07-21 15:59:32 UTC
Created attachment 1898534 [details]
These iwl firmware updates broke my internet connection

Description of problem:
WiFi connected  but no internet after the 20220708 iwl firmware release. 
Running " sudo dnf downgrade iwl\* " restores internet connection

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Michael Kronenberg 2022-07-21 16:00:28 UTC
Created attachment 1898535 [details]
Description here mentions my RTL 8822ce wifi card

Comment 2 Peter Robinson 2022-07-22 19:06:32 UTC
It's not the iwl firmware, you're going to have to provide more debug.

Comment 3 Michael Kronenberg 2022-07-23 01:50:58 UTC
This line was mentioned in the description of the update. I included a screenshot in the ticket.

"rtw88: 8822c: Update normal firmware to v9.9.13"

Realtek doesn't make a 8822c card. Search results for 8822c return the 8822ce, the card that I have in my laptop.

Please let me know how I can help. Not sure what you mean by "provide more debug"

Comment 4 Peter Robinson 2022-07-23 09:26:33 UTC
(In reply to Michael Kronenberg from comment #3)
> This line was mentioned in the description of the update. I included a
> screenshot in the ticket.
> 
> "rtw88: 8822c: Update normal firmware to v9.9.13"
> 
> Realtek doesn't make a 8822c card. Search results for 8822c return the
> 8822ce, the card that I have in my laptop.
> 
> Please let me know how I can help. Not sure what you mean by "provide more
> debug"

Are you sure it's not the linux-firmware package then? There's a number of packages in the update if you look that cover all sort of hardware.

The commit is directly from the Realtek commit updating the firmware:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=7eef50f83896dc574582d69a36a643722365b04c

Comment 5 Michael Kronenberg 2022-07-24 22:53:27 UTC
That makes sense. How do I submit a bug to that team?

Comment 6 Peter Robinson 2022-07-25 06:53:38 UTC
(In reply to Michael Kronenberg from comment #5)
> That makes sense. How do I submit a bug to that team?

If you look at the component you're already here, you're going to have include a lot more detailed information though.

1) the line from the output from lspci for the wireless card
2) the kernel your running (uname -a)
3) the appropriate lines (please not the whole lot) from dmesg for the card when it's working with the old firmware, and when it doesn't with the new firmware.
4) the make/model of the HW

Comment 7 Michael Kronenberg 2022-07-25 17:51:36 UTC
Thank you for all the help. I hope this is the output you are looking for.

Bug Behavior Update:
the downgrade of iwl\* stopped working so I upgraded again. I have booted the computer approximately 10 times since then. One time the wifi card worked, the other times it did not.

lspci output for wireless card
0000:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter

kernel info output of (uname -a)
Linux fedora 5.18.13-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 22 14:03:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

dmesg output

These two lines repeat frequently:
[ 2220.966377] rtw_8822ce 0000:01:00.0: failed to write TX skb to HCI
[ 2220.966382] rtw_8822ce 0000:01:00.0: failed to pusk skb, ret -28

5 lines that repeat 3 times
[ 7293.636131] rtw_8822ce 0000:01:00.0: timed out to flush pci tx ring[1]
[ 7293.636245] rtw_8822ce 0000:01:00.0: timed out to flush pci tx ring[3]
[ 7293.746112] rtw_8822ce 0000:01:00.0: timed out to flush queue 0
[ 7293.856116] rtw_8822ce 0000:01:00.0: timed out to flush queue 1
[ 7293.965971] rtw_8822ce 0000:01:00.0: timed out to flush queue 2

Another line that repeats frequently
[ 7602.944127] rtw_8822ce 0000:01:00.0: failed to get tx report from firmware

Hardware Info
HP Laptop 14-dq2xxx

Comment 8 Peter Robinson 2022-07-25 18:41:05 UTC
So it looks like I have either the same, or very similar PCIe card in my Jetson Xavier-NX and it works fine with the latest firmware:

0004:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter

dmesg:
[   17.705965] rtw_8822ce 0004:01:00.0: enabling device (0000 -> 0003)
[   17.748664] rtw_8822ce 0004:01:00.0: Firmware version 9.9.4, H2C version 15
[   17.763867] rtw_8822ce 0004:01:00.0: Firmware version 9.9.13, H2C version 15
[   17.934706] rtw_8822ce 0004:01:00.0 wlP4p1s0: renamed from wlan0

What do you see on the working rev of the firmware?

Comment 9 Michael Kronenberg 2022-07-25 19:47:39 UTC
I no longer has a working rev of the firmware

I found these lines related to what you just posted

[   41.450317] rtw_8822ce 0000:01:00.0: enabling device (0000 -> 0003)
[   41.456571] rtw_8822ce 0000:01:00.0: Firmware version 9.9.4, H2C version 15
[   41.462128] rtw_8822ce 0000:01:00.0: Firmware version 9.9.13, H2C version 15
[   41.485910] rtw_8822ce 0000:01:00.0 wlo1: renamed from wlan0

Comment 10 Michael Schlechtinger 2022-07-29 11:21:51 UTC
My notebook also uses the RTL8822CE wifi adapter from Realtek. I can confirm that the latest version of the linux-firmware Package breaks the internet connection. I reverted the file rtw8822c_fw.bin.xz to the former version and now it‘s working again.

Comment 11 Peter Robinson 2022-07-29 11:28:41 UTC
I am going to need more information than "it breaks it" as I have one that works fine with the update.

Comment 12 Michael Schlechtinger 2022-07-29 12:00:25 UTC
Created attachment 1900141 [details]
dmesg of working version

Comment 13 Michael Schlechtinger 2022-07-29 12:00:49 UTC
Created attachment 1900142 [details]
dmesg of broken version

Comment 14 Michael Schlechtinger 2022-07-29 12:02:36 UTC
I checked the dmesg output with the current broken version and the former working version. When using the current version of linux-firmware there is this line in the log:

[   21.868857] rtw_8822ce 0000:01:00.0: failed to do dpk calibration

Comment 15 Peter Robinson 2022-07-29 12:08:35 UTC
(In reply to Michael Schlechtinger from comment #12)
> Created attachment 1900141 [details]
> dmesg of working version

Please just put the pertinent bits in a comment in the bug as it's much easier to read than having to download an load separate files.

Please also include current kernel and explicit details of the HW as requested above.

Comment 16 Michael Schlechtinger 2022-07-29 12:15:31 UTC
1) the line from the output from lspci for the wireless card
> 01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter

2) the kernel your running (uname -a)
> Linux matebook 5.18.13-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 22 14:03:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

3) the appropriate lines (please not the whole lot) from dmesg for the card when it's working with the old firmware, and when it doesn't with the new firmware.
> Record in log with current broken firmware:
> [   21.868857] rtw_8822ce 0000:01:00.0: failed to do dpk calibration

4) the make/model of the HW
> Huawei Matebook / HUAWEI HN-WX9X

Comment 17 Michael Kronenberg 2022-07-29 15:06:57 UTC
Hey Michael Schlechtinger:

How did you downgrade that file?

Comment 18 Peter Robinson 2022-07-29 15:29:41 UTC
dnf downgrade linux-firmware\*

Comment 19 Michael Schlechtinger 2022-07-29 15:41:49 UTC
Since I had no internet connection I downloaded the RPM of the former version on another computer, extracted it, backed up the current rtw8822c_fw.bin.xz and replaced it with the old one. After rebooting internet was operational again.

Comment 20 lostmoonofsaturn 2022-07-29 19:37:20 UTC
I've just done an install of the 27 July Live Respin of Fedora Workstation, which includes the linux-firmware update at issue, on an HP Dev One, which uses the rtw_8822ce card.

Running live before installation, I was able to log into local wifi and browse, etc.  The login attempt initially displayed a "failed to connect" notice, then connected successfully without intervention from me.

After completing the install on the Dev One, the wifi connection was live on the first boot.  I completed Gnome's initial setup routine and ran through Settings per usual, without adjusting the wifi or network settings BUT including changing the hostname in Settings.

When I closed Settings and opened Firefox, wifi was turned off. I clicked the wifi 'Turn On" option.  Gnome indicated wifi was working. I.e, the wifi signal strength icon in the top bar displayed as expected.  The dropdown menu displayed the correct local wifi network name. Opening "wifi settings" I saw the connection was enabled,  "Default route" and "DNS" were both shown as "192.168.1.1".  The IPv4 and IPv6 tabs displayed "Automatic (DHCP); "DNS" and "ROUTES" were set to "Automatic" and no addresses were displayed.

However, DNS was apparently not working, at least Firefox could not find URL's and pings to google.com and elsewhere in the terminal timed out with no returns. A restart had no impact.

From the logs, I can offer:

"rtw_8822ce 0000:01:00.0: time out to flush queue 2" [repeated 16 times]

"rtw_8822ce 0000:01:00.0: Firmware version 9.9.4, H2C version 15" [repeated 3 times]

"Failed to send hostname reply:  Transport endpoint is not connected" [repeated 3 times] 

This behavior and the log entries duplicate what I saw on a previous F36 Workstation install after updating to the latest linux-firmware package.

(I need to use this unit so I will be removing the Fedora install.)

Comment 21 Peter Robinson 2022-08-12 10:09:15 UTC
*** Bug 2117866 has been marked as a duplicate of this bug. ***

Comment 22 Fedora Update System 2022-10-13 16:55:52 UTC
FEDORA-2022-f82d476e16 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f82d476e16

Comment 23 Fedora Update System 2022-10-13 16:55:54 UTC
FEDORA-2022-20532b3b71 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-20532b3b71

Comment 24 Fedora Update System 2022-10-13 16:55:56 UTC
FEDORA-2022-844ec1d905 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-844ec1d905

Comment 25 Fedora Update System 2022-10-14 10:30:58 UTC
FEDORA-2022-20532b3b71 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-20532b3b71`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-20532b3b71

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 26 Fedora Update System 2022-10-14 14:37:54 UTC
FEDORA-2022-f82d476e16 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-f82d476e16`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f82d476e16

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 27 Fedora Update System 2022-10-14 14:37:57 UTC
FEDORA-2022-844ec1d905 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-844ec1d905`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-844ec1d905

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 28 Fedora Update System 2022-10-16 21:13:40 UTC
FEDORA-2022-9e934e6bb4 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-9e934e6bb4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-9e934e6bb4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 29 Fedora Update System 2022-10-16 22:27:33 UTC
FEDORA-2022-844ec1d905 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 30 Fedora Update System 2022-10-29 19:41:32 UTC
FEDORA-2022-f82d476e16 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 31 Fedora Update System 2022-11-10 22:19:17 UTC
FEDORA-2022-9e934e6bb4 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.