Bug 2040360 - networking stops working after enabling enhanced networking on aws m4.large instances
Summary: networking stops working after enabling enhanced networking on aws m4.large i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-13 14:48 UTC by Michael Nguyen
Modified: 2022-02-02 01:15 UTC (History)
25 users (show)

Fixed In Version: kernel-5.15.17-200.fc35 kernel-5.15.18-100.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-30 01:34:12 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
System Log from AWS (62.50 KB, text/plain)
2022-01-13 14:48 UTC, Michael Nguyen
no flags Details

Description Michael Nguyen 2022-01-13 14:48:30 UTC
Created attachment 1850604 [details]
System Log from AWS

1. Please describe the problem:
After enabling enhanced networking in an AWS m4.large instance, the networking seems to stop working.   From CI runs, this does not seem to affect m5.large instances.

This bug was originally found in Fedora CoreOS which has enhanced networking enabled by default which preventing the installation of Fedora CoreOS.   See https://github.com/coreos/fedora-coreos-tracker/issues/1066


2. What is the Version-Release number of the kernel:

kernel-core-5.15.13-200.fc35.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

From the CI runs in Fedora CoreOS, it was working on `5.15.10-200.fc35.x86_64` and stopped working on `5.15.10-200.fc35.x86_64`.

There were two CVEs fixed between these versions that seem to be related to networking:
- https://bugzilla.redhat.com/show_bug.cgi?id=2031199
- https://bugzilla.redhat.com/show_bug.cgi?id=2034940


4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

This problem happens intermittently.  It is sometimes successful but most of the time it does not.

- Launch AMI ami-08b4ee602f76bff79 with m4.large instance type on us-east-1 in AWS (latest from https://alt.fedoraproject.org/cloud/)
- SSH to instance and run `dnf -y update`
- Reboot and verify that SSH login to the instance is still possible.
- Stop the instance in AWS
- Run `aws ec2 modify-instance-attribute --instance-id <instance-id> --sriov-net-support simple` to enable enhance networking
- Start the instance in AWS
- Login is no longer possible through SSH

Note: There is no way to disable enhanced networking once it is enabled in AWS.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

From testing it seems like this can be reproduced ~50%-60% of the time in the rawhide kernel


6. Are you running any modules that not shipped with directly Fedora's kernel?:

No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Since networking is down after starting, I can't run any additional commands.  I only have the system log from AWS which is attached.




Additional Notes:

The difference between enhanced networking and out of the box Fedora is the network driver--ixgbevf and vif, respectively.

Original Issue:
https://github.com/coreos/fedora-coreos-tracker/issues/1066

Comment 1 Dusty Mabe 2022-01-23 20:57:08 UTC
The offending commit seems to be 5.15.X commit d8888cdabedf which maps to mainline commit 83dbf898a2d4. More info at https://github.com/coreos/fedora-coreos-tracker/issues/1066#issuecomment-1019560658

Comment 3 Fedora Update System 2022-01-27 21:12:35 UTC
FEDORA-2022-a84329335e has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a84329335e

Comment 4 Fedora Update System 2022-01-27 21:12:38 UTC
FEDORA-2022-aaa4e47375 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-aaa4e47375

Comment 5 Fedora Update System 2022-01-28 01:15:26 UTC
FEDORA-2022-a84329335e has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-a84329335e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a84329335e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2022-01-29 05:05:10 UTC
FEDORA-2022-aaa4e47375 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-aaa4e47375`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-aaa4e47375

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2022-01-30 01:34:12 UTC
FEDORA-2022-aaa4e47375 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Fedora Update System 2022-01-31 01:56:24 UTC
FEDORA-2022-48acd4718d has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-48acd4718d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-48acd4718d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2022-02-02 01:15:12 UTC
FEDORA-2022-48acd4718d has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.