Bug 1584216

Summary:	Fedora 28 on AWS does not get proper MTU set from DHCP
Product:	[Fedora] Fedora	Reporter:	Luca BRUNO <lucab>
Component:	kernel	Assignee:	Kernel Maintainer List <kernel-maint>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	28	CC:	airlied, andrew.jeddeloh, bgilbert, bskeggs, dustymabe, ewk, fedora-kernel-xen, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, labbott, linville, mchehab, mjg59, steved
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	kernel-4.17.3-100.fc27 kernel-4.17.3-200.fc28	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-07-01 01:34:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Luca BRUNO 2018-05-30 13:47:51 UTC

Description of problem:

Booting a Fedora 28 Cloud image on AWS, the default network interface (eth0) gets up with a default MTU of 1500. However, the DCHP advertises 9001 as the proper MTU for that interface, and I can confirm that on older releases (F27-cloud) the MTU was correctly set to 9001.

# Components versions:

```
# cat /etc/os-release 
NAME=Fedora
VERSION="28 (Cloud Edition)"
ID=fedora
VERSION_ID=28
PRETTY_NAME="Fedora 28 (Cloud Edition)"
```

```
# uname -r
4.16.0-0.rc4.git0.1.fc28.x86_64
```

Actual results:

``
# ip addr show eth0 | grep mtu
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
```

# Expected results

Previously on Fedora 27:

```
# cat /etc/os-release 
NAME=Fedora
VERSION="27 (Cloud Edition)"
ID=fedora
VERSION_ID=27
PRETTY_NAME="Fedora 27 (Cloud Edition)"
```

```
# uname -r
4.13.15-300.fc27.x86_64
```

```
# ip addr show eth0 | grep mtu
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
```


# Additional info:

I'm coming here from these seemingly related bugs:
 * https://github.com/coreos/bugs/issues/2443
 * https://github.com/systemd/systemd/issues/9102

This kernel commit also showed up as interesting during investigation, but I'm not familiar enough with Xen drivers to follow the flow:
 * https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=f599c64fdf7d9c108e8717fb04bc41c680120da4

Comment 1 Laura Abbott 2018-05-30 14:08:32 UTC

4.16.0-0.rc4.git0.1.fc28.x86_64 is old, please update to the latest kernel version and test that.

Comment 2 Dusty Mabe 2018-05-30 14:18:57 UTC

hey @luca - you have a few options:

1. dnf update -y to get latest F28 kernel
2. There are no recent F28 cloud AMIs (since F28 release) but there are updated F28 Atomic Host images. You can boot one of those if you like. AMIs below.
3. You can book a rawhide AMI to see an even newer kernel

Fedora-AtomicHost-28-20180530.0.x86_64        eu-west-2            ami-4d6d812a    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ap-northeast-1       ami-2628d359    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        eu-central-1         ami-74e8dd9f    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        us-west-1            ami-3c42585c    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        us-west-2            ami-e9087291    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ap-southeast-2       ami-ffd7059d    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ca-central-1         ami-57f27133    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ap-southeast-1       ami-af2618d3    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        sa-east-1            ami-3c015850    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ap-northeast-2       ami-1c832872    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        eu-west-1            ami-46e4de3f    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        ap-south-1           ami-f46e409b    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        eu-west-2            ami-2413ff43    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ap-northeast-1       ami-7329d20c    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        eu-central-1         ami-17e9dcfc    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        us-west-1            ami-62425802    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        us-west-2            ami-31156f49    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ap-southeast-2       ami-5cd5073e    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ca-central-1         ami-78ef6c1c    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ap-southeast-1       ami-0229177e    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        sa-east-1            ami-a50f56c9    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ap-northeast-2       ami-518c273f    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        eu-west-1            ami-a3e4deda    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        ap-south-1           ami-8b90bfe4    hvm           standard
Fedora-AtomicHost-28-20180530.0.x86_64        us-east-1            ami-efbdd490    hvm           gp2
Fedora-AtomicHost-28-20180530.0.x86_64        us-east-1            ami-c34a23bc    hvm           standard\




Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-west-2            ami-6d00ec0a    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-northeast-1       ami-fd768e82    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-central-1         ami-ba4d7851    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-west-1            ami-38afb458    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-west-2            ami-2c96ed54    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-southeast-2       ami-0d85576f    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ca-central-1         ami-e5e76481    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-southeast-1       ami-a890afd4    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 sa-east-1            ami-e2e5bb8e    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-northeast-2       ami-30bc175e    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-west-1            ami-fe784287    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-south-1           ami-cdbd92a2    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-west-2            ami-f906ea9e    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-northeast-1       ami-05758d7a    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-central-1         ami-73427798    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-west-1            ami-d4afb4b4    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-west-2            ami-5497ec2c    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-southeast-2       ami-3c8a585e    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ca-central-1         ami-c1e764a5    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-southeast-1       ami-7193ac0d    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 sa-east-1            ami-79e5bb15    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-northeast-2       ami-23b3184d    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 eu-west-1            ami-a87943d1    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 ap-south-1           ami-c5bd92aa    hvm           standard
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-east-1            ami-ac2d43d3    hvm           gp2
Fedora-Cloud-Base-Rawhide-20180529.n.0.x86_64 us-east-1            ami-1cd6b963    hvm           standard

Comment 3 Luca BRUNO 2018-05-30 14:50:39 UTC

dnf-upgraded that same instance to `4.16.11-300.fc28.x86_64` and rebooted, still observing the same behavior as per the original report.

Comment 4 Luca BRUNO 2018-05-30 16:51:07 UTC

From an additional out-of-band suggestion: after upgrading the kernel (and only that) on the F27 instance to `4.16.12-200.fc27.x86_64` I see the same behavior as F28, MTU is capped to 1500.

Comment 5 Dusty Mabe 2018-05-30 16:57:04 UTC

@luca how do you reproduce this issue? I might be able to help you find what kernel it was introduced in.

Comment 6 Luca BRUNO 2018-05-30 18:02:42 UTC

Jumbo MTU is the default configuration from AWS DHCP. At least on F27, just booting a node without any further configuration lets cloud-init/dhclient pick up the option from DHCP and reflect it on the interface.

Comment 7 Benjamin Gilbert 2018-05-31 02:09:42 UTC

Reverting https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=f599c64fdf7d9c108e8717fb04bc41c680120da4 fixes the problem on Container Linux.

Comment 8 Dusty Mabe 2018-05-31 13:25:20 UTC

(In reply to Benjamin Gilbert from comment #7)
> Reverting
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/
> commit/?id=f599c64fdf7d9c108e8717fb04bc41c680120da4 fixes the problem on
> Container Linux.

Can we report that upstream and get this problem fixed?

Comment 9 Andrew Jeddeloh 2018-06-06 18:54:49 UTC

I reported this upstream on netdev:
https://marc.info/?l=linux-netdev&m=152830901817853&w=2

Comment 10 Dusty Mabe 2018-06-21 20:35:13 UTC

Laura,

The kernel you gave me today with upstream fixes [1] for this seems to have worked well!

[root@ip-10-0-123-104 ~]# ip addr show eth0                                                                                                           
                                                                                                                                                      
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 12:58:af:20:38:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.123.104/16 brd 10.0.255.255 scope global dynamic noprefixroute eth0
       valid_lft 3571sec preferred_lft 3571sec
    inet6 fe80::1058:afff:fe20:38b8/64 scope link
       valid_lft forever preferred_lft forever
[root@ip-10-0-123-104 ~]#
[root@ip-10-0-123-104 ~]# rpm -q kernel
kernel-4.17.2-201.xenfix.fc28.x86_64

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=27768255

Comment 11 Laura Abbott 2018-06-21 21:19:56 UTC

Great! I pushed this to F28, I expect rawhide will pick this up shortly.

Comment 12 Fedora Update System 2018-06-26 19:27:05 UTC

kernel-4.17.3-100.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2a0f8b2c9d

Comment 13 Fedora Update System 2018-06-26 19:28:41 UTC

kernel-4.17.3-200.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d92fde52d7

Comment 14 Fedora Update System 2018-06-28 12:31:11 UTC

kernel-4.17.3-100.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2a0f8b2c9d

Comment 15 Fedora Update System 2018-06-28 15:03:10 UTC

kernel-4.17.3-200.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d92fde52d7

Comment 16 Fedora Update System 2018-07-01 01:34:08 UTC

kernel-4.17.3-100.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2018-07-01 02:37:41 UTC

kernel-4.17.3-200.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Dusty Mabe 2018-07-01 22:05:28 UTC

looks fixed in kernel-4.17.3-200.fc28 

```
[fedora@ip-10-0-24-118 ~]$ ip -4 a show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 10.0.24.118/16 brd 10.0.255.255 scope global dynamic noprefixroute eth0
       valid_lft 3409sec preferred_lft 3409sec
[fedora@ip-10-0-24-118 ~]$ rpm -q kernel 
kernel-4.17.3-200.fc28.x86_64
```