Bug 1414194

Summary:	kernel 4.9.3 breaks container networking under OpenShift
Product:	[Fedora] Fedora	Reporter:	Joel Diaz <jdiaz>
Component:	kernel	Assignee:	Kernel Maintainer List <kernel-maint>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	25	CC:	cz172638, gansalmon, ichavero, itamar, jonathan, kernel-maint, labbott, madhu.chinakonda, mchehab
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1414469 (view as bug list)		Environment:
Last Closed:	2017-04-11 15:04:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1414469, 1422172

Description Joel Diaz 2017-01-18 01:35:32 UTC

Description of problem:
I recently started to be unable to communicate (TCP) from one container to another container under OpenShift origin. After much digging around, I've found that kernel 4.9.3-200.fc25.x86_64 is where the problems have started. I've confirmed that an earlier kernel 4.8.16-300.fc25.x86_64 works.

Version-Release number of selected component (if applicable):
kernel-4.9.3-200.fc25.x86_64

How reproducible:
100%

Steps to Reproduce:
Unfortunately I don't know how else to reproduce this, so bear with the intricate setup.
1. Download/untar OpenShift Origin from https://github.com/openshift/origin/releases/download/v1.3.2/openshift-origin-client-tools-v1.3.2-ac1d579-linux-64bit.tar.gz'
1a) Install docker, and add the following line to /etc/sysconfig/docker:
INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'
1b) sudo systemctl start docker
2. sudo firewall-cmd --add-service=dns (so that containers can do DNS resolution from one container to the next)
3. sudo ./openshift-origin.../oc cluster up
4. Connect to openshift web url using your browser (provided as output from 'oc cluster up')
5. Log in (user: developer, password: developer), select the "My Project" project, and 'Add to project' the 'mysql-ephemeral' container
6. 'Add to project' a second 'mysql-ephemeral' container. Call the "Database Service Name" 'mysql2' to differentiate it (just need a container with the mysql client installed)
7. From the first mysql container, see the environment variables that were set for MYSQL_USER and MYSQL_PASSWORD. You can find this in the web UI through Applications->Pods then viewing the "Environment" tab on the first mysql container.
8. From the second mysql container, go to the terminal: Applications->Pods->second mysql container, then "Terminal" tab.
9. From the second mysql container shell: mysql -u<MYSQL_USER var from first mysql pod> -p<MYSQL_PASSWORD from first mysql pod> -hmysql sampledb

Actual results:
connecting to the mysql pod fails

Expected results:
connecting to the mysql pod succeeds

Additional info:
performing the above steps with kernel-4.8.16-300.fc25.x86_64 works just fine.

Comment 1 Laura Abbott 2017-01-18 01:54:12 UTC

Can you try the scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=17316113 ? This fixed an issue with kubernetes networking (see https://bugzilla.redhat.com/show_bug.cgi?id=1414068)

Comment 2 Joel Diaz 2017-01-18 02:12:29 UTC

(In reply to Laura Abbott from comment #1)
> Can you try the scratch build at
> https://koji.fedoraproject.org/koji/taskinfo?taskID=17316113 ? This fixed an
> issue with kubernetes networking (see
> https://bugzilla.redhat.com/show_bug.cgi?id=1414068)

Kernel 4.9.4-202.rhbz1414068.fc25 does let the kubernetes inter-container networking work again.

Comment 3 Justin M. Forbes 2017-04-11 14:57:00 UTC

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 25 kernel bugs.

Fedora 25 has now been rebased to 4.10.9-200.fc25.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26.

If you experience different issues, please open a new bug report for those.

Comment 4 Joel Diaz 2017-04-11 15:04:58 UTC

Works with kernel-4.9.14-200.fc25.x86_64