Bug 1414194 - kernel 4.9.3 breaks container networking under OpenShift
Summary: kernel 4.9.3 breaks container networking under OpenShift
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1414469 1422172
TreeView+ depends on / blocked
 
Reported: 2017-01-18 01:35 UTC by Joel Diaz
Modified: 2019-01-09 12:54 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1414469 (view as bug list)
Environment:
Last Closed: 2017-04-11 15:04:58 UTC
Type: Bug


Attachments (Terms of Use)

Description Joel Diaz 2017-01-18 01:35:32 UTC
Description of problem:
I recently started to be unable to communicate (TCP) from one container to another container under OpenShift origin. After much digging around, I've found that kernel 4.9.3-200.fc25.x86_64 is where the problems have started. I've confirmed that an earlier kernel 4.8.16-300.fc25.x86_64 works.

Version-Release number of selected component (if applicable):
kernel-4.9.3-200.fc25.x86_64

How reproducible:
100%

Steps to Reproduce:
Unfortunately I don't know how else to reproduce this, so bear with the intricate setup.
1. Download/untar OpenShift Origin from https://github.com/openshift/origin/releases/download/v1.3.2/openshift-origin-client-tools-v1.3.2-ac1d579-linux-64bit.tar.gz'
  1a) Install docker, and add the following line to /etc/sysconfig/docker:
        INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'
  1b) sudo systemctl start docker
2. sudo firewall-cmd --add-service=dns (so that containers can do DNS resolution from one container to the next)
3. sudo ./openshift-origin.../oc cluster up
4. Connect to openshift web url using your browser (provided as output from 'oc cluster up')
5. Log in (user: developer, password: developer), select the "My Project" project, and 'Add to project' the 'mysql-ephemeral' container
6. 'Add to project' a second 'mysql-ephemeral' container. Call the "Database Service Name" 'mysql2' to differentiate it (just need a container with the mysql client installed)
7. From the first mysql container, see the environment variables that were set for MYSQL_USER and MYSQL_PASSWORD. You can find this in the web UI through Applications->Pods then viewing the "Environment" tab on the first mysql container.
8. From the second mysql container, go to the terminal: Applications->Pods->second mysql container, then "Terminal" tab.
9. From the second mysql container shell: mysql -u<MYSQL_USER var from first mysql pod> -p<MYSQL_PASSWORD from first mysql pod> -hmysql sampledb


Actual results:
connecting to the mysql pod fails

Expected results:
connecting to the mysql pod succeeds

Additional info:
performing the above steps with kernel-4.8.16-300.fc25.x86_64 works just fine.

Comment 1 Laura Abbott 2017-01-18 01:54:12 UTC
Can you try the scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=17316113 ? This fixed an issue with kubernetes networking (see https://bugzilla.redhat.com/show_bug.cgi?id=1414068)

Comment 2 Joel Diaz 2017-01-18 02:12:29 UTC
(In reply to Laura Abbott from comment #1)
> Can you try the scratch build at
> https://koji.fedoraproject.org/koji/taskinfo?taskID=17316113 ? This fixed an
> issue with kubernetes networking (see
> https://bugzilla.redhat.com/show_bug.cgi?id=1414068)

Kernel 4.9.4-202.rhbz1414068.fc25 does let the kubernetes inter-container networking work again.

Comment 3 Justin M. Forbes 2017-04-11 14:57:00 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 25 kernel bugs.

Fedora 25 has now been rebased to 4.10.9-200.fc25.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26.

If you experience different issues, please open a new bug report for those.

Comment 4 Joel Diaz 2017-04-11 15:04:58 UTC
Works with kernel-4.9.14-200.fc25.x86_64


Note You need to log in before you can comment on or make changes to this bug.