Bug 1420182

Summary:	[3.5] conntrack executable not found on $PATH during cluster horizontal run
Product:	OpenShift Container Platform	Reporter:	Mike Fiedler <mifiedle>
Component:	Installer	Assignee:	Giuseppe Scrivano <gscrivan>
Status:	CLOSED ERRATA	QA Contact:	Johnny Liu <jialiu>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.5.0	CC:	aos-bugs, jokerman, mifiedle, mmccomas
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	OCP 3.4 and 3.3 introduced a requirement on the conntrack executable but this dependency was not enforced at install time so service proxy management may have failed post installation. The installer now ensures that conntrack is installed.	Story Points:	---
Clone Of:
Clones:	1420393 1420395 (view as bug list)		Environment:
Last Closed:	2017-04-12 18:49:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1420393, 1420395

Description Mike Fiedler 2017-02-08 04:05:46 UTC

Description of problem:

While scaling a 300 node cluster up to 1000 projects and 4000 running pods, the following node Error level message popped a few times in the log on the master.   I need to check other nodes to see if they hit it, too.

Feb  7 22:56:58 192 atomic-openshift-node: E0207 22:56:58.045551    3115 proxier.go:755] conntrack return with error: Error looking for path of conntrack: exec: "conntrack": executable file not found in $PATH
Feb  7 22:56:58 192 atomic-openshift-node: E0207 22:56:58.045600    3115 proxier.go:755] conntrack return with error: Error looking for path of conntrack: exec: "conntrack": executable file not found in $PATH



Version-Release number of selected component (if applicable): 3.5.0.17


How reproducible:   Unknown,  saw a few instances during this test and will investigate other node logs


Expected:  No Error or Fatal messages.

Comment 2 Mike Fiedler 2017-02-08 04:07:45 UTC

This cluster was installed with the latest (as of 7 Feb) master branch of openshift-ansible/playbooks/byo/config.yml

Comment 3 Ben Bennett 2017-02-08 14:14:36 UTC

We need to make sure that the libnetfilter_conntrack RPM is installed on any node running openshift-node.

Comment 4 Scott Dodson 2017-02-08 14:39:51 UTC

/usr/sbin/conntrack is provided by conntrack-tools need to ensure that's installed on all nodes.

Comment 5 Scott Dodson 2017-02-08 14:41:40 UTC

Including verifying availability on atomic host and/or the containerized node image. Currently libnetfilter_conntrack is required there but not conntrack-tools

Comment 6 Giuseppe Scrivano 2017-02-09 10:43:54 UTC

change for the container version:

https://github.com/openshift/origin/pull/12885

not containerized version:

https://github.com/openshift/openshift-ansible/pull/3305

Comment 8 Johnny Liu 2017-02-22 14:13:29 UTC

Verified this bug with openshift-ansible-3.5.13-1.git.0.562e91d.el7.noarch, and PASS.

For rpm install,the following step is shown in node install.
TASK [openshift_node : Install conntrack-tools package] ************************

After installation, check rpm package.
# rpm -q conntrack-tools
conntrack-tools-1.4.3-1.el7.x86_64

For containerized install:
# docker run --rm -ti --entrypoint rpm openshift3/node:v3.5.0.32  -q conntrack-tools
conntrack-tools-1.4.3-1.el7.x86_64

Comment 10 errata-xmlrpc 2017-04-12 18:49:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903