Bug 1161623

Summary:	Apache vhosts config missing, truncated
Product:	OpenShift Container Platform	Reporter:	Brenton Leanhardt <bleanhar>
Component:	Containers	Assignee:	Brenton Leanhardt <bleanhar>
Status:	CLOSED ERRATA	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	high
Version:	2.2.0	CC:	adellape, anli, bmeng, jhonce, jokerman, libra-bugs, libra-onpremise-devel, mmccomas, pruan, sten
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-openshift-origin-common-1.29.1.1-1, rubygem-openshift-origin-node-1.32.3.1-1, rubygem-openshift-origin-frontend-apache-vhost-0.11.3.1-1	Doc Type:	Bug Fix
Doc Text:	Apache vhost configuration creation was not always handled in an atomic manner. In environments with a high number of concurrent front end modifications, invalid configuration files could be created. This bug fix makes Apache vhost configurations atomic by ensuring that concurrent modifications do not result in broken configuration files. As a result, concurrent application creation, updates, and deletion no longer results in broken Apache vhost configuration files. Additionally, a Watchman plug-in has been added to check for this type of error.	Story Points:	---
Clone Of:	1160752	Environment:
Last Closed:	2014-12-10 13:25:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1160752
Bug Blocks:

Description Brenton Leanhardt 2014-11-07 13:59:09 UTC

+++ This bug was initially created as a clone of Bug #1160752 +++

Description of problem: Since release 2.0.52, apache vhosts configurations are missing or incompleted. 


Version-Release number of selected component (if applicable): rubygem-openshift-origin-node-1.31.9-1.el6oso.noarch


How reproducible: sometimes


Steps to Reproduce:
Has been shown to happen on existing apps, possibly new apps. 

Actual results:
Gears return 404, 400, 302

Expected results:
Apache vhosts config should be complete and working

Additional info:
Attempted fix: 
find /etc/httpd/conf.d/openshift/ -type d -empty - match out that list, | xargs rhc-fix-missing-frontend -b
This fixed ~990 out of 10900+ empty apache dirs, didn't help with gears missing in /etc/httpd/conf.d/openshift/ or with incomplete vhost entries.

--- Additional comment from openshift-github-bot on 2014-11-06 21:18:38 EST ---

Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/5b062cabb7ef125fc3743eea1b8fc2bbb17041c3
Bug 1160752 - Make apache-vhost more atomic

* Write one buffer to file rather than a number of write
* Truncate conf files rather than removing them to ease up httpd
  failures on restart
* Have watchman XXX plugin clean up "deleted" configurations

https://github.com/openshift/origin-server/commit/8710485cb06793ae67e4bac9aab349a41e0b19f2
Bug 1160752 - Add Watchman plugin to clean up vhost configurations

https://github.com/openshift/origin-server/commit/d2299d1c917227e9dad313ee467dda74fc88a9e7
Bug 1160752 - Make Watchman frontend cleanup period configurable

--- Additional comment from Meng Bo on 2014-11-07 05:01:03 EST ---

Do parallel creating/deleting and accessing the gears on node, will not cause httpd workers hang.

And after the watchman plugin introduced, the old httpd config files will not be deleted when the gear is deleting.
Set the time to two days past, the watchman will delete the configs automatically.

Move bug to verified.

Comment 3 Anping Li 2014-11-25 10:42:37 UTC

Verified and pass on puddle-2-2-2014-11-24
1) Do parallel creating/deleting and accessing the gears on node for one day.
   No httpd worker hang was found.
2) diff gears and the httpd configure files.
   All gears has httpd configure files.
   There are extra httpd configure files for deleted gears.
3) set FRONTEND_CLEANUP_PERIOD=10 and restart watchman

4) wating several minutes, and those deleted gears httpd configure files are deleted.

Comment 5 errata-xmlrpc 2014-12-10 13:25:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-1979.html