Bug 1161623 - Apache vhosts config missing, truncated
Summary: Apache vhosts config missing, truncated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: ---
Assignee: Brenton Leanhardt
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1160752
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-07 13:59 UTC by Brenton Leanhardt
Modified: 2015-04-02 02:21 UTC (History)
10 users (show)

Fixed In Version: rubygem-openshift-origin-common-1.29.1.1-1, rubygem-openshift-origin-node-1.32.3.1-1, rubygem-openshift-origin-frontend-apache-vhost-0.11.3.1-1
Doc Type: Bug Fix
Doc Text:
Apache vhost configuration creation was not always handled in an atomic manner. In environments with a high number of concurrent front end modifications, invalid configuration files could be created. This bug fix makes Apache vhost configurations atomic by ensuring that concurrent modifications do not result in broken configuration files. As a result, concurrent application creation, updates, and deletion no longer results in broken Apache vhost configuration files. Additionally, a Watchman plug-in has been added to check for this type of error.
Clone Of: 1160752
Environment:
Last Closed: 2014-12-10 13:25:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1979 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.2.2 bug fix and enhancement update 2014-12-10 18:23:46 UTC

Description Brenton Leanhardt 2014-11-07 13:59:09 UTC
+++ This bug was initially created as a clone of Bug #1160752 +++

Description of problem: Since release 2.0.52, apache vhosts configurations are missing or incompleted. 


Version-Release number of selected component (if applicable): rubygem-openshift-origin-node-1.31.9-1.el6oso.noarch


How reproducible: sometimes


Steps to Reproduce:
Has been shown to happen on existing apps, possibly new apps. 

Actual results:
Gears return 404, 400, 302

Expected results:
Apache vhosts config should be complete and working

Additional info:
Attempted fix: 
find /etc/httpd/conf.d/openshift/ -type d -empty - match out that list, | xargs rhc-fix-missing-frontend -b
This fixed ~990 out of 10900+ empty apache dirs, didn't help with gears missing in /etc/httpd/conf.d/openshift/ or with incomplete vhost entries.

--- Additional comment from openshift-github-bot on 2014-11-06 21:18:38 EST ---

Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/5b062cabb7ef125fc3743eea1b8fc2bbb17041c3
Bug 1160752 - Make apache-vhost more atomic

* Write one buffer to file rather than a number of write
* Truncate conf files rather than removing them to ease up httpd
  failures on restart
* Have watchman XXX plugin clean up "deleted" configurations

https://github.com/openshift/origin-server/commit/8710485cb06793ae67e4bac9aab349a41e0b19f2
Bug 1160752 - Add Watchman plugin to clean up vhost configurations

https://github.com/openshift/origin-server/commit/d2299d1c917227e9dad313ee467dda74fc88a9e7
Bug 1160752 - Make Watchman frontend cleanup period configurable

--- Additional comment from Meng Bo on 2014-11-07 05:01:03 EST ---

Do parallel creating/deleting and accessing the gears on node, will not cause httpd workers hang.

And after the watchman plugin introduced, the old httpd config files will not be deleted when the gear is deleting.
Set the time to two days past, the watchman will delete the configs automatically.

Move bug to verified.

Comment 3 Anping Li 2014-11-25 10:42:37 UTC
Verified and pass on puddle-2-2-2014-11-24
1) Do parallel creating/deleting and accessing the gears on node for one day.
   No httpd worker hang was found.
2) diff gears and the httpd configure files.
   All gears has httpd configure files.
   There are extra httpd configure files for deleted gears.
3) set FRONTEND_CLEANUP_PERIOD=10 and restart watchman

4) wating several minutes, and those deleted gears httpd configure files are deleted.

Comment 5 errata-xmlrpc 2014-12-10 13:25:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-1979.html


Note You need to log in before you can comment on or make changes to this bug.