Bug 1173026

Summary: watchman cleans up frontend config dirs but not files referencing them
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: ContainersAssignee: Brenton Leanhardt <bleanhar>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.2.0CC: agrimm, jokerman, libra-bugs, libra-onpremise-devel, mmccomas, rpenta, yadu
Target Milestone: ---Keywords: NeedsTestCase
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-origin-node-util-1.32.4.1-1 Doc Type: Bug Fix
Doc Text:
Cause: Previously, watchman would delete frontend configuration directories for secondary gears before the gears were removed. Consequence: Watchman could yield a broken frontend configuration. Fix: Watchman now only removes configuration directories which have no references in other configuration files. Result: Watchman no longer prematurely removes frontend configurations for secondary gears.
Story Points: ---
Clone Of: 1172323 Environment:
Last Closed: 2015-01-08 15:34:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1169306, 1172323    
Bug Blocks:    

Description Gaoyun Pei 2014-12-11 10:25:38 UTC
The same issue on ose-2.2/2014-12-10.1, openshift-origin-node-util-1.32.2.1-1.el6op.noarch

Frontend plugin will remove empty httpd config dir of scale-up gear when the conf file of this gear still existing. And this leads to httpd restarts failure.

+++ This bug was initially created as a clone of Bug #1172323 +++

Description of problem:

in commit ef1e1cf2 (which was written to fix BZ 1169306), code was added to the watchman frontend plugin which removes empty config directories.  However, this code does not remove the conf files which reference those directories.  The result is a broken apache configuration.


Version-Release number of selected component (if applicable):
openshift-origin-node-util-1.32.2-1.el6oso.noarch

How reproducible:
Always

Steps to Reproduce:
1. Restart watchman on a node with empty config directories for secondary gears
2. Restart apache

Actual results:

Starting httpd: httpd.worker: Syntax error on line 222 of /etc/httpd/conf/httpd.conf: Syntax error on line 75 of /etc/httpd/conf.d/000000_default.conf: Syntax error on line 72 of /etc/httpd/conf.d/openshift/52a00fa7e0b8cde0c70003ee_lara_0_52a00fa7e0b8cde0c70003ee.conf: Include directory '/etc/httpd/conf.d/openshift/52a00fa7e0b8cde0c70003ee_lara_52a00fa7e0b8cde0c70003ee' not found

Expected results:

A working apache configuration

--- Additional comment from Andy Grimm on 2014-12-09 15:29:37 EST ---

I should have actually read BZ 1169306 before I wrote this report.  I suppose the problem should be stated differently -- the issue should probably be stated as "frontend plugin is remove conf directories for secondary gears which have not yet been deleted", or something to that effect.

--- Additional comment from openshift-github-bot on 2014-12-10 18:18:30 EST ---

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/057462d055d6443dc89213866d5b5f7c1f51e644
Bug 1172323 - Only remove unreferenced empty directories from the frontend configuration

--- Additional comment from Yan Du on 2014-12-11 02:28:36 EST ---

Test on devenv_5350

Issue have been fixed. openshift-watchman and apache
can be restart normally on a node with empty config directories for secondary gears. and after restart openshift-watchman restart, the conf directories for secondary gears will be removed.

[root@ip-10-81-188-254 ~]# service openshift-watchman restart
Stopping Watchman
Starting Watchman
[root@ip-10-81-188-254 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: [Thu Dec 11 07:24:11 2014] [warn] module php5_module is already loaded, skipping
[Thu Dec 11 07:24:11 2014] [warn] module passenger_module is already loaded, skipping
[Thu Dec 11 07:24:11 2014] [warn] module passenger_module is already loaded, skipping
Warning: DocumentRoot [/var/www/html/binaryartifacts] does not exist
[Thu Dec 11 07:24:11 2014] [warn] NameVirtualHost *:443 has no VirtualHosts
[Thu Dec 11 07:24:11 2014] [warn] NameVirtualHost *:80 has no VirtualHosts
                                                           [  OK  ]

--- Additional comment from Yan Du on 2014-12-11 03:52:38 EST ---

Steps to verified:

1. Create a scalable app with db embedded
2. Scale up once
3. Manually mpty the config files in scale up gear  
4. service openshift-watchman restart
5. service httpd restart

After step4, the empty db config directory will be removed, but the empty scale up gear directory will not be removed. 

And in Step5, apache service can be restart normally without any error.

Comment 3 Gaoyun Pei 2014-12-12 01:11:43 UTC
Verify this bug with openshift-origin-node-util-1.32.4.1-1.el6op.noarch.

1. Create a scalable app with db embedded
2. Scale up the app once
[root@broker ~]# rhc app show bugzz --gears
ID         State   Cartridges   Size  SSH URL
---------- ------- ------------ ----- ------------------------------------------
jj-bugzz-1 started python-3.3   small jj-bugzz-1.com.cn
                   haproxy-1.4
jj-bugzz-2 started python-3.3   small jj-bugzz-2.com.cn
jj-bugzz-3 started mongodb-2.4  small jj-bugzz-3.com.cn

3. Got the following httpd conf files and dirs, manually empty the config files in scale up gear: jj-bugzz-2_jj_jj-bugzz-2
-rw-r--r--.  1 root root    2451 Dec 11 17:57 jj-bugzz-1_jj_0_bugzz.conf
drwxr-xr-x.  2 root root    4096 Dec 11 17:57 jj-bugzz-1_jj_bugzz
-rw-r--r--.  1 root root    2471 Dec 11 17:59 jj-bugzz-2_jj_0_jj-bugzz-2.conf
drwxr-xr-x.  2 root root    4096 Dec 11 17:59 jj-bugzz-2_jj_jj-bugzz-2
drwxr-xr-x.  2 root root    4096 Dec 11 18:01 jj-bugzz-3_jj_jj-bugzz-3

4. Set the system time to 2 days later, service openshift-watchman restart
The empty conf dir of scale up gear was not deleted.
The empty conf dir of db gear was deleted.

5. service httpd could restart successfully.

Comment 5 errata-xmlrpc 2015-01-08 15:34:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0019.html