Bug 1008845

Summary: [origin_runtime_245] It is still trying to use mod_rewrite method when setting the OPENSHIFT_FRONTEND_HTTP_PLUGINS to vhost in node.conf
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: ContainersAssignee: Rob Millner <rmillner>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: dmcphers, mfisher
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-17 13:29:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Meng Bo 2013-09-17 07:34:32 UTC
Description of problem:
Create app after set the node use vhost frontend plugin will fail with error Unexpected error: No such file or directory - /etc/httpd/conf.d/openshift/nodes.txt

Version-Release number of selected component (if applicable):
devenv_3786

How reproducible:
always

Steps to Reproduce:
1. Launch instance
2. Install the vhost plugin
rpm -e --nodeps rubygem-openshift-origin-frontend-apache-mod-rewrite.noarch rubygem-openshift-origin-frontend-nodejs-websocket.noarch
yum install rubygem-openshift-origin-frontend-apache-vhost.noarch
3. Modify the 000000_default.conf and comment the openshift_route.include lines
#  include conf.d/openshift_route.include
4. Modify the node.conf to use the vhost plugin
OPENSHIFT_FRONTEND_HTTP_PLUGINS=openshift-origin-frontend-apache-vhost
5. Restart httpd and mcollective
6. Create app

Actual results:
App creation will fail with the error:

Unexpected error: No such file or directory - /etc/httpd/conf.d/openshift/nodes.txt

Expected results:
App creation should get success.

Additional info:
It worked well on the fork_ami_828

Error log in platform.log during app creation.
<--snip-->
September 17 02:20:22 WARN V2CartModel#connect_frontend: No such file or directory - /etc/httpd/conf.d/openshift/nodes.txt
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:807:in `size'
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:807:in `compare_file'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:221:in `flush'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apache-mod-rewrite-0.1.3/lib/openshift/runtime/frontend/http/plugins/apache-mod-rewrite.rb:72:in `connect'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:555:in `block in call_plugins'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:551:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:551:in `call_plugins'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:325:in `connect'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:953:in `block (2 levels) in connect_frontend'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:930:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:930:in `block in connect_frontend'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:929:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:929:in `connect_frontend'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:293:in `block in configure'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:126:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:126:in `apply_profile'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:44:in `block (2 levels) in <class:Cgroups>'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:254:in `configure'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:18:in `configure'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:767:in `block in oo_configure'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:290:in `with_container_from_args'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:766:in `oo_configure'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:136:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:103:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
September 17 02:20:22 ERROR Unexpected error during configure: No such file or directory - /etc/httpd/conf.d/openshift/nodes.txt (Errno::ENOENT)
  /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:807:in `size'
  /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:807:in `compare_file'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:221:in `flush'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:240:in `close'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apachedb-0.1.3/lib/openshift/runtime/frontend/http/plugins/apachedb.rb:265:in `open'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-frontend-apache-mod-rewrite-0.1.3/lib/openshift/runtime/frontend/http/plugins/apache-mod-rewrite.rb:72:in `connect'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:555:in `block in call_plugins'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:551:in `map'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:551:in `call_plugins'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/frontend_httpd.rb:325:in `connect'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:953:in `block (2 levels) in connect_frontend'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:930:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:930:in `block in connect_frontend'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:929:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:929:in `connect_frontend'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:293:in `block in configure'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:126:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:126:in `apply_profile'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/utils/cgroups.rb:44:in `block (2 levels) in <class:Cgroups>'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/v2_cart_model.rb:254:in `configure'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.15.0/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:18:in `configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:767:in `block in oo_configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:290:in `with_container_from_args'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:766:in `oo_configure'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:136:in `execute_action'
  /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:103:in `cartridge_do_action'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
  /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
  /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
<--snip-->

Comment 1 Rob Millner 2013-09-19 19:35:05 UTC
I can't reproduce the bug on devenv_3802.  


Apps were created for python-2.6, 2.7 and 3.3 all with non-scaled, scaled, jenkins, and scaled+jenkins.  All scaled apps were scaled out to three gears.  Aliases for foo.example.com and bar.example.com were added to the python 2.7 scaled+jenkins app.  SSL cert+key was added for foo.example.com.

(scripts from the python 2.7 project)

Verify you can talk to the apps and get the correct SSL certs from foo.example.com.

ex:
vi /etc/hosts   # add "127.0.0.2 foo.example.com"
curl -vvv -k https://foo.example.com/


Here's the sequence I used to test it.  The yum repository has an older, buggy build of the vhost package but the one in /root/devenv-local.

# oo-frontend-plugin-modify --save > file
Backing up 25 frontends.

# oo-frontend-plugin-modify --delete
Delete all web frontend configuration? (yes/no)
yes

# rpm --nodeps -e rubygem-openshift-origin-frontend-apache-mod-rewrite

# rpm -Uvh devenv-local/rubygem-openshift-origin-frontend-apache-vhost-*.el6.noarch.rpm
Preparing...                ########################################### [100%]
   1:rubygem-openshift-origi########################################### [100%]


# vi /etc/httpd/conf.d/000000_default.conf
      Comment out the following line from both vhosts:
      # include conf.d/openshift_route.include

# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: [Thu Sep 19 14:59:12 2013] [warn] module passenger_module is already loaded, skipping
[Thu Sep 19 14:59:12 2013] [warn] NameVirtualHost *:443 has no VirtualHosts
[Thu Sep 19 14:59:12 2013] [warn] NameVirtualHost *:80 has no VirtualHosts
                                                           [  OK  ]

# vi /etc/openshift/node.conf

     Change the OPENSHIFT_FRONTEND_HTTP_PLUGINS line at the bottom to use vhost instead of mod-rewrite.
     OPENSHIFT_FRONTEND_HTTP_PLUGINS=openshift-origin-frontend-apache-vhost,openshift-origin-frontend-nodejs-websocket


# service ruby193-mcollective restart
Shutting down mcollective:                                 [  OK  ]
Starting mcollective:                                      [  OK  ]

# oo-frontend-plugin-modify --restore < file  # takes a few minutes...


Verify apps are reachable again and you get the correct cert.
# curl -vvv -k https://foo.example.com/

Verify that broker is reachable
# rhc domain show
...
You have 13 applications in your domain.

Comment 2 Rob Millner 2013-09-19 19:40:11 UTC
On the same node, I deleted applications+aliases and rebuilt them.  The script to add aliases failed with the following error:

ould not add job 'newpysj27-build' in Jenkins server:
   create_job status: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /createItem was not found on this server.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at jenkins-rmillner0115.dev.rhcloud.com Port 443</address>
</body></html>
build config does not exist
You'll need to correct this error before attempting to embed the Jenkins client again.


Could not add job 'newpysj27-build' in Jenkins server:
   create_job status: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /createItem was not found on this server.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at jenkins-rmillner0115.dev.rhcloud.com Port 443</address>
</body></html>
build config does not exist
You'll need to correct this error before attempting to embed the Jenkins client again.

Alias foo.example.com can't be found in application .

Comment 3 Rob Millner 2013-09-19 21:08:48 UTC
Pull request to fix the jenkins error.  This will require an updated version of the vhost package.

https://github.com/openshift/origin-server/pull/3671

Comment 4 Meng Bo 2013-09-23 08:05:01 UTC
The mcollective service was moved to ruby193-mcollective.

It works for me too.

Close this bug and use the bug 1010047 to tracking the alias issue.