Bug 968943

Summary: Fail to deploy zend app via zend sever console due to broken symlink in /usr/local/zend/bin/apachectl
Product: OpenShift Online Reporter: joycezhang <jinzhang>
Component: ContainersAssignee: Vojtech Vitek <vvitek>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: dmcphers, hripps, vvitek, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 04:14:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
HelloWorld.zpk none

Description joycezhang 2013-05-30 10:29:15 UTC
Description of problem:
Broken symlink is shown for apachctl of zend cartridge as below:
[root@ip-10-154-163-16 ~]# ls -lh /usr/local/zend/bin/apachectl 
lrwxrwxrwx. 1 root root 65 May 30 04:31 /usr/local/zend/bin/apachectl -> /usr/libexec/stickshift/cartridges/zend-5.6/info/bin/httpd_ctl.sh

Version-Release number of selected component (if applicable):
devenv_3290

How reproducible:
always

Steps to Reproduce:
1.SSH into instance and check symlink of usr/local/zend/bin/apachectl 


Actual results:
Broken symlink is returned.

Expected results:
The deprecated symlink should not exist.
[root@ip-10-154-163-16 ~]# ls -lh /usr/local/zend/bin/apachectl 
lrwxrwxrwx. 1 root root 65 May 30 04:31 /usr/local/zend/bin/apachectl 

Additional info:

Comment 1 Vojtech Vitek 2013-05-30 16:36:24 UTC
/usr/local/zend/bin/apachectl -> /usr/libexec/stickshift/cartridges/zend-5.6/info/bin/httpd_ctl.sh

This file is owned by root:root and was created during the %install of the zend cartridge RPM.

Only new zend cartridge RPM update can fix this.

Comment 2 Vojtech Vitek 2013-05-30 16:42:03 UTC
The intention is to overwrite the /usr/local/zend/bin/apachectl symlink with new value pointing to /sandbox/zend/bin/apachectl during the RPM update (%install).

From the gear's perspective (or the migration perspective), we must create the /sandbox/zend/bin/apachectl symlink pointing to the <GEAR-HOME>/zend/bin/control and make sure the control script behaves the same way as the apachectl does normally.

Comment 3 Vojtech Vitek 2013-05-30 16:46:13 UTC
Seems like this is the only symlink issue from the RPM installation perspective:

$ find /usr/local/zend/ -type l -exec ls -Al {} \; | grep -v -e '.so' -e _original
lrwxrwxrwx. 1 root root 17 May 26 12:44 /usr/local/zend/tmp -> /sandbox/zend/tmp
lrwxrwxrwx. 1 root root 17 May 26 12:44 /usr/local/zend/var -> /sandbox/zend/var
lrwxrwxrwx. 1 root root 3 May 26 00:25 /usr/local/zend/bin/php-cli -> php
lrwxrwxrwx. 1 root root 65 May 26 12:45 /usr/local/zend/bin/apachectl -> /usr/libexec/stickshift/cartridges/zend-5.6/info/bin/httpd_ctl.sh
lrwxrwxrwx. 1 root root 17 May 26 12:44 /usr/local/zend/etc -> /sandbox/zend/etc
lrwxrwxrwx. 1 root root 30 May 26 12:44 /usr/local/zend/gui/lighttpd/tmp -> /sandbox/zend/gui/lighttpd/tmp
lrwxrwxrwx. 1 root root 31 May 26 12:44 /usr/local/zend/gui/lighttpd/logs -> /sandbox/zend/gui/lighttpd/logs
lrwxrwxrwx. 1 root root 24 May 26 00:24 /usr/local/zend/gui/lighttpd/htdocs/ZendServer -> /usr/local/zend/gui/html
lrwxrwxrwx. 1 root root 30 May 26 12:44 /usr/local/zend/gui/lighttpd/etc -> /sandbox/zend/gui/lighttpd/etc
lrwxrwxrwx. 1 root root 34 May 26 12:44 /usr/local/zend/gui/application/data -> /sandbox/zend/gui/application/data

Comment 4 Xiaoli Tian 2013-05-31 07:12:54 UTC
Hi, Vojtech

Do you think this bug will affect zend function too much (we are not sure for this), since it seems the basic function is not affected? 

We just thought it's a broken symlink  which is already deprecated while filing the bugs, so set it to low severity.

If it will affect function too much, we need to raise its severity to medium at least and it needs to be fixed before sprint 28 release ( which may mean you have to  make a stage pull request for this bug)

Thanks

Comment 5 Vojtech Vitek 2013-05-31 13:32:41 UTC
I guess we could face some troubles in Zend Server deployment (through the Zend Server Console). But I couldn't actually reproduce any issues yet. Let me dig into that a bit more..

This is the configuration we could have troubles with:

zend-5.6/files/user-files/etc/zdd.ini:zend_deployment.webserver.apache.ctl=/usr/local/zend/bin/apachectl

zend/versions/5.6/configuration/user-files/etc/zdd.ini:zend_deployment.webserver.apache.ctl=/usr/local/zend/bin/apachectl

Comment 6 Vojtech Vitek 2013-05-31 13:55:20 UTC
Feedback from David Lowes, the Zend developer:
> then there will be problems with deployment of applications using
> the deployment feature
> the Zend Server gui uses a different mechanism so it should not have problems
> existing applications will continue to work
> users will have trouble deploying new applications using Zend Server's
> deployment mechanism
> since it modifies apache's settings and requires restart
> http://www.zend.com/en/products/server/deployment
> this is part of the added value that Zend Server provides over the distro's php
> we should not release without it working properly

Comment 7 Vojtech Vitek 2013-05-31 14:53:37 UTC
HOTFIX:

1. Edit zend/versions/5.6/configuration/user-files/etc/zdd.ini
 - zend_deployment.webserver.apache.ctl=/usr/local/zend/bin/apachectl
 + zend_deployment.webserver.apache.ctl=$OPENSHIFT_ZEND_DIR/bin/control

2. Create /sandbox/zend/bin/apachectl symlink that points to
 $OPENSHIFT_ZEND_DIR/bin/control

3. Update the v2 migration script (apply both 1. and 2.)

------------

AFTER FIX

4. Raise new low-severity bugzilla against wrong "apachectl" command while SSH'd into gear (the /usr/local/zend/bin/apachectl won't work)

5. Wait after all v1 apps get migrated to v2 format

6. Fix the /usr/local/zend/bin/apachectl symlink to point to /sandbox/zend/bin/apachectl

7. Optionally change the zend/versions/5.6/configuration/user-files/etc/zdd.ini back to
- zend_deployment.webserver.apache.ctl=$OPENSHIFT_ZEND_DIR/bin/control
+ zend_deployment.webserver.apache.ctl=/usr/local/zend/bin/apachectl

Comment 9 Vojtech Vitek 2013-05-31 17:29:43 UTC
master hotfix PR: https://github.com/openshift/li/pull/1503
stage hotfix PR: https://github.com/openshift/li/pull/1504

Comment 10 openshift-github-bot 2013-05-31 17:53:51 UTC
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/754aa6600e468a19b33c21a53c365fcda32ad74a
Fix Zend apachectl deployment settings

Fixes bug 968943
https://bugzilla.redhat.com/show_bug.cgi?id=968943

Comment 11 joycezhang 2013-06-03 09:21:37 UTC
Verified this bug on both STG and devenv-stage_356 as below:

1. Check apachectl status after migrating from devenv-stage_355 to devenv-stage_356, it can work well as below

[zendm3-dmtest12.dev.rhcloud.com bin]\> ls -lh /sandbox/zend/bin/apachectl 
lrwxrwxrwx. 1 root root 60 Jun  3 03:50 /sandbox/zend/bin/apachectl -> /var/lib/openshift/259200256249816428838912/zend/bin/control

sandbox/zend/bin/apachectl status
Application is running
177 177 177 177 0 0 14811 0 --:--:-- --:--:-- --:--:-- 0 Total Accesses: 0 Total kBytes: 0 Uptime: 426 ReqPerSec: 0 BytesPerSec: 0 BusyWorkers: 1 IdleWorkers: 0 Scoreboard: W...........................................................
[zendm3-dmtest12.dev.rhcloud.com bin]\> ls -lh /sandbox/zend/bin/apachectl 

2. On STG, we found the migration script haven't been run as /sandbox/zend/bin/apachectl doesn't exist.

Keep this bug on_qa, will check on STG after next time migration.

Comment 12 joycezhang 2013-06-03 10:05:52 UTC
Open another bug 970006 for tracking apachectl can't work issue.

Comment 13 Vojtech Vitek 2013-06-03 13:13:15 UTC
Created attachment 756299 [details]
HelloWorld.zpk

For QE:

How to test the Zend Server deployment:
1. Create Zend app
2. Open Zend Server Console at <app-url>/ZendServer
3. Click at Applications -> Deploy Application button
4. Upload HelloWorld.zpk and proceed the deployment with non-root path, such as '/HelloWorld/'
5. Make sure the HelloWorld application deploys successfully
6. Test the application at path from 4, such as <app-url>/HelloWorld/

Note: Nice deployment video tutorial: http://www.zend.com/en/products/server/deployment

Comment 14 Xiaoli Tian 2013-06-04 01:41:39 UTC
Comments for reminding QE: We need to add cases to cover deploy zend app via zend server console according to comment 13.

Comment 15 joycezhang 2013-06-04 07:01:25 UTC
@Vojtech Vitek  Thanks for your detailed steps provided about the Zend Server deployment.
We had such kind of cases to test Zend Server deployment for Drupal before, and no errors met when using Drupal data.

For the hellowword.zpk, we checked on STG(devenv-stage_357) on both new created and old zend apps, it can be deployed and accessed correctly.

But On STG(devenv-stage_357), we still couldn't find /sandbox/zend/bin/apachectl as below, could you please help double confirm if the migration script has been run on STG?

[zend1-stgtest1.stg.rhcloud.com 51a816592587c8a6cb0002a0]\> /sandbox/zend/bin/apachectl
bash: /sandbox/zend/bin/apachectl: No such file or directory

Comment 16 joycezhang 2013-06-04 07:05:04 UTC
@Vojtech Vitek more info need to update, it's just happened for the old zend apps. For the new created zend apps, /sandbox/zend/bin/apachectl could work well.

Comment 17 joycezhang 2013-06-04 07:41:44 UTC
@Vojtech Vitek  We also found the <defaut-server> can't show the value sometimes after the deployment in Dashboard of ZendServer as below

Recent Events
Id 	Generated by Rule 	Count 	Last Occ. 	Severity 	Origin
 
000002 	Slow Request Execution (Absolute) 	1 	04-Jun 01:43 	Warning 	http://zend1-stgtest1.stg.rhcloud.com/test1/
000003 	Application Deployed Successfully 	1 	04-Jun 02:16 	Info 	http://<default-server>/helloword
000001 	Application Deployed Successfully 	1 	04-Jun 01:40 	Info 	http://<default-server>/test1


Is that correct result?

Comment 18 Vojtech Vitek 2013-06-04 11:44:30 UTC
@zhangjing are you talking about /sandbox/zend/bin/apachectl in "root" context on STG? That would be OK, as we need the symlink only from "gear" perspective.

Actually, we don't need that symlink at all right now - it's low priority and it can be solved together with bug 970006.

Comment 19 Vojtech Vitek 2013-06-04 11:46:45 UTC
Ad Slow Request Execution - Should be fine, as it's just Warning. See http://files.zend.com/help/previous-version/Zend-Server-4-Community-Edition/slow_request_execution.htm
I'll double check this with Zend guys.

Comment 20 joycezhang 2013-06-05 09:34:49 UTC
Move it as verified according to comment 18. Thanks.

Comment 21 Vojtech Vitek 2013-06-06 16:14:58 UTC
Just for record.

Feedback from David Lowes, the Zend developer:
> Slow request execution is a warning that can be safely ignored
> (the thresholds are configurable).