Bug 1037704

Summary: Scaling doesn't work properly - PHP
Product: OpenShift Online Reporter: info
Component: ImageAssignee: Paul Morie <pmorie>
Status: CLOSED NOTABUG QA Contact: libra bugs <libra-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.xCC: dmcphers, info, jhadvig, jhonce, vvitek
Target Milestone: ---Keywords: Reopened, SupportQuestion, UpcomingRelease
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-03 17:30:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description info 2013-12-03 15:55:11 UTC
Description of problem:

Besides my application not being able to support more than 5 concurrent connections (seems like resources are not there really), the scaling doesn't happen because without reaching the 15 connections limits the haproxy doesn't start a new gear. Ok, I set the second gear manually, but then all traffic is still routed to the local gear. When it goes 503, the second gear doesn't have the repo on it. When the local gear comes back up, it's like the second: empty.



How reproducible:


Steps to Reproduce:
1. Set scaling to minimum 2 maximum all available gears
2. Open 6 pages of the app so that the local-gear goes down (yes, 6 concurrent requests will get "Unable to allocate memory for pool" error!) - you may need to test with an app that shuts down before the scaling happens by haproxy.
3. The second gear stays up but has no content - only the default repo (health_check.php and index.php)
4. When the local-gear comes back up from the very critical 4-5 connections, it seems to behave like the second, i.e. default repo - this stays like this until you manually shut down the second gear but limiting the scaling from 1 minimum to 1 max. - haproxy shows that the second gear is up since more minutes than the local gear, which I guess shouldn't happen??)
5. Everything back to normal when gear min = max = 1

Comment 1 Paul Morie 2013-12-03 20:29:56 UTC
If we could get some additional info it would help to diagnose the issue with your app.  Would you provide:

1. Your app's name
2. Your openshift domain
3. The email address associated w/ the account

Thanks

Comment 2 Paul Morie 2013-12-03 20:33:18 UTC
Additionally, what is your app trying to do?  Are there any per-connection operations that could explain the performance issues you're seeing?

Comment 3 info 2013-12-03 23:01:20 UTC
app name is rain which you can find at rain-dogshouse.rhcloud.com and email address is info [a t ] pempo.it
The app is just a simple scalable Wordpress (scalable) installation which I used to test the platform and gears'capacity. The code is probably not perfect but the scaling behaviour i'm showing is probably not related to that. 
And yes, I used the explicit database address and port. I can send you the full logs pointing out the times where I caused the bug.

Comment 4 Jakub Hadvig 2014-02-10 17:06:02 UTC
Hi,

so the problem really is that for now our cartridges are not "scaleable", cause when your app is scaled-up, new gears are added to your application and HAproxy decides which of the gear will serve incoming request and action are made in the application(e.g. uploaded static files). But those changes aren't synced with other gear, so after that gear is scaled-down some of the gears are destroyed and you lose those changes(uploaded static files, etc.) 

But if you want there is a workaround for the wordpress:
https://www.openshift.com/blogs/how-to-automatically-scale-wordpress-on-openshift

So I suggest to close this one as WONTFIX

Comment 5 info 2014-02-10 17:23:53 UTC
Hello Jakub,

I do know that uploaded static files which do not go into the data dir will not be synced if they are on secondary gears. In this case I can run the very minimal risk as uploads can made happen on the primary gear and also loosing some static uploaded files on a replicated gear does not imply the behaviour I described above.

This has nothing to do with what happens in my case, which is that if I manually set the gears to 2 minimum no content of the primary gear gets copied on the second, to make it really short. I need to set it manually because the resources are too low to handle more than a few cunc connections, rather than 15 which is when the haproxy scales automatically.

Comment 6 Paul Morie 2014-02-11 15:32:03 UTC
Hi-

What content on the primary gear are you expecting to be synced to the second gear?

P

Comment 7 Vojtech Vitek 2014-02-12 15:26:30 UTC
> "Unable to allocate memory for pool" error
points to the APC memory failure, meaning that the PHP application consumes too much gear memory. This might be related to (or duplicate of) bug 1001400.

Comment 8 Vojtech Vitek 2014-02-12 16:38:53 UTC
Trello cards for APC:
1. Disable/Edit APC settings manually: https://trello.com/c/1VcpoZTf
2. Disable APC automatically via DEVELOPMENT MODE: https://trello.com/c/lbjDOA6S

Comment 9 Dan McPherson 2014-02-12 20:53:01 UTC
Closing as upstream based on the trello cards referenced.

Comment 10 info 2014-02-12 21:43:18 UTC
I do not understand why it's closed. This is not related to the APC issue, which is only why the gear doesn't scale automatically. The problem is that if I manually set the gears to minimum 2 the content is not replicated on the second gear. When you ask what content should be synced, well, everything, otherwise how is the app supposed to work?

Comment 11 Dan McPherson 2014-02-12 21:55:10 UTC
The content that is synced to the new gears is the deployments and dependencies directories.  Are you expecting something other than that to be synced?

Comment 12 info 2014-02-12 23:10:58 UTC
no.

Comment 13 info 2014-02-13 07:40:01 UTC
For example, try to connect to pempo.it now. If you connect to the main gear you see a typical wp installation, on the second gear you get the default openshift index.php

Comment 14 Paul Morie 2014-02-13 19:54:51 UTC
It doesn't look like rain-doghouse has php scaled up at the moment.  Would you scale up to 2 so we can take a look at the logs?

Thanks,

Paul

Comment 15 info 2014-02-14 13:12:48 UTC
I have set minimum 2 maximum all available at rain-dogshouse
I did the same for pempo.it which is another test project

they both hand the correct installation if the connection is routed to the main gear, and the standard openshift php installation files if on the second gear, which in the rain-dogshouse case is:
gear-52fdf88b500446c4a5000249-dogshouse

Comment 16 info 2014-03-01 11:36:41 UTC
I suppose this bug cannot be fixed? How comes as it is very severe as an issue?

Comment 17 Paul Morie 2014-03-03 17:30:38 UTC
It looks like you're editing index.php in app-root/runtime/repo directly.  In order to get your changes deployed correctly into the app (ie, so that auto-scaling will correctly scale up the new gear with your application's code), you need to git clone your application's repo and deploy your changes using git push.

See 'How do I use OpenShift Online?' in the following link:

https://www.openshift.com/products/online

Comment 18 Dan McPherson 2014-03-04 02:40:01 UTC
Another getting started guide:

https://www.openshift.com/get-started