Bug 1267325

Verified that this impacts Enterprise 2.2.6/7

+++ This bug was initially created as a clone of Bug #1261540 +++

Description of problem:

I have an app on rhcloud.com that's close to its quota.  During builds and when ssh-ing it tells me:
`Warning: Gear 55e73c8c0c1e66c54c00000d is using 93.9% of inodes allowed`
That's fine.

But I saw this nonsense in haproxy_ctld.log:
```
E, [2015-09-08T13:12:44.608192 #408384] ERROR -- : Failed to get stats from Warning:
E, [2015-09-08T13:12:44.608655 #408384] ERROR -- : Failed to get stats from Gear
E, [2015-09-08T13:12:44.609350 #408384] ERROR -- : Failed to get stats from 55e73c8c0c1e66c54c00000d
E, [2015-09-08T13:12:44.609828 #408384] ERROR -- : Failed to get stats from is
E, [2015-09-08T13:12:44.610308 #408384] ERROR -- : Failed to get stats from using
E, [2015-09-08T13:12:44.610412 #408384] ERROR -- : Failed to get stats from 93.9%
E, [2015-09-08T13:12:44.610906 #408384] ERROR -- : Failed to get stats from of
E, [2015-09-08T13:12:44.611436 #408384] ERROR -- : Failed to get stats from inodes
E, [2015-09-08T13:12:44.612098 #408384] ERROR -- : Failed to get stats from allowed
```
Digging into the source I see `get_remote_sessions_count()` is called with URLs from app_haproxy_status_urls.conf, which turns out already corrupted:
```
[mathdown468cf5aaf3prod-cben.rhcloud.com 55e73c8c0c1e66c54c00000d]\> cat haproxy/conf/app_haproxy_status_urls.conf 
Warning:
Gear
55e73c8c0c1e66c54c00000d
is
using
93.9%
of
inodes
allowed
```

From there I'm not sure where to debug...
This is a throwaway app, feel free to inspect or mutate it in any way, or tell me what to check.

Version-Release number of selected component (if applicable):
?

How reproducible:
Tried once, reproduced.

Steps to Reproduce:
1. create scalable app
2. dd if=/dev/zero ... to fill up to >90% quota.
   (the original discovery was for inodes quota, this repro is for size quota)
3. restart
4. scale to 2 gears
5. or maybe I restarted here?  not sure.
6. profit: 

Actual results:
```
[tmp-mathdown.rhcloud.com 55ef20d37628e14a5300019c]\> cat haproxy/conf/app_haproxy_status_urls.conf 
Warning:
Gear
55ef20d37628e14a5300019c
is
using
94.4%
of
disk
quota
[tmp-mathdown.rhcloud.com 55ef20d37628e14a5300019c]\> tail app-root/logs/haproxy_ctld.log 
E, [2015-09-09T11:02:35.844932 #280135] ERROR -- : Failed to get stats from quota
E, [2015-09-09T11:02:40.847626 #280135] ERROR -- : Failed to get stats from Warning:
E, [2015-09-09T11:02:40.849385 #280135] ERROR -- : Failed to get stats from Gear
E, [2015-09-09T11:02:40.849737 #280135] ERROR -- : Failed to get stats from 55ef20d37628e14a5300019c
E, [2015-09-09T11:02:40.850333 #280135] ERROR -- : Failed to get stats from is
E, [2015-09-09T11:02:40.850822 #280135] ERROR -- : Failed to get stats from using
E, [2015-09-09T11:02:40.850912 #280135] ERROR -- : Failed to get stats from 94.4%
E, [2015-09-09T11:02:40.851988 #280135] ERROR -- : Failed to get stats from of
E, [2015-09-09T11:02:40.852337 #280135] ERROR -- : Failed to get stats from disk
E, [2015-09-09T11:02:40.852719 #280135] ERROR -- : Failed to get stats from quota
```

Expected results:
```
[prod-mathdown.rhcloud.com 55ee99710c1e662f8b000013]\> cat haproxy/conf/app_haproxy_status_urls.conf 
cat: haproxy/conf/app_haproxy_status_urls.conf: No such file or directory
[prod-mathdown.rhcloud.com 55ee99710c1e662f8b000013]\> tail app-root/logs/haproxy_ctld.log
I, [2015-09-08T04:19:04.773447 #492161]  INFO -- : Starting haproxy_ctld
I, [2015-09-08T04:29:09.878269 #492161]  INFO -- : remove-gear - capacity: 0.0% gear_count: 2 sessions: 0 remove_thresh: 31.5%
I, [2015-09-08T04:29:11.263448 #492161]  INFO -- : remove-gear - exit_code: 1  output: Cannot remove gear because min limit '2' reached.
```
(not sure why app_haproxy_status_urls.conf is missing, that's also a 2-gear app.)

Additional info:

--- Additional comment from John W. Lamb on 2015-09-29 11:22:06 EDT ---

I was able to reproduce this in Online. Working on it now.