Bug 833039 - scalable app become not available after move app gear for a scalable app.
scalable app become not available after move app gear for a scalable app.
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-18 08:52 EDT by Johnny Liu
Modified: 2015-05-14 21:58 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-25 14:26:38 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2012-06-18 08:52:28 EDT
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.set up multi node
2.create scalable app, here is the geare info.
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <data>
    <gear>
      <uuid>eefee7c05b83402e84adfd4a7489234d</uuid>
      <components>
        <component>
          <connection-url>mysql://eefee7c05b-jialiu.dev.rhcloud.com:38141/</connection-url>
          <password>M6V95A171-ul</password>
          <username>admin</username>
          <database-name>willdestroy1</database-name>
          <name>mysql-5.1</name>
          <internal-port>3306</internal-port>
          <proxy-port>38141</proxy-port>
          <proxy-host>eefee7c05b-jialiu.dev.rhcloud.com</proxy-host>
        </component>
      </components>
      <git-url>ssh://eefee7c05b83402e84adfd4a7489234d@eefee7c05b-jialiu.dev.rhcloud.com/~/git/eefee7c05b.git/</git-url>
    </gear>
    <gear>
      <uuid>7ac5c18a984e4450800af1470b5f19f4</uuid>
      <components>
        <component>
          <name>ruby-1.8</name>
          <internal-port>8080</internal-port>
          <proxy-port>38131</proxy-port>
          <proxy-host>willdestroy1-jialiu.dev.rhcloud.com</proxy-host>
        </component>
        <component>
          <name>haproxy-1.4</name>
          <internal-port nil="true"></internal-port>
          <proxy-port nil="true"></proxy-port>
          <proxy-host nil="true"></proxy-host>
        </component>
        <component>
          <job-url>https://jenkins-jialiu.dev.rhcloud.com/job/willdestroy1-build/</job-url>
          <name>jenkins-client-1.4</name>
          <internal-port nil="true"></internal-port>
          <proxy-port nil="true"></proxy-port>
          <proxy-host nil="true"></proxy-host>
        </component>
      </components>
      <git-url>ssh://7ac5c18a984e4450800af1470b5f19f4@willdestroy1-jialiu.dev.rhcloud.com/~/git/willdestroy1.git/</git-url>
    </gear>
    <gear>
      <uuid>223018c7a8fc40029d20a061bd39abba</uuid>
      <components>
        <component>
          <name>ruby-1.8</name>
          <internal-port>8080</internal-port>
          <proxy-port>38136</proxy-port>
          <proxy-host>223018c7a8-jialiu.dev.rhcloud.com</proxy-host>
        </component>
      </components>
      <git-url>ssh://223018c7a8fc40029d20a061bd39abba@willdestroy1-jialiu.dev.rhcloud.com/~/git/willdestroy1.git/</git-url>
    </gear>
  </data>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
  </supported-api-versions>
  <status>ok</status>
  <type>gears</type>
  <version>1.0</version>
  <messages/>
</response>

3.move ruby-1.8 gear (223018c7a8fc40029d20a061bd39abba) from one node to another
4.access app url
  
Actual results:
app url will show haproxy-status page.

Expected results:
app url should show app index page.

Additional info:
Dig more, found the haproxy config file is not updated after move operation.
# vim /var/lib/stickshift/7ac5c18a984e4450800af1470b5f19f4//haproxy-1.4/conf/haproxy.cfg
<--snip-->
listen express 127.1.254.1:8080
    cookie GEAR insert indirect nocache
    option httpchk GET /
    balance leastconn
    server  filler 127.1.254.2:8080 backup
    server gear-223018c7a8-jialiu 10.110.81.169:38136 check fall 2 rise 3 inter 2000 cookie 223018c7a8-jialiu


10.110.81.169 is the source node IP.
the targe node ip should be "10.72.198.196".
Comment 1 Johnny Liu 2012-06-18 08:54:36 EDT
The above testing is runing against devenv_1850
Comment 2 Rajat Chopra 2012-06-18 20:52:03 EDT
Race condition in haproxy_ctld and set_gear_endpoints hook as the DNS entry does not propagate quickly.

Now, even if the entry does not get properly updated initially, the ctld daemon should fix it within a cycle of 5 minutes. Thus, the app should restore itself in less than 300 seconds.
Comment 3 Rajat Chopra 2012-06-18 20:53:58 EDT
Fixed the race condition with rev#a90239b1d42d3052e02eda9e27f6e15d8601f7b9
Comment 4 Rony Gong 2012-06-19 04:08:46 EDT
verified on devenv_1857,
listen express 127.1.251.129:8080
    cookie GEAR insert indirect nocache
    option httpchk GET /
    balance leastconn
    server  filler 127.1.251.130:8080 backup
    server gear-e84fa30398-qgong1 10.92.241.213:38111 check fall 2 rise 3 inter 2000 cookie e84fa30398-qgong1


the 10.92.241.213 it the destination node ip

Note You need to log in before you can comment on or make changes to this bug.