Bug 965816 - adding the same cartridge immediately after remove of the cartridge from any jboss apps will generate a HTTP 500 error.
adding the same cartridge immediately after remove of the cartridge from any ...
Status: CLOSED WORKSFORME
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Bill DeCoste
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-21 15:37 EDT by Peter Ruan
Modified: 2015-05-14 19:19 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-05-28 11:41:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Peter Ruan 2013-05-21 15:37:02 EDT
Description of problem:
adding the same cartridge immediately after remove of the cartridge from any jboss apps will generate a HTTP 500 error.

Version-Release number of selected component (if applicable):
latest devenv and OSE

How reproducible:
always.

Steps to Reproduce:
1. create any jboss app for example  # rhc app create rwhsm1yvzj jbossews-2.0
2. add postgresql to it.  # rhc cartridge add postgresql-8.4 -a rwhsm1yvzj
3. delete the cart # rhc cartridge remove postgresql-8.4 -a rwhsm1yvzj 
4. add the cart back to the app #  rhc cartridge add postgresql-8.4 -a rwhsm1yvzj

Actual results:

    Given the "postgresql" cartridge is removed from the app                     # features/step_definitions/cartridge.rb:22
      /XDEBUG[2013-05-21 12:16:43]/ Calling step: 'I remove the "postgresql-8.4" cartridge from the "rwhsm1yvzj" app'
      /DEBUG/Removing the postgresql-8.4 cartridge from the rwhsm1yvzj app
      /DEBUG/EXEC: rhc cartridge remove postgresql-8.4 -a rwhsm1yvzj --confirm -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120  
      /INFO/Removing postgresql-8.4 from 'rwhsm1yvzj' ... removed
      /XDEBUG[2013-05-21 12:17:20]/ Calling step: 'the exit code should be "0"'
    And the "postgresql" cartridge is added to the app                           # features/step_definitions/cartridge.rb:22
      /XDEBUG[2013-05-21 12:17:20]/ Calling step: 'I add the "postgresql-8.4" cartridge to the "rwhsm1yvzj" app'
      /DEBUG/EXEC: rhc cartridge add postgresql-8.4 -a rwhsm1yvzj -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120  
      /INFO/Adding postgresql-8.4 to application 'rwhsm1yvzj' ... The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://ec2-54-225-21-160.compute-1.amazonaws.com/broker/rest/domains/pruand/applications/rwhsm1yvzj/cartridges
      /WARN/Retrying rhc_cartridge_add after non zero exit code.
      /DEBUG/EXEC: rhc cartridge add postgresql-8.4 -a rwhsm1yvzj -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120  
      /INFO/Adding postgresql-8.4 to application 'rwhsm1yvzj' ... The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://ec2-54-225-21-160.compute-1.amazonaws.com/broker/rest/domains/pruand/applications/rwhsm1yvzj/cartridges
      /WARN/Retrying rhc_cartridge_add after non zero exit code.
      /DEBUG/EXEC: rhc cartridge add postgresql-8.4 -a rwhsm1yvzj -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120  
      /INFO/Adding postgresql-8.4 to application 'rwhsm1yvzj' ... The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://ec2-54-225-21-160.compute-1.amazonaws.com/broker/rest/domains/pruand/applications/rwhsm1yvzj/cartridges
      /WARN/Retrying rhc_cartridge_add after non zero exit code.
      /DEBUG/EXEC: rhc cartridge add postgresql-8.4 -a rwhsm1yvzj -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120  
      /INFO/Adding postgresql-8.4 to application 'rwhsm1yvzj' ... The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://ec2-54-225-21-160.compute-1.amazonaws.com/broker/rest/domains/pruand/applications/rwhsm1yvzj/cartridges

Expected results:
success

Additional info:
Comment 1 Peter Ruan 2013-05-21 16:22:30 EDT
NOTE: when I run the steps manually, I do not see the problem.  Perhaps it's a timing issue?

the example steps are the follow:   

Usage: rhc cartridge-status <cartridge> [--namespace NAME] [--app NAME]
Pass '--help' to see the full list of options
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge status postgresql-8.4 -a rwhsm1yvzj
RESULT:
Postgres is running
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge stop postgresql-8.4 -a rwhsm1yvzj 
RESULT:
postgresql-8.4 stopped
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge status postgresql-8.4 -a rwhsm1yvzj
RESULT:
Postgres is stopped
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge status postgresql-8.4 -a rwhsm1yvzj
RESULT:
Postgres is stopped
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge start postgresql-8.4 -a rwhsm1yvzj
RESULT:
postgresql-8.4 started
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge reload postgresql-8.4 -a rwhsm1yvzj
RESULT:
postgresql-8.4 reloaded
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge stop postgresql-8.4 -a rwhsm1yvzj
RESULT:
postgresql-8.4 stopped
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge start postgresql-8.4 -a rwhsm1yvzj 
RESULT:
postgresql-8.4 started
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge remove postgresql-8.4 -a rwhsm1yvzj
Removing a cartridge is a destructive operation that may result in loss of data associated with the cartridge.

Are you sure you wish to remove postgresql-8.4 from 'rwhsm1yvzj'? (yes|no): y

Removing postgresql-8.4 from 'rwhsm1yvzj' ... removed
[peter@unused-16-138 <OSTEST_ONPREM> rwhsm1yvzj]# rhc cartridge add postgresql-8.4 -a rwhsm1yvzj -l 'pruan@redhat.com' -p 'vostok08'  --insecure --timeout 120 -d
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Request GET https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200  648 ms
DEBUG: Request GET https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/domains
DEBUG:    code 200  153 ms
DEBUG: Getting all cartridges
DEBUG: Request GET https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/cartridges
DEBUG:    code 200  388 ms
Adding postgresql-8.4 to application 'rwhsm1yvzj' ... DEBUG: Getting all domains
DEBUG: Request GET https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/domains/pruanc/applications/rwhsm1yvzj?include=cartridges
DEBUG:    code 200  182 ms
DEBUG: Adding cartridge rwhsm1yvzj
DEBUG: Request POST https://ec2-50-16-21-34.compute-1.amazonaws.com/broker/rest/domains/pruanc/applications/rwhsm1yvzj/cartridges
DEBUG:    code 201 37916 ms
Success

postgresql-8.4 (PostgreSQL Database 8.4)
----------------------------------------
  Gears:          Located with jbossews-2.0
  Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
  Database Name:  rwhsm1yvzj
  Password:       yVulhbS5jshi
  Username:       adminw9f11te

RESULT:
Added postgresql-8.4 to application rwhsm1yvzj
Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
Comment 2 Peter Ruan 2013-05-21 18:45:37 EDT
Will need to wait for pmorie's checkin to retest this test scenario.  The workaround for now is to restart the app prior to adding the cartridge back.


03:19:16 PM) jhonce: bdecoste: pmorie was adding a fix for nuke any running processes before teardown.  I'm guessing something in postgres didn't stop when expected is holding onto the port so setup is failing, then stop/teardown are unhappy as well
(03:19:49 PM) jhonce: bdecoste: CLOSE_WAIT is the clue, someone is holding a socket open
(03:20:42 PM) bdecoste: jhonce: lol the dreaded CLOSE_WAIT. Having DHL flashbacks
(03:20:48 PM) jhonce: bdecoste: do a ps -ef |grep 501
(03:21:28 PM) jhonce: bdecoste: Oh Yeah
(03:21:32 PM) bdecoste: jhonce: it's in an ok state now. So the socket eventually closed and made it happy
(03:21:53 PM) pruan: jhonce: the same test is run against different platform (php/ruby...) but only jboss apps failed...
(03:21:56 PM) jhonce: bdecoste: Yup yup dirty environment
Comment 3 Peter Ruan 2013-05-21 18:47:00 EDT
Paul,
  Can you please advise as to when your 'nuke' feature is available? 

Thanks,
-peter
Comment 4 Paul Morie 2013-05-28 11:41:58 EDT
I wasn't able to reproduce this issue - see the cucumber tests I used at: https://github.com/pmorie/origin-server/commit/a1887d47f6b26b9de5fc9bd08151d6129e939372

Please re-open with instructions if you can reproduce.

Note You need to log in before you can comment on or make changes to this bug.