Bug 976113 - Timeout checking state of app during migration isn't being ignored by the migration
Timeout checking state of app during migration isn't being ignored by the mig...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-19 20:13 EDT by Dan McPherson
Modified: 2015-05-14 19:22 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-24 10:58:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan McPherson 2013-06-19 20:13:01 EDT
This error was observed by the migration and reported as an error (Seems we should be printing a warning and ignoring):

Migrating app 'zd' gear 'zd' with uuid '51c17dd803ef64dc4200003e' on node 'ex-sml-node1.int.rhcloud.com' for user: wsun+1@redhat.com
Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'wsun+1@redhat.com' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version' after 2 tries with exception: Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'wsun+1@redhat.com' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version
["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:570:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:568:in `each'", "/usr/bin/rhc-admin-migrate:568:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:716:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'wsun+1@redhat.com' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version
Migrating on node...
***time_migrate_on_node_measured_from_broker=65247***
Migrate on node output:
 Beginning 2.0.29 migration for 51c17dd803ef64dc4200003e
Inspecting gear at /var/lib/openshift/51c17dd803ef64dc4200003e
Migrating gear at /var/lib/openshift/51c17dd803ef64dc4200003e
Refreshing cartridge redhat:zend:5.6:0.0.1, ignoring cartridge version.
Marking step zend_update_ident complete
Starting gear with uuid '51c17dd803ef64dc4200003e' on node 'ex-sml-node1'
***time_restart=0***
Resetting quota blocks: 1048576  inodes: 40000
Validating gear 51c17dd803ef64dc4200003e post-migration
Pre-migration state: started
Caught an exception during internal migration steps: Timeout::Error
/opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:146:in `rescue in rbuf_fill'
/opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:140:in `rbuf_fill'
/opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:122:in `readuntil'
/opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:132:in `readline'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:2563:in `read_status_line'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:2552:in `read_new'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1320:in `block in transport_request'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1317:in `catch'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1317:in `transport_request'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1294:in `request'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1287:in `block in request'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1285:in `request'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate.rb:381:in `validate_gear'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate.rb:95:in `migrate'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/libra.rb:77:in `migrate_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
Migrate on node exit code: 1
Comment 1 openshift-github-bot 2013-06-20 15:50:11 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/f730210e873e10afa05991d3f0972b10126da5ba
Fix bug 976113: handle timeout in migration validate_gear
Comment 2 Paul Morie 2013-06-20 16:26:27 EDT
Fixed in devenv 3397
Comment 3 Meng Bo 2013-06-21 07:41:34 EDT
Checked on devenv_3398, it will take more times to retry the timeout issue now.

1.Add sleep code to the post_start hook for my app.
2.Migrate the app and measure the time.

# time rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version
/usr/bin/rhc-admin-migrate:139:in `rescue in migrate_gear': Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version (RuntimeError)
["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:719:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version
Migrating on node...
***time_migrate_on_node_measured_from_broker=240489***
Migrate on node exit code: 1
	from /usr/bin/rhc-admin-migrate:38:in `migrate_gear'
	from /usr/bin/rhc-admin-migrate:719:in `<main>'

real	4m15.610s
user	0m9.293s
sys	0m1.161s


Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.