This error was observed by the migration and reported as an error (Seems we should be printing a warning and ignoring): Migrating app 'zd' gear 'zd' with uuid '51c17dd803ef64dc4200003e' on node 'ex-sml-node1.int.rhcloud.com' for user: wsun+1 Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'wsun+1' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version' after 2 tries with exception: Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'wsun+1' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version ["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:570:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:568:in `each'", "/usr/bin/rhc-admin-migrate:568:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:716:in `<main>'"] Output: Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'wsun+1' --migrate-gear '51c17dd803ef64dc4200003e' --app-name 'zd' --version '2.0.29' --ignore-cartridge-version Migrating on node... ***time_migrate_on_node_measured_from_broker=65247*** Migrate on node output: Beginning 2.0.29 migration for 51c17dd803ef64dc4200003e Inspecting gear at /var/lib/openshift/51c17dd803ef64dc4200003e Migrating gear at /var/lib/openshift/51c17dd803ef64dc4200003e Refreshing cartridge redhat:zend:5.6:0.0.1, ignoring cartridge version. Marking step zend_update_ident complete Starting gear with uuid '51c17dd803ef64dc4200003e' on node 'ex-sml-node1' ***time_restart=0*** Resetting quota blocks: 1048576 inodes: 40000 Validating gear 51c17dd803ef64dc4200003e post-migration Pre-migration state: started Caught an exception during internal migration steps: Timeout::Error /opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:146:in `rescue in rbuf_fill' /opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:140:in `rbuf_fill' /opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:122:in `readuntil' /opt/rh/ruby193/root/usr/share/ruby/net/protocol.rb:132:in `readline' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:2563:in `read_status_line' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:2552:in `read_new' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1320:in `block in transport_request' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1317:in `catch' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1317:in `transport_request' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1294:in `request' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1287:in `block in request' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start' /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:1285:in `request' /opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate.rb:381:in `validate_gear' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate.rb:95:in `migrate' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/libra.rb:77:in `migrate_action' /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch' Migrate on node exit code: 1
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/f730210e873e10afa05991d3f0972b10126da5ba Fix bug 976113: handle timeout in migration validate_gear
Fixed in devenv 3397
Checked on devenv_3398, it will take more times to retry the timeout issue now. 1.Add sleep code to the post_start hook for my app. 2.Migrate the app and measure the time. # time rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version /usr/bin/rhc-admin-migrate:139:in `rescue in migrate_gear': Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version (RuntimeError) ["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:719:in `<main>'"] Output: Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '0471845cda5e11e2935f12313d04cca9' --app-name 'timeout' --version '2.0.29' --ignore-cartridge-version Migrating on node... ***time_migrate_on_node_measured_from_broker=240489*** Migrate on node exit code: 1 from /usr/bin/rhc-admin-migrate:38:in `migrate_gear' from /usr/bin/rhc-admin-migrate:719:in `<main>' real 4m15.610s user 0m9.293s sys 0m1.161s Move bug to verified.