Bug 1111086
| Summary: | RHEV-H Upgrade failed: Timeout during node upgrade | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | cshao <cshao> | ||||||
| Component: | ovirt-engine-webadmin-portal | Assignee: | Alon Bar-Lev <alonbl> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Pavel Stehlik <pstehlik> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.4.0 | CC: | acathrow, alonbl, bazulay, cshao, dougsland, ecohen, fdeutsch, gklein, gouyang, guasun, hadong, huiwa, iheim, jboggs, leiwang, oourfali, pstehlik, rbarry, Rhev-m-bugs, yaniwang, ybronhei, ycui, yeylon | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 3.4.2 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | infra | ||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-07-07 06:56:02 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
please attach the deploy log - should be under /var/log/ovirt-engine/host-deploy/ , the full path is written in engine's audit log Created attachment 911345 [details]
host-deploy.log
Alon did we do anyting with the user engine uses to login to the node ? The name of the user within the public key has no effect, it is just a comment. From engine log I see the issue is: 2014-06-19 14:36:23,045 ERROR [org.ovirt.engine.core.bll.OVirtNodeUpgrade] (org.ovirt.thread.pool-4-thread-38) [13daca56] Timeout during node 10.66.8.153 upgrade: javax.naming.TimeLimitExceededException: SSH session timeout host 'root.8.153' at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:499) [utils.jar:] Which is probably bug#1093882, which is timing issue, which will be solved properly in 3.5.0, see bug#1094123. The strange fact in this log is that this happens almost every attempt, while we see this very seldom. Please try to download[1] and replace it at where symlink of [2] points, please backup whatever you have first, and report back if it helps. If it does there is better method to keep this survive updates until 3.5.0, but first let's see the effect. [1] http://search.maven.org/remotecontent?filepath=org/apache/sshd/sshd-core/0.11.0/sshd-core-0.11.0.jar [2] /usr/share/ovirt-engine/modules/org/apache/sshd/main/sshd-core.jar (In reply to Alon Bar-Lev from comment #4) > The name of the user within the public key has no effect, it is just a > comment. From engine log I see the issue is: > > 2014-06-19 14:36:23,045 ERROR [org.ovirt.engine.core.bll.OVirtNodeUpgrade] > (org.ovirt.thread.pool-4-thread-38) [13daca56] Timeout during node > 10.66.8.153 upgrade: javax.naming.TimeLimitExceededException: SSH session > timeout host 'root.8.153' > at > org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:499) > [utils.jar:] > > > Which is probably bug#1093882, which is timing issue, which will be solved > properly in 3.5.0, see bug#1094123. > > The strange fact in this log is that this happens almost every attempt, > while we see this very seldom. > > Please try to download[1] and replace it at where symlink of [2] points, > please backup whatever you have first, and report back if it helps. If it > does there is better method to keep this survive updates until 3.5.0, but > first let's see the effect. > > [1] > http://search.maven.org/remotecontent?filepath=org/apache/sshd/sshd-core/0. > 11.0/sshd-core-0.11.0.jar > [2] /usr/share/ovirt-engine/modules/org/apache/sshd/main/sshd-core.jar The upgrade via ssh public-key can successful according your steps. 1. download the package 2. replace it 3. ovirt-engine restart Hi Shaochen, > > The upgrade via ssh public-key can successful according your steps. > 1. download the package > 2. replace it > 3. ovirt-engine restart Perfect, thanks for the update. > Please try to download[1] and replace it at where symlink of [2] points, please > backup whatever you have first, and report back if it helps. If it does there is > better method to keep this survive updates until 3.5.0, but first let's see the > effect. Hi Alon, Based on the log (as you already pointed) and now with confirmation that updating sshd-core jar solves the issue (the same as bz#1093882) I am moving this bug to you, if this is incorrect move back to me. Please let us know your suggestion for 3.4 users that face it. Thanks! This bug on downstream and bz#1093882 is upstream, do we want to fix it on downstream 3.4.z or only 3.5. Thanks :) (In reply to shaochen from comment #7) > This bug on downstream and bz#1093882 is upstream, do we want to fix it on > downstream 3.4.z or only 3.5. > > Thanks :) It is already fixed in 3.5. Is this a production environment? The only thing in this regard that was changed since rhevm-3.1 is the jboss version, so it is quite surprising that this pops up without me able to explain why. (In reply to Alon Bar-Lev from comment #8) > Is this a production environment? (In reply to Alon Bar-Lev from comment #10) > (In reply to Alon Bar-Lev from comment #8) > > Is this a production environment? This is not a production environment. QE reproduced this bug in our testing environment, but I think the bug can also be reproduced in customer's production environment. Thanks! *** This bug has been marked as a duplicate of bug 1093882 *** The bug 1093882 is for 3.5. I still encounter this issue on rhevm 3.4.z(rhevm-webadmin-portal-3.4.2-1.1.el6ev.noarch) + rhev-hypervisor6-6.5-20140930.1 So this issue will impact rhevm 3.4.z, do we need to fix it on 3.4.3? Pavel, could you reproduce and trace this bug on RHEV side? Thanks Ying We cannot reproduce this issue on 3.4.2 and webadmin-portal-3.4.2-1.1.el6ev.noarch and rhev-hypervisor6-6.5-20140930.1.el6ev.noarch.rpm Pavel, Do you test register rhevh to rhevm, then upgrade previous rhevh from rhevm via ssh Publickey? I recheck my test env, and can reproduce this problem on different my env. as original bug description error. Thanks Ying The ovirt-engine is just comment, it is not used for anything. The fact that authorized keys are installed at ~root makes it effect root user. (In reply to Alon Bar-Lev from comment #17) > The ovirt-engine is just comment, it is not used for anything. The fact that > authorized keys are installed at ~root makes it effect root user. yes, I missed see comment 4 and 5 to debug this issue. Thanks to explain here again. Please add any further comments to bug 1093882 (because this bug was closed as a dup thereof). (In reply to Fabian Deutsch from comment #22) > Please add any further comments to bug 1093882 (because this bug was closed > as a dup thereof). this bug is downstream bug, we can not discuss it in upstream bug 1093882. I will work with Pavel on IRC to ensure this issue firstly, or send email if needed later. Re-test this issue on a clean RHEVM ENV, the issue has gone. RHEVH Upgrade via RHEVM can succeed. Sorry for the Inconvenience. Test version: rhev-hypervisor6-6.5-20140821.1.el6ev ovirt-node-3.0.1-18.el6.14.noarch vdsm-4.14.13-2.el6ev.x86_64 rhev-hypervisor6-6.5-20140930.1 RHEVM AV12.1 Red Hat Enterprise Virtualization Manager Version: 3.4.3-1.1.el6ev Test steps: 1. Install rhev-hypervisor6-6.5-20140821.1.el6ev 2. Register RHEV-H to RHEVM av. 3. Maintenance the hypervisor. 4. Upgrade to rhev-hypervisor6-6.5-20140930.1 via SSH Publickey. Test result: RHEV-H Upgrade can successful via SSH Publickey in RHEV-M side. Thanks! |
Created attachment 910299 [details] ovirt-engine.log Description of problem: RHEV-H Upgrade failed via SSH Public-key in RHEV-M webadmin. RHEV-M send Public-Key to RHEV-H via ovirt-engine user when upgrading, but actually there is no ovirt-engine user at all in RHEV-M side, so the upgrade will fail. but if we manually modify the user from ovirt-engine to ovirt, the upgrade can successful. in RHEV-H: # cat /root/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0iFp1dSW+ZqZeW/hXO9C *** ovirt-engine In RHEV-M: # id ovirt-engine id: ovirt-engine: No such user # id ovirt uid=108(ovirt) gid=108(ovirt) groups=108(ovirt) RHEVM Fail log: =============== Host installation failed. SSH session timeout host "root.8.153" Failed to install Host. Processing stoped due to timeout. Version-Release number of selected component (if applicable): rhev-hypervisor6-6.5-20140618.0.el6ev vdsm-4.14.7-3.el6ev.x86_64 ovirt-node-3.0.1-18.el6_5.10.noarch ovirt-node-plugin-vdsm-0.1.1-22.el6ev.noarch RHEVM av9.6 How reproducible: 100% Steps to Reproduce: 1. Install RHEV-H 6.5-20140618.0.el6ev 2. Register RHEV-H to RHEVM av. 3. Maintenance the hypervisor. 4. Upgrade via SSH Publickey. Actual results: RHEV-H Upgrade failed via SSH Publickey in RHEV-M webadmin. Expected results: RHEV-H Upgrade can successful via SSH Publickey in RHEV-M side. Additional info: We can't sure which component is correct for this bug, so we set to ovirt-engine-webadmin-portal first, please free move it if need.