Bug 1026088
| Summary: | Deploying storage node fails if announcement of another node failed | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Other] RHQ Project | Reporter: | John Sanda <jsanda> | ||||||||||||
| Component: | Core Server | Assignee: | John Sanda <jsanda> | ||||||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Mike Foley <mfoley> | ||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||
| Priority: | unspecified | ||||||||||||||
| Version: | 4.9 | CC: | ahovsepy, hrupp | ||||||||||||
| Target Milestone: | GA | ||||||||||||||
| Target Release: | RHQ 4.10 | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | |||||||||||||||
| : | 1026108 (view as bug list) | Environment: | |||||||||||||
| Last Closed: | 2014-04-23 12:29:44 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Bug Depends On: | |||||||||||||||
| Bug Blocks: | 1026108 | ||||||||||||||
| Attachments: |
|
||||||||||||||
This issue could also occur during the UNANNOUNCE and REMOVE_MAINTENANCE phase of undeployment and the ADD_MAINTENANCE phase of deployment. I have committed a fix to master. Code has been refactored to query for the storage node being deployed by address (instead of operation mode) to ensure we avoid a NonUniqueResultException. master commit hash: f330eb0d The commit cited in comment 2 included a change that could result in trying to query for the node under deployment to soon, resulting in a NoResultException. It would not fail the deployment, but added a lot of noise in the logs that can and should be avoided. I have gone ahead and fixed when the query is executed. master commit hash: 7035640df verified in master : d3ea23b verification scenario is: * ANNOUNCE Fail During the installation of N1 removing rhq-storage-auth config so N2 cannot be connected to N1. Getting the rhq-storage-auth back and installing N3, which connects to N1. * BOOTSTRAP Fail i was unale to reproduce a scenario in which bootstrap would fail - it takes ~ 1-2 secs to start and finish bootstrap after announce *ADD_MAINTENANCE fail killing nodes during the add maintenance, so N2 failed installing N3 and connectint for N1 Please get corresponding screenshots attached. Created attachment 821634 [details]
storageAnnounceError
Created attachment 821635 [details]
storageNodesAfterAnnounceFailed
Created attachment 821636 [details]
storageNewNodesAfterAnnounceFailed
Created attachment 821637 [details]
alert-addToMaintFailure
Created attachment 821638 [details]
storageNewNodesAfterAddToMainFailure
Bulk closing of 4.10 issues. If an issue is not solved for you, please open a new BZ (or clone the existing one) with a version designator of 4.10. |
Description of problem: Start with a single node cluster. Deploy a second node, N2. Suppose the deployment fails during the announce phase which means N2 will have an operation mode of ANNOUNCE. Now try to deploy a third node, N3. Deployment of N3 will fail as well with an exception like, 09:54:52,310 ERROR [org.rhq.enterprise.server.storage.StorageNodeOperationsHandlerBean] (EJB default - 3) Aborting storage node deployment due to unexpected error while announcing cluster nodes.: javax.ejb.EJBException: javax.persistence.NonUniqueResultException: result returns more than one elements . . . at org.rhq.enterprise.server.storage.StorageNodeOperationsHandlerLocal$$$view137.handleAnnounce(Unknown Source) [rhq-server.jar:4.10.0-SNAPSHOT] at org.rhq.enterprise.server.storage.StorageNodeOperationsHandlerBean.handleOperationUpdateIfNecessary(StorageNodeOperationsHandlerBean.java:304) [rhq-server.jar:4.10.0-SNAPSHOT] . . . Caused by: javax.persistence.NonUniqueResultException: result returns more than one elements at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:293) [hibernate-entitymanager-4.2.0.CR1.jar:4.2.0.CR1] at org.rhq.enterprise.server.storage.StorageNodeOperationsHandlerBean.findStorageNodeByMode(StorageNodeOperationsHandlerBean.java:769) [rhq-server.jar:4.10.0-SNAPSHOT] at org.rhq.enterprise.server.storage.StorageNodeOperationsHandlerBean.handleAnnounce(StorageNodeOperationsHandlerBean.java:391) [rhq-server.jar:4.10.0-SNAPSHOT] When the announce operation finishes on N1 and the agent reports the results to the server, the server fetches N3 using JPA's Query.getSingleResult() and the result set includes both N2 and N3. The filtering in the query is too broad. The (un)deployment code was designed and intended to allow deployments in scenarios like this where a previous deployment of another node may have failed and it is easier/faster to deploy a completely different node rather than retrying the failed deployment. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: