Bug 1666085
| Summary: | "Multiple nodes match the same number of attributes" error, overcloud deployment fails, manual database cleanup fixed it | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | J Turock <jturock> | ||||||
| Component: | openstack-ironic-inspector | Assignee: | Dmitry Tantsur <dtantsur> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | mlammon | ||||||
| Severity: | low | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 13.0 (Queens) | CC: | bfournie, dtantsur, jturock, mburns, slinaber | ||||||
| Target Milestone: | z6 | Keywords: | Triaged, ZStream | ||||||
| Target Release: | 13.0 (Queens) | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | openstack-ironic-inspector-7.2.3-1.el7ost | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2019-04-30 17:24:46 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Hi! Could you please attach your instackenv.json (without passwords) and /var/log/ironic-inspector/inspector.log for the relevant time span? I'll attach them to this bugzilla. Thanks! Created attachment 1521098 [details]
instackenv.json, removed passwords
Created attachment 1521099 [details]
ironic inspector logs from all relevant time periods
I was mucking around quite a bit so other errors may be in the logs as well.
Found potential root cause: 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service [-] Periodic clean up of node cache failed: OverflowError: date value out of range 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service Traceback (most recent call last): 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironic_inspector/wsgi_service.py", line 203, in periodic_clean_up 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service if node_cache.clean_up(): 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py", line 883, in clean_up 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service threshold = timeutils.utcnow() - datetime.timedelta(seconds=timeout) 2019-01-08 10:54:57.477 20324 ERROR ironic_inspector.wsgi_service OverflowError: date value out of range Okay, I see that at some point in time the clean up timeout was set to 36000000000000 seconds, which is.. a lot :) Is the problem still reproducible after you set it back to 3600 and restarted? Sorry about that! I had set the timeout back to 3600 and restarted when I got the OverflowError. (I also built a totally new undercloud from scratch, and I have yet to replicate the database errors there) Okay, I believe your problem was actually caused by the timeout value overflowing. I have put up a patch to prevent this kind of situations in the future. Thanks! J Turock, I am curious to know if the deployment was first successful then follow your steps or just like you stated in original description? We did not see this issue and have not heard back from the submitter. We are marking it verified. If you find it to still be an issue, please re-open or file a new bug. Thank you. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0937 |
Version-Release number of selected component (if applicable): Red Hat OpenStack Platform 13 (latest stable), fully up-to-date 4 physical servers, various flavors of Fujitsu Undercloud is a VM on VMware ESXi 6.5, running RHEL 7.6, fully up-to-date How reproducible: Happens every time for me Steps to Reproduce: - Introspect some nodes - openstack baremetal node delete those nodes - Re-import, then re-introspect the same nodes - Try to deploy overcloud Actual results: 2019-01-08 11:52:14.856 6622 ERROR ironic_inspector.utils [req-79b14bcf-0a64-4c32-9bd9-37ca4aecaf72 - - - - -] Multiple nodes match the same number of attributes {'bmc_address': u'192.168.1.114', 'mac': [u'b4:96:91:2a:5b:d0', u'b4:96:91:2a:5b:d1', u'90:1b:0e:e7:f5:cc', u'90:1b:0e:e7:f5:cd', u'90:1b:0e:f5:7e:78', u'90:1b:0e:f5:7e:79', u'90:1b:0e:f5:7e:7a', u'90:1b:0e:f5:7e:7b']}: [u'7795cf95-30dd-4f4d-8cbf-f532a7fb9a29', u'8974d263-99d6-4858-a4c0-ce7725c4fc36', u'04fe769d-cc32-4d17-80ab-de8f629e23b7'] Then, the deployment fails. Expected results: Overcloud deploys successfully Additional info: I'm happy to attach any logs or config files you may find useful. I managed to manually delete the nodes from the ironic-inspector database, and this fixed the problem. mysql -u root -p # enter root password, found in /home/stack/undercloud-passwords.conf > use ironic-inspector; > show tables; > select * from nodes; # noted that several old nodes were in this table # matched the UUIDs of my nodes from "openstack baremetal node list" # deleted all entries in this table that didn't match the current nodes, for example: > delete from nodes where uuid = '04fe769d-cc32-4d17-80ab-de8f629e23b7'