Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1848101

Summary: Ironic agent got stuck in ramdisk and failed to provision baremetal node
Product: Red Hat OpenStack Reporter: Derek Li <derli>
Component: openstack-ironic-python-agentAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Alistair Tonner <atonner>
Severity: high Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: bfournie, dtantsur, jkreger, mburns, rpittau, slinaber
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-24 17:35:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
provisioning-was-done-after-two-hours
none
ironic-deployment-log none

Comment 2 Bob Fournier 2020-06-19 01:18:42 UTC
Note that the error in the attached screenshot is "TypeError: cannot concatenate 'str' and 'tuple' objects" in journal.py.  We've had an indication of a similar error in https://bugzilla.redhat.com/show_bug.cgi?id=1848006 but as of yet don't have any more details.

Comment 4 Derek Li 2020-06-19 04:34:16 UTC
Created attachment 1698040 [details]
provisioning-was-done-after-two-hours

Comment 5 Derek Li 2020-06-19 04:36:02 UTC
Created attachment 1698041 [details]
ironic-deployment-log

Comment 6 Bob Fournier 2020-07-14 14:29:17 UTC
Hi Derek - any update to this or way to get more info?

Comment 11 Julia Kreger 2020-07-28 18:15:07 UTC
Greetings Derek,

I've looked through the agent code multiple times as well as some of the libraries we invoke and use, and I've not found what is causing the underlying failure. But I do think I've got a fix that should at least tell us what the original failure was before the logging failure. I've proposed that upstream into review.

From what I can tell, something is failing with an exception while the collectors are being loaded. This ends up going into the exception handling code where you just start to see the method "self.logger.error" being invoked. Deep down in the logging, the journald logging code fails and raises TypeError which is what is seen.

So hopefully, the patch I've proposed upstream can help us at least identify what is going on.

Comment 12 Julia Kreger 2020-08-24 17:35:24 UTC
We've been able to identify that the root cause of this issue is actually masking of a connectivity failure. This has been fixed upstream and the patch has been merged downstream for future release. Marking as a duplicate.

*** This bug has been marked as a duplicate of bug 1862105 ***