Bug 1334274

Summary:	AttributeErrors from ioprocess during shutdown when vdsm tests finish
Product:	[oVirt] vdsm	Reporter:	Piotr Kliczewski <pkliczew>
Component:	Core	Assignee:	Nir Soffer <nsoffer>
Status:	CLOSED UPSTREAM	QA Contact:	Elad <ebenahar>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.18.0	CC:	amureini, bronhaim, bugs, dougsland, gklein, nsoffer, ratamir, smizrahi, tnisan, ybronhei
Target Milestone:	ovirt-4.1.0-alpha	Flags:	rule-engine: ovirt-4.1+ rule-engine: planning_ack+ tnisan: devel_ack+ ratamir: testing_ack+
Target Release:	4.19.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-25 08:45:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Storage	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Piotr Kliczewski 2016-05-09 10:30:00 UTC

There is bunch of AttributeErrors seen when vdsm log fails.


Exception AttributeError: "'NoneType' object has no attribute
'write'"Exception in thread ioprocess communication (15770) (most
likely raised during interpreter shutdown): in <bound method
IOProcess.__del__ of <ioprocess.IOProcess object at 0x7fa0bb84a510>>
ignored
Exception AttributeError:
"'NoneType' object has no attribute 'wrTraceback (most recent call last):
ite'"  File "/usr/lib64/python2.7/threading.py", line 804, in
__bootstrap_inner in <bound method IOProcess.__del__ of
<ioprocess.IOProcess object at 0x7fa07f7b5b10>> ignored

Exception AttributeError  File "/usr/lib64/python2.7/threading.py",
line 757, in run
:   File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py",
line 180, in _communicate"'NoneType' object has no attribute 'write'"
 in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f831950>><type 'exceptions.AttributeError'>: 'NoneType' object
has no attribute 'close' ignored
Exception
AttributeError: "'NoneType' object has no attribute 'write'" in <bound
method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f84fe90>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f866850>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f82fbd0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f7b5310>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0bb84a350>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0bb919550>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f861690>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f8342d0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0b8491610>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0b8187ed0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0bb800ad0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f861a10>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0bb791090>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa0bb774650>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f834b10>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f77fb50>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f84ad90>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f84f690>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f60f510>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f84a4d0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f82f150>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'write'"
in <bound method IOProcess.__del__ of <ioprocess.IOProcess object at
0x7fa07f831290>> ignored

Comment 1 Nir Soffer 2016-06-03 18:48:42 UTC

The root cause is using __del__ in ioprocess for resources management.
This is bad idea in general, but we cannot fix it in ioprocess because old
vdsm versions depend on this.

The reason we see this when running vdsm tests is vdsm does not manage the
lifetime of ioprocess during the tests. Tests that use ioprocess should
clean up the created instances when the test is done.

Moving the bug back to vdsm.

Comment 2 Red Hat Bugzilla Rules Engine 2016-06-03 18:48:49 UTC

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 3 Red Hat Bugzilla Rules Engine 2016-06-03 18:50:06 UTC

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 4 Allon Mureinik 2016-07-06 17:04:14 UTC

High risk for a questionable reward. At this point in time, it isn't work the risk for 4.0.z.

Comment 5 Nir Soffer 2016-07-16 21:25:19 UTC

The latest patch is simple enough to backport to 4.0. This patch does not improve
the scale issues, but it fixes the warnings during shutdown.

This patch also fixes the issue of stuck tests we see randomly on the CI, caused
by the ioprocess threads automatic close.

We will open a separate bug for the scale issues.

Comment 6 Nir Soffer 2016-07-19 20:39:33 UTC

Merged in master.

Comment 7 Raz Tamir 2016-11-08 07:16:00 UTC

Hey Nir,
Can we get a clear steps to reproduce?

Comment 8 Nir Soffer 2016-11-08 07:36:52 UTC

This happens every time you run "make check" before the fix. We never seen this in
the application itself since stderr is redirected to /dev/null.

Comment 9 Sandro Bonazzola 2016-12-12 13:58:31 UTC

The fix for this issue should be included in oVirt 4.1.0 beta 1 released on December 1st. If not included please move back to modified.

Comment 10 Elad 2016-12-22 13:02:51 UTC

Nir, can you provide an exact flow?

Comment 11 Nir Soffer 2016-12-22 13:05:41 UTC

(In reply to Elad from comment #10)
> Nir, can you provide an exact flow?

This is not visible in vdsm (the errors are sent to /dev/null). The only way
to reproduce this is to run vdsm tests before and after this patch.

Comment 12 Nir Soffer 2016-12-22 13:07:05 UTC

Elad, see also comment 8.

Comment 13 Elad 2016-12-22 13:55:24 UTC

Raz, to me it sounds like CLOSED UPSTREAM. What do you think?

Comment 14 Raz Tamir 2016-12-25 08:43:30 UTC

Yes, close it with resolution UPSTREAM