Bug 1414604

Summary: IOProcessClient WARNING Timeout waiting for communication thread for client
Product: [oVirt] ovirt-distribution Reporter: bill.james <bill.james>
Component: ioprocessAssignee: Nir Soffer <nsoffer>
ioprocess sub component: General QA Contact: Lilach Zitnitski <lzitnits>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: amureini, bill.james, bronhaim, bugs, dougsland, nsoffer, ratamir, saggi, tnisan
Version: ioprocess-0.15.0Flags: rule-engine: ovirt-4.2+
rule-engine: planning_ack+
tnisan: devel_ack+
ratamir: testing_ack+
Target Milestone: ovirt-4.2.0   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ioprocess-1.0.2-1.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-22 09:58:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm.log, supervdsm.log, messages none

Description bill.james@j2.com 2017-01-19 01:04:23 UTC
Description of problem:
vdsm.log has these entries:
> Thread-720062::DEBUG::2017-01-13
> 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) Starting
> communication thread for client ioprocess-5874
> Thread-720062::WARNING::2017-01-13
> 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) Timeout
> waiting for communication thread for client ioprocess-5874


Version-Release number of selected component (if applicable):
vdsm-4.18.11-1.el7.centos.x86_64
python-ioprocess-0.16.1-1.el7.noarch
ioprocess-0.16.1-1.el7.x86_64


How reproducible:
> We have an ovirt system with 3 clusters, all running centos7.
> ovirt engine is running on separate host,
> ovirt-engine-3.6.4.1-1.el7.centos.noarch
> 2 of the clusters are running newer version of ovirt, 3 nodes each,
> ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64,
> vdsm-4.18.11-1.el7.centos.x86_64.
> 1 cluster is still running the older version,
> ovirt-engine-3.6.4.1-1.el7.centos.noarch.


Steps to Reproduce:
1. happens just while running even before we had active Vms on cluster.
2. Only "DMZ" cluster is showing these timeouts. Not sure if firewall maybe causing issues?
3.

Actual results:


Expected results:


Additional info:

Comment 1 Yaniv Kaul 2017-01-19 06:53:22 UTC
Can you attach complete logs? both vdsm, supervdsm and /var/log/messages?

Comment 2 bill.james@j2.com 2017-01-19 18:15:29 UTC
Created attachment 1242541 [details]
vdsm.log, supervdsm.log, messages

Comment 3 Martin Perina 2017-01-25 11:51:47 UTC
Nir, could you please take a look?

Comment 4 Nir Soffer 2017-01-25 14:00:05 UTC
Moving to ioprocess, IOProcessClient is part of ioprocess, not vdsm.

Comment 5 Fedora End Of Life 2017-11-16 19:20:29 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 6 Nir Soffer 2017-11-16 20:28:59 UTC
Patch was modified long time ago, but gerrit did not update the bug since the bug
should be under ovirt.

Comment 7 Allon Mureinik 2017-11-19 08:03:48 UTC
Resetting target milestone for 4.2.1 - let's make sure we tag, build and ship this.

Comment 8 Allon Mureinik 2017-11-19 08:04:27 UTC
(In reply to Allon Mureinik from comment #7)
> Resetting target milestone for 4.2.1 - let's make sure we tag, build and
> ship this.

4.2.0, actually. Just clicked that we just did a beta for oVirt, not a GA.

Comment 9 Lilach Zitnitski 2017-11-28 09:41:05 UTC
Nir, are those errors appear only on DMZ clusters as mentioned in the first paragraph, or is there a different way to reproduce them?

Comment 10 Nir Soffer 2017-11-28 10:00:12 UTC
We don't know how to reproduce the errors.

The change in ioprocess was only increasing the default timeout because it was too
low.

Note that the change is include only in ioprocess 1.0 - and this version is not 
released yet, so we cannot test it yet.

Comment 11 Fedora Update System 2018-01-30 17:03:45 UTC
ioprocess-1.0.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 12 Fedora Update System 2018-01-31 22:45:26 UTC
ioprocess-1.0.0-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 13 Fedora Update System 2018-02-04 13:25:58 UTC
ioprocess-1.0.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 16 Fedora Update System 2018-02-04 20:18:44 UTC
ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 17 Lilach Zitnitski 2018-02-19 09:17:56 UTC
Full tier1 automation ran using ioprocess-1.0.2-1.el7ev.x86_64, no errors or regressions occurred related to the new ioprocess.

moving to verify.

Comment 18 Sandro Bonazzola 2018-02-22 09:58:27 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 19 Fedora Update System 2018-03-06 17:18:56 UTC
ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.