Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 484029[details]
vdsm logs.
Description of problem:
running on small scale system, with new meta-data, at some point, rhevm decided to deactivate storage domain due too problematic state (high latency), and sent deactivate storage domain, vdsm started to process this command, but never returns valid response code (greped all logs), and looks like this:
------------------------------------------------------------------------------
Thread-86543::INFO::2011-03-12 15:36:31,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d5
3-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=146)
Thread-86543::DEBUG::2011-03-12 15:36:31,684::task::491::TaskManager.Task::(_debug) Task f3e5ac66-b788-414f-a57f-ff9b35e7c97c: moving from state init -> state
preparing
------------------------------------------------------------------------------
then, I see few logs regarding this thread, at some point it goes to sleep for 2 minutes (didn't manage to acquire resource), and then i see the following:
------------------------------------------------------------------------------
Thread-86543::INFO::2011-03-12 15:38:18,908::sp::942::Storage.StoragePool::(deactivateSD) sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000
------------------------------------------------------------------------------
no return response what so ever, towards the end of log, i get the following errors, but non over that specific SD:
------------------------------------------------------------------------------
- RuntimeError: _handleRequests._checkForMail - Could not read mailbox
- AttributeError: 'NoneType' object has no attribute 'partial'
[root@rhev-i32c-01 vdsm]# zgrep deactivateStorage /var/log/vdsm/vdsm.log.* |grep aa6a8d53-8c0a-4be1-865e-452948c2ef83 | grep Run
/var/log/vdsm/var/log/vdsm/vdsm.log.27.gz:Thread-86543::INFO::2011-03-12 15:36:31,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=146)
/var/log/vdsm/vdsm.log.42.gz:Thread-63070::INFO::2011-03-12 00:27:12,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=141)
result:
vg is activate, but has no link in '/rhev/data-center/mnt/blockSD/', backend rollbacked command, and 'thinks' vg (domain) is up, meaning, totally a mess.
setup:
1) fcp
2) 31 storage domains
3) vm load - 194
Comment 1RHEL Program Management
2011-04-04 02:12:42 UTC
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.
Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 2Eduardo Warszawski
2011-04-07 09:53:08 UTC
The attached logs are not from this bug.
Haim, please add them or reproduce.
(In reply to comment #2)
> The attached logs are not from this bug.
> Haim, please add them or reproduce.
small chances to reproduce - will re-open in case i'll hit it again
Created attachment 484029 [details] vdsm logs. Description of problem: running on small scale system, with new meta-data, at some point, rhevm decided to deactivate storage domain due too problematic state (high latency), and sent deactivate storage domain, vdsm started to process this command, but never returns valid response code (greped all logs), and looks like this: ------------------------------------------------------------------------------ Thread-86543::INFO::2011-03-12 15:36:31,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d5 3-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=146) Thread-86543::DEBUG::2011-03-12 15:36:31,684::task::491::TaskManager.Task::(_debug) Task f3e5ac66-b788-414f-a57f-ff9b35e7c97c: moving from state init -> state preparing ------------------------------------------------------------------------------ then, I see few logs regarding this thread, at some point it goes to sleep for 2 minutes (didn't manage to acquire resource), and then i see the following: ------------------------------------------------------------------------------ Thread-86543::INFO::2011-03-12 15:38:18,908::sp::942::Storage.StoragePool::(deactivateSD) sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 ------------------------------------------------------------------------------ no return response what so ever, towards the end of log, i get the following errors, but non over that specific SD: ------------------------------------------------------------------------------ - RuntimeError: _handleRequests._checkForMail - Could not read mailbox - AttributeError: 'NoneType' object has no attribute 'partial' [root@rhev-i32c-01 vdsm]# zgrep deactivateStorage /var/log/vdsm/vdsm.log.* |grep aa6a8d53-8c0a-4be1-865e-452948c2ef83 | grep Run /var/log/vdsm/var/log/vdsm/vdsm.log.27.gz:Thread-86543::INFO::2011-03-12 15:36:31,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=146) /var/log/vdsm/vdsm.log.42.gz:Thread-63070::INFO::2011-03-12 00:27:12,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=aa6a8d53-8c0a-4be1-865e-452948c2ef83 spUUID=a8e3a5e0-1437-4dfb-9ac5-c6835227a074 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=141) result: vg is activate, but has no link in '/rhev/data-center/mnt/blockSD/', backend rollbacked command, and 'thinks' vg (domain) is up, meaning, totally a mess. setup: 1) fcp 2) 31 storage domains 3) vm load - 194