Bug 1295381 - ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'unhashable type: 'dict'' - trying to restart agent
ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'unhashable typ...
Status: CLOSED NOTABUG
Product: ovirt-hosted-engine-ha
Classification: oVirt
Component: Agent (Show other bugs)
1.3.3.3
Unspecified Unspecified
unspecified Severity medium (vote)
: ---
: ---
Assigned To: Martin Sivák
Ilanit Stein
sla
:
Depends On:
Blocks: RHEV3.6Upgrade
  Show dependency treegraph
 
Reported: 2016-01-04 04:59 EST by Jiri Belka
Modified: 2016-01-25 06:05 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-25 06:05:00 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: SLA
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)

  None (edit)
Description Jiri Belka 2016-01-04 04:59:06 EST
Description of problem:

MainThread::ERROR::2016-01-04 10:23:27,916::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'unhashable type: 'dict'' - trying to restart agent
MainThread::WARNING::2016-01-04 10:23:32,921::agent::208::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'

Uff, it is ERROR level and it has no information value for an user observing agent.log.

Either make it different level or extend the message to contain useful value. Thx.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch

How reproducible:
no idea

Steps to Reproduce:
1. no idea
2.
3.

Actual results:
MainThread::ERROR::2016-01-04 10:23:27,916::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::

Expected results:
ERROR level msg should have information value as it is ERROR, serious issue

Additional info:
Either make it different level or extend the message to contain useful value.
Comment 2 Martin Sivák 2016-01-04 05:04:11 EST
Simone, this is related to the upgrade code according to the log.

jbelka: It is a language error the user can do nothing about. Can you retest with the latest version? I believe we released 1.3.3.5 or 6 before Christmas.
Comment 3 Simone Tiraboschi 2016-01-04 11:09:29 EST
The hosted engine-storage domain was created inside a custom folder within the NFS export which is not the typical configuration:
the export was 10.34.63.202:/mnt/export/nfs/lv2 while hosted-engine was deployed in 10.34.63.202:/mnt/export/nfs/lv2/_brq-setup and _brq-setup that was owned by root:root

It seams that current VDSM complains about that and unmount the storage so the upgrade was failing.

MainProcess|Thread-15::WARNING::2016-01-04 14:17:03,320::fileUtils::84::Storage.fileUtils::(validateAccess) Permission denied for directory: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv2___brq-setup with permissions: 7
MainProcess|Thread-15::ERROR::2016-01-04 14:17:03,321::supervdsmServer::118::SuperVdsm.ServerCallback::(wrapper) Error in validateAccess
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 116, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 302, in validateAccess
    kwargs=kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 290, in _runAs
    raise err
OSError: [Errno 13] Permission denied

The same configuration was working with VDSM 4.16.

Fixing folder ownership to root:root and upgrading ovirt-hosted-engine-ha to 1.3.3.6 allowed to conclude the upgrade.
Comment 4 Simone Tiraboschi 2016-01-05 05:09:56 EST
(In reply to Simone Tiraboschi from comment #3)
> Fixing folder ownership to root:root

Sorry, to vdsm:kvm
Comment 5 Doron Fediuck 2016-01-25 06:05:00 EST
OK,
looking into this the user manually added _brq-setup subdirectory with the wrong ownership (root:root) and entered the whole '10.34.63.202:/mnt/export/nfs/lv2/_brq-setup' at hosted-engine setup.

This is something which wasn't properly checked previously. Specifically vdsm 4.16 was working with that configuration while vdsm 4.17 explicitly or better check NFS folder permission.

Based on the above this is not a bug and closing accordingly.

Note You need to log in before you can comment on or make changes to this bug.