Bug 1295381

Summary:	ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'unhashable type: 'dict'' - trying to restart agent
Product:	[oVirt] ovirt-hosted-engine-ha	Reporter:	Jiri Belka <jbelka>
Component:	Agent	Assignee:	Martin Sivák <msivak>
Status:	CLOSED NOTABUG	QA Contact:	Ilanit Stein <istein>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	1.3.3.3	CC:	bugs, dfediuck, jbelka, stirabos
Target Milestone:	---	Flags:	rule-engine: planning_ack? rule-engine: devel_ack? rule-engine: testing_ack?
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	sla
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-01-25 11:05:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	SLA	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1285700

Description Jiri Belka 2016-01-04 09:59:06 UTC

Description of problem:

MainThread::ERROR::2016-01-04 10:23:27,916::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'unhashable type: 'dict'' - trying to restart agent
MainThread::WARNING::2016-01-04 10:23:32,921::agent::208::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'

Uff, it is ERROR level and it has no information value for an user observing agent.log.

Either make it different level or extend the message to contain useful value. Thx.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch

How reproducible:
no idea

Steps to Reproduce:
1. no idea
2.
3.

Actual results:
MainThread::ERROR::2016-01-04 10:23:27,916::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::

Expected results:
ERROR level msg should have information value as it is ERROR, serious issue

Additional info:
Either make it different level or extend the message to contain useful value.

Comment 2 Martin Sivák 2016-01-04 10:04:11 UTC

Simone, this is related to the upgrade code according to the log.

jbelka: It is a language error the user can do nothing about. Can you retest with the latest version? I believe we released 1.3.3.5 or 6 before Christmas.

Comment 3 Simone Tiraboschi 2016-01-04 16:09:29 UTC

The hosted engine-storage domain was created inside a custom folder within the NFS export which is not the typical configuration:
the export was 10.34.63.202:/mnt/export/nfs/lv2 while hosted-engine was deployed in 10.34.63.202:/mnt/export/nfs/lv2/_brq-setup and _brq-setup that was owned by root:root

It seams that current VDSM complains about that and unmount the storage so the upgrade was failing.

MainProcess|Thread-15::WARNING::2016-01-04 14:17:03,320::fileUtils::84::Storage.fileUtils::(validateAccess) Permission denied for directory: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv2___brq-setup with permissions: 7
MainProcess|Thread-15::ERROR::2016-01-04 14:17:03,321::supervdsmServer::118::SuperVdsm.ServerCallback::(wrapper) Error in validateAccess
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 116, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 302, in validateAccess
    kwargs=kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 290, in _runAs
    raise err
OSError: [Errno 13] Permission denied

The same configuration was working with VDSM 4.16.

Fixing folder ownership to root:root and upgrading ovirt-hosted-engine-ha to 1.3.3.6 allowed to conclude the upgrade.

Comment 4 Simone Tiraboschi 2016-01-05 10:09:56 UTC

(In reply to Simone Tiraboschi from comment #3)
> Fixing folder ownership to root:root

Sorry, to vdsm:kvm

Comment 5 Doron Fediuck 2016-01-25 11:05:00 UTC

OK,
looking into this the user manually added _brq-setup subdirectory with the wrong ownership (root:root) and entered the whole '10.34.63.202:/mnt/export/nfs/lv2/_brq-setup' at hosted-engine setup.

This is something which wasn't properly checked previously. Specifically vdsm 4.16 was working with that configuration while vdsm 4.17 explicitly or better check NFS folder permission.

Based on the above this is not a bug and closing accordingly.