Bug 1858230 - Errors related to GlusterHook.read seen in engine.log
Summary: Errors related to GlusterHook.read seen in engine.log
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Gluster
Version: ---
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.4.3
: 4.40.35
Assignee: Kaustav Majumder
QA Contact: SATHEESARAN
URL:
Whiteboard:
Depends On:
Blocks: 1858228
TreeView+ depends on / blocked
 
Reported: 2020-07-17 10:09 UTC by SATHEESARAN
Modified: 2020-11-11 06:46 UTC (History)
5 users (show)

Fixed In Version: vdsm-4.40.35
Clone Of: 1858228
Environment:
Last Closed: 2020-11-11 06:42:02 UTC
oVirt Team: Gluster
Embargoed:
pm-rhel: ovirt-4.4+


Attachments (Terms of Use)
engine.log (1.21 MB, application/octet-stream)
2020-07-17 10:11 UTC, SATHEESARAN
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 111503 0 master MERGED gluster: Fixed py3 type errors in gluster/hooks.py 2021-01-28 11:45:31 UTC

Description SATHEESARAN 2020-07-17 10:09:26 UTC
Description of problem:
-----------------------
While checking the engine.log post RHHI-V deployment, there are few errors related to GlusterHooks operations. This is not causing any functional issues.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
RHV Manager version - 4.4.1-11
RHHI-V 1.8


How reproducible:
--------------------
Always

Steps to Reproduce:
--------------------
1. Complete hyperconverged deployment
2. Check engine.log for errors

Actual results:
----------------
1. Some repetitive  errors related to 'GetGlusterHookContentVDSCommand' are seen
2. Exception related to reading glusterhook.read is also seen


Expected results:
-----------------
No errors should be seen in engine.log related to gluster hooks

--- Additional comment from SATHEESARAN on 2020-07-17 10:07:42 UTC ---

Some of the errors looks like Python porting problem

<snip>
2020-07-16 13:39:56,697Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Failed in 'GetGlusterHookContentVDS' method
2020-07-16 13:39:56,697Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Failed in 'GetGlusterHookContentVDS' method
2020-07-16 13:39:56,697Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}]
2020-07-16 13:39:56,697Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a05e46d8>> with arguments: ('gsync-create', 'POST', '56glusterd-geo-rep-create-post.sh') error: a bytes-like object is required, not 'str'"}]
2020-07-16 13:39:56,699Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Command 'GetGlusterHookContentVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, GlusterHookVDSParameters:{hostId='577aa8ad-a5bb-47db-b664-caf8738efed2'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a05e46d8>> with arguments: ('gsync-create', 'POST', '56glusterd-geo-rep-create-post.sh') error: a bytes-like object is required, not 'str'"}, code = -32603
2020-07-16 13:39:56,699Z INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] FINISH, GetGlusterHookContentVDSCommand, return: , log id: 47f11e70
2020-07-16 13:39:56,702Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2842) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command GetGlusterHookContentVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}
2020-07-16 13:39:56,702Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Command 'GetGlusterHookContentVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, GlusterHookVDSParameters:{hostId='577aa8ad-a5bb-47db-b664-caf8738efed2'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}, code = -32603
2020-07-16 13:39:56,702Z INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] FINISH, GetGlusterHookContentVDSCommand, return: , log id: 105fbd65
2020-07-16 13:39:57,698Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2844) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699438>> with arguments: ('delete', 'POST', '57glusterfind-delete-post') error: a bytes-like object is required, not 'str'"}]

</snip>


Some are run time exceptions
<snip>
2020-07-16 13:39:59,685Z ERROR [org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28)
 [] Exception in sync: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.common.errors.EngineException: EngineExcepti
on: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal 
JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with a
rguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}, code = -32603 (Failed with error unex
pected and code 16)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil.invokeAll(ThreadPoolUtil.java:196)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.saveHookContent(GlusterHookSyncJob.java:234)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.saveNewHooks(GlusterHookSyncJob.java:226)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.addOrUpdateHooks(GlusterHookSyncJob.java:189)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.refreshHooksInCluster(GlusterHookSyncJob.java:85)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.refreshHooks(GlusterHookSyncJob.java:54)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.scheduling.OvirtGlusterSchedulingService$1.run(OvirtGlusterSchedulingService.java:221)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
</snip>

Comment 1 SATHEESARAN 2020-07-17 10:11:57 UTC
Created attachment 1701528 [details]
engine.log

Comment 2 SATHEESARAN 2020-10-30 14:25:54 UTC
Verified with RHV 4.4.3.8-0.1.el8ev

No such errors or tracebacks seen in engine.log post RHHI-V deployment

Comment 3 Sandro Bonazzola 2020-11-11 06:42:02 UTC
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.