Bug 1858228

Summary: Errors related to GlusterHook.read seen in engine.log
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: SATHEESARAN <sasundar>
Component: rhhiAssignee: Kaustav Majumder <kmajumde>
Status: CLOSED CURRENTRELEASE QA Contact: SATHEESARAN <sasundar>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.5CC: godas, kmajumde, rhs-bugs, sabose
Target Milestone: ---Keywords: ZStream
Target Release: RHHI-V 1.8.z Batch Update 2   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1858230 (view as bug list) Environment:
Last Closed: 2020-12-01 06:58:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1858230    
Bug Blocks:    
Attachments:
Description Flags
engine.log none

Description SATHEESARAN 2020-07-17 10:05:04 UTC 
Description of problem:
-----------------------
While checking the engine.log post RHHI-V deployment, there are few errors related to GlusterHooks operations. This is not causing any functional issues.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
RHV Manager version - 4.4.1-11
RHHI-V 1.8


How reproducible:
--------------------
Always

Steps to Reproduce:
--------------------
1. Complete hyperconverged deployment
2. Check engine.log for errors

Actual results:
----------------
1. Some repetitive  errors related to 'GetGlusterHookContentVDSCommand' are seen
2. Exception related to reading glusterhook.read is also seen


Expected results:
-----------------
No errors should be seen in engine.log related to gluster hooks
Comment 1 SATHEESARAN 2020-07-17 10:07:42 UTC 
Some of the errors looks like Python porting problem

<snip>
2020-07-16 13:39:56,697Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Failed in 'GetGlusterHookContentVDS' method
2020-07-16 13:39:56,697Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Failed in 'GetGlusterHookContentVDS' method
2020-07-16 13:39:56,697Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}]
2020-07-16 13:39:56,697Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a05e46d8>> with arguments: ('gsync-create', 'POST', '56glusterd-geo-rep-create-post.sh') error: a bytes-like object is required, not 'str'"}]
2020-07-16 13:39:56,699Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] Command 'GetGlusterHookContentVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, GlusterHookVDSParameters:{hostId='577aa8ad-a5bb-47db-b664-caf8738efed2'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a05e46d8>> with arguments: ('gsync-create', 'POST', '56glusterd-geo-rep-create-post.sh') error: a bytes-like object is required, not 'str'"}, code = -32603
2020-07-16 13:39:56,699Z INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2845) [] FINISH, GetGlusterHookContentVDSCommand, return: , log id: 47f11e70
2020-07-16 13:39:56,702Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2842) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command GetGlusterHookContentVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}
2020-07-16 13:39:56,702Z ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] Command 'GetGlusterHookContentVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, GlusterHookVDSParameters:{hostId='577aa8ad-a5bb-47db-b664-caf8738efed2'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with arguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}, code = -32603
2020-07-16 13:39:56,702Z INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2842) [] FINISH, GetGlusterHookContentVDSCommand, return: , log id: 105fbd65
2020-07-16 13:39:57,698Z WARN  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHookContentVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2844) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699438>> with arguments: ('delete', 'POST', '57glusterfind-delete-post') error: a bytes-like object is required, not 'str'"}]

</snip>


Some are run time exceptions
<snip>
2020-07-16 13:39:59,685Z ERROR [org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28)
 [] Exception in sync: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.common.errors.EngineException: EngineExcepti
on: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetGlusterHookContentVDS, error = Internal 
JSON-RPC error: {'reason': "Attempt to call function: <bound method GlusterHook.read of <vdsm.gluster.apiwrapper.GlusterHook object at 0x7fa9a0699128>> with a
rguments: ('add-brick', 'PRE', '28Quota-enable-root-xattr-heal.sh') error: a bytes-like object is required, not 'str'"}, code = -32603 (Failed with error unex
pected and code 16)
        at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil.invokeAll(ThreadPoolUtil.java:196)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.saveHookContent(GlusterHookSyncJob.java:234)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.saveNewHooks(GlusterHookSyncJob.java:226)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.addOrUpdateHooks(GlusterHookSyncJob.java:189)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.refreshHooksInCluster(GlusterHookSyncJob.java:85)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.gluster.GlusterHookSyncJob.refreshHooks(GlusterHookSyncJob.java:54)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.scheduling.OvirtGlusterSchedulingService$1.run(OvirtGlusterSchedulingService.java:221)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.glassfish.javax.enterprise.concurrent.0.redhat-1//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
</snip>
Comment 2 SATHEESARAN 2020-07-17 10:12:21 UTC 
Created attachment 1701529 [details]
engine.log
Comment 4 SATHEESARAN 2020-10-30 14:28:08 UTC 
Verified with RHV 4.4.3.8-0.1.el8ev
No such errors or tracebacks seen in engine.log post RHHI-V deployment