Bug 1679406

+++ This bug was initially created as a clone of Bug #1649054 +++

Description of problem:
glustereventsd does not start due to an import error in utils.py

Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/sbin/glustereventsd", line 24, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import handlers
Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import utils
Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     from .eventsapiconf import (LOG_FILE,
Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not loaded, cannot perform relative import

Installed Pyython version is 3.5.2

--- Additional comment from Mrten on 2018-12-04 17:12:09 UTC ---

same as Bug 1650017

--- Additional comment from Mrten on 2018-12-04 17:25:31 UTC ---

This is a fix for the first bug:

--- utils.py.old	2018-12-04 18:19:18.846463181 +0100
+++ utils.py	2018-12-04 18:21:09.830149998 +0100
@@ -26,13 +26,13 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
+from eventsapiconf import (LOG_FILE,
                            WEBHOOKS_FILE,
                            DEFAULT_CONFIG_FILE,
                            CUSTOM_CONFIG_FILE,
                            UUID_FILE,
                            CERTS_DIR)
-from . import eventtypes
+import eventtypes


and you need this one as well:

--- glustereventsd.py.old	2018-12-04 18:25:06.841481182 +0100
+++ glustereventsd.py	2018-12-04 18:22:51.529863014 +0100
@@ -28,7 +28,7 @@
 from utils import logger, PidFile, PidFileLockFailed, boolify


-class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
+class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):

     def handle(self):
         data = self.request[0].strip()
@@ -98,7 +98,7 @@

     # Start the Eventing Server, UDP Server
     try:
-        server = SocketServer.ThreadingUDPServer(
+        server = socketserver.ThreadingUDPServer(
             (SERVER_ADDRESS, port),
             GlusterEventsRequestHandler)
     except socket.error as e:

--- Additional comment from Mrten on 2018-12-04 17:28:51 UTC ---

but you still have "glustereventsd[20980]: Unable to get Port details from Config"

in the log :(

--- Additional comment from Mrten on 2018-12-04 17:36:28 UTC ---

default config file does not exist:

root@gluster-3:# less /etc/glusterfs/eventsconfig.json
/etc/glusterfs/eventsconfig.json: No such file or directory

--- Additional comment from Mrten on 2018-12-04 17:38:39 UTC ---

inserting this:

https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json

into /etc/glusterfs/eventsconfig.json makes it start without problems

--- Additional comment from Mrten on 2018-12-05 09:24:13 UTC ---

haha, but now geo-replication is broken

[2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap
    tf(*aargs)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon
    slave_host, master, suuid, slavenodes)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 325, in monitor
    brick_path=w[0]['dir'])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 602, in gf_event
    from events.gf_event import gf_event as gfevent
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in <module>
    from .utils import logger, setup_logger, get_config
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module>
    from eventsapiconf import (LOG_FILE,
ImportError: No module named 'eventsapiconf'

--- Additional comment from Mrten on 2018-12-05 09:31:22 UTC ---

better patch:

--- utils.py.old	2018-12-04 18:19:18.846463181 +0100
+++ utils.py	2018-12-05 10:30:03.306848363 +0100
@@ -26,13 +26,25 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
-                           WEBHOOKS_FILE,
-                           DEFAULT_CONFIG_FILE,
-                           CUSTOM_CONFIG_FILE,
-                           UUID_FILE,
-                           CERTS_DIR)
-from . import eventtypes
+try:
+    from .eventsapiconf import (LOG_FILE,
+                                WEBHOOKS_FILE,
+                                DEFAULT_CONFIG_FILE,
+                                CUSTOM_CONFIG_FILE,
+                                UUID_FILE,
+                                CERTS_DIR)
+except ImportError:
+    from eventsapiconf import (LOG_FILE,
+                               WEBHOOKS_FILE,
+                               DEFAULT_CONFIG_FILE,
+                               CUSTOM_CONFIG_FILE,
+                               UUID_FILE,
+                               CERTS_DIR)
+
+try:
+  from . import eventtypes
+except ImportError:
+  import eventtypes

--- Additional comment from dchang0 on 2018-12-12 08:06:26 UTC ---

Hi, Mrten--

I still have the exact same bug AFTER applying the patches you provided (the patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for utils.py).

I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code. The patches appear to have been applied correctly but attempting to start glustereventsd still gets this error:

Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.
Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server...
-- Subject: Unit glustereventsd.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has begun starting up.
Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd service: glustereventsdTraceback (most recent call last):
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     from .eventsapiconf import (LOG_FILE,
Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import
Dec 11 16:50:55 datastore2c glustereventsd[710]:  failed!
Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call last):
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     from .eventsapiconf import (LOG_FILE,
Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1
Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.


I can provide whatever info you need to help debug this. Thanks!

--- Additional comment from Arano-kai on 2018-12-28 01:27 UTC ---

Hi dchang0.
According to your logs, patch is not apply.
Here the compiled from Mrten's answers patch that you need apply with following cmd:
`patch -p 0 -d [libdir] < glustereventsd.patch`
Where [libdir]:
- Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
- Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
Also, there is missing file that you can get by `wget 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`

--- Additional comment from dchang0 on 2018-12-28 01:44:44 UTC ---

(In reply to Arano-kai from comment #9)
> Created attachment 1517163 [details]
> glustereventsd.py and utils.py fixup
> 
> Hi dchang0.
> According to your logs, patch is not apply.
> Here the compiled from Mrten's answers patch that you need apply with
> following cmd:
> `patch -p 0 -d [libdir] < glustereventsd.patch`
> Where [libdir]:
> - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
> - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
> Also, there is missing file that you can get by `wget
> 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/
> eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`

Thanks. I will check the files in question manually and reapply them if needed and report back.

eventsconfig.json was not missing for me (I built completely from scratch, and that file was included and installed in the proper location).

--- Additional comment from Arano-kai on 2018-12-28 01:49 UTC ---

Woops, forget to add 'SystemError' to catcher.
Here the fixed version.

--- Additional comment from dchang0 on 2018-12-28 04:14:37 UTC ---

This doesn't matter now, since you've changed at least the glustereventsd.py patch, but I'm posting these anyway for diagnostic purposes.

The source code was built in /root/glusterfs-5.1. The install with the patches applied is in /usr/local...

-----


root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(


-----


root@datastore2c:/home/pi# diff  /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py
29,35c29,42
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except ImportError:
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
36a44,47
> try:
>     from . import eventtypes
> except ImportError:
>     import eventtypes


-----

So it looks like I did apply the patches before, but they were probably bad/broken patches.

Anyway, I will replace the patched files with the original source, then apply your new patches and see what happens and report back.

--- Additional comment from dchang0 on 2018-12-28 04:52:57 UTC ---

Okay, I just put the original files from source back (using make install) and then applied your new patches from Attachment #1517165 [details]. I'm getting a different error (probably an easier one to solve).

Strangely, the patch command does not create the .orig files. Perhaps my patch command is broken or non-standard...


Here are the diffs:

-----

root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(

------

root@datastore2c:/usr/local/libexec# diff  /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py
29,35c29,46
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except (ImportError, SystemError):
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
> try:
>     from . import eventtypes
> except (ImportError, SystemError):
>     import eventtypes


-----

Here is the new error when attempting to run:

service glustereventsd start

It appears to be a simple path error (can't find the gluster command).



root@datastore2c:/usr/local/libexec# journalctl -xe
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     import utils
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/utils.py", line 37, in <module>
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     from eventsapiconf import (LOG_FILE,
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in <modul
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     CUSTOM_CONFIG_FILE = get_glusterd_workdir() + CUSTOM_CONFIG_FILE_TO_SYNC
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 22, in get_gl
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     universal_newlines = True)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     restore_signals, start_new_session)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     raise child_exception_type(errno_num, err_msg)
Dec 27 20:46:15 datastore2c glustereventsd[14033]: FileNotFoundError: [Errno 2] No such file or directory: 'gluster'
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1
Dec 27 20:46:15 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'.


-----

The gluster command is at this path:

root@datastore2c:/usr/local/libexec# which gluster
/usr/local/sbin/gluster


I confirm that it runs insofar as it can put me into the interactive command line mode and respond to simple commands like help.

Thanks for your help so far. Let me know what I can do to help.

--- Additional comment from dchang0 on 2018-12-28 08:20:18 UTC ---

BTW, if what I'm reporting in Comment 13 is now a new bug, please let me know and I will start a different bug report for it. Unless it is actually better to treat these two errors as one bug.  Thanks!

--- Additional comment from Arano-kai on 2018-12-28 10:28:31 UTC ---

It is probably you hit by https://stackoverflow.com/q/5658622, that is some $PATH inconsistency from python view and shell view.
Since systemd used and units run in isolated env, you can try to explicitly set $PATH by overriding glustereventsd.service:
# echo $PATH
<current shell PATH>
# systemctl edit glustereventsd.service

[Service]
Environment=PATH=<current shell PATH>

--- Additional comment from Arano-kai on 2018-12-28 11:18:07 UTC ---

As for Comment 12, patch is right and applied. That is not right is python version:
# python3 -V
Python 3.5.3
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>SystemError<<<: Parent module '' not loaded, cannot perform relative import

# python3 -V
Python 3.7.2rc1
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>ImportError<<<: attempted relative import with no known parent package

So we need catch SystemError for older python.

--- Additional comment from dchang0 on 2018-12-29 05:14:01 UTC ---

(In reply to Arano-kai from comment #15)
> It is probably you hit by https://stackoverflow.com/q/5658622, that is some
> $PATH inconsistency from python view and shell view.
> Since systemd used and units run in isolated env, you can try to explicitly
> set $PATH by overriding glustereventsd.service:
> # echo $PATH
> <current shell PATH>
> # systemctl edit glustereventsd.service
> 
> [Service]
> Environment=PATH=<current shell PATH>

Thanks for the help! I ended up editing the PATH= line in the file /etc/init.d/glustereventsd

It seems to work now (starts without errors). I have not tested the actual functionality yet but will do so soon enough.

I would say that with the patches you provided in attachment 1517165 [details], this bug is fixed completely for me. Thanks again!

--- Additional comment from Sahina Bose on 2019-02-14 07:10:46 UTC ---

Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted to a 5.x release?