Description of problem: glustereventsd does not start due to an import error in utils.py Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/sbin/glustereventsd", line 24, in <module> Nov 12 20:21:04 s002 glustereventsd[28140]: import handlers Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in <module> Nov 12 20:21:04 s002 glustereventsd[28140]: import utils Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module> Nov 12 20:21:04 s002 glustereventsd[28140]: from .eventsapiconf import (LOG_FILE, Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not loaded, cannot perform relative import Installed Pyython version is 3.5.2
same as Bug 1650017
This is a fix for the first bug: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-04 18:21:09.830149998 +0100 @@ -26,13 +26,13 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, +from eventsapiconf import (LOG_FILE, WEBHOOKS_FILE, DEFAULT_CONFIG_FILE, CUSTOM_CONFIG_FILE, UUID_FILE, CERTS_DIR) -from . import eventtypes +import eventtypes and you need this one as well: --- glustereventsd.py.old 2018-12-04 18:25:06.841481182 +0100 +++ glustereventsd.py 2018-12-04 18:22:51.529863014 +0100 @@ -28,7 +28,7 @@ from utils import logger, PidFile, PidFileLockFailed, boolify -class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): +class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): def handle(self): data = self.request[0].strip() @@ -98,7 +98,7 @@ # Start the Eventing Server, UDP Server try: - server = SocketServer.ThreadingUDPServer( + server = socketserver.ThreadingUDPServer( (SERVER_ADDRESS, port), GlusterEventsRequestHandler) except socket.error as e:
but you still have "glustereventsd[20980]: Unable to get Port details from Config" in the log :(
default config file does not exist: root@gluster-3:# less /etc/glusterfs/eventsconfig.json /etc/glusterfs/eventsconfig.json: No such file or directory
inserting this: https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json into /etc/glusterfs/eventsconfig.json makes it start without problems
haha, but now geo-replication is broken [2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap tf(*aargs) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon slave_host, master, suuid, slavenodes) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 325, in monitor brick_path=w[0]['dir']) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 602, in gf_event from events.gf_event import gf_event as gfevent File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in <module> from .utils import logger, setup_logger, get_config File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module> from eventsapiconf import (LOG_FILE, ImportError: No module named 'eventsapiconf'
better patch: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-05 10:30:03.306848363 +0100 @@ -26,13 +26,25 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, - WEBHOOKS_FILE, - DEFAULT_CONFIG_FILE, - CUSTOM_CONFIG_FILE, - UUID_FILE, - CERTS_DIR) -from . import eventtypes +try: + from .eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) +except ImportError: + from eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) + +try: + from . import eventtypes +except ImportError: + import eventtypes
Hi, Mrten-- I still have the exact same bug AFTER applying the patches you provided (the patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for utils.py). I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code. The patches appear to have been applied correctly but attempting to start glustereventsd still gets this error: Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server... -- Subject: Unit glustereventsd.service has begun start-up -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has begun starting up. Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd service: glustereventsdTraceback (most recent call last): Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in <module> Dec 11 16:50:55 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module> Dec 11 16:50:55 datastore2c glustereventsd[710]: import utils Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module> Dec 11 16:50:55 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:55 datastore2c glustereventsd[710]: failed! Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call last): Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in <module> Dec 11 16:50:56 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module> Dec 11 16:50:56 datastore2c glustereventsd[710]: import utils Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module> Dec 11 16:50:56 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1 Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server. -- Subject: Unit glustereventsd.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has failed. -- -- The result is failed. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. I can provide whatever info you need to help debug this. Thanks!
Created attachment 1517163 [details] glustereventsd.py and utils.py fixup Hi dchang0. According to your logs, patch is not apply. Here the compiled from Mrten's answers patch that you need apply with following cmd: `patch -p 0 -d [libdir] < glustereventsd.patch` Where [libdir]: - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) Also, there is missing file that you can get by `wget 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`
(In reply to Arano-kai from comment #9) > Created attachment 1517163 [details] > glustereventsd.py and utils.py fixup > > Hi dchang0. > According to your logs, patch is not apply. > Here the compiled from Mrten's answers patch that you need apply with > following cmd: > `patch -p 0 -d [libdir] < glustereventsd.patch` > Where [libdir]: > - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" > - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) > Also, there is missing file that you can get by `wget > 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/ > eventsconfig.json' -O /etc/glusterfs/eventsconfig.json` Thanks. I will check the files in question manually and reapply them if needed and report back. eventsconfig.json was not missing for me (I built completely from scratch, and that file was included and installed in the proper location).
Created attachment 1517165 [details] glustereventsd.py and utils.py fixup Woops, forget to add 'SystemError' to catcher. Here the fixed version.
This doesn't matter now, since you've changed at least the glustereventsd.py patch, but I'm posting these anyway for diagnostic purposes. The source code was built in /root/glusterfs-5.1. The install with the patches applied is in /usr/local... ----- root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ----- root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,42 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except ImportError: > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) 36a44,47 > try: > from . import eventtypes > except ImportError: > import eventtypes ----- So it looks like I did apply the patches before, but they were probably bad/broken patches. Anyway, I will replace the patched files with the original source, then apply your new patches and see what happens and report back.
Okay, I just put the original files from source back (using make install) and then applied your new patches from Attachment #1517165 [details]. I'm getting a different error (probably an easier one to solve). Strangely, the patch command does not create the .orig files. Perhaps my patch command is broken or non-standard... Here are the diffs: ----- root@datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ------ root@datastore2c:/usr/local/libexec# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,46 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except (ImportError, SystemError): > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > try: > from . import eventtypes > except (ImportError, SystemError): > import eventtypes ----- Here is the new error when attempting to run: service glustereventsd start It appears to be a simple path error (can't find the gluster command). root@datastore2c:/usr/local/libexec# journalctl -xe Dec 27 20:46:15 datastore2c glustereventsd[14033]: import utils Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/utils.py", line 37, in <module> Dec 27 20:46:15 datastore2c glustereventsd[14033]: from eventsapiconf import (LOG_FILE, Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in <modul Dec 27 20:46:15 datastore2c glustereventsd[14033]: CUSTOM_CONFIG_FILE = get_glusterd_workdir() + CUSTOM_CONFIG_FILE_TO_SYNC Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 22, in get_gl Dec 27 20:46:15 datastore2c glustereventsd[14033]: universal_newlines = True) Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/lib/python3.5/subprocess.py", line 676, in __init__ Dec 27 20:46:15 datastore2c glustereventsd[14033]: restore_signals, start_new_session) Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child Dec 27 20:46:15 datastore2c glustereventsd[14033]: raise child_exception_type(errno_num, err_msg) Dec 27 20:46:15 datastore2c glustereventsd[14033]: FileNotFoundError: [Errno 2] No such file or directory: 'gluster' Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1 Dec 27 20:46:15 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server. -- Subject: Unit glustereventsd.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has failed. -- -- The result is failed. Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state. Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. ----- The gluster command is at this path: root@datastore2c:/usr/local/libexec# which gluster /usr/local/sbin/gluster I confirm that it runs insofar as it can put me into the interactive command line mode and respond to simple commands like help. Thanks for your help so far. Let me know what I can do to help.
BTW, if what I'm reporting in Comment 13 is now a new bug, please let me know and I will start a different bug report for it. Unless it is actually better to treat these two errors as one bug. Thanks!
It is probably you hit by https://stackoverflow.com/q/5658622, that is some $PATH inconsistency from python view and shell view. Since systemd used and units run in isolated env, you can try to explicitly set $PATH by overriding glustereventsd.service: # echo $PATH <current shell PATH> # systemctl edit glustereventsd.service [Service] Environment=PATH=<current shell PATH>
As for Comment 12, patch is right and applied. That is not right is python version: # python3 -V Python 3.5.3 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>SystemError<<<: Parent module '' not loaded, cannot perform relative import # python3 -V Python 3.7.2rc1 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>ImportError<<<: attempted relative import with no known parent package So we need catch SystemError for older python.
(In reply to Arano-kai from comment #15) > It is probably you hit by https://stackoverflow.com/q/5658622, that is some > $PATH inconsistency from python view and shell view. > Since systemd used and units run in isolated env, you can try to explicitly > set $PATH by overriding glustereventsd.service: > # echo $PATH > <current shell PATH> > # systemctl edit glustereventsd.service > > [Service] > Environment=PATH=<current shell PATH> Thanks for the help! I ended up editing the PATH= line in the file /etc/init.d/glustereventsd It seems to work now (starts without errors). I have not tested the actual functionality yet but will do so soon enough. I would say that with the patches you provided in attachment 1517165 [details], this bug is fixed completely for me. Thanks again!
Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted to a 5.x release?
REVIEW: https://review.gluster.org/22262 (eventsapi: Fix Python3 compatibility issues) posted (#1) for review on release-5 by Aravinda VK
REVIEW: https://review.gluster.org/22262 (eventsapi: Fix Python3 compatibility issues) merged (#1) on release-5 by Aravinda VK
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.5, please open a new bug report. glusterfs-5.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-March/000119.html [2] https://www.gluster.org/pipermail/gluster-users/