Bug 1212670

Summary: swift fails to start properly due to "ImportError: No module named pyeclib.ec_iface"
Product: [Community] RDO Reporter: wes hayutin <whayutin>
Component: openstack-swiftAssignee: Alan Pevec (Fedora) <apevec>
Status: CLOSED CURRENTRELEASE QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: unspecified    
Version: trunkCC: apevec, derekh, pasik, yeylon, zaitcev
Target Milestone: ---   
Target Release: Kilo   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1212148    
Bug Blocks:    
Attachments:
Description Flags
controller logs none

Description wes hayutin 2015-04-17 02:32:00 UTC 
Description of problem:

Apr 17 01:59:01 packstack swift-object-updater: File "/usr/bin/swift-object-updater", line 17, in <module>
Apr 17 01:59:01 packstack swift-object-updater: from swift.obj.updater import ObjectUpdater
Apr 17 01:59:01 packstack swift-object-updater: File "/usr/lib/python2.7/site-packages/swift/obj/updater.py", line 32, in <module>
Apr 17 01:59:01 packstack swift-object-replicator: Traceback (most recent call last):
Apr 17 01:59:01 packstack swift-object-replicator: File "/usr/bin/swift-object-replicator", line 17, in <module>
Apr 17 01:59:01 packstack swift-object-replicator: from swift.obj.replicator import ObjectReplicator
Apr 17 01:59:01 packstack swift-object-replicator: File "/usr/lib/python2.7/site-packages/swift/obj/replicator.py", line 40, in <module>
Apr 17 01:59:01 packstack swift-object-updater: from swift.common.storage_policy import split_policy_string, PolicyError
Apr 17 01:59:01 packstack swift-object-updater: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:01 packstack swift-object-replicator: from swift.obj.diskfile import (DiskFileManager, get_hashes, get_data_dir,
Apr 17 01:59:01 packstack swift-object-replicator: File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 54, in <module>
Apr 17 01:59:01 packstack swift-object-updater: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:01 packstack swift-object-updater: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:01 packstack swift-object-replicator: from swift.common.request_helpers import is_sys_meta
Apr 17 01:59:01 packstack swift-object-replicator: File "/usr/lib/python2.7/site-packages/swift/common/request_helpers.py", line 29, in <module>
Apr 17 01:59:01 packstack swift-object-replicator: from swift.common.storage_policy import POLICIES
Apr 17 01:59:01 packstack swift-object-replicator: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:01 packstack swift-object-replicator: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:01 packstack swift-object-replicator: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:01 packstack swift-object-auditor: Traceback (most recent call last):
Apr 17 01:59:01 packstack swift-object-auditor: File "/usr/bin/swift-object-auditor", line 17, in <module>
Apr 17 01:59:01 packstack swift-object-auditor: from swift.obj.auditor import ObjectAuditor
Apr 17 01:59:01 packstack swift-object-auditor: File "/usr/lib/python2.7/site-packages/swift/obj/auditor.py", line 25, in <module>
Apr 17 01:59:01 packstack systemd: openstack-swift-object-updater.service: main process exited, code=exited, status=1/FAILURE
Apr 17 01:59:01 packstack systemd: Unit openstack-swift-object-updater.service entered failed state.
Apr 17 01:59:01 packstack swift-object-auditor: from swift.obj import diskfile
Apr 17 01:59:01 packstack swift-object-auditor: File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 54, in <module>
Apr 17 01:59:01 packstack swift-object-auditor: from swift.common.request_helpers import is_sys_meta
Apr 17 01:59:01 packstack swift-object-auditor: File "/usr/lib/python2.7/site-packages/swift/common/request_helpers.py", line 29, in <module>
Apr 17 01:59:01 packstack swift-object-auditor: from swift.common.storage_policy import POLICIES
Apr 17 01:59:01 packstack swift-object-auditor: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:01 packstack swift-object-auditor: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:01 packstack swift-object-auditor: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack systemd: openstack-swift-object-auditor.service: main process exited, code=exited, status=1/FAILURE
Apr 17 01:59:02 packstack systemd: Unit openstack-swift-object-auditor.service entered failed state.
Apr 17 01:59:02 packstack swift-account-reaper: Traceback (most recent call last):
Apr 17 01:59:02 packstack swift-account-reaper: File "/usr/bin/swift-account-reaper", line 17, in <module>
Apr 17 01:59:02 packstack swift-account-reaper: from swift.account.reaper import AccountReaper
Apr 17 01:59:02 packstack swift-account-reaper: File "/usr/lib/python2.7/site-packages/swift/account/reaper.py", line 36, in <module>
Apr 17 01:59:02 packstack avahi-daemon[556]: Server startup complete. Host name is packstack-6.local. Local service cookie is 3385100875.
Apr 17 01:59:02 packstack systemd: openstack-swift-object-replicator.service: main process exited, code=exited, status=1/FAILURE
Apr 17 01:59:02 packstack systemd: Unit openstack-swift-object-replicator.service entered failed state.
Apr 17 01:59:02 packstack swift-account-reaper: from swift.common.storage_policy import POLICIES, PolicyError
Apr 17 01:59:02 packstack swift-account-reaper: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:02 packstack swift-account-reaper: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:02 packstack swift-account-reaper: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack swift-container-server: Error trying to load config from /etc/swift/container-server.conf: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack systemd: openstack-swift-account-reaper.service: main process exited, code=exited, status=1/FAILURE
Apr 17 01:59:02 packstack systemd: Unit openstack-swift-account-reaper.service entered failed state.
Apr 17 01:59:02 packstack swift-proxy-server: Error trying to load config from /etc/swift/proxy-server.conf: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack swift-container-replicator: Traceback (most recent call last):
Apr 17 01:59:02 packstack swift-container-replicator: File "/usr/bin/swift-container-replicator", line 17, in <module>
Apr 17 01:59:02 packstack swift-container-replicator: from swift.container.replicator import ContainerReplicator
Apr 17 01:59:02 packstack swift-container-replicator: File "/usr/lib/python2.7/site-packages/swift/container/replicator.py", line 27, in <module>
Apr 17 01:59:02 packstack swift-container-replicator: from swift.common.storage_policy import POLICIES
Apr 17 01:59:02 packstack swift-container-replicator: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:02 packstack swift-container-replicator: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:02 packstack swift-container-replicator: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack swift-object-server: Traceback (most recent call last):
Apr 17 01:59:02 packstack swift-object-server: File "/usr/bin/swift-object-server", line 20, in <module>
Apr 17 01:59:02 packstack swift-object-server: from swift.obj import server
Apr 17 01:59:02 packstack swift-object-server: File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 43, in <module>
Apr 17 01:59:02 packstack swift-object-server: from swift.obj import ssync_receiver
Apr 17 01:59:02 packstack swift-object-server: File "/usr/lib/python2.7/site-packages/swift/obj/ssync_receiver.py", line 27, in <module>
Apr 17 01:59:02 packstack swift-object-server: from swift.common import request_helpers
Apr 17 01:59:02 packstack swift-object-server: File "/usr/lib/python2.7/site-packages/swift/common/request_helpers.py", line 29, in <module>
Apr 17 01:59:02 packstack swift-object-server: from swift.common.storage_policy import POLICIES
Apr 17 01:59:02 packstack swift-object-server: File "/usr/lib/python2.7/site-packages/swift/common/storage_policy.py", line 22, in <module>
Apr 17 01:59:02 packstack swift-object-server: from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
Apr 17 01:59:02 packstack swift-object-server: ImportError: No module named pyeclib.ec_iface
Apr 17 01:59:02 packstack systemd: openstack-swift-proxy.service: main process exited, code=exited, status=1/FAILURE
Apr 17 01:59:02 packstack systemd: Unit openstack-swift-proxy.service entered failed state.
Comment 1 wes hayutin 2015-04-17 02:34:25 UTC 
Created attachment 1015431 [details]
controller logs

see logs for details..

The error is found in /var/log/messages
Comment 2 Alan Pevec (Fedora) 2015-04-17 09:55:25 UTC 
While Fedora review is performed, I'll upload draft versions of python-pyeclib + deps to RDO Kilo repo.
Comment 3 Alan Pevec (Fedora) 2015-04-17 12:15:39 UTC 
BTW this is an unconditional import while it should be loaded only if EC storage-policy is configured!
Comment 4 Alan Pevec (Fedora) 2015-04-17 13:24:46 UTC 
> upload draft versions of python-pyeclib + deps to RDO Kilo repo.

I've uploaded python-pyeclib and liberasurecode with few quickfixes to make them build and install, which together with https://review.gerrithub.io/230457 should fix this issue.

BUT pyeclib is not fully functional yet, see bug 1212148 comment 8
Luckily EC storage-policy is not configured by default so working import is enough to unblock RDO CI.
Comment 5 Alan Pevec (Fedora) 2015-04-17 13:30:18 UTC 
http://trunk.rdoproject.org/centos70/ad/66/ad66801915c0b87f3ba3b6648d473d601deac1e9_af64a80a/
Comment 6 Alan Pevec (Fedora) 2015-04-17 14:40:25 UTC 
CI passed with repo in comment 5