Bug 1331977

Summary:	cannot start swift properly with EC configured
Product:	Red Hat OpenStack	Reporter:	Mike Abrams <mabrams>
Component:	openstack-swift	Assignee:	Pete Zaitcev <zaitcev>
Status:	CLOSED ERRATA	QA Contact:	Mike Abrams <mabrams>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	derekh, jschluet, pgrist, scohen, srevivo, zaitcev
Target Milestone:	rc	Keywords:	Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	liberasurecode-1.2.0-2.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 15:33:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Mike Abrams 2016-04-30 21:19:16 UTC

Description of problem:
cannot start swift properly with EC configured:

How reproducible:
sudo swift-init restart all

Steps to Reproduce:
1. install OSP
2. configure EC
[storage-policy:1]
name = deepfreeze10-4
policy_type = erasure_coding
ec_type = liberasurecode_rs_vand
ec_num_data_fragments = 10
ec_num_parity_fragments = 4
ec_object_segment_size = 1048576
3. sudo swift-init restart all

Actual results:

[stack@instack swift]$ sudo swift-init restart all
Signal container-updater  pid: 2104  signal: 15
Signal account-auditor  pid: 2105  signal: 15
Signal object-replicator  pid: 2106  signal: 15
No object-replicator running
Signal container-sync  pid: 2107  signal: 15
No container-sync running
Signal container-replicator  pid: 2108  signal: 15
No container-replicator running
Signal object-auditor  pid: 2110  signal: 15
No object-auditor running
Signal object-expirer  pid: 2116  signal: 15
No object-expirer running
Signal container-auditor  pid: 2117  signal: 15
Signal container-server  pid: 2118  signal: 15
No container-server running
Signal object-reconstructor  pid: 2122  signal: 15
No object-reconstructor running
Signal object-server  pid: 2125  signal: 15
No object-server running
Signal account-reaper  pid: 2127  signal: 15
No account-reaper running
Signal proxy-server  pid: 2129  signal: 15
No proxy-server running
Signal account-replicator  pid: 2130  signal: 15
Signal object-updater  pid: 2132  signal: 15
No object-updater running
Signal container-reconciler  pid: 2135  signal: 15
No container-reconciler running
Signal account-server  pid: 2137  signal: 15
No account-server running
container-auditor (2117) appears to have stopped
account-replicator (2130) appears to have stopped
container-updater (2104) appears to have stopped
account-auditor (2105) appears to have stopped
Starting container-updater...(/etc/swift/container-server.conf)
Starting account-auditor...(/etc/swift/account-server.conf)
Starting object-replicator...(/etc/swift/object-server.conf)
Starting container-sync...(/etc/swift/container-server.conf)
Starting container-replicator...(/etc/swift/container-server.conf)
Starting object-auditor...(/etc/swift/object-server.conf)
Starting object-expirer...(/etc/swift/object-expirer.conf)
Starting container-auditor...(/etc/swift/container-server.conf)
Starting container-server...(/etc/swift/container-server.conf)
Starting object-reconstructor...(/etc/swift/object-server.conf)
Starting object-server...(/etc/swift/object-server.conf)
Starting account-reaper...(/etc/swift/account-server.conf)
Starting proxy-server...(/etc/swift/proxy-server.conf)
Starting account-replicator...(/etc/swift/account-server.conf)
Starting object-updater...(/etc/swift/object-server.conf)
Starting container-reconciler...(/etc/swift/container-reconciler.conf)
Starting account-server...(/etc/swift/account-server.conf)
liberasurecode[2389]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2391]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2392]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2393]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2395]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2397]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2398]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2404]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2406]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2411]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2414]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2415]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

liberasurecode[2416]: liberasurecode_backend_open: dynamic linking error libJerasure.so: cannot open shared object file: No such file or directory
ERROR: Invalid Storage Policy Configuration in /etc/swift/swift.conf (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 2)

[stack@instack swift]$

Version-Release number of selected component (if applicable):
liberasurecode-1.1.0-1.el7ost.x86_64

Expected results:
services restart without error

Additional info:
liberasurecode installs the following files:
[vagrant@localhost ~]$ rpm -ql liberasurecode | grep lib64
/usr/lib64/libXorcode.so.1
/usr/lib64/libXorcode.so.1.0.1
/usr/lib64/liberasurecode.so.1
/usr/lib64/liberasurecode.so.1.1.0
/usr/lib64/liberasurecode_rs_vand.so.1
/usr/lib64/liberasurecode_rs_vand.so.1.0.1
/usr/lib64/libnullcode.so.1
/usr/lib64/libnullcode.so.1.0.1

So I created a link to fix this:

ln -s /usr/lib64/liberasurecode_rs_vand.so.1
/usr/lib64/liberasurecode_rs_vand.so

And then it works.

I don't know the linking details of pyeclib or liberasurecode, but I
think something needs to be changed in one of them?

(thanks Christian!)

Comment 2 Pete Zaitcev 2016-05-04 06:03:44 UTC

The symlink is not how you're supposed to do this. In Fedora, the .so
comes with -devel package. In normal runtime, the package opens a
versioned object because that's what provides a measure of binary
compatibility. But of course if PyECLib just runs a dlopen(),
not much I can do.

I suppose I could add a requirement that pulls liberasurecode-devel.
Or, perhaps add a note to sample config.

Comment 4 Pete Zaitcev 2016-10-21 04:31:25 UTC

I moved the .so into the main package in liberasurecode-1.2.0-2.el7ost,
marking this as MODIFIED. Mike, I need your explicit confirmation that
this worked to resolve the issues.

Comment 8 errata-xmlrpc 2016-12-14 15:33:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html