Description of problem: openstack-swift-proxy.service journal shows the following errors: liberasurecode[29188]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory liberasurecode[29188]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory liberasurecode[29188]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory liberasurecode[29188]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Version-Release number of selected component (if applicable): openstack-swift-container-2.6.0-1.el7ost.noarch openstack-swift-proxy-2.6.0-1.el7ost.noarch python-swiftclient-3.0.0-1.el7ost.noarch openstack-swift-2.6.0-1.el7ost.noarch openstack-swift-object-2.6.0-1.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-swift-account-2.6.0-1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with latest OSPd9 images Actual results: The service gets started but I'm not sure if these errors affect swift functionality. Expected results: There are no errors during openstack-swift-proxy startup. Additional info:
I don't have a Jerasure installed (stock RHEL OSP9 + self-built Swift 2.7.0), and it does not throw errors like this. I have a EC policy, but forced our supported EC type: [root@rhev-a24c-01 ~]# more /etc/swift/swift.conf [swift-hash] swift_hash_path_suffix = *************** [swift-constraints] max_meta_count = 90 max_meta_overall_size = 4096 [storage-policy:0] name = policy0 policy_type = replication default = yes [storage-policy:1] name = ec0603 policy_type = erasure_coding ec_type = liberasurecode_rs_vand ec_num_data_fragments = 6 ec_num_parity_fragments = 3 ec_object_segment_size = 1048576 [root@rhev-a24c-01 ~]# Unfortunately I think Jerasure is still the default, so you want that ec_type if you have an EC policy. Otherwise it's an annoying, but harmless message.
This is how the swift conf looks on a default overcloud deployment: [root@ctrl-r00-00 heat-admin]# cat /etc/swift/swift.conf [swift-hash] swift_hash_path_suffix = ADqUsM7H6VDsk2RcBq9a6YWTP [swift-constraints] max_header_size=8192 Even though it's not affecting functionality, I think we should adjust the conf file via the installer so we get a clean log.
Per conversation with Pete, he believes that this is likely specific to RHOS 9.0. As such, moving this back to 9.0 with an async release target and assigning to him.
this bug was fixed in the latest release of liberasurecode. There's a bz tracking the packaging of the new version. It is bug #1380824. Once that gets completed and verified, we should be able to close this bug too.
(In reply to Thiago da Silva from comment #6) > this bug was fixed in the latest release of liberasurecode. I presume Thiago meant 1.2.0, which is the latest at this time.
Pete, correct. I meant 1.2.0 https://github.com/openstack/liberasurecode/releases/tag/1.2.0
This bug seems to still exist in RHOSP 10 using single controller (POC) deployment. When using HA/multi controller deployment, the service openstack-swift-reconstructor doesn't exist and thus this issue doesn't occur. Installed Packages Name : openstack-swift-object Arch : noarch Version : 2.10.2 Release : 2.el7ost Size : 23 k Repo : installed From repo : rhos-10.0-signed Summary : Object services for Swift URL : http://launchpad.net/swift License : ASL 2.0 Description : OpenStack Object Storage (Swift) aggregates commodity servers to work together : in clusters for reliable, redundant, and large-scale storage of static objects. : : This package contains the openstack-swift object server. [root@overcloud-controller-0 system]# yum info liberasurecode Installed Packages Name : liberasurecode Arch : x86_64 Version : 1.2.0 Release : 2.el7ost Size : 126 k Repo : installed From repo : rhos-10.0-signed Summary : Erasure Code API library written in C with pluggable backends URL : https://bitbucket.org/tsg-/liberasurecode/ License : BSD and CRC32 Description : An API library for Erasure Code, written in C. It provides a number : of pluggable backends, such as Intel ISA-L library. [root@overcloud-controller-0 system]# systemctl list-units openstack-swift* UNIT LOAD ACTIVE SUB DESCRIPTION openstack-swift-account-auditor.service loaded active running OpenStack Object Storage (swift) - Account Auditor openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper openstack-swift-account-replicator.service loaded active running OpenStack Object Storage (swift) - Account Replicat openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server openstack-swift-container-auditor.service loaded active running OpenStack Object Storage (swift) - Container Audito openstack-swift-container-reconciler.service loaded active running OpenStack Object Storage (swift) - Container Reconc openstack-swift-container-replicator.service loaded active running OpenStack Object Storage (swift) - Container Replic openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Update openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server openstack-swift-object-auditor.service loaded active running OpenStack Object Storage (swift) - Object Auditor openstack-swift-object-expirer.service loaded active running OpenStack Object Storage (swift) - Object Expirer ● openstack-swift-object-reconstructor.service loaded failed failed OpenStack Object Storage (swift) - Object Reconstru openstack-swift-object-replicator.service loaded active running OpenStack Object Storage (swift) - Object Replicato openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 16 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [root@overcloud-controller-0 system]# systemctl status openstack-swift-object-reconstructor -l ● openstack-swift-object-reconstructor.service - OpenStack Object Storage (swift) - Object Reconstructor Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-reconstructor.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2017-10-07 04:01:58 UTC; 8min ago Process: 452728 ExecStart=/usr/bin/swift-object-reconstructor /etc/swift/object-server.conf (code=exited, status=1/FAILURE) Main PID: 452728 (code=exited, status=1/FAILURE) Oct 07 04:01:58 overcloud-controller-0 systemd[1]: Started OpenStack Object Storage (swift) - Object Reconstructor. Oct 07 04:01:58 overcloud-controller-0 systemd[1]: Starting OpenStack Object Storage (swift) - Object Reconstructor... Oct 07 04:01:58 overcloud-controller-0 liberasurecode[452728]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 07 04:01:58 overcloud-controller-0 liberasurecode[452728]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 07 04:01:58 overcloud-controller-0 liberasurecode[452728]: liberasurecode_instance_create: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 07 04:01:58 overcloud-controller-0 liberasurecode[452728]: liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 07 04:01:58 overcloud-controller-0 swift-object-reconstructor[452728]: Unable to find object-reconstructor config section in /etc/swift/object-server.conf Oct 07 04:01:58 overcloud-controller-0 systemd[1]: openstack-swift-object-reconstructor.service: main process exited, code=exited, status=1/FAILURE Oct 07 04:01:58 overcloud-controller-0 systemd[1]: Unit openstack-swift-object-reconstructor.service entered failed state. Oct 07 04:01:58 overcloud-controller-0 systemd[1]: openstack-swift-object-reconstructor.service failed. The error messages though seem harmless because after I resolved what seems to be separate issue "Unable to find object-reconstructor config section in /etc/swift/object-server.conf" by adding this section to the config file the service restarted correctly [root@overcloud-controller-0 system]# systemctl status openstack-swift-object-reconstructor -l ● openstack-swift-object-reconstructor.service - OpenStack Object Storage (swift) - Object Reconstructor Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-reconstructor.service; disabled; vendor preset: disabled) Active: active (running) since Sat 2017-10-07 04:19:52 UTC; 3s ago Main PID: 540828 (swift-object-re) CGroup: /system.slice/openstack-swift-object-reconstructor.service └─540828 /usr/bin/python2 /usr/bin/swift-object-reconstructor /etc/swift/object-server.conf Oct 07 04:19:52 overcloud-controller-0 systemd[1]: Started OpenStack Object Storage (swift) - Object Reconstructor. Oct 07 04:19:52 overcloud-controller-0 systemd[1]: Starting OpenStack Object Storage (swift) - Object Reconstructor... Oct 07 04:19:52 overcloud-controller-0 liberasurecode[540828]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 07 04:19:52 overcloud-controller-0 liberasurecode[540828]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 07 04:19:52 overcloud-controller-0 liberasurecode[540828]: liberasurecode_instance_create: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 07 04:19:52 overcloud-controller-0 liberasurecode[540828]: liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 07 04:19:52 overcloud-controller-0 object-server[540828]: Starting object reconstructor in daemon mode. Oct 07 04:19:52 overcloud-controller-0 object-server[540828]: Starting object reconstruction pass. Oct 07 04:19:52 overcloud-controller-0 object-server[540828]: Nothing reconstructed for 0.000293970108032 seconds. Oct 07 04:19:52 overcloud-controller-0 object-server[540828]: Object reconstruction complete. (0.00 minutes)
seem meet this bug too. and version of liberasurecode is 1.2 [root@swift yum.repos.d]# rpm -qa | grep liberasurecode liberasurecode-1.2.0-2.el7.x86_64 [root@swift yum.repos.d]# systemctl status openstack-swift-object openstack-swift-container openstack-swift-account.service ● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-10-20 10:53:41 CST; 34min ago Main PID: 12704 (swift-object-se) CGroup: /system.slice/openstack-swift-object.service ├─12704 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf ├─12747 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf ├─12748 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf └─12749 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf Oct 20 10:53:41 swift systemd[1]: Started OpenStack Object Storage (swift) - Object Server. Oct 20 10:53:41 swift systemd[1]: Starting OpenStack Object Storage (swift) - Object Server... Oct 20 10:53:41 swift liberasurecode[12704]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12704]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12704]: liberasurecode_instance_create: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12704]: liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift object-server[12704]: Started child 12747 Oct 20 10:53:41 swift object-server[12704]: Started child 12748 Oct 20 10:53:41 swift object-server[12704]: Started child 12749 ● openstack-swift-container.service - OpenStack Object Storage (swift) - Container Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-container.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-10-20 10:53:41 CST; 34min ago Main PID: 12707 (swift-container) CGroup: /system.slice/openstack-swift-container.service ├─12707 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf ├─12731 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf └─12732 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf Oct 20 10:53:41 swift systemd[1]: Started OpenStack Object Storage (swift) - Container Server. Oct 20 10:53:41 swift systemd[1]: Starting OpenStack Object Storage (swift) - Container Server... Oct 20 10:53:41 swift liberasurecode[12707]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12707]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12707]: liberasurecode_instance_create: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12707]: liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift container-server[12707]: Started child 12731 Oct 20 10:53:41 swift container-server[12707]: Started child 12732 ● openstack-swift-account.service - OpenStack Object Storage (swift) - Account Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-account.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2017-10-20 10:53:41 CST; 34min ago Process: 12705 ExecStart=/usr/bin/swift-account-server /etc/swift/account-server.conf (code=exited, status=1/FAILURE) Main PID: 12705 (code=exited, status=1/FAILURE) Oct 20 10:53:41 swift systemd[1]: Started OpenStack Object Storage (swift) - Account Server. Oct 20 10:53:41 swift systemd[1]: Starting OpenStack Object Storage (swift) - Account Server... Oct 20 10:53:41 swift liberasurecode[12705]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12705]: liberasurecode_instance_create: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12705]: liberasurecode_instance_create: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift liberasurecode[12705]: liberasurecode_instance_create: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory Oct 20 10:53:41 swift systemd[1]: openstack-swift-account.service: main process exited, code=exited, status=1/FAILURE Oct 20 10:53:41 swift systemd[1]: Unit openstack-swift-account.service entered failed state. Oct 20 10:53:41 swift systemd[1]: openstack-swift-account.service failed.
I'm facing the same issue on a fresh RHOP10 deployment (3 CTRL + 16 Computes)
(In reply to Marius Cornea from comment #0) > The service gets started but I'm not sure if these errors affect swift > functionality. This doesn't affect Swift functionality (at least not the default one). The errors are due to missing additional 3rd party libraries; these can be used instead of the default EC libraries if wanted & provided. However, these are not included in OSP, thus these warnings are ok. I just checked functionality & service startup on latest OSP10z9, and this looks all good to me using the following rpms: python-pyeclib-1.3.1-1.el7ost.x86_64 liberasurecode-1.2.0-2.el7ost.x86_64 openstack-swift-account-2.10.2-3.el7ost.noarch openstack-swift-object-2.10.2-3.el7ost.noarch openstack-swift-container-2.10.2-3.el7ost.noarch If a service doesn't start, it is very unlikely due to these errors (except when configuring Swift EC to use these 3rd party libraries and not installing them). Closing this BZ therefore.
Closed worksforme, nothing to test/automate per close loop.