Bug 1167156

Hi everyone,
I'm seeing this with a fresh install of RDO Liberty with packstack.
In my dmesg, I noticed this:
[root@osp3 log]# dmesg -T|grep -i seg
[Wed Nov 25 15:16:12 2015] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[Wed Nov 25 16:00:33 2015] cinder-api[7442]: segfault at 7f15026d3760 ip 00007f15026d3760 sp 00007ffd22a6d478 error 15 in multiarray.so[7f15026d2000+d000]
[Wed Nov 25 16:01:37 2015] cinder-api[7439]: segfault at 7f15026d3760 ip 00007f15026d3760 sp 00007ffd22a6d478 error 15 in multiarray.so[7f15026d2000+d000]
[Wed Nov 25 16:03:30 2015] cinder-api[121099]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844418 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:04:07 2015] cinder-api[121491]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844358 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:06:55 2015] cinder-api[123452]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844358 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:10:22 2015] cinder-api[129300]: segfault at 7f36f9c4e760 ip 00007f36f9c4e760 sp 00007ffd038536b8 error 15 in multiarray.so[7f36f9c4d000+d000]

Here's my package information:
openstack-neutron-openvswitch-7.0.0-2.el7.noarch
openstack-dashboard-8.0.0-1.el7.noarch
openstack-nova-api-12.0.0-1.el7.noarch
openstack-nova-scheduler-12.0.0-1.el7.noarch
openstack-ceilometer-common-5.0.0-1.el7.noarch
openstack-ceilometer-alarm-5.0.0-1.el7.noarch
openstack-packstack-puppet-2015.2-0.1.dev1654.gcbbf46e.el7.noarch
openstack-neutron-common-7.0.0-2.el7.noarch
openstack-packstack-2015.2-0.1.dev1654.gcbbf46e.el7.noarch
openstack-ceilometer-api-5.0.0-1.el7.noarch
openstack-nova-console-12.0.0-1.el7.noarch
openstack-cinder-7.0.0-1.el7.noarch
openstack-nova-conductor-12.0.0-1.el7.noarch
openstack-neutron-ml2-7.0.0-2.el7.noarch
openstack-swift-plugin-swift3-1.7-4.el7.noarch
openstack-utils-2014.2-1.el7.noarch
openstack-ceilometer-compute-5.0.0-1.el7.noarch
openstack-swift-account-2.5.0-1.el7.noarch
openstack-ceilometer-central-5.0.0-1.el7.noarch
openstack-selinux-0.6.41-1.el7.noarch
openstack-swift-object-2.5.0-1.el7.noarch
openstack-nova-common-12.0.0-1.el7.noarch
openstack-nova-novncproxy-12.0.0-1.el7.noarch
openstack-neutron-7.0.0-2.el7.noarch
openstack-swift-2.5.0-1.el7.noarch
openstack-ceilometer-notification-5.0.0-1.el7.noarch
openstack-keystone-8.0.0-1.el7.noarch
openstack-nova-cert-12.0.0-1.el7.noarch
openstack-ceilometer-collector-5.0.0-1.el7.noarch
openstack-nova-compute-12.0.0-1.el7.noarch
openstack-swift-container-2.5.0-1.el7.noarch
openstack-ceilometer-polling-5.0.0-1.el7.noarch
openstack-puppet-modules-7.0.0-1.el7.noarch
openstack-swift-proxy-2.5.0-1.el7.noarch
openstack-glance-11.0.0-1.el7.noarch

# file coredump 
coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dis'


# cat core_backtrace 
{   "signal": 11
,   "executable": "/usr/bin/python2.7"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 139728211752800
                ,   "build_id": "de5043ccb8fcffec55983392b1bf608a3524b13c"
                ,   "build_id_offset": 3327840
                ,   "file_name": "/usr/lib64/python2.7/site-packages/numpy/core/multiarray.so"
                } ]
        } ]
}

# file coredump 
coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dis'


# cat core_backtrace 
{   "signal": 11
,   "executable": "/usr/bin/python2.7"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 139728211752800
                ,   "build_id": "de5043ccb8fcffec55983392b1bf608a3524b13c"
                ,   "build_id_offset": 3327840
                ,   "file_name": "/usr/lib64/python2.7/site-packages/numpy/core/multiarray.so"
                } ]
        } ]
}

#  nova --debug volume-create --display-name i1v1 1
DEBUG (session:198) REQ: curl -g -i -X GET http://10.0.128.166:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (connectionpool:383) "GET /v2.0 HTTP/1.1" 200 229
DEBUG (session:215) RESP: [200] content-length: 229 content-encoding: gzip vary: X-Auth-Token,Accept-Encoding server: Apache/2.4.6 (Red Hat Enterprise Linux) connection: close date: Wed, 25 Nov 2015 21:45:15 GMT content-type: application/json x-openstack-request-id: req-b67ba33f-53db-4fd6-a949-11a349ddec5b 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.0.128.166:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:86) Making authentication request to http://10.0.128.166:5000/v2.0/tokens
INFO (connectionpool:238) Resetting dropped connection: 10.0.128.166
DEBUG (connectionpool:383) "POST /v2.0/tokens HTTP/1.1" 200 989
DEBUG (session:198) REQ: curl -g -i -X GET http://10.0.128.166:8774/v2/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ca8c58454f98563cbafa1452a5dbf7babd154230"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (connectionpool:383) "GET /v2/ HTTP/1.1" 200 377
DEBUG (session:215) RESP: [200] date: Wed, 25 Nov 2015 21:45:16 GMT connection: keep-alive content-type: application/json content-length: 377 x-compute-request-id: req-f00cb5a3-309f-455c-b2b1-50c223a4de45 
RESP BODY: {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.0.128.166:8774/v2/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}

WARNING: Command volume-create is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
DEBUG (session:198) REQ: curl -g -i -X POST http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ca8c58454f98563cbafa1452a5dbf7babd154230" -d '{"volume": {"display_name": "i1v1", "imageRef": null, "availability_zone": null, "volume_type": null, "display_description": null, "snapshot_id": null, "size": 1}}'
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (shell:905) Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 903, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 830, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 2124, in do_volume_create
    imageRef=args.image_id)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 79, in create
    return self._create('/volumes', body, 'volume')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 386, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 430, in _send_request
    raise exceptions.ConnectionRefused(msg)
ConnectionRefused: Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes
ERROR (ConnectionRefused): Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes

# cinder list
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
| ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
You have new mail in /var/spool/mail/root

Somehow, I found a workaround:

# mv /usr/lib64/python2.7/site-packages/numpy /root

(move numpy out of the way)

After this, I was able to create a cinder volume:

[stack@osp3 local(keystone_admin)]$ cinder create --volume_type iscsi --display_name my-test 1
+---------------------------------------+--------------------------------------+
|                Property               |                Value                 |
+---------------------------------------+--------------------------------------+
|              attachments              |                  []                  |
|           availability_zone           |                 nova                 |
|                bootable               |                false                 |
|          consistencygroup_id          |                 None                 |
|               created_at              |      2015-11-25T23:12:05.000000      |
|              description              |                 None                 |
|               encrypted               |                False                 |
|                   id                  | bbcb3926-474c-44fa-b371-c3e0c6c42e2d |
|                metadata               |                  {}                  |
|            migration_status           |                 None                 |
|              multiattach              |                False                 |
|                  name                 |               my-test                |
|         os-vol-host-attr:host         |                 None                 |
|     os-vol-mig-status-attr:migstat    |                 None                 |
|     os-vol-mig-status-attr:name_id    |                 None                 |
|      os-vol-tenant-attr:tenant_id     |   c75e4b6fcb6b4d14bb1a53b9b94f88d4   |
|   os-volume-replication:driver_data   |                 None                 |
| os-volume-replication:extended_status |                 None                 |
|           replication_status          |               disabled               |
|                  size                 |                  1                   |
|              snapshot_id              |                 None                 |
|              source_volid             |                 None                 |
|                 status                |               creating               |
|                user_id                |   6040218d2b234806a90e54c48cf7358d   |
|              volume_type              |                iscsi                 |
+---------------------------------------+--------------------------------------+
[stack@osp3 local(keystone_admin)]$ cinder list
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+
|                  ID                  |   Status  | Migration Status |   Name  | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+
| bbcb3926-474c-44fa-b371-c3e0c6c42e2d | available |        -         | my-test |  1   |    iscsi    |  false   |    False    |             |
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+

This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.

Re-Opening as I am seeing this again with RDO Newton-3.
Again, moving numpy out of the way solved the issue.[stack@osp1 ~

(keystone_admin)]$ cinder service-list
ERROR: Unable to establish connection to http://10.0.128.164:8776/v2/39e70c8bc5364ddb84818935e1438f28/os-services

[root@osp1 cinder(keystone_admin)]# mv /usr/lib64/python2.7/site-packages/numpy /root

[stack@osp1 ~(keystone_admin)]$ cinder service-list
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host     | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | osp1     | nova | enabled | up    | 2016-12-14T13:17:37.000000 | -               |
| cinder-scheduler | osp1     | nova | enabled | up    | 2016-12-14T13:17:16.000000 | -               |
| cinder-volume    | osp1@lvm | nova | enabled | up    | 2016-12-14T13:17:38.000000 | -               |
+------------------+----------+------+---------+-------+----------------------------+-----------------+

This is on an all-in-one packstack install from 2016/12/12. (Dev 12th 2016)

This is on a fully patched RHEL7.3 x86_64 and numpy is from RedHat:
[stack@osp1 ~(keystone_admin)]$ rpm -qi numpy
Name        : numpy
Epoch       : 1
Version     : 1.7.1
Release     : 11.el7
Architecture: x86_64
Install Date: Tue 13 Dec 2016 11:27:29 AM EST
Group       : Development/Languages
Size        : 13350837
License     : BSD and Python
Signature   : RSA/SHA256, Wed 23 Sep 2015 02:47:49 AM EDT, Key ID 199e2f91fd431d51
Source RPM  : numpy-1.7.1-11.el7.src.rpm
Build Date  : Fri 08 Aug 2014 04:15:11 AM EDT
Build Host  : x86-021.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.numpy.org/
Summary     : A fast multidimensional array facility for Python
Description :
NumPy is a general-purpose array-processing package designed to
efficiently manipulate large multi-dimensional arrays of arbitrary
records without sacrificing too much speed for small multi-dimensional
arrays.  NumPy is built on the Numeric code base and adds features
introduced by numarray as well as an extended C-API and the ability to
create arrays of arbitrary type.

There are also basic facilities for discrete fourier transform,
basic linear algebra and random number generation. Also included in
this package is a version of f2py that works properly with NumPy.

Can you provide a debug-level cinder api log when this occurs?

Thanks!

As soon as I move numpy back to its original location, the following happens:
cinder services die and keep being restarted.
In /var/log/cinder/api.log, I get these lines over and over:

2016-12-14 10:22:51.893 4214 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:22:51.894 4214 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:51.917 4214 WARNING oslo_config.cfg [-] Option "enable_v1_api" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:51.917 4214 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
2016-12-14 10:22:54.185 4507 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:22:54.186 4507 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:54.205 4507 WARNING oslo_config.cfg [-] Option "enable_v1_api" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:54.206 4507 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
^C

With DEBUG enabled in cinder.conf, here's what I get:

2016-12-14 10:39:44.680 109929 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:39:44.681 109929 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:39:44.712 109929 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-12-14 10:39:44.712 109929 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-12-14 10:39:44.713 109929 DEBUG oslo.service.wsgi [-] Loading app osapi_volume from /etc/cinder/api-paste.ini load_app /usr/lib/python2.7/site-packages/oslo_service/wsgi.py:352
2016-12-14 10:39:44.715 109929 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
2016-12-14 10:39:47.238 110204 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:39:47.238 110204 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:39:47.257 110204 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-12-14 10:39:47.257 110204 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-12-14 10:39:47.257 110204 DEBUG oslo.service.wsgi [-] Loading app osapi_volume from /etc/cinder/api-paste.ini load_app /usr/lib/python2.7/site-packages/oslo_service/wsgi.py:352
2016-12-14 10:39:47.259 110204 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
^C

One more thing that I noticed when I ran packstack again with GNOCCHI enabled:

10.0.128.164_controller.pp:                       [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.0.128.164_controller.pp
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: ImportError: C extension: No module named numpy not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.

Is it possible that it is gnocchi that pulls down numpy, which made cinder segfault?

[root@osp1 ~]# rpm -e numpy
error: Failed dependencies:
        numpy is needed by (installed) python2-scipy-0.17.0-2.el7.x86_64
        numpy >= 1.6 is needed by (installed) python-numexpr-2.3-4.el7.x86_64
        numpy is needed by (installed) python-tables-3.1.1-2.el7.x86_64
        numpy is needed by (installed) python-matplotlib-1.2.0-15.el7.x86_64
        numpy is needed by (installed) python-pandas-0.17.0-2.el7.x86_64
        numpy is needed by (installed) python-gnocchi-3.0.0-1.el7.noarch
        numpy(x86-64) is needed by (installed) python-Bottleneck-0.6.0-4.el7.x86_64

numpy seems required by python-gnocchi..

Problem seems to be in the adddocs part of numpy (at least the interaction with my test system). Removing the following two files makes the problem go away:

/usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc
/usr/lib64/python2.7/site-packages/numpy/add_newdocs.py

output below:

[root@osp1 numpy(keystone_admin)]# cinder service-list
ERROR: Unable to establish connection to http://10.0.128.164:8776/v2/39e70c8bc5364ddb84818935e1438f28/os-services: HTTPConnectionPool(host='10.0.128.164', port=8776): Max retries exceeded with url: /v2/39e70c8bc5364ddb84818935e1438f28/os-services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3463490>: Failed to establish a new connection: [Errno 111] Connection refused',))

[root@osp1 numpy(keystone_admin)]# rm /usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc
rm: remove regular file ‘/usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc’? y

[root@osp1 numpy(keystone_admin)]# rm /usr/lib64/python2.7/site-packages/numpy/add_newdocs.py
rm: remove regular file ‘/usr/lib64/python2.7/site-packages/numpy/add_newdocs.py’? y

[root@osp1 numpy(keystone_admin)]# cinder service-list
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host     | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | osp1     | nova | enabled | up    | 2016-12-14T23:45:03.000000 | -               |
| cinder-scheduler | osp1     | nova | enabled | up    | 2016-12-14T23:45:05.000000 | -               |
| cinder-volume    | osp1@lvm | nova | enabled | down  | 2016-12-14T23:40:08.000000 | -               |
+------------------+----------+------+---------+-------+----------------------------+-----------------+

Disregard the previous comment, -anything- that breaks numpy will work around the issue.
My knowledge of python imports is limited but I've figured that commenting out a single line in __init__.py from numpy makes the segfault go away:

# diff -ru /usr/lib64/python2.7/site-packages/numpy.orig /usr/lib64/python2.7/site-packages/numpy
diff -ru /usr/lib64/python2.7/site-packages/numpy.orig/__init__.py /usr/lib64/python2.7/site-packages/numpy/__init__.py
--- /usr/lib64/python2.7/site-packages/numpy.orig/__init__.py   2013-04-07 01:04:05.000000000 -0400
+++ /usr/lib64/python2.7/site-packages/numpy/__init__.py        2016-12-14 22:37:08.170209848 -0500
@@ -147,7 +147,7 @@
     from core import *
     import compat
     import lib
-    from lib import *
+    #from lib import *
     import linalg
     import fft
     import polynomial

This probably invalidates the whole numpy module but I have no idea on how to debug this further at this point..

Gdb?

Regards,
Vincent

I ran cinder-api under gdb and got this:

[root@osp1 ~]# gdb /usr/bin/python2
[...]
(gdb) set args /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
(gdb) r
Starting program: /usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Detaching after fork from child process 35441.
Detaching after fork from child process 35443.
Detaching after fork from child process 35445.
Detaching after fork from child process 35455.
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2016-12-14 23:09:22.376 35433 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 23:09:22.377 35433 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 23:09:22.422 35433 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.

Program received signal SIGSEGV, Segmentation fault.
0x00007fffd8788760 in PyArray_API () from /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
Missing separate debuginfos, use: debuginfo-install PyQt4-4.10.1-13.el7.x86_64 PyYAML-3.10-11.el7.x86_64 atlas-3.10.1-10.el7.x86_64 attica-0.4.2-1.el7.x86_64 blas-3.4.2-5.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 dbus-libs-1.6.12-17.el7.x86_64 dbusmenu-qt-0.9.2-7.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fontconfig-2.10.95-10.el7.x86_64 freetype-2.4.11-12.el7.x86_64 gamin-0.1.10-16.el7.x86_64 glib2-2.46.2-4.el7.x86_64 gmp-6.0.0-12.el7_1.x86_64 gobject-introspection-1.42.0-1.el7.x86_64 kdelibs-4.14.8-5.el7_2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 lapack-3.4.2-5.el7.x86_64 libICE-1.0.9-2.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.14-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.1-2.1.el7.x86_64 libXft-2.3.2-2.el7.x86_64 libXi-1.7.4-2.el7.x86_64 libXinerama-1.1.3-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libXrandr-1.4.2-2.el7.x86_64 libXrender-0.9.8-2.1.el7.x86_64 libXtst-1.2.2-2.1.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgfortran-4.8.5-11.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libquadmath-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64 libutempter-1.1.6-4.el7.x86_64 libuuid-2.23.2-33.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 libyaml-0.1.4-11.el7_0.x86_64 m2crypto-0.21.1-17.el7.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.3-2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.21.3-1.1.el7_3.x86_64 numpy-1.7.1-11.el7.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 postgresql-libs-9.2.18-1.el7.x86_64 pygobject3-base-3.14.0-3.el7.x86_64 pykde4-4.10.5-4.el7.x86_64 python-lxml-3.2.1-4.el7.x86_64 python-markupsafe-0.11-10.el7.x86_64 python-msgpack-0.4.6-3.el7.x86_64 python-netifaces-0.10.4-3.el7.x86_64 python-psutil-1.2.1-1.el7.x86_64 python-psycopg2-2.5.1-3.el7.x86_64 python-simplejson-3.5.3-5.el7.x86_64 python-sqlalchemy-1.0.11-1.el7.x86_64 python-wrapt-1.10.8-2.el7.x86_64 python2-cffi-1.5.2-1.el7.x86_64 python2-crypto-2.6.1-9.el7.x86_64 python2-cryptography-1.3.1-3.el7.x86_64 python2-greenlet-0.4.9-1.el7.x86_64 python2-scipy-0.17.0-2.el7.x86_64 qca2-2.0.3-7.el7.x86_64 qt-4.8.5-13.el7.x86_64 qt-x11-4.8.5-13.el7.x86_64 sip-4.14.6-4.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64

The begining of the backtrace shows this:

(gdb) bt
#0  0x00007fffd8788760 in PyArray_API () at /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
#1  0x00007fffd5574c0f in init_fblas () at /usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so
#2  0x00007ffff7b0bb69 in _PyImport_LoadDynamicModule (name=name@entry=0x5dfc390 "scipy.linalg._fblas", pathname=pathname@entry=0x5dfe280 "/usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so", fp=<optimized out>)
    at /usr/src/debug/Python-2.7.5/Python/importdl.c:53
#3  0x00007ffff7b099a3 in load_module (name=name@entry=0x5dfc390 "scipy.linalg._fblas", fp=<optimized out>, pathname=pathname@entry=0x5dfe280 "/usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so", type=<optimized out>, loader=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/import.c:1915
#4  0x00007ffff7b09c41 in import_submodule (mod=mod@entry=<module at remote 0x5d34478>, subname=subname@entry=0x5d33924 "_fblas", fullname=fullname@entry=0x5dfc390 "scipy.linalg._fblas") at /usr/src/debug/Python-2.7.5/Python/import.c:2700
#5  0x00007ffff7b0a13f in ensure_fromlist (mod=mod@entry=<module at remote 0x5d34478>, fromlist=fromlist@entry=('_fblas',), buf=buf@entry=0x5dfc390 "scipy.linalg._fblas", buflen=12, recursive=recursive@entry=0)
    at /usr/src/debug/Python-2.7.5/Python/import.c:2606
#6  0x00007ffff7b0a97a in PyImport_ImportModuleLevel (locals=<optimized out>, level=<optimized out>, fromlist=('_fblas',), globals=<optimized out>, name=0x0) at /usr/src/debug/Python-2.7.5/Python/import.c:2269
#7  0x00007ffff7b0a97a in PyImport_ImportModuleLevel (name=<optimized out>, globals=<optimized out>, locals=<optimized out>, fromlist=('_fblas',), level=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/import.c:2288
#8  0x00007ffff7aedb1f in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/bltinmodule.c:49

Let me know if you'd like more information.
Regards,

I am still investigating.
With an unmodified numpy, the problem disappeared when I removed the following rpms (they have some payload under /usr/{lib,lib64}/python2.7/site-packages)

cmpi-bindings-pywbem-0.9.5-6.el7.x86_64
dbus-python-1.1.1-9.el7.i686
farstream-python-0.1.2-8.el7.x86_64
file /usr/lib64/python2.7/site-packages/log_picker is not owned by any package
file /usr/lib64/python2.7/site-packages/numpy.orig is not owned by any package
file /usr/lib64/python2.7/site-packages/pyanaconda is not owned by any package
firstboot-19.12-1.el7.x86_64
fontforge-20120731b-12.el7.i686
fontforge-20120731b-12.el7.x86_64
gnome-abrt-0.3.4-8.el7.x86_64
gnome-tweak-tool-3.14.3-2.el7.noarch
gstreamer-python-0.10.22-6.el7.x86_64
gupnp-igd-python-0.2.2-3.el7.x86_64
hplip-libs-3.15.9-2.el7.i686
hplip-libs-3.15.9-2.el7.x86_64
iotop-0.6-2.el7.noarch
langtable-python-0.0.31-3.el7.noarch
liblouis-python-2.5.2-10.el7.noarch
libstoragemgmt-python-1.3.4-1.el7.noarch
libuser-python-0.60-7.el7_1.x86_64
lvm2-python-libs-2.02.166-1.el7_3.1.x86_64
mpich-3.0-3.0.4-10.el7.i686
mpich-3.0-3.0.4-10.el7.x86_64
mpich-3.2-3.2-2.el7.i686
mpich-3.2-3.2-2.el7.x86_64
mvapich2-2.0-2.0a-6.el7.i686
mvapich2-2.0-2.0a-6.el7.x86_64
mvapich2-2.0-psm-2.0a-6.el7.x86_64
mvapich2-2.2-2.2-0.3.rc1.el7.i686
mvapich2-2.2-2.2-0.3.rc1.el7.x86_64
mvapich2-2.2-psm-2.2-0.3.rc1.el7.x86_64
openmpi-1.10.3-3.el7.i686
openmpi-1.10.3-3.el7.x86_64
orca-3.6.3-4.el7.x86_64
PackageKit-1.0.7-6.el7.x86_64
pykde4-4.10.5-4.el7.i686
pykde4-4.10.5-4.el7.x86_64
pyldb-1.1.26-1.el7.i686
pyldb-1.1.26-1.el7.x86_64
pyorbit-2.24.0-15.el7.x86_64
PyQt4-4.10.1-13.el7.i686
PyQt4-4.10.1-13.el7.x86_64
pytalloc-2.1.6-1.el7.i686
pywbem-0.7.0-25.20130827svn625.el7.noarch
setroubleshoot-3.2.27.2-3.el7.x86_64
setroubleshoot-server-3.2.27.2-3.el7.x86_64
sip-4.14.6-4.el7.i686
sip-4.14.6-4.el7.x86_64
speech-dispatcher-python-0.7.1-15.el7.x86_64
system-config-date-1.10.6-2.el7.noarch
system-config-kdump-2.0.13-15.el7.noarch
system-config-keyboard-base-1.4.0-4.el7.noarch
system-config-printer-libs-1.4.1-19.el7.noarch
systemd-python-219-30.el7_3.6.x86_64
wireshark-1.10.14-10.el7.i686
wireshark-1.10.14-10.el7.x86_64
yum-langpacks-0.4.2-7.el7.noarch

I am currently trying to isolate this further..

This doesn't make any sense...

if I remove the following two rpms, then the segfault goes away.

pykde4-4.10.5-4.el7.i686
pykde4-4.10.5-4.el7.x86_64

I'll be uploading a small video shortly to demo this issue.

Created attachment 1232272 [details]
Video showing how the removal of pykde4 resolves the issue of segfault in numpy (multiarray.so)

I'am attempting to re-route this issue to 'numpy' in RHEL.

The issue can be reproduced simply with the following python code:

import dl
import sys
sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)
import scipy.linalg

It is fixed by this upstream commit:
https://github.com/numpy/numpy/commit/a226daa75816577ac9c734133e600e5eebd9aaa9

Created attachment 1234818 [details]
Backported upstream patch

*** Bug 1547531 has been marked as a duplicate of this bug. ***

Hello Nikola,

could you please have a look at the description of this bug (Doc Text field) above and tell me whether it is correct content-wise?

I am not sure what kind of crash it is neither am I sure what f2py refers to (a library, a package, a command, ...?)

Could you also tell me how this bug influenced the end user?

Thank you

Jaroslav

Hello Jaroslav,

the doc text seems fine, only "PyArray API" should be "PyArray_API", it's literally name of a symbol.
Also there is no "kernel crash", just segmentation fault in numpy/core/multiarray.so.

f2py is "Fortran to Python interface generator", it builds Python modules from Fortran sources.

The PyArray_API symbol is present in multiarray module, which is part of numpy, but before the fix it was also present in all modules generated by f2py. And every generated module also imports multiarray module from numpy. A lot of generated modules are part of scipy, and when RTLD_GLOBAL flag was set (that's not usually a case), importing such modules lead to a conflict and a crash.

(In reply to Nikola Forró from comment #33)
> Hello Jaroslav,
> 
> the doc text seems fine, only "PyArray API" should be "PyArray_API", it's
> literally name of a symbol.
> Also there is no "kernel crash", just segmentation fault in
> numpy/core/multiarray.so.
> 
> f2py is "Fortran to Python interface generator", it builds Python modules
> from Fortran sources.
> 
> The PyArray_API symbol is present in multiarray module, which is part of
> numpy, but before the fix it was also present in all modules generated by
> f2py. And every generated module also imports multiarray module from numpy.
> A lot of generated modules are part of scipy, and when RTLD_GLOBAL flag was
> set (that's not usually a case), importing such modules lead to a conflict
> and a crash.

Thank you Nikola,

I've updated the doc text above.

Jaroslav

*** Bug 1533516 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0659