Bug 1167156 - cinder-api[14407]: segfault at 7fc84636f7e0 ip 00007fc84636f7e0 sp 00007fff3110a468 error 15 in multiarray.so[7fc846369000+d000]
Summary: cinder-api[14407]: segfault at 7fc84636f7e0 ip 00007fc84636f7e0 sp 00007fff31...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: numpy   
(Show other bugs)
Version: 7.3
Hardware: x86_64 Linux
medium
medium
Target Milestone: rc
: 7.3
Assignee: Nikola Forró
QA Contact: Mirosław Zalewski
Jaroslav Klech
URL:
Whiteboard:
Keywords: Patch, Reopened
: 1533516 1547531 (view as bug list)
Depends On:
Blocks: 1420851 1115589 1391165 1465906 1465925 1531036
TreeView+ depends on / blocked
 
Reported: 2014-11-24 03:27 UTC by Viorel Tabara
Modified: 2018-04-10 08:35 UTC (History)
9 users (show)

Fixed In Version: numpy-1.7.1-12.el7
Doc Type: Bug Fix
Doc Text:
Importing python modules generated by *f2py* now works properly Previously, when dynamic linking loader was configured to load symbols globally, a segmentation fault occurred when importing any python module generated by the *f2py* utility. This update renames the `PyArray_API` symbol to `_npy_f2py_ARRAY_API`, which prevents potential conflicts with the same symbol in the multiarray module. As a result, importing modules generated by *f2py* no longer leads to a segmentation fault.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 08:34:47 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Video showing how the removal of pykde4 resolves the issue of segfault in numpy (multiarray.so) (2.22 MB, application/octet-stream)
2016-12-15 16:25 UTC, Vincent S. Cojot
no flags Details
Backported upstream patch (1.22 KB, patch)
2016-12-22 15:30 UTC, Nikola Forró
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0659 None None None 2018-04-10 08:35 UTC

Description Viorel Tabara 2014-11-24 03:27:04 UTC
Description of problem:
=======================

Working with packages from:

   https://repos.fedorapeople.org/repos/openstack/openstack-juno/fedora-20/

I cannot create Cinder volumes as `cinder-api` is segfault-ing.

This is a fresh `packstack` install as documented at:

   https://openstack.redhat.com/Quickstart



How reproducible:
=================

Always.



Steps to Reproduce:
===================

1. Install using `packstack`.

2. Create a Cinder volume with either:

      nova --debug volume-create --display-name i1v1 1
      
   or:

      cinder --debug create --display-name i1v1 1
      
3. List the volumes:

      nova volume-list



Actual results:
===============

The volume isn't created.

After the REQ and RESP debug outputs showing the `curl` requests and responses 
the following error:

   DEBUG (shell:803) HTTPConnectionPool(host='192.168.0.9', port=8776): Max retries exceeded with url: /v1/727bc2ee4d1a46fba7be2ca689a1e4a6/volumes (Caused by <
   class 'httplib.BadStatusLine'>: '')
   Traceback (most recent call last):
     File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main
       OpenStackComputeShell().main(argv)
     File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main
       args.func(self.cs, args)
     File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 1837, in do_volume_create
       imageRef=args.image_id)
     File "/usr/lib/python2.7/site-packages/novaclient/v1_1/volumes.py", line 71, in create
       return self._create('/volumes', body, 'volume')
     File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create
       _resp, body = self.api.client.post(url, body=body)
     File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post
       return self._cs_request(url, 'POST', **kwargs)
     File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request
       resp, body = self._time_request(url, method, **kwargs)
     File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request
       resp, body = self.request(url, method, **kwargs)
     File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 410, in request
       **kwargs)
     File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
       return session.request(method=method, url=url, **kwargs)
     File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
       resp = self.send(prep, **send_kwargs)
     File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
       r = adapter.send(request, **kwargs)
     File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
       raise ConnectionError(e)
   ConnectionError: HTTPConnectionPool(host='192.168.0.9', port=8776): Max retries exceeded with url: /v1/727bc2ee4d1a46fba7be2ca689a1e4a6/volumes (Caused by <c
   lass 'httplib.BadStatusLine'>: '')
   ERROR (ConnectionError): HTTPConnectionPool(host='192.168.0.9', port=8776): Max retries exceeded with url: /v1/727bc2ee4d1a46fba7be2ca689a1e4a6/volumes (Caus
   ed by <class 'httplib.BadStatusLine'>: '')


In /var/log/messages (or /var/log/kernel):

   Nov 22 23:16:22 stuff kernel: cinder-api[22114]: segfault at 7fbda26367e0 ip 00007fbda26367e0 sp 00007fff7069d3c8 error 15 in multiarray.so[7fbda2630000+d000]
 

Expected results:
=================

Successfully create a Cinder volume.



Additional info:
================

I can provide the 'coredump` file.

Here's the abrt info:

   Directory:      /var/tmp/abrt/ccpp-2014-11-22-23:16:22-22114
   abrt_version:   2.2.2
   analyzer:       CCpp
   architecture:   x86_64
   cmdline:        /usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
   component:      openstack-cinder
   count:          1
   executable:     /usr/bin/python2.7
   hostname:       stuff.can.local
   kernel:         3.16.7-200.fc20.x86_64
   last_occurrence: 1416723382
   os_release:     Fedora release 20 (Heisenbug)
   package:        openstack-cinder-2014.2-1.fc22
   pid:            22114
   pkg_arch:       noarch
   pkg_epoch:      0
   pkg_name:       openstack-cinder
   pkg_release:    1.fc22
   pkg_version:    2014.2
   pwd:            /
   reason:         python2.7 killed by SIGSEGV
   runlevel:       N 5
   time:           Sat 22 Nov 2014 11:16:22 PM MST
   type:           CCpp
   uid:            165
   username:       cinder
   uuid:           c9b2d07d3a08e30d5bb602411e7628ac38ac2755

   coredump:       Binary file, 90284032 bytes
   dso_list:       Text file, 16184 bytes
   maps:           Text file, 74532 bytes

   cgroup:
   :10:hugetlb:/
   :9:perf_event:/
   :8:blkio:/
   :7:net_cls,net_prio:/
   :6:freezer:/
   :5:devices:/system.slice
   :4:memory:/
   :3:cpu,cpuacct:/
   :2:cpuset:/
   :1:name=systemd:/system.slice/openstack-cinder-api.service

   core_backtrace:
   :{   "signal": 11
   :,   "executable": "/usr/bin/python2.7"
   :,   "stacktrace":
   :      [ {   "crash_thread": true
   :        ,   "frames":
   :              [ {   "address": 140452449970144
   :                ,   "build_id": "357625781bbad706320b03674e8aef5cc461c5be"
   :                ,   "build_id_offset": 3446752
   :                ,   "file_name": "/usr/lib64/python2.7/site-packages/numpy/core/multiarray.so"
   :                } ]
   :        } ]
   :}

   environ:
   :PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
   :LANG=en_US.UTF-8
   :HOME=/var/lib/cinder
   :LOGNAME=cinder
   :USER=cinder
   :SHELL=/sbin/nologin

   limits:
   :Limit                     Soft Limit           Hard Limit           Units
   :Max cpu time              unlimited            unlimited            seconds
   :Max file size             unlimited            unlimited            bytes
   :Max data size             unlimited            unlimited            bytes
   :Max stack size            8388608              unlimited            bytes
   :Max core file size        0                    unlimited            bytes
   :Max resident set          unlimited            unlimited            bytes
   :Max processes             15325                15325                processes
   :Max open files            1024                 4096                 files
   :Max locked memory         65536                65536                bytes
   :Max address space         unlimited            unlimited            bytes
   :Max file locks            unlimited            unlimited            locks
   :Max pending signals       15325                15325                signals
   :Max msgqueue size         819200               819200               bytes
   :Max nice priority         0                    0
   :Max realtime priority     0                    0
   :Max realtime timeout      unlimited            unlimited            us

   open_fds:
   :0:/dev/null
   :pos:	0
   :flags:	0100000
   :mnt_id:	17
   :1:socket:[100424744]
   :pos:	0
   :flags:	02
   :mnt_id:	7
   :2:socket:[100424744]
   :pos:	0
   :flags:	02
   :mnt_id:	7
   :3:/var/log/cinder/api.log
   :pos:	2396879
   :flags:	0102001
   :mnt_id:	36
   :4:pipe:[100424546]
   :pos:	0
   :flags:	04000
   :mnt_id:	8
   :5:anon_inode:[eventpoll]
   :pos:	0
   :flags:	02
   :mnt_id:	9
   :6:anon_inode:[eventpoll]
   :pos:	0
   :flags:	02
   :mnt_id:	9
   :tfd:        4 events:       1b data:  276d10000000004
   :tfd:        7 events:       1b data:  276d10000000007
   :7:socket:[100425820]
   :pos:	0
   :flags:	04002
   :mnt_id:	7
   :8:socket:[100532334]
   :pos:	0
   :flags:	04002
   :mnt_id:	7
   :9:/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :10:socket:[100532371]
   :pos:	0
   :flags:	02
   :mnt_id:	7
   :11:/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/compiler.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :12:/usr/lib/python2.7/site-packages/taskflow/types/graph.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :13:/usr/lib/python2.7/site-packages/networkx/__init__.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :14:/usr/lib/python2.7/site-packages/networkx/linalg/__init__.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :15:/usr/lib/python2.7/site-packages/networkx/linalg/algebraicconnectivity.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :16:/usr/lib64/python2.7/site-packages/scipy/linalg/__init__.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :17:/usr/lib64/python2.7/site-packages/scipy/linalg/misc.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :18:/usr/lib64/python2.7/site-packages/scipy/linalg/blas.py
   :pos:	0
   :flags:	0100000
   :mnt_id:	36
   :19:/usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so
   :pos:	0
   :flags:	0100000
   :mnt_id:	36

   os_info:
   :NAME=Fedora
   :VERSION="20 (Heisenbug)"
   :ID=fedora
   :VERSION_ID=20
   :PRETTY_NAME="Fedora 20 (Heisenbug)"
   :ANSI_COLOR="0;34"
   :CPE_NAME="cpe:/o:fedoraproject:fedora:20"
   :HOME_URL="https://fedoraproject.org/"
   :BUG_REPORT_URL="https://bugzilla.redhat.com/"
   :REDHAT_BUGZILLA_PRODUCT="Fedora"
   :REDHAT_BUGZILLA_PRODUCT_VERSION=20
   :REDHAT_SUPPORT_PRODUCT="Fedora"
   :REDHAT_SUPPORT_PRODUCT_VERSION=20

   proc_pid_status:
   :Name:	cinder-api
   :State:	S (sleeping)
   :Tgid:	22114
   :Ngid:	0
   :Pid:	22114
   :PPid:	21732
   :TracerPid:	0
   :Uid:	165	165	165	165
   :Gid:	165	165	165	165
   :FDSize:	64
   :Groups:	99 165
   :VmPeak:	  663440 kB
   :VmSize:	  663440 kB
   :VmLck:	       0 kB
   :VmPin:	       0 kB
   :VmHWM:	   93184 kB
   :VmRSS:	   93184 kB
   :VmData:	  147968 kB
   :VmStk:	     136 kB
   :VmExe:	       4 kB
   :VmLib:	   84544 kB
   :VmPTE:	    1112 kB
   :VmSwap:	       0 kB
   :Threads:	1
   :SigQ:	0/15325
   :SigPnd:	0000000000000000
   :ShdPnd:	0000000000000000
   :SigBlk:	0000000000000000
   :SigIgn:	0000000001001002
   :SigCgt:	0000000180004001
   :CapInh:	0000000000000000
   :CapPrm:	0000000000000000
   :CapEff:	0000000000000000
   :CapBnd:	0000003fffffffff
   :Seccomp:	0
   :Cpus_allowed:	f
   :Cpus_allowed_list:	0-3
   :Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
   :Mems_allowed_list:	0
   :voluntary_ctxt_switches:	480
   :nonvoluntary_ctxt_switches:	20

Comment 1 Vincent S. Cojot 2015-11-25 21:15:29 UTC
Hi everyone,
I'm seeing this with a fresh install of RDO Liberty with packstack.
In my dmesg, I noticed this:
[root@osp3 log]# dmesg -T|grep -i seg
[Wed Nov 25 15:16:12 2015] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[Wed Nov 25 16:00:33 2015] cinder-api[7442]: segfault at 7f15026d3760 ip 00007f15026d3760 sp 00007ffd22a6d478 error 15 in multiarray.so[7f15026d2000+d000]
[Wed Nov 25 16:01:37 2015] cinder-api[7439]: segfault at 7f15026d3760 ip 00007f15026d3760 sp 00007ffd22a6d478 error 15 in multiarray.so[7f15026d2000+d000]
[Wed Nov 25 16:03:30 2015] cinder-api[121099]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844418 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:04:07 2015] cinder-api[121491]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844358 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:06:55 2015] cinder-api[123452]: segfault at 7f5b2267e760 ip 00007f5b2267e760 sp 00007ffca4844358 error 15 in multiarray.so[7f5b2267d000+d000]
[Wed Nov 25 16:10:22 2015] cinder-api[129300]: segfault at 7f36f9c4e760 ip 00007f36f9c4e760 sp 00007ffd038536b8 error 15 in multiarray.so[7f36f9c4d000+d000]

Here's my package information:
openstack-neutron-openvswitch-7.0.0-2.el7.noarch
openstack-dashboard-8.0.0-1.el7.noarch
openstack-nova-api-12.0.0-1.el7.noarch
openstack-nova-scheduler-12.0.0-1.el7.noarch
openstack-ceilometer-common-5.0.0-1.el7.noarch
openstack-ceilometer-alarm-5.0.0-1.el7.noarch
openstack-packstack-puppet-2015.2-0.1.dev1654.gcbbf46e.el7.noarch
openstack-neutron-common-7.0.0-2.el7.noarch
openstack-packstack-2015.2-0.1.dev1654.gcbbf46e.el7.noarch
openstack-ceilometer-api-5.0.0-1.el7.noarch
openstack-nova-console-12.0.0-1.el7.noarch
openstack-cinder-7.0.0-1.el7.noarch
openstack-nova-conductor-12.0.0-1.el7.noarch
openstack-neutron-ml2-7.0.0-2.el7.noarch
openstack-swift-plugin-swift3-1.7-4.el7.noarch
openstack-utils-2014.2-1.el7.noarch
openstack-ceilometer-compute-5.0.0-1.el7.noarch
openstack-swift-account-2.5.0-1.el7.noarch
openstack-ceilometer-central-5.0.0-1.el7.noarch
openstack-selinux-0.6.41-1.el7.noarch
openstack-swift-object-2.5.0-1.el7.noarch
openstack-nova-common-12.0.0-1.el7.noarch
openstack-nova-novncproxy-12.0.0-1.el7.noarch
openstack-neutron-7.0.0-2.el7.noarch
openstack-swift-2.5.0-1.el7.noarch
openstack-ceilometer-notification-5.0.0-1.el7.noarch
openstack-keystone-8.0.0-1.el7.noarch
openstack-nova-cert-12.0.0-1.el7.noarch
openstack-ceilometer-collector-5.0.0-1.el7.noarch
openstack-nova-compute-12.0.0-1.el7.noarch
openstack-swift-container-2.5.0-1.el7.noarch
openstack-ceilometer-polling-5.0.0-1.el7.noarch
openstack-puppet-modules-7.0.0-1.el7.noarch
openstack-swift-proxy-2.5.0-1.el7.noarch
openstack-glance-11.0.0-1.el7.noarch

Comment 2 Vincent S. Cojot 2015-11-25 21:41:30 UTC
# file coredump 
coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dis'


# cat core_backtrace 
{   "signal": 11
,   "executable": "/usr/bin/python2.7"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 139728211752800
                ,   "build_id": "de5043ccb8fcffec55983392b1bf608a3524b13c"
                ,   "build_id_offset": 3327840
                ,   "file_name": "/usr/lib64/python2.7/site-packages/numpy/core/multiarray.so"
                } ]
        } ]
}

Comment 3 Vincent S. Cojot 2015-11-25 21:41:39 UTC
# file coredump 
coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dis'


# cat core_backtrace 
{   "signal": 11
,   "executable": "/usr/bin/python2.7"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 139728211752800
                ,   "build_id": "de5043ccb8fcffec55983392b1bf608a3524b13c"
                ,   "build_id_offset": 3327840
                ,   "file_name": "/usr/lib64/python2.7/site-packages/numpy/core/multiarray.so"
                } ]
        } ]
}

Comment 4 Vincent S. Cojot 2015-11-25 21:46:05 UTC
#  nova --debug volume-create --display-name i1v1 1
DEBUG (session:198) REQ: curl -g -i -X GET http://10.0.128.166:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (connectionpool:383) "GET /v2.0 HTTP/1.1" 200 229
DEBUG (session:215) RESP: [200] content-length: 229 content-encoding: gzip vary: X-Auth-Token,Accept-Encoding server: Apache/2.4.6 (Red Hat Enterprise Linux) connection: close date: Wed, 25 Nov 2015 21:45:15 GMT content-type: application/json x-openstack-request-id: req-b67ba33f-53db-4fd6-a949-11a349ddec5b 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.0.128.166:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:86) Making authentication request to http://10.0.128.166:5000/v2.0/tokens
INFO (connectionpool:238) Resetting dropped connection: 10.0.128.166
DEBUG (connectionpool:383) "POST /v2.0/tokens HTTP/1.1" 200 989
DEBUG (session:198) REQ: curl -g -i -X GET http://10.0.128.166:8774/v2/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ca8c58454f98563cbafa1452a5dbf7babd154230"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (connectionpool:383) "GET /v2/ HTTP/1.1" 200 377
DEBUG (session:215) RESP: [200] date: Wed, 25 Nov 2015 21:45:16 GMT connection: keep-alive content-type: application/json content-length: 377 x-compute-request-id: req-f00cb5a3-309f-455c-b2b1-50c223a4de45 
RESP BODY: {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.0.128.166:8774/v2/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}

WARNING: Command volume-create is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
DEBUG (session:198) REQ: curl -g -i -X POST http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ca8c58454f98563cbafa1452a5dbf7babd154230" -d '{"volume": {"display_name": "i1v1", "imageRef": null, "availability_zone": null, "volume_type": null, "display_description": null, "snapshot_id": null, "size": 1}}'
INFO (connectionpool:203) Starting new HTTP connection (1): 10.0.128.166
DEBUG (shell:905) Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 903, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 830, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 2124, in do_volume_create
    imageRef=args.image_id)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 79, in create
    return self._create('/volumes', body, 'volume')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 386, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 430, in _send_request
    raise exceptions.ConnectionRefused(msg)
ConnectionRefused: Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes
ERROR (ConnectionRefused): Unable to establish connection to http://10.0.128.166:8776/v1/c75e4b6fcb6b4d14bb1a53b9b94f88d4/volumes

# cinder list
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
| ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
You have new mail in /var/spool/mail/root

Comment 5 Vincent S. Cojot 2015-11-25 23:13:46 UTC
Somehow, I found a workaround:

# mv /usr/lib64/python2.7/site-packages/numpy /root

(move numpy out of the way)

After this, I was able to create a cinder volume:

[stack@osp3 local(keystone_admin)]$ cinder create --volume_type iscsi --display_name my-test 1
+---------------------------------------+--------------------------------------+
|                Property               |                Value                 |
+---------------------------------------+--------------------------------------+
|              attachments              |                  []                  |
|           availability_zone           |                 nova                 |
|                bootable               |                false                 |
|          consistencygroup_id          |                 None                 |
|               created_at              |      2015-11-25T23:12:05.000000      |
|              description              |                 None                 |
|               encrypted               |                False                 |
|                   id                  | bbcb3926-474c-44fa-b371-c3e0c6c42e2d |
|                metadata               |                  {}                  |
|            migration_status           |                 None                 |
|              multiattach              |                False                 |
|                  name                 |               my-test                |
|         os-vol-host-attr:host         |                 None                 |
|     os-vol-mig-status-attr:migstat    |                 None                 |
|     os-vol-mig-status-attr:name_id    |                 None                 |
|      os-vol-tenant-attr:tenant_id     |   c75e4b6fcb6b4d14bb1a53b9b94f88d4   |
|   os-volume-replication:driver_data   |                 None                 |
| os-volume-replication:extended_status |                 None                 |
|           replication_status          |               disabled               |
|                  size                 |                  1                   |
|              snapshot_id              |                 None                 |
|              source_volid             |                 None                 |
|                 status                |               creating               |
|                user_id                |   6040218d2b234806a90e54c48cf7358d   |
|              volume_type              |                iscsi                 |
+---------------------------------------+--------------------------------------+
[stack@osp3 local(keystone_admin)]$ cinder list
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+
|                  ID                  |   Status  | Migration Status |   Name  | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+
| bbcb3926-474c-44fa-b371-c3e0c6c42e2d | available |        -         | my-test |  1   |    iscsi    |  false   |    False    |             |
+--------------------------------------+-----------+------------------+---------+------+-------------+----------+-------------+-------------+

Comment 6 Chandan Kumar 2016-05-19 16:11:12 UTC
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.

Comment 7 Vincent S. Cojot 2016-12-14 13:20:30 UTC
Re-Opening as I am seeing this again with RDO Newton-3.
Again, moving numpy out of the way solved the issue.[stack@osp1 ~

(keystone_admin)]$ cinder service-list
ERROR: Unable to establish connection to http://10.0.128.164:8776/v2/39e70c8bc5364ddb84818935e1438f28/os-services

[root@osp1 cinder(keystone_admin)]# mv /usr/lib64/python2.7/site-packages/numpy /root

[stack@osp1 ~(keystone_admin)]$ cinder service-list
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host     | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | osp1     | nova | enabled | up    | 2016-12-14T13:17:37.000000 | -               |
| cinder-scheduler | osp1     | nova | enabled | up    | 2016-12-14T13:17:16.000000 | -               |
| cinder-volume    | osp1@lvm | nova | enabled | up    | 2016-12-14T13:17:38.000000 | -               |
+------------------+----------+------+---------+-------+----------------------------+-----------------+

This is on an all-in-one packstack install from 2016/12/12. (Dev 12th 2016)

This is on a fully patched RHEL7.3 x86_64 and numpy is from RedHat:
[stack@osp1 ~(keystone_admin)]$ rpm -qi numpy
Name        : numpy
Epoch       : 1
Version     : 1.7.1
Release     : 11.el7
Architecture: x86_64
Install Date: Tue 13 Dec 2016 11:27:29 AM EST
Group       : Development/Languages
Size        : 13350837
License     : BSD and Python
Signature   : RSA/SHA256, Wed 23 Sep 2015 02:47:49 AM EDT, Key ID 199e2f91fd431d51
Source RPM  : numpy-1.7.1-11.el7.src.rpm
Build Date  : Fri 08 Aug 2014 04:15:11 AM EDT
Build Host  : x86-021.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.numpy.org/
Summary     : A fast multidimensional array facility for Python
Description :
NumPy is a general-purpose array-processing package designed to
efficiently manipulate large multi-dimensional arrays of arbitrary
records without sacrificing too much speed for small multi-dimensional
arrays.  NumPy is built on the Numeric code base and adds features
introduced by numarray as well as an extended C-API and the ability to
create arrays of arbitrary type.

There are also basic facilities for discrete fourier transform,
basic linear algebra and random number generation. Also included in
this package is a version of f2py that works properly with NumPy.

Comment 8 Eric Harney 2016-12-14 13:57:50 UTC
Can you provide a debug-level cinder api log when this occurs?

Thanks!

Comment 9 Vincent S. Cojot 2016-12-14 15:24:19 UTC
As soon as I move numpy back to its original location, the following happens:
cinder services die and keep being restarted.
In /var/log/cinder/api.log, I get these lines over and over:

2016-12-14 10:22:51.893 4214 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:22:51.894 4214 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:51.917 4214 WARNING oslo_config.cfg [-] Option "enable_v1_api" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:51.917 4214 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
2016-12-14 10:22:54.185 4507 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:22:54.186 4507 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:54.205 4507 WARNING oslo_config.cfg [-] Option "enable_v1_api" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:22:54.206 4507 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
^C

Comment 10 Vincent S. Cojot 2016-12-14 15:40:25 UTC
With DEBUG enabled in cinder.conf, here's what I get:

2016-12-14 10:39:44.680 109929 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:39:44.681 109929 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:39:44.712 109929 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-12-14 10:39:44.712 109929 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-12-14 10:39:44.713 109929 DEBUG oslo.service.wsgi [-] Loading app osapi_volume from /etc/cinder/api-paste.ini load_app /usr/lib/python2.7/site-packages/oslo_service/wsgi.py:352
2016-12-14 10:39:44.715 109929 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
2016-12-14 10:39:47.238 110204 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 10:39:47.238 110204 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 10:39:47.257 110204 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-12-14 10:39:47.257 110204 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-12-14 10:39:47.257 110204 DEBUG oslo.service.wsgi [-] Loading app osapi_volume from /etc/cinder/api-paste.ini load_app /usr/lib/python2.7/site-packages/oslo_service/wsgi.py:352
2016-12-14 10:39:47.259 110204 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.
^C

Comment 11 Vincent S. Cojot 2016-12-14 21:08:14 UTC
One more thing that I noticed when I ran packstack again with GNOCCHI enabled:

10.0.128.164_controller.pp:                       [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.0.128.164_controller.pp
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: ImportError: C extension: No module named numpy not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.

Is it possible that it is gnocchi that pulls down numpy, which made cinder segfault?

Comment 12 Vincent S. Cojot 2016-12-14 21:09:47 UTC
[root@osp1 ~]# rpm -e numpy
error: Failed dependencies:
        numpy is needed by (installed) python2-scipy-0.17.0-2.el7.x86_64
        numpy >= 1.6 is needed by (installed) python-numexpr-2.3-4.el7.x86_64
        numpy is needed by (installed) python-tables-3.1.1-2.el7.x86_64
        numpy is needed by (installed) python-matplotlib-1.2.0-15.el7.x86_64
        numpy is needed by (installed) python-pandas-0.17.0-2.el7.x86_64
        numpy is needed by (installed) python-gnocchi-3.0.0-1.el7.noarch
        numpy(x86-64) is needed by (installed) python-Bottleneck-0.6.0-4.el7.x86_64

numpy seems required by python-gnocchi..

Comment 13 Vincent S. Cojot 2016-12-14 23:46:44 UTC
Problem seems to be in the adddocs part of numpy (at least the interaction with my test system). Removing the following two files makes the problem go away:

/usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc
/usr/lib64/python2.7/site-packages/numpy/add_newdocs.py

output below:

[root@osp1 numpy(keystone_admin)]# cinder service-list
ERROR: Unable to establish connection to http://10.0.128.164:8776/v2/39e70c8bc5364ddb84818935e1438f28/os-services: HTTPConnectionPool(host='10.0.128.164', port=8776): Max retries exceeded with url: /v2/39e70c8bc5364ddb84818935e1438f28/os-services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3463490>: Failed to establish a new connection: [Errno 111] Connection refused',))

[root@osp1 numpy(keystone_admin)]# rm /usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc
rm: remove regular file ‘/usr/lib64/python2.7/site-packages/numpy/add_newdocs.pyc’? y

[root@osp1 numpy(keystone_admin)]# rm /usr/lib64/python2.7/site-packages/numpy/add_newdocs.py
rm: remove regular file ‘/usr/lib64/python2.7/site-packages/numpy/add_newdocs.py’? y

[root@osp1 numpy(keystone_admin)]# cinder service-list
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host     | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+----------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | osp1     | nova | enabled | up    | 2016-12-14T23:45:03.000000 | -               |
| cinder-scheduler | osp1     | nova | enabled | up    | 2016-12-14T23:45:05.000000 | -               |
| cinder-volume    | osp1@lvm | nova | enabled | down  | 2016-12-14T23:40:08.000000 | -               |
+------------------+----------+------+---------+-------+----------------------------+-----------------+

Comment 14 Vincent S. Cojot 2016-12-15 03:41:08 UTC
Disregard the previous comment, -anything- that breaks numpy will work around the issue.
My knowledge of python imports is limited but I've figured that commenting out a single line in __init__.py from numpy makes the segfault go away:

# diff -ru /usr/lib64/python2.7/site-packages/numpy.orig /usr/lib64/python2.7/site-packages/numpy
diff -ru /usr/lib64/python2.7/site-packages/numpy.orig/__init__.py /usr/lib64/python2.7/site-packages/numpy/__init__.py
--- /usr/lib64/python2.7/site-packages/numpy.orig/__init__.py   2013-04-07 01:04:05.000000000 -0400
+++ /usr/lib64/python2.7/site-packages/numpy/__init__.py        2016-12-14 22:37:08.170209848 -0500
@@ -147,7 +147,7 @@
     from core import *
     import compat
     import lib
-    from lib import *
+    #from lib import *
     import linalg
     import fft
     import polynomial

This probably invalidates the whole numpy module but I have no idea on how to debug this further at this point..

Gdb?

Regards,
Vincent

Comment 15 Vincent S. Cojot 2016-12-15 04:10:22 UTC
I ran cinder-api under gdb and got this:

[root@osp1 ~]# gdb /usr/bin/python2
[...]
(gdb) set args /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
(gdb) r
Starting program: /usr/bin/python2 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Detaching after fork from child process 35441.
Detaching after fork from child process 35443.
Detaching after fork from child process 35445.
Detaching after fork from child process 35455.
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2016-12-14 23:09:22.376 35433 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-14 23:09:22.377 35433 WARNING oslo_config.cfg [-] Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-12-14 23:09:22.422 35433 WARNING cinder.api [-] The v1 api is deprecated and is not under active development. You should set enable_v1_api=false and enable_v3_api=true in your cinder.conf file.

Program received signal SIGSEGV, Segmentation fault.
0x00007fffd8788760 in PyArray_API () from /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
Missing separate debuginfos, use: debuginfo-install PyQt4-4.10.1-13.el7.x86_64 PyYAML-3.10-11.el7.x86_64 atlas-3.10.1-10.el7.x86_64 attica-0.4.2-1.el7.x86_64 blas-3.4.2-5.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 dbus-libs-1.6.12-17.el7.x86_64 dbusmenu-qt-0.9.2-7.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fontconfig-2.10.95-10.el7.x86_64 freetype-2.4.11-12.el7.x86_64 gamin-0.1.10-16.el7.x86_64 glib2-2.46.2-4.el7.x86_64 gmp-6.0.0-12.el7_1.x86_64 gobject-introspection-1.42.0-1.el7.x86_64 kdelibs-4.14.8-5.el7_2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 lapack-3.4.2-5.el7.x86_64 libICE-1.0.9-2.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.14-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.1-2.1.el7.x86_64 libXft-2.3.2-2.el7.x86_64 libXi-1.7.4-2.el7.x86_64 libXinerama-1.1.3-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libXrandr-1.4.2-2.el7.x86_64 libXrender-0.9.8-2.1.el7.x86_64 libXtst-1.2.2-2.1.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgfortran-4.8.5-11.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libquadmath-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64 libutempter-1.1.6-4.el7.x86_64 libuuid-2.23.2-33.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 libyaml-0.1.4-11.el7_0.x86_64 m2crypto-0.21.1-17.el7.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.3-2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.21.3-1.1.el7_3.x86_64 numpy-1.7.1-11.el7.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 postgresql-libs-9.2.18-1.el7.x86_64 pygobject3-base-3.14.0-3.el7.x86_64 pykde4-4.10.5-4.el7.x86_64 python-lxml-3.2.1-4.el7.x86_64 python-markupsafe-0.11-10.el7.x86_64 python-msgpack-0.4.6-3.el7.x86_64 python-netifaces-0.10.4-3.el7.x86_64 python-psutil-1.2.1-1.el7.x86_64 python-psycopg2-2.5.1-3.el7.x86_64 python-simplejson-3.5.3-5.el7.x86_64 python-sqlalchemy-1.0.11-1.el7.x86_64 python-wrapt-1.10.8-2.el7.x86_64 python2-cffi-1.5.2-1.el7.x86_64 python2-crypto-2.6.1-9.el7.x86_64 python2-cryptography-1.3.1-3.el7.x86_64 python2-greenlet-0.4.9-1.el7.x86_64 python2-scipy-0.17.0-2.el7.x86_64 qca2-2.0.3-7.el7.x86_64 qt-4.8.5-13.el7.x86_64 qt-x11-4.8.5-13.el7.x86_64 sip-4.14.6-4.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64

Comment 16 Vincent S. Cojot 2016-12-15 04:11:47 UTC
The begining of the backtrace shows this:

(gdb) bt
#0  0x00007fffd8788760 in PyArray_API () at /usr/lib64/python2.7/site-packages/numpy/core/multiarray.so
#1  0x00007fffd5574c0f in init_fblas () at /usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so
#2  0x00007ffff7b0bb69 in _PyImport_LoadDynamicModule (name=name@entry=0x5dfc390 "scipy.linalg._fblas", pathname=pathname@entry=0x5dfe280 "/usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so", fp=<optimized out>)
    at /usr/src/debug/Python-2.7.5/Python/importdl.c:53
#3  0x00007ffff7b099a3 in load_module (name=name@entry=0x5dfc390 "scipy.linalg._fblas", fp=<optimized out>, pathname=pathname@entry=0x5dfe280 "/usr/lib64/python2.7/site-packages/scipy/linalg/_fblas.so", type=<optimized out>, loader=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/import.c:1915
#4  0x00007ffff7b09c41 in import_submodule (mod=mod@entry=<module at remote 0x5d34478>, subname=subname@entry=0x5d33924 "_fblas", fullname=fullname@entry=0x5dfc390 "scipy.linalg._fblas") at /usr/src/debug/Python-2.7.5/Python/import.c:2700
#5  0x00007ffff7b0a13f in ensure_fromlist (mod=mod@entry=<module at remote 0x5d34478>, fromlist=fromlist@entry=('_fblas',), buf=buf@entry=0x5dfc390 "scipy.linalg._fblas", buflen=12, recursive=recursive@entry=0)
    at /usr/src/debug/Python-2.7.5/Python/import.c:2606
#6  0x00007ffff7b0a97a in PyImport_ImportModuleLevel (locals=<optimized out>, level=<optimized out>, fromlist=('_fblas',), globals=<optimized out>, name=0x0) at /usr/src/debug/Python-2.7.5/Python/import.c:2269
#7  0x00007ffff7b0a97a in PyImport_ImportModuleLevel (name=<optimized out>, globals=<optimized out>, locals=<optimized out>, fromlist=('_fblas',), level=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/import.c:2288
#8  0x00007ffff7aedb1f in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/bltinmodule.c:49

Let me know if you'd like more information.
Regards,

Comment 17 Vincent S. Cojot 2016-12-15 14:55:01 UTC
I am still investigating.
With an unmodified numpy, the problem disappeared when I removed the following rpms (they have some payload under /usr/{lib,lib64}/python2.7/site-packages)

cmpi-bindings-pywbem-0.9.5-6.el7.x86_64
dbus-python-1.1.1-9.el7.i686
farstream-python-0.1.2-8.el7.x86_64
file /usr/lib64/python2.7/site-packages/log_picker is not owned by any package
file /usr/lib64/python2.7/site-packages/numpy.orig is not owned by any package
file /usr/lib64/python2.7/site-packages/pyanaconda is not owned by any package
firstboot-19.12-1.el7.x86_64
fontforge-20120731b-12.el7.i686
fontforge-20120731b-12.el7.x86_64
gnome-abrt-0.3.4-8.el7.x86_64
gnome-tweak-tool-3.14.3-2.el7.noarch
gstreamer-python-0.10.22-6.el7.x86_64
gupnp-igd-python-0.2.2-3.el7.x86_64
hplip-libs-3.15.9-2.el7.i686
hplip-libs-3.15.9-2.el7.x86_64
iotop-0.6-2.el7.noarch
langtable-python-0.0.31-3.el7.noarch
liblouis-python-2.5.2-10.el7.noarch
libstoragemgmt-python-1.3.4-1.el7.noarch
libuser-python-0.60-7.el7_1.x86_64
lvm2-python-libs-2.02.166-1.el7_3.1.x86_64
mpich-3.0-3.0.4-10.el7.i686
mpich-3.0-3.0.4-10.el7.x86_64
mpich-3.2-3.2-2.el7.i686
mpich-3.2-3.2-2.el7.x86_64
mvapich2-2.0-2.0a-6.el7.i686
mvapich2-2.0-2.0a-6.el7.x86_64
mvapich2-2.0-psm-2.0a-6.el7.x86_64
mvapich2-2.2-2.2-0.3.rc1.el7.i686
mvapich2-2.2-2.2-0.3.rc1.el7.x86_64
mvapich2-2.2-psm-2.2-0.3.rc1.el7.x86_64
openmpi-1.10.3-3.el7.i686
openmpi-1.10.3-3.el7.x86_64
orca-3.6.3-4.el7.x86_64
PackageKit-1.0.7-6.el7.x86_64
pykde4-4.10.5-4.el7.i686
pykde4-4.10.5-4.el7.x86_64
pyldb-1.1.26-1.el7.i686
pyldb-1.1.26-1.el7.x86_64
pyorbit-2.24.0-15.el7.x86_64
PyQt4-4.10.1-13.el7.i686
PyQt4-4.10.1-13.el7.x86_64
pytalloc-2.1.6-1.el7.i686
pywbem-0.7.0-25.20130827svn625.el7.noarch
setroubleshoot-3.2.27.2-3.el7.x86_64
setroubleshoot-server-3.2.27.2-3.el7.x86_64
sip-4.14.6-4.el7.i686
sip-4.14.6-4.el7.x86_64
speech-dispatcher-python-0.7.1-15.el7.x86_64
system-config-date-1.10.6-2.el7.noarch
system-config-kdump-2.0.13-15.el7.noarch
system-config-keyboard-base-1.4.0-4.el7.noarch
system-config-printer-libs-1.4.1-19.el7.noarch
systemd-python-219-30.el7_3.6.x86_64
wireshark-1.10.14-10.el7.i686
wireshark-1.10.14-10.el7.x86_64
yum-langpacks-0.4.2-7.el7.noarch

I am currently trying to isolate this further..

Comment 18 Vincent S. Cojot 2016-12-15 16:04:09 UTC
This doesn't make any sense...

if I remove the following two rpms, then the segfault goes away.

pykde4-4.10.5-4.el7.i686
pykde4-4.10.5-4.el7.x86_64

I'll be uploading a small video shortly to demo this issue.

Comment 19 Vincent S. Cojot 2016-12-15 16:25 UTC
Created attachment 1232272 [details]
Video showing how the removal of pykde4 resolves the issue of segfault in numpy (multiarray.so)

Comment 20 Vincent S. Cojot 2016-12-19 22:08:59 UTC
I'am attempting to re-route this issue to 'numpy' in RHEL.

Comment 22 Nikola Forró 2016-12-22 15:27:52 UTC
The issue can be reproduced simply with the following python code:

import dl
import sys
sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)
import scipy.linalg

It is fixed by this upstream commit:
https://github.com/numpy/numpy/commit/a226daa75816577ac9c734133e600e5eebd9aaa9

Comment 23 Nikola Forró 2016-12-22 15:30 UTC
Created attachment 1234818 [details]
Backported upstream patch

Comment 31 Nikola Forró 2018-02-21 14:00:35 UTC
*** Bug 1547531 has been marked as a duplicate of this bug. ***

Comment 32 Jaroslav Klech 2018-03-22 13:50:02 UTC
Hello Nikola,

could you please have a look at the description of this bug (Doc Text field) above and tell me whether it is correct content-wise?

I am not sure what kind of crash it is neither am I sure what f2py refers to (a library, a package, a command, ...?)

Could you also tell me how this bug influenced the end user?

Thank you

Jaroslav

Comment 33 Nikola Forró 2018-03-22 14:55:13 UTC
Hello Jaroslav,

the doc text seems fine, only "PyArray API" should be "PyArray_API", it's literally name of a symbol.
Also there is no "kernel crash", just segmentation fault in numpy/core/multiarray.so.

f2py is "Fortran to Python interface generator", it builds Python modules from Fortran sources.

The PyArray_API symbol is present in multiarray module, which is part of numpy, but before the fix it was also present in all modules generated by f2py. And every generated module also imports multiarray module from numpy. A lot of generated modules are part of scipy, and when RTLD_GLOBAL flag was set (that's not usually a case), importing such modules lead to a conflict and a crash.

Comment 34 Jaroslav Klech 2018-03-22 17:06:16 UTC
(In reply to Nikola Forró from comment #33)
> Hello Jaroslav,
> 
> the doc text seems fine, only "PyArray API" should be "PyArray_API", it's
> literally name of a symbol.
> Also there is no "kernel crash", just segmentation fault in
> numpy/core/multiarray.so.
> 
> f2py is "Fortran to Python interface generator", it builds Python modules
> from Fortran sources.
> 
> The PyArray_API symbol is present in multiarray module, which is part of
> numpy, but before the fix it was also present in all modules generated by
> f2py. And every generated module also imports multiarray module from numpy.
> A lot of generated modules are part of scipy, and when RTLD_GLOBAL flag was
> set (that's not usually a case), importing such modules lead to a conflict
> and a crash.

Thank you Nikola,

I've updated the doc text above.

Jaroslav

Comment 35 Mirosław Zalewski 2018-04-03 13:39:00 UTC
*** Bug 1533516 has been marked as a duplicate of this bug. ***

Comment 37 errata-xmlrpc 2018-04-10 08:34:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0659


Note You need to log in before you can comment on or make changes to this bug.