Bug 1305891

Summary: skyring crashed with long traceback
Product: [Red Hat Storage] Red Hat Storage Console Reporter: Daniel Horák <dahorak>
Component: coreAssignee: Nishanth Thomas <nthomas>
core sub component: provisioning QA Contact: Daniel Horák <dahorak>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: unspecified CC: mbukatov, mkudlej, sankarshan
Version: 2Keywords: TestBlocker
Target Milestone: ---   
Target Release: 2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhscon-ceph-0.0.23-1.el7scon.x86_64, rhscon-core-0.0.24-1.el7scon.x86_64, rhscon-ui-0.0.39-1.el7scon.noarch Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-19 05:33:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Another skyring.log with crash. none

Description Daniel Horák 2016-02-09 14:09:28 UTC 
Created attachment 1122426 [details]
Skyring log

Description of problem:
  Skyring crashed when I was creating few storage pools with very long traceback (see log in attachment).

Version-Release number of selected component (if applicable):
  rhscon-ceph-0.0.6-2.el7.x86_64
  rhscon-core-0.0.8-2.el7.x86_64
  rhscon-ui-0.0.11-1.el7.noarch


How reproducible:
  occasionally 

Steps to Reproduce:
  This is probably not 100% sure reproducer as it happened only twice what I saw.
1. I had 10 nodes (3 nodes as Monitors).
2. I created ceph cluster via USM web UI.
3. Then I tried to create few storage pools via API - sending POST request to:
    http://usm-server....rh.com:8080/api/v1/clusters/d0ebc8c7-feab-47af-a702-21b96cb41ceb/storages
with following data {"replicas": 3, "options": {"pgnum": "128"}, "size": "2GB", "name": "TestStorageXX", "type": "replicated"}

Actual results:
  Skyring crash.

Expected results:
  Skyring should not crash.

Additional info:
  I noticed following line in the log file few lines before the traceback:
    Too many open files (signaler.cpp:234)
  I'm not sure if it is related or not and what might cause it.

  If I'll find some more detailed information or clear reproducer, I'll comment it later.
Comment 1 Daniel Horák 2016-02-22 08:49:02 UTC 
Created attachment 1129184 [details]
Another skyring.log with crash.

Happened again twice on Friday and once today. I still have no clear reproducible scenario... basically run USM cluster for a while and perform some actions...
Comment 2 Martin Bukatovic 2016-02-24 09:09:38 UTC 
I noticed a similar crash as well.

~~~
# systemctl status skyringd
● skyringd.service - skyring server
   Loaded: loaded (/usr/lib/systemd/system/skyringd.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2016-02-23 21:09:36 CET; 12h ago
  Process: 31684 ExecStop=/etc/init.d/skyringd stop (code=exited, status=0/SUCCESS)
 Main PID: 4922 (code=exited, status=2)

Feb 22 20:29:01 mbukatov-usm2-server.example.com systemd[1]: Starting skyring server...
Feb 22 20:29:01 mbukatov-usm2-server.example.com skyringd[4916]: Starting skyring:
Feb 22 20:29:01 mbukatov-usm2-server.example.com systemd[1]: Started skyring server.
Feb 22 20:29:20 mbukatov-usm2-server.example.com systemd[1]: Started skyring server.
Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: skyringd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: Unit skyringd.service entered failed state.
Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: skyringd.service failed.
~~~

Package versions:

~~~
# rpm -qa 'rhscon*'
rhscon-core-0.0.8-4.el7.x86_64
rhscon-ceph-0.0.6-4.el7.x86_64
rhscon-ui-0.0.14-1.el7.noarch
~~~

Looking into /var/log/skyring/skyring.log and filtering interesting pieces:

~~~
[negroni] Started GET /api/v1/unmanaged_nodes
[negroni] Completed 200 OK in 86.174234ms
[negroni] Started GET /api/v1/unmanaged_nodes
Too many open files (epoll.cpp:41)
1
~~~

~~~
SIGABRT: abort
PC=0x7f9cfee685f7
signal arrived during cgo execution

goroutine 5791 [syscall, locked to thread]:
runtime.cgocall_errno(0x40f240, 0xc208200b78, 0x7f9c00000000)
        /usr/lib/golang/src/runtime/cgocall.go:130 +0xf5 fp=0xc208200b58 sp=0xc208200b30
github.com/sbinet/go-python._Cfunc_PyObject_CallObject(0x2210668, 0x7f9cfb220050, 0x0)
        /builddir/.skyring_build/golang/gopath/src/github.com/sbinet/go-python/:1172 +0x44 fp=0xc208200b78 sp=0xc208200b58
github.com/sbinet/go-python.(*PyObject).CallObject(0xc20802c1b0, 0xc209f6e3c0, 0x8)
        /builddir/.skyring_build/golang/gopath/src/github.com/sbinet/go-python/object.go:298 +0x38 fp=0xc208200ba8 sp=0xc208200b78
github.com/skyrings/skyring-common/tools/gopy.(*PyFunction).Call(0xc20802c1b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring-common/tools/gopy/gopy.go:23 +0x184 fp=0xc208200cd0 sp=0xc208200ba8
github.com/skyrings/skyring/backend/salt.Salt.GetNodes(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/backend/salt/salt.go:174 +0x110 fp=0xc208200d48 sp=0xc208200cd0
github.com/skyrings/skyring/backend/salt.(*Salt).GetNodes(0xf69028, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        <autogenerated>:9 +0xdf fp=0xc208200df0 sp=0xc208200d48
github.com/skyrings/skyring/nodemanager/saltnodemanager.SaltNodeManager.GetUnmanagedNodes(0xc209e20c00, 0x0, 0x0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/nodemanager/saltnodemanager/salt_node_manager.go:147 +0x58 fp=0xc208200f78 sp=0xc208200df0
github.com/skyrings/skyring/nodemanager/saltnodemanager.(*SaltNodeManager).GetUnmanagedNodes(0xf69028, 0x41d267, 0x0, 0x0)
        <autogenerated>:3 +0xaa fp=0xc208200fb0 sp=0xc208200f78
github.com/skyrings/skyring/apps/skyring.(*App).GET_UnmanagedNodes(0xc20808e120, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/nodes.go:332 +0x57 fp=0xc2082010c8 sp=0xc208200fb0
github.com/skyrings/skyring/apps/skyring.*App.GET_UnmanagedNodes·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/routes.go:117 +0x45 fp=0xc2082010f0 sp=0xc2082010c8
net/http.HandlerFunc.ServeHTTP(0xc20808e1e0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /usr/lib/golang/src/net/http/server.go:1265 +0x41 fp=0xc208201110 sp=0xc2082010f0
github.com/gorilla/mux.(*Router).ServeHTTP(0xc20800d2c0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/gorilla/mux/mux.go:100 +0x297 fp=0xc208201218 sp=0xc208201110
github.com/codegangsta/negroni.func·001(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ce0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:41 +0x57 fp=0xc208201250 sp=0xc208201218
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc20808e4b0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ce0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc208201278 sp=0xc208201250
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc20808e4b0, 0xc20801fca0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc2082012c0 sp=0xc208201278
github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc2082012f8 sp=0xc2082012c0
github.com/skyrings/skyring/apps/skyring.(*App).LoginRequired(0xc20808e120, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:383 +0x20a fp=0xc208201378 sp=0xc2082012f8
github.com/skyrings/skyring/apps/skyring.*App.LoginRequired·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:168 +0x4f fp=0xc2082013a8 sp=0xc208201378
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc20808e490, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082013d0 sp=0xc2082013a8
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc20808e490, 0xc20801fc80, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201418 sp=0xc2082013d0
github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0xc2080b94a0, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:73 +0x13d fp=0xc208201488 sp=0xc208201418
github.com/gorilla/mux.(*Router).ServeHTTP(0xc20800d270, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/gorilla/mux/mux.go:100 +0x297 fp=0xc208201590 sp=0xc208201488
github.com/codegangsta/negroni.func·001(0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8c60)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:41 +0x57 fp=0xc2082015c8 sp=0xc208201590
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc2080eec80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8c60)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082015f0 sp=0xc2082015c8
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc2080eec80, 0xc2080d7f80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201638 sp=0xc2082015f0
github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201670 sp=0xc208201638
github.com/skyrings/skyring/apps/skyring.(*App).LoggingContext(0xc20808e120, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:411 +0x688 fp=0xc208201788 sp=0xc208201670
github.com/skyrings/skyring/apps.Application.LoggingContext·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:178 +0x61 fp=0xc2082017c8 sp=0xc208201788
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc2080d7c40, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082017f0 sp=0xc2082017c8
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc2080d7c40, 0xc2080d7f60, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201838 sp=0xc2082017f0
github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201870 sp=0xc208201838
github.com/codegangsta/negroni.(*Static).ServeHTTP(0xc2081b7710, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8300)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/static.go:49 +0x2f4 fp=0xc2082019b8 sp=0xc208201870
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f110, 0xc2081b7710, 0xc2080d7f40, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201a00 sp=0xc2082019b8
github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201a38 sp=0xc208201a00
github.com/codegangsta/negroni.(*Logger).ServeHTTP(0xc20802c768, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de82c0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/logger.go:25 +0x1ce fp=0xc208201b40 sp=0xc208201a38
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f0e8, 0xc20802c768, 0xc2080d7f20, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201b88 sp=0xc208201b40
github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201bc0 sp=0xc208201b88
github.com/codegangsta/negroni.(*Recovery).ServeHTTP(0xc2080d7a80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de82a0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/recovery.go:45 +0x8c fp=0xc208201bf8 sp=0xc208201bc0
github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f0c0, 0xc2080d7a80, 0xc2080d7d00, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201c40 sp=0xc208201bf8
github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0xc2081b7770, 0x7f9cffe31080, 0xc209e50000, 0xc209c21790)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:73 +0x13d fp=0xc208201cb0 sp=0xc208201c40
net/http.serverHandler.ServeHTTP(0xc2081d7aa0, 0x7f9cffe31080, 0xc209e50000, 0xc209c21790)
        /usr/lib/golang/src/net/http/server.go:1703 +0x19a fp=0xc208201d08 sp=0xc208201cb0
net/http.(*conn).serve(0xc2081ce640)
        /usr/lib/golang/src/net/http/server.go:1204 +0xb57 fp=0xc208201fd8 sp=0xc208201d08
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208201fe0 sp=0xc208201fd8
created by net/http.(*Server).Serve
        /usr/lib/golang/src/net/http/server.go:1751 +0x35e
~~~

Tons of other golang stack traces like this follows:

~~~
goroutine 1 [chan receive, 1478 minutes]:
main.start()
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:214 +0x15e3
main.func·002(0xc2080bc900)
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:127 +0x1c6
github.com/codegangsta/cli.(*App).Run(0xc2080bc7e0, 0xc20800a000, 0x1, 0x1, 0x0, 0x0)
        /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/cli/app.go:180 +0x1207
main.main()
        /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:130 +0x7fd
~~~
Comment 4 Martin Bukatovic 2016-02-24 09:29:10 UTC 
Additional info for comment 2:

In case the number of open files is actually related, see:

~~~
# ulimit -n
1024
~~~
Comment 6 Martin Kudlej 2016-03-07 09:39:13 UTC 
*** Bug 1310065 has been marked as a duplicate of this bug. ***
Comment 9 Daniel Horák 2016-07-20 08:27:06 UTC 
We didn't experienced this issue for the last weeks neither during  manual or automated testing, so closing this Bug as VERIFIED.

Last time tested on:
  rhscon-ceph-0.0.33-1.el7scon.x86_64
  rhscon-core-selinux-0.0.34-1.el7scon.noarch
  rhscon-core-0.0.34-1.el7scon.x86_64
  rhscon-ui-0.0.48-1.el7scon.noarch

>> VERIFIED