Created attachment 1122426 [details] Skyring log Description of problem: Skyring crashed when I was creating few storage pools with very long traceback (see log in attachment). Version-Release number of selected component (if applicable): rhscon-ceph-0.0.6-2.el7.x86_64 rhscon-core-0.0.8-2.el7.x86_64 rhscon-ui-0.0.11-1.el7.noarch How reproducible: occasionally Steps to Reproduce: This is probably not 100% sure reproducer as it happened only twice what I saw. 1. I had 10 nodes (3 nodes as Monitors). 2. I created ceph cluster via USM web UI. 3. Then I tried to create few storage pools via API - sending POST request to: http://usm-server....rh.com:8080/api/v1/clusters/d0ebc8c7-feab-47af-a702-21b96cb41ceb/storages with following data {"replicas": 3, "options": {"pgnum": "128"}, "size": "2GB", "name": "TestStorageXX", "type": "replicated"} Actual results: Skyring crash. Expected results: Skyring should not crash. Additional info: I noticed following line in the log file few lines before the traceback: Too many open files (signaler.cpp:234) I'm not sure if it is related or not and what might cause it. If I'll find some more detailed information or clear reproducer, I'll comment it later.
Created attachment 1129184 [details] Another skyring.log with crash. Happened again twice on Friday and once today. I still have no clear reproducible scenario... basically run USM cluster for a while and perform some actions...
I noticed a similar crash as well. ~~~ # systemctl status skyringd ● skyringd.service - skyring server Loaded: loaded (/usr/lib/systemd/system/skyringd.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-02-23 21:09:36 CET; 12h ago Process: 31684 ExecStop=/etc/init.d/skyringd stop (code=exited, status=0/SUCCESS) Main PID: 4922 (code=exited, status=2) Feb 22 20:29:01 mbukatov-usm2-server.example.com systemd[1]: Starting skyring server... Feb 22 20:29:01 mbukatov-usm2-server.example.com skyringd[4916]: Starting skyring: Feb 22 20:29:01 mbukatov-usm2-server.example.com systemd[1]: Started skyring server. Feb 22 20:29:20 mbukatov-usm2-server.example.com systemd[1]: Started skyring server. Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: skyringd.service: main process exited, code=exited, status=2/INVALIDARGUMENT Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: Unit skyringd.service entered failed state. Feb 23 21:09:36 mbukatov-usm2-server.example.com systemd[1]: skyringd.service failed. ~~~ Package versions: ~~~ # rpm -qa 'rhscon*' rhscon-core-0.0.8-4.el7.x86_64 rhscon-ceph-0.0.6-4.el7.x86_64 rhscon-ui-0.0.14-1.el7.noarch ~~~ Looking into /var/log/skyring/skyring.log and filtering interesting pieces: ~~~ [negroni] Started GET /api/v1/unmanaged_nodes [negroni] Completed 200 OK in 86.174234ms [negroni] Started GET /api/v1/unmanaged_nodes Too many open files (epoll.cpp:41) 1 ~~~ ~~~ SIGABRT: abort PC=0x7f9cfee685f7 signal arrived during cgo execution goroutine 5791 [syscall, locked to thread]: runtime.cgocall_errno(0x40f240, 0xc208200b78, 0x7f9c00000000) /usr/lib/golang/src/runtime/cgocall.go:130 +0xf5 fp=0xc208200b58 sp=0xc208200b30 github.com/sbinet/go-python._Cfunc_PyObject_CallObject(0x2210668, 0x7f9cfb220050, 0x0) /builddir/.skyring_build/golang/gopath/src/github.com/sbinet/go-python/:1172 +0x44 fp=0xc208200b78 sp=0xc208200b58 github.com/sbinet/go-python.(*PyObject).CallObject(0xc20802c1b0, 0xc209f6e3c0, 0x8) /builddir/.skyring_build/golang/gopath/src/github.com/sbinet/go-python/object.go:298 +0x38 fp=0xc208200ba8 sp=0xc208200b78 github.com/skyrings/skyring-common/tools/gopy.(*PyFunction).Call(0xc20802c1b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring-common/tools/gopy/gopy.go:23 +0x184 fp=0xc208200cd0 sp=0xc208200ba8 github.com/skyrings/skyring/backend/salt.Salt.GetNodes(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/backend/salt/salt.go:174 +0x110 fp=0xc208200d48 sp=0xc208200cd0 github.com/skyrings/skyring/backend/salt.(*Salt).GetNodes(0xf69028, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) <autogenerated>:9 +0xdf fp=0xc208200df0 sp=0xc208200d48 github.com/skyrings/skyring/nodemanager/saltnodemanager.SaltNodeManager.GetUnmanagedNodes(0xc209e20c00, 0x0, 0x0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/nodemanager/saltnodemanager/salt_node_manager.go:147 +0x58 fp=0xc208200f78 sp=0xc208200df0 github.com/skyrings/skyring/nodemanager/saltnodemanager.(*SaltNodeManager).GetUnmanagedNodes(0xf69028, 0x41d267, 0x0, 0x0) <autogenerated>:3 +0xaa fp=0xc208200fb0 sp=0xc208200f78 github.com/skyrings/skyring/apps/skyring.(*App).GET_UnmanagedNodes(0xc20808e120, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/nodes.go:332 +0x57 fp=0xc2082010c8 sp=0xc208200fb0 github.com/skyrings/skyring/apps/skyring.*App.GET_UnmanagedNodes·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/routes.go:117 +0x45 fp=0xc2082010f0 sp=0xc2082010c8 net/http.HandlerFunc.ServeHTTP(0xc20808e1e0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /usr/lib/golang/src/net/http/server.go:1265 +0x41 fp=0xc208201110 sp=0xc2082010f0 github.com/gorilla/mux.(*Router).ServeHTTP(0xc20800d2c0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/gorilla/mux/mux.go:100 +0x297 fp=0xc208201218 sp=0xc208201110 github.com/codegangsta/negroni.func·001(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ce0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:41 +0x57 fp=0xc208201250 sp=0xc208201218 github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc20808e4b0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ce0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc208201278 sp=0xc208201250 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc20808e4b0, 0xc20801fca0, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc2082012c0 sp=0xc208201278 github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc2082012f8 sp=0xc2082012c0 github.com/skyrings/skyring/apps/skyring.(*App).LoginRequired(0xc20808e120, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:383 +0x20a fp=0xc208201378 sp=0xc2082012f8 github.com/skyrings/skyring/apps/skyring.*App.LoginRequired·fm(0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:168 +0x4f fp=0xc2082013a8 sp=0xc208201378 github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc20808e490, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790, 0xc209de8ca0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082013d0 sp=0xc2082013a8 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc20808e490, 0xc20801fc80, 0x7f9cffe31118, 0xc209bbabc0, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201418 sp=0xc2082013d0 github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0xc2080b94a0, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:73 +0x13d fp=0xc208201488 sp=0xc208201418 github.com/gorilla/mux.(*Router).ServeHTTP(0xc20800d270, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/gorilla/mux/mux.go:100 +0x297 fp=0xc208201590 sp=0xc208201488 github.com/codegangsta/negroni.func·001(0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8c60) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:41 +0x57 fp=0xc2082015c8 sp=0xc208201590 github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc2080eec80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8c60) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082015f0 sp=0xc2082015c8 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc2080eec80, 0xc2080d7f80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201638 sp=0xc2082015f0 github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201670 sp=0xc208201638 github.com/skyrings/skyring/apps/skyring.(*App).LoggingContext(0xc20808e120, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/apps/skyring/skyring.go:411 +0x688 fp=0xc208201788 sp=0xc208201670 github.com/skyrings/skyring/apps.Application.LoggingContext·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:178 +0x61 fp=0xc2082017c8 sp=0xc208201788 github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc2080d7c40, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de83c0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:24 +0x4b fp=0xc2082017f0 sp=0xc2082017c8 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2ac78, 0xc2080d7c40, 0xc2080d7f60, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201838 sp=0xc2082017f0 github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201870 sp=0xc208201838 github.com/codegangsta/negroni.(*Static).ServeHTTP(0xc2081b7710, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de8300) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/static.go:49 +0x2f4 fp=0xc2082019b8 sp=0xc208201870 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f110, 0xc2081b7710, 0xc2080d7f40, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201a00 sp=0xc2082019b8 github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201a38 sp=0xc208201a00 github.com/codegangsta/negroni.(*Logger).ServeHTTP(0xc20802c768, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de82c0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/logger.go:25 +0x1ce fp=0xc208201b40 sp=0xc208201a38 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f0e8, 0xc20802c768, 0xc2080d7f20, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201b88 sp=0xc208201b40 github.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0x57 fp=0xc208201bc0 sp=0xc208201b88 github.com/codegangsta/negroni.(*Recovery).ServeHTTP(0xc2080d7a80, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790, 0xc209de82a0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/recovery.go:45 +0x8c fp=0xc208201bf8 sp=0xc208201bc0 github.com/codegangsta/negroni.middleware.ServeHTTP(0x7f9cffe2f0c0, 0xc2080d7a80, 0xc2080d7d00, 0x7f9cffe31118, 0xc209bba280, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:33 +0xb8 fp=0xc208201c40 sp=0xc208201bf8 github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0xc2081b7770, 0x7f9cffe31080, 0xc209e50000, 0xc209c21790) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/negroni/negroni.go:73 +0x13d fp=0xc208201cb0 sp=0xc208201c40 net/http.serverHandler.ServeHTTP(0xc2081d7aa0, 0x7f9cffe31080, 0xc209e50000, 0xc209c21790) /usr/lib/golang/src/net/http/server.go:1703 +0x19a fp=0xc208201d08 sp=0xc208201cb0 net/http.(*conn).serve(0xc2081ce640) /usr/lib/golang/src/net/http/server.go:1204 +0xb57 fp=0xc208201fd8 sp=0xc208201d08 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208201fe0 sp=0xc208201fd8 created by net/http.(*Server).Serve /usr/lib/golang/src/net/http/server.go:1751 +0x35e ~~~ Tons of other golang stack traces like this follows: ~~~ goroutine 1 [chan receive, 1478 minutes]: main.start() /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:214 +0x15e3 main.func·002(0xc2080bc900) /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:127 +0x1c6 github.com/codegangsta/cli.(*App).Run(0xc2080bc7e0, 0xc20800a000, 0x1, 0x1, 0x0, 0x0) /builddir/.skyring_build/golang/gopath/src/github.com/codegangsta/cli/app.go:180 +0x1207 main.main() /builddir/.skyring_build/golang/gopath/src/github.com/skyrings/skyring/main.go:130 +0x7fd ~~~
Additional info for comment 2: In case the number of open files is actually related, see: ~~~ # ulimit -n 1024 ~~~
*** Bug 1310065 has been marked as a duplicate of this bug. ***
We didn't experienced this issue for the last weeks neither during manual or automated testing, so closing this Bug as VERIFIED. Last time tested on: rhscon-ceph-0.0.33-1.el7scon.x86_64 rhscon-core-selinux-0.0.34-1.el7scon.noarch rhscon-core-0.0.34-1.el7scon.x86_64 rhscon-ui-0.0.48-1.el7scon.noarch >> VERIFIED