Bug 1198222

Summary: pcsd: GUI fails if orphaned resource is present in a cluster
Product: Red Hat Enterprise Linux 7 Reporter: Radek Steiger <rsteiger>
Component: pcsAssignee: Ondrej Mular <omular>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.1CC: cfeist, cluster-maint, tojeline
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcs-0.9.142-2.el7 Doc Type: Bug Fix
Doc Text:
Cause: Orphaned resource is present in cluster. Consequence: In web UI all nodes are displayed as offline and no resources are shown. Fix: Orphaned resources are handled properly. Result: Web UI works normally even though there is orphaned resource in cluster.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 09:35:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Radek Steiger 2015-03-03 15:22:36 UTC
> Description of problem:

If an orphaned resource exists in a cluster status (e.g. orphaned="true" in crm_mon -X output), GUI fails to work properly due to an attempt on parsing non-existent attributes.

Orphaned resources are not very common, but whenever they appear, pcs GUI should be able to handle them properly. We can deal with them in the command line:

# pcs resource
 dummy1	(ocf::heartbeat:Dummy):	 ORPHANED Started (unmanaged) 
# pcs resource show dummy1
Error: unable to find resource 'dummy1'

# pcs resource cleanup
All resources/stonith devices successfully cleaned up
# pcs resource
#

User can use the cleanup as a temporary 'fix'.


> Version-Release number of selected component (if applicable):
pcs-0.9.137-10.el7


>How reproducible:
Easily 


> Steps to Reproduce:

1. Create a resource
2. Make it orphaned by removing the corresponding primitive tag with 'pcs cluster edit'
3. Open the cluster in a GUI


> Actual results:

Nodes are reported to be stopped, no resources nor any other information is shown.


> Expected results:

GUI keeps on working properly.


> Additional info:

from /var/log/pcsd.log

I, [2015-03-03T15:55:52.088631 #604]  INFO -- : Running: /usr/sbin/pcs acl show
NoMethodError - undefined method `each_pair' for nil:NilClass:
  /usr/lib/pcsd/remote.rb:518:in `block in node_status'
  /usr/lib/pcsd/remote.rb:514:in `each'
  /usr/lib/pcsd/remote.rb:514:in `node_status'
  /usr/lib/pcsd/remote.rb:15:in `remote'
  /usr/lib/pcsd/pcsd.rb:185:in `block in <top (required)>'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in `context'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in `context'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/logger.rb:15:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/show_exceptions.rb:21:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
  /usr/lib/pcsd/vendor/bundle/ruby/gems/rack-1.5.2/lib/rack/handler/webrick.rb:60:in `service'
  /usr/share/ruby/webrick/httpserver.rb:138:in `service'
  /usr/share/ruby/webrick/httpserver.rb:94:in `run'
  /usr/share/ruby/webrick/server.rb:295:in `block in start_thread'
virt-041.cluster-qe.lab.eng.brq.redhat.com - - [03/Mar/2015:15:55:51 CET] "GET /remote/status HTTP/1.1" 500 166517
- -> /remote/status

Comment 2 Tomas Jelinek 2015-06-24 13:59:21 UTC
Patch in upstream:
https://github.com/feist/pcs/commit/fac6f7a1d058ba699ee90f0b4423900fcfef0395

Comment 4 Tomas Jelinek 2015-07-13 12:44:21 UTC
Before Fix:
[root@rh71-node1 ~]# rpm -q pcs
pcs-0.9.141-1.el7.x86_64
[root@rh71-node1:~]# pcs resource create delay delay startdelay=0 stopdelay=10
[root@rh71-node1:~]# pcs resource delete delay --force
Deleting Resource - delay
[root@rh71-node1:~]# pcs resource | grep delay
 delay  (ocf::heartbeat:Delay):  ORPHANED Started rh71-node2

Open the cluster in a GUI. Nodes are reported to be stopped, no resources nor any other information is shown.



After Fix:
[root@rh71-node1:~]# rpm -q pcs
pcs-0.9.142-2.el7.x86_64
[root@rh71-node1:~]# pcs resource create delay delay startdelay=0 stopdelay=10
[root@rh71-node1:~]# pcs resource delete delay --force
Deleting Resource - delay
[root@rh71-node1:~]# pcs resource | grep delay
 delay  (ocf::heartbeat:Delay):  ORPHANED Started rh71-node2

Open the cluster in a GUI. GUI works properly.

Comment 7 errata-xmlrpc 2015-11-19 09:35:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-2290.html