Bug 1343661

Summary: Rebase clufter component [RHEL 7.3]
Product: Red Hat Enterprise Linux 7 Reporter: Jan Pokorný [poki] <jpokorny>
Component: clufterAssignee: Jan Pokorný [poki] <jpokorny>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: medium Docs Contact: Steven J. Levine <slevine>
Priority: high    
Version: 7.1CC: borgan, cfeist, cluster-qe, extras-qa, jpokorny, lkocman, lkuprova, mlisik, nobody, phagara, royoung, rsteiger, rwilliam, salmy, tlavigne, vpavlin
Target Milestone: rcKeywords: Rebase
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: clufter-0.59.5-2.el7 Doc Type: Technology Preview
Doc Text:
_clufter_ rebased to version 0.59.5 The _clufter_ packages, available as a Technology Preview, have been upgraded to upstream version 0.59.5, which provides a number of bug fixes, new features, and user experience enhancements over the previous version. Among the notable updates are the following: * When converting the old cluster stack configuration into files for a Pacemaker stack or into the respective sequence of `pcs` commands with the `ccs2pcs` and `ccs2pcscmd` families of `clufter` commands, monitor action is properly propagated or added. * When converting configuration files for the Pacemaker stack using the `corosync.conf` file, either as a byproduct of converting CMAN-based configuration or with first-class input such as the `*2pcscmd{,-needle}` families of commands, the cluster name is propagated correctly. Previously, the cluster name was mistakenly dropped, resulting in a command that confused the name of the first cluster node for the name of the cluster as in, for example, "pcs cluster setup --start --name node1 node2 node3". * When converting CMAN-based configuration into the parallel configuration for a Pacemaker stack with the `ccs2pcs` family of commands, accidentally broken values of attributes marked as having an ID type in the schema no longer occur. * When converting either CMAN or Pacemaker stack specific configuration into the respective sequence of `pcs` commands with the `*2pcscmd` families of commands, the `clufter` tool no longer suggests "pcs cluster cib file --config", which does not currently work for subsequent local-modification `pcs` commands. Instead it suggests "pcs cluster cib file". * The `clufter` tool outputs now may vary significantly depending on the specified distribution target since the tool now aligns the output with what the respective environment, such as the `pcs` version, can support. Because of this, your distribution or setup may not be supported, and you should not expect that one sequence of `pcs` commands that the `clufter` tool produces is portable to a completely different environment. * The `clufter` tool now supports several new features of the `pcs` tool, including quorum devices. Additionally, the `clufter` tool supports older features recently added to the `pcs` tool, including ticket constraints, and resource sets for colocation and order constraints.
Story Points: ---
Clone Of: 1269964
: 1367536 1387424 (view as bug list) Environment:
Last Closed: 2016-11-04 07:52:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1269964, 1270740, 1272570, 1272592, 1300014, 1300050, 1328078, 1353934, 1357945    
Bug Blocks: 1367536, 1387424    

Description Jan Pokorný [poki] 2016-06-07 15:37:12 UTC
+++ This bug was initially created as a clone of Bug #1269964 +++

Upstream development of clufter has progressed (since 7.2 package till 7.3
package + something more likely to come) so this asks for a rebase.

More specifically, clufter received plentiful bug fixes and enhancements
compared to previous RHEL minor release (7.2/clufter-0.56.2),
notably commits:

filters/coro2pcscmd[cluster,corosync]: make pcs --wait check silent
filters/configuration[cib2pcscmd]: avoid "pcs cluster cib <file> --config"

Both of which are relevant to [bug 1328078].

Comment 1 Jan Pokorný [poki] 2016-06-08 15:42:58 UTC
In fact, it should have been
"compared to previous RHEL minor release (7.2/clufter-0.50.4)"
hence following new features and bugfixes are forthcoming:

0.50.5:
http://oss.clusterlabs.org/pipermail/users/2015-December/001981.html

0.55.0:
http://oss.clusterlabs.org/pipermail/users/2015-December/001981.html

0.56.0:
http://oss.clusterlabs.org/pipermail/users/2016-February/002214.html

0.56.1:
http://oss.clusterlabs.org/pipermail/users/2016-February/002271.html

0.56.2:
http://oss.clusterlabs.org/pipermail/users/2016-March/002534.html

Comment 13 Steven J. Levine 2016-08-17 16:21:43 UTC
Poki:

For the release note description of this rebase, could you provide a summary of the updates and fixes for the rebase in the doc text field?  I'm not really sure what to call out.

Thanks,

Steven

Comment 14 Jan Pokorný [poki] 2016-09-08 17:33:36 UTC
Steven, sure, here's what I've compiled from the proposed sources
([comment 1] + [comment 9]), leaving some less important items out,
modifying some of them (denoted with initial 'E' instead of '.')
and marking some especially important with '!':


Bug fixes
=========

! ccs2pcs* conversion now properly propagates or adds monitor action
  [resolves: rhbz#1272570]

. ccs2pcs* conversion now (suitably) aborts when it reaches exclusive
  resource group
  [related: rhbz#1206640 (rhbz#1272191)]

. ccs2pcs* conversion now does not take undeployed (contained in
  <resources> and not referenced elsewhere) into account and is
  more careful about carrying about parent service group auxiliary
  reference for further processing
  [related/resolves: rhbz#1272592]

! pcs2pcscmd{,-needle} now propagates cluster name correctly,
  previously it was mistakenly dropped resulting in confusing
  first cluster node for that
  (e.g., pcs cluster setup --start --name node1 node2 node3)

. ccs2pcs* conversion now propagates two_node (and internally
  also expected_votes) properties of cman as defined in cluster.conf
  correctly

. help screens/manual pages for *2pcs* commands now documents
  "cib" arguments correctly

. pcs2pcscmd* conversion now correctly includes cmd-wrap filter
  resulting in the output line-wrapped per expectation

. "assisted recovery" now works on systems without /dev/tty as well
  as on systems for which open-modify[open+close] (final close yet
  to come) of particular file won't reliably discover mtime change;
  now strict open-close-modify is used instead (and mtime check in
  a was-file-changed test is preceded with a file size comparison for
  good measure), making intermittent failures in test runs disappear

. clufter is now capable of handling command options as unicode
  (relates to the usage as a library, original discovery thanks
  to pcs) and "the magic interpolation" of the command inputs now
  works at places where it was supposed to but unfortunately did not

! ccs2pcs* commands no longer generate accidentally broken values
  of attributes marked as having an ID type in the schema
  [resolves: rhbz#1300050]

. ccs2pcs* commands now translate notion of recovery/relocate recover
  policy of the resource group as supported by RGManager into the
  parallel expression in Pacemaker universe;  the same applies to
  __independent_subtree=2 at the resource level and empty restricted
  failover domain (that is referred to from existing resource group)

. ccs2pcs* commands now propagate stop timeout of the vm original
  resource agent

. *2pcscmd* commands now support group meta attributes properly

. *2pcscmd* commands no longer emit bogus properties of the operations
  (id, name, interval) as these are position-fixed values in the
  respective pcs syntax, hence not requiring explicit key=value
  treatment

E with {cib,pcs}2pcscmd* commands, clufter no longer chokes on
  validation failures (unless --nocheck provided) due to source CIB
  file using newer "validate-with" validation version specification
  than the only supported so far (pacemaker-1.2.rng) or possibly
  using a syntax not compatible with that; now also 2.0, 2.3, 2.4, and
  2.5 versions are supported
  [resolves: rhbz#1300014]

. with {cib,pcs}2pcscmd* commands, clufter no longer chokes on
  validation failures (unless --nocheck provided) due to source CIB
  not containing "status" section (which is normally the case with
  implicit input located in /var/lib/pacemaker/cib/cib.xml);
  now the bundled, compacted schemas mark this section optional
  and also the recipe to distill such format from pacemaker native
  schemas ensures the same assumption holds even if not pre-existed
  [resolves: rhbz#1269964, comment 9]
  [see also: https://github.com/ClusterLabs/pacemaker/pull/957]
. internal representations of command + options/arguments was fixed
  in several ways so as to provide correct outcomes in both general
  (previously, some options could be duplicated while overwriting
  other options/arguments, and standalone negative numbers were
  considered options) and pcs (--wait=X cannot be decoupled the same
  way option parsers can usually cope with, as pcs built-in parser
  treats this specifically) cases

! with *2pcscmd* commands, clufter no longer suggests
  "pcs cluster cib <file> --config" that doesn't currently work for
  subsequent local-modification pcs commands (which is the purpose
  together with sequence-crowning cib-push in this context), so
  rather use mere "pcs cluster cib <file>"
  [resolves: rhbz#1328078]

. with [cp]cs2pcscmd commands, clufter no longer suggests
  "pcs cluster start --all --wait=-1"  as part of the emitted command
  sequence  (last option decides, through a failure, whether pcs accepts
  a numeric argument there, which would then make the rest of sequence
  use this recent, more elegant provision of pcs instead of "sleep")
  without suppressing both standard and error outputs so as to prevent
  unnecessary clutter with newer, compatible versions of pcs

. with *2pcscmd* commands, clufter would previously emit doubled
  "pcs" at the beginning for the command defining simple order
  constraint

. with *2pcscmd* commands, clufter would previously omit and/or
  logic operators between each pair of atomic expressions
  forming a rule for location constraint

. with  *2pcscmd* commands, clufter would previously disregard
  master/slave roles correctly encoded with a capitalized first
  letter in CIB for colocation and location constraints

. {ccs,pcs}2pcscmd* would previously exceed a recursion limit due
  to not catching the bottom of the recursion properly

. {ccs,pcs}2pcscmd* commands would previously emit an incorrectly
  quoted command (in the self-check the following sequence of
  commands is indeed being run on the to-be-clustered machine)
  unless -g or --noguidance option was used;
  in turn the respective internals received a considerable
  overhaul to be able to cope with nested commands
  (command/process substitution) better;
  for users of previous releases, the remedy is to use this -g switch
  or to pass --noop=cmd-wrap to suppress the faulty filter from
  the pipeline (at the expense of not-so-easy-to-consume output)


New features
============

. [cp]cs2pcscmd* commands now supports "--wait" parameter to pcs
  command for starting the cluster and prefers it to static "sleep"
  when possible (pcs version recent enough)
  [see also: rhbz#1229822]

. ccs2pcs* conversion is now more careful when translating
  corosync logging subsystems (e.g., CONFDB -> CMAP)

. *2pcscmd* conversion now handles clone/master resources;
  likewise, node properties, utilization (per node or resource)
  and {op,rsc}_defaults are now covered when converting CIB
  content into sequence of pcs commands
  [resolves/related: rhbz#1270740]

. with *2pcscmd* commands, clufter now supports resource sets
  for colocation and order constraints

. with *2pcscmd* commands, clufter now supports ticket contraints

. pcs2pcscmd* commands are now aware of quorum device configured
  in corosync.conf and are able to emit respective configuration
  commands using pcs tool

. {cib,pcs}2pcscmd* commands are now aware of configured alert
  handlers in CIB and able to emit respective configuration
  commands using pcs tool



New behavior
============

. as pcs currently does not support templates [rhbz#1281359],
  a new filter, cib-meld, is included in *2pcscmd conversion and its
  purpose is to mimic Pacemaker internal desugaring where suitable

. cmd-wrap and (new) cib-revitalize (translating deprecated cluster
  properties into preferred notation for now) filters are directly
  exposed as commands with an assumption they are useful on their own

E clufter now relies on the specified distribution target (which
  basically boils down to snapshot of the supported features, as
  opposed to passing zillion extra parameters expressing the same)
  stronger than ever;
  this has several implications: do not expect that one sequence
  of pcs commands at the clufter's output is portable to completely
  different environment, and your distribution/setup may not be
  supported

E there is now --list-dists option to clufter that is intended
  mainly to suggest as to which --dist option values (note this
  has an increased importance as mentioned) are supported

. *2pcscmd commands now first emit a comment block containing key
  pieces of information about the run, such as a current date,
  library version, the overall command that was executed, and
  importantly (more and more), the target system specification
  (this utilizes a new, dedicated cmd-annotate filter)


User friendliness
=================

. (suppresible) progress info/notes/warnings output was further
  unified and color is used more widely to distinguish semantic
  meaning if possible

. user-assisted validation failure recovery was made preoccupied
  users friendly (bail out when the launched editor is closed
  without a change two times in row) as opposed to neverending
  loop preventing possibly baffled user to leave gracefully;
  also one more hint is given in the help section of the edited
  document facilitating the recovery

. help screens and man pages for *2pcscmd* commands now warn
  againts using --tmp-cib '' (empty string) as it means resorting
  to shot-by-shot semantics, as opposed to accumulate-and-push
  (desirable), and this can lead to unexpected inconsistencies

. *2pcscmd commands will now colorize the output (very plain support,
  more to be expected) if either a terminal is used as a sink (and
  colors not explicitly forbidden) or if this is enforced; so far
  only shell comments and some parts of pcs syntax are supported,
  but even in this form, it should help users to wrap their heads
  (eyes) around what can be considered quite a complex output
  from the first sight

E when *2pcscmd or cmd-wrap commands colorize the output, "meta"
  words for the shell are distinguished as well

Comment 15 Jan Pokorný [poki] 2016-09-08 17:41:32 UTC
More pleasant to read, bugzilla friendly version
(s/rhbz#\([0-9]*\)/[bug \1]/g):

Bug fixes
=========

! ccs2pcs* conversion now properly propagates or adds monitor action
  [resolves: [bug 1272570]]

. ccs2pcs* conversion now (suitably) aborts when it reaches exclusive
  resource group
  [related: [bug 1206640] ([bug 1272191])]

. ccs2pcs* conversion now does not take undeployed (contained in
  <resources> and not referenced elsewhere) into account and is
  more careful about carrying about parent service group auxiliary
  reference for further processing
  [related/resolves: [bug 1272592]]

! pcs2pcscmd{,-needle} now propagates cluster name correctly,
  previously it was mistakenly dropped resulting in confusing
  first cluster node for that
  (e.g., pcs cluster setup --start --name node1 node2 node3)

. ccs2pcs* conversion now propagates two_node (and internally
  also expected_votes) properties of cman as defined in cluster.conf
  correctly

. help screens/manual pages for *2pcs* commands now documents
  "cib" arguments correctly

. pcs2pcscmd* conversion now correctly includes cmd-wrap filter
  resulting in the output line-wrapped per expectation

. "assisted recovery" now works on systems without /dev/tty as well
  as on systems for which open-modify[open+close] (final close yet
  to come) of particular file won't reliably discover mtime change;
  now strict open-close-modify is used instead (and mtime check in
  a was-file-changed test is preceded with a file size comparison for
  good measure), making intermittent failures in test runs disappear

. clufter is now capable of handling command options as unicode
  (relates to the usage as a library, original discovery thanks
  to pcs) and "the magic interpolation" of the command inputs now
  works at places where it was supposed to but unfortunately did not

! ccs2pcs* commands no longer generate accidentally broken values
  of attributes marked as having an ID type in the schema
  [resolves: [bug 1300050]]

. ccs2pcs* commands now translate notion of recovery/relocate recover
  policy of the resource group as supported by RGManager into the
  parallel expression in Pacemaker universe;  the same applies to
  __independent_subtree=2 at the resource level and empty restricted
  failover domain (that is referred to from existing resource group)

. ccs2pcs* commands now propagate stop timeout of the vm original
  resource agent

. *2pcscmd* commands now support group meta attributes properly

. *2pcscmd* commands no longer emit bogus properties of the operations
  (id, name, interval) as these are position-fixed values in the
  respective pcs syntax, hence not requiring explicit key=value
  treatment

E with {cib,pcs}2pcscmd* commands, clufter no longer chokes on
  validation failures (unless --nocheck provided) due to source CIB
  file using newer "validate-with" validation version specification
  than the only supported so far (pacemaker-1.2.rng) or possibly
  using a syntax not compatible with that; now also 2.0, 2.3, 2.4, and
  2.5 versions are supported
  [resolves: [bug 1300014]]

. with {cib,pcs}2pcscmd* commands, clufter no longer chokes on
  validation failures (unless --nocheck provided) due to source CIB
  not containing "status" section (which is normally the case with
  implicit input located in /var/lib/pacemaker/cib/cib.xml);
  now the bundled, compacted schemas mark this section optional
  and also the recipe to distill such format from pacemaker native
  schemas ensures the same assumption holds even if not pre-existed
  [resolves: [bug 1269964], comment 9]
  [see also: https://github.com/ClusterLabs/pacemaker/pull/957]
. internal representations of command + options/arguments was fixed
  in several ways so as to provide correct outcomes in both general
  (previously, some options could be duplicated while overwriting
  other options/arguments, and standalone negative numbers were
  considered options) and pcs (--wait=X cannot be decoupled the same
  way option parsers can usually cope with, as pcs built-in parser
  treats this specifically) cases

! with *2pcscmd* commands, clufter no longer suggests
  "pcs cluster cib <file> --config" that doesn't currently work for
  subsequent local-modification pcs commands (which is the purpose
  together with sequence-crowning cib-push in this context), so
  rather use mere "pcs cluster cib <file>"
  [resolves: [bug 1328078]]

. with [cp]cs2pcscmd commands, clufter no longer suggests
  "pcs cluster start --all --wait=-1"  as part of the emitted command
  sequence  (last option decides, through a failure, whether pcs accepts
  a numeric argument there, which would then make the rest of sequence
  use this recent, more elegant provision of pcs instead of "sleep")
  without suppressing both standard and error outputs so as to prevent
  unnecessary clutter with newer, compatible versions of pcs

. with *2pcscmd* commands, clufter would previously emit doubled
  "pcs" at the beginning for the command defining simple order
  constraint

. with *2pcscmd* commands, clufter would previously omit and/or
  logic operators between each pair of atomic expressions
  forming a rule for location constraint

. with  *2pcscmd* commands, clufter would previously disregard
  master/slave roles correctly encoded with a capitalized first
  letter in CIB for colocation and location constraints

. {ccs,pcs}2pcscmd* would previously exceed a recursion limit due
  to not catching the bottom of the recursion properly

. {ccs,pcs}2pcscmd* commands would previously emit an incorrectly
  quoted command (in the self-check the following sequence of
  commands is indeed being run on the to-be-clustered machine)
  unless -g or --noguidance option was used;
  in turn the respective internals received a considerable
  overhaul to be able to cope with nested commands
  (command/process substitution) better;
  for users of previous releases, the remedy is to use this -g switch
  or to pass --noop=cmd-wrap to suppress the faulty filter from
  the pipeline (at the expense of not-so-easy-to-consume output)


New features
============

. [cp]cs2pcscmd* commands now supports "--wait" parameter to pcs
  command for starting the cluster and prefers it to static "sleep"
  when possible (pcs version recent enough)
  [see also: [bug 1229822]]

. ccs2pcs* conversion is now more careful when translating
  corosync logging subsystems (e.g., CONFDB -> CMAP)

. *2pcscmd* conversion now handles clone/master resources;
  likewise, node properties, utilization (per node or resource)
  and {op,rsc}_defaults are now covered when converting CIB
  content into sequence of pcs commands
  [resolves/related: [bug 1270740]]

. with *2pcscmd* commands, clufter now supports resource sets
  for colocation and order constraints

. with *2pcscmd* commands, clufter now supports ticket contraints

. pcs2pcscmd* commands are now aware of quorum device configured
  in corosync.conf and are able to emit respective configuration
  commands using pcs tool

. {cib,pcs}2pcscmd* commands are now aware of configured alert
  handlers in CIB and able to emit respective configuration
  commands using pcs tool



New behavior
============

. as pcs currently does not support templates [[bug 1281359]],
  a new filter, cib-meld, is included in *2pcscmd conversion and its
  purpose is to mimic Pacemaker internal desugaring where suitable

. cmd-wrap and (new) cib-revitalize (translating deprecated cluster
  properties into preferred notation for now) filters are directly
  exposed as commands with an assumption they are useful on their own

E clufter now relies on the specified distribution target (which
  basically boils down to snapshot of the supported features, as
  opposed to passing zillion extra parameters expressing the same)
  stronger than ever;
  this has several implications: do not expect that one sequence
  of pcs commands at the clufter's output is portable to completely
  different environment, and your distribution/setup may not be
  supported

E there is now --list-dists option to clufter that is intended
  mainly to suggest as to which --dist option values (note this
  has an increased importance as mentioned) are supported

. *2pcscmd commands now first emit a comment block containing key
  pieces of information about the run, such as a current date,
  library version, the overall command that was executed, and
  importantly (more and more), the target system specification
  (this utilizes a new, dedicated cmd-annotate filter)


User friendliness
=================

. (suppresible) progress info/notes/warnings output was further
  unified and color is used more widely to distinguish semantic
  meaning if possible

. user-assisted validation failure recovery was made preoccupied
  users friendly (bail out when the launched editor is closed
  without a change two times in row) as opposed to neverending
  loop preventing possibly baffled user to leave gracefully;
  also one more hint is given in the help section of the edited
  document facilitating the recovery

. help screens and man pages for *2pcscmd* commands now warn
  againts using --tmp-cib '' (empty string) as it means resorting
  to shot-by-shot semantics, as opposed to accumulate-and-push
  (desirable), and this can lead to unexpected inconsistencies

. *2pcscmd commands will now colorize the output (very plain support,
  more to be expected) if either a terminal is used as a sink (and
  colors not explicitly forbidden) or if this is enforced; so far
  only shell comments and some parts of pcs syntax are supported,
  but even in this form, it should help users to wrap their heads
  (eyes) around what can be considered quite a complex output
  from the first sight

E when *2pcscmd or cmd-wrap commands colorize the output, "meta"
  words for the shell are distinguished as well

Comment 25 errata-xmlrpc 2016-11-04 07:52:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2526.html

Comment 26 Steven J. Levine 2016-11-28 14:34:56 UTC
Very slight update to the doc text to respond to question from translator confused about the meaning of "confused the first cluster node for the cluster name".  I think the example itself makes it clear and I explained what this meant to the translator for her purposes, but I thought it would help to refer to "the name of the first cluster node" as a parallel phrase to "the name of the cluster".