Bug 1526494

Summary: Rebase clufter component [RHEL 6.10]
Product: Red Hat Enterprise Linux 6 Reporter: Jan Pokorný [poki] <jpokorny>
Component: clufterAssignee: Jan Pokorný [poki] <jpokorny>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: medium Docs Contact: Steven J. Levine <slevine>
Priority: high    
Version: 6.10CC: aherr, cfeist, jpokorny, mkolaja, mlisik, slevine
Target Milestone: rcKeywords: Rebase
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: clufter-0.77.1-1.el6 Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
_clufter_ rebased to version 0.77.1 The _clufter_ packages have been upgraded to upstream version 0.77.1, which provides a number of bug fixes, new features, and user experience enhancements over the previous version. Among the notable updates are the following: * When producing `pcs` commands, the `clufter` tool now supports a preferred ability to generate `pcs` commands that will update only the modifications made to a configuration by means of a differential update rather than a pushing a wholesale update of the entire configuration. Likewise when applicable, the `clufter` tool now supports instructing the `pcs` tool to configure user permissions (ACLs). For this to work across the instances of various major versions of the document schemas, `clufter` gained the notion of internal on-demand format upgrades, mirroring the internal mechanics of `pacemaker`. Similarly, `clufter` is now capable of configuring the `bundle` feature. * In any script-like output sequence such as that produced with the `ccs2pcscmd` and `pcs2pcscmd` families of `clufter` commands, the intended shell interpreter is now emitted in a valid form, so that the respective commented line can be honored by the operating system. * When using `clufter` to translate an existing configuration with the `pcs2pcscmd-needle` command in the case where the `corosync.conf` equivalent omits the `cluster_name` option (which is not the case with standard pcs-initiated configurations), the contained `pcs cluster setup` invocation no longer causes cluster misconfiguration with the name of the first given node interpreted as the required cluster name specification. The same invocation will now include the `--encryption 0|1` switch when available, in order to reflect the original configuration accurately. * All `clufter` commands having a sequence of `pcs` commands at the output, meaning they are passed through a post-processing to improve readability (unless disabled with `--noop=cmd-wrap`), no longer have the issue that some characters with special meaning in shell language were not being quoted, which changed their interpretation. * The `clufter` tool now also covers some additional recently added means of configuration as facilitated with `pcs` (heuristics for a quorum device, meta attributes for top-level `bundle` resource units) when producing the sequence of configuring `pcs` commands to reflect existing configurations when applicable. On the `corosync` configuration interfacing side, the format parser no longer misinterprets commented-out lines with spaces or tabulators in front of the respective delimiter, and support for some mechanically introduced options was reconsidered under closer examination of what `pcs` actually handles. For information on the capabilities of *clufter*, see the `clufter(1)` man page or the output of the "clufter -h" command. For examples of *clufter* usage, see the following Red Hat Knowledgebase article: https://access.redhat.com/articles/2810031.
Story Points: ---
Clone Of: 1367536 Environment:
Last Closed: 2018-06-19 05:23:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Pokorný [poki] 2017-12-15 16:02:38 UTC
+++ This bug was initially created as a clone of Bug #1367536 +++

+++ This bug was initially created as a clone of Bug #1387424 +++

Upstream development of clufter has progressed  so this asks for
a rebase.

NOTE:
It's understood that under normal circumstances, packages are not
receiving any updates sans the critical fixes, but this component begs
to differ as it also stands for a RHEL 7 migration technology, and RHEL
7 itself is a slightly moving target across minor releases, which is
in turn being gradually accommodated by this tool.  And because it's
reasonable to allow for tailored migration up to the latest RHEL 7 minor
version as of the particular RHEL 6 release (6.10 here), the request
to proceed with the rebase is deemed justified here.

More specifically, clufter received plentiful bug fixes and enhancements
compared to previous RHEL minor release (6.9/clufter-0.59.8) as
currently, clufter is at version 0.77.0.

Comment 8 Jan Pokorný [poki] 2018-04-19 15:21:42 UTC
Regarding changelog items (improving upon the procedure over that
of [bug 1387424 comment 12]):

$ git tag --no-merged v0.59.8
> v0.70.0
> v0.75.0
> v0.76.0
> v0.77.0
> v0.77.1

$ git tag -n9999 --no-merged v0.59.8 | sed 's|^v[^ ]*[ ]*|\n&\n-------\n|'
> 
> v0.70.0         
> -------
> bug fix, enhancement and, foremostly, Python 3 compatibility release
>     
>     - bug fixes:
>       . since v0.59.7, {ccs,pcs}2pcscmd commands were meant to emit
>         hashbangs so as to make it clear which shell (e.g. bash if its
>         extensions are relied upon) the suggested scripts should be run
>         under, but alas, exclamation mark has been accidentally missing
>         [related rhbz#1381531]
>       . invocations like "clufter ccs2pcscmd | bash -x", despite being
>         discouraged (review is needed for any nontrivial case, anyway),
>         if "pcs cluster auth" part interactively prompting credentials
>         was part of the run, they would be obtained from the piped script
>         under interpreter's execution itself due to the way the descriptors
>         are inherited; now there is an explicit stdin/stdout redirect
>         against /dev/tty in place to prevent such undesired side-effect
>     - functional changes:
>       . owing to the internal overhaul, *2pcscmd commands now properly
>         adapt to whether the output is directed to a terminal or not
>         (including piping to another command) with regards to colorizing
>         and text-wrapping for reasonable line lengths unless overridden
>         with explicit switches
>       . commands dealing with XML format (at some internal, not necessarily
>         user-facing phase; covers all *2pcs* ones) are now proofed so as
>         to preserve XML attribute ordering, leading to consistent results
>         (desired way beyond internal test suite)
>       . better diagnostics are now provided for some previously neglected
>         error conditions
>     - feature highlight: Python 3 support (-> minor version bump so high)
>       . beside Python 2.6, clufter project now strives to support 3.2+
>         as well; brand new compatibility with the latter was casually
>         and unit tested (the suite was also adapted to support both),
>         but some corner cases may have not been hit yet, hence this
>         class of bug reports is especially appreciated
>       . for the clufter as a library users: under Python 3, "bytestring"
>         protocol now, due to a strict dichotomy, indeed means "passed as
>         bytes (or compatible) type" (if the contrary is observed, it is
>         a bug to be fixed)
>       . *2pcscmd commands now state also the Python interpreter in the
>         initial informative comment block to ease the troubleshooting
>         in the light of newly introduced dual Python 2/3 support
>       . so-called external plugins are now implicitly searched also in
>         additional, Python interpreter agnostic path intended primarily
>         for plugins maintaining dual Python 2/3 compatibility (in the same
>         vein as core clufter as of this release)
>       . meta-spec prescribing a suggested RPM packaging in a generic way
>         now encompasses the above point as well as it offers python3-clufter
>         subpackage, and in turn factors some shared files to other new
>         subpackages (clufter-bin, clufter-common)
> 
> v0.75.0         
> -------
> enhancement and Python 3 compatibility improving release
>     
>     - enhancements:
>       . when using *2pcscmd commands, one can now specify --set-exec
>         option to make executable; note this is explicitly discouraged
>         as one should always take precautions prior to running anything
>         executable from a semirandom source and this would only make
>         it easier to skip that step and accidentally run the generated
>         script right away
>       . with {cib,pcs}2pcscmd commands, the generated sequence is now
>         adapted to whether pcs is able to natively cope with "external/*"
>         specification of stonith resources where applicable
>       . borrowing directly from pacemaker project, the upgrade between
>         1.x and 2.x lines of the schema for a CIB instance is now
>         available as fmt-cib-1to2 filter, arranged to be implicitly
>         triggered so as to bridge supplied vs. required format
>         discrepancies (see first internal enhancement point below)
>     - feature extensions:
>       . {cib,pcs}2pcscmd* commands are now aware of configured ACL
>         definitions in CIB and able to emit respective configuration
>         commands using pcs tool; thanks to implicit format upgrade
>         arrangement with fmt-cib-1to2 filter (see above), it can cope
>         with "old ACL style" of configuration at the input as well
>         [resolves: rhbz#1440876]
>       . likewise {cib,pcs}2pcscmd* commands now instruct sufficiently
>         recent target pcs to perform push of the CIB changes (accumulated
>         throughout the generated script) as a differential update rather
>         than propagating a complete CIB copy, which remains the default
>         otherwise (both unless push-by-push is applied, despite warnings)
>     - internal enhancements:
>       . format hierarchies were noticeably rethought with a notion of
>         upgradeability in mind -- something natural for the versioned,
>         compatibility-sensitive formats such as CIB, the prime target
>         (see also the mentioned upgrade filter); with a proper
>         arrangement these upgrades are performed implicitly on the
>         format instances being exchanged between the filters, should
>         the target filter impose that on validation failure basis
>       . Python 3 compatibility received further polishing
>       . meta specfile holding the suggested RPM packaging moved "nano"
>         fallback editor dependency over to -cli package, resulting in
>         a bit more lightweight pcs installation, as that itself depends
>         just on python*-clufter
>         [resolves: PGissue#1]
> 
> v0.76.0         
> -------
> bug fix and enhancements (support for pcs commands defining bundles)
>     
>     - bug fixes:
>       . some messages emitted in conversion corner cases are meant to
>         terminate the whole process, which previously may have not been
>         the case
>     - feature extensions:
>       . {cib,pcs}2pcscmd* commands are now aware of configured bundle
>         definitions in CIB and able to emit respective configuration
>         commands using pcs tool, provided that the target versions of
>         relevant packages already support that
>     - internal enhancements:
>       . the logic behind reusing pacemaker's own schemas is now available
>         as a proper script that now also -- rationally -- only extracts the
>         lowest and highest available minor version per a major one of the CIB
>         schema files, which further corresponds to what is now statically
>         carried with the project distribution and, foremost, what decreases
>         the number of the worst-case validations
>       . some warnings possibly emitted during the conversions now share
>         a uniform style
> 
> v0.77.0         
> -------
> bug fix and extension release (qdevice heuristics, bundle meta attrs)
>     
>     - bug fixes:
>       . practically every built-in command accepts "magic files" in place
>         of regular file specifications as arguments (also advertised at
>         the bottom of "clufter COMMAND -h"), but when it denotes the input
>         and happens to be resolved as standard input, a problem would
>         previously arise under Python 3, because this stream is implicitly
>         switched in text mode, whereas the data to be fetched are expected
>         to be raw bytes by the established framework (not enforced in Py2)
>       . pcs2pcscmd-needle command would previously lead to an incorrect
>         "pcs cluster setup" command in the resulting sequence when the
>         "cluster_name" parameter in the "totem" section of the input
>         corosync configuration was not (very legitimately) specified,
>         because the name of the first given node (if present) would
>         be mistakenly used where the cluster name specification was
>         expected as it was effectively skipped in the respective syntax;
>         now an artificial name is supplied instead if need be
>         (currently, pcs cannot create nameless clusters like that)
>       . all commands having sequence of pcs commands on the output,
>         hence getting post-processed (line-wrapped and generally
>         prettified) with the aim to get them human-friendly, might
>         previously suffer from some characters with special meaning
>         in shell language not being quoted (nor escaped for that matter)
>         properly (despite they might have been prior to this post-process),
>         leading to various, possibly subtle misbehaviours when executed
>         (for instance part of the intended standalone command being run in
>         the backgrounded subshell while the rest getting interpreted as
>         as a whole new, likely not fulfillable, command, if the particular
>         unquoted chunk in the case at hand contained '&' character); trivial
>         instances of this discrepancy should no longer occur, and for extra
>         confidence (or as a workaround with older versions), one can always
>         use "--noop=cmd-wrap" to suppress (tiny bit erratic) post-processing
>         (thanks to Madkiss, seconded by lge at #clusterlabs)
>     - feature extensions:
>       . {ccs,cib,pcs}2pcscmd* commands now make use of "--encryption 0|1"
>         switch finally added to "pcs cluster setup" command, incl. dealing
>         with a hiccup of introducing that support unconditionally (0.9.158),
>         only to return back to original default of no encryption, but with
>         a possibility to enable it (0.9.159 and counting), depending
>         on the pcs versions of the target distribution
>       . pcs2pcscmd-needle command is now aware of configured heuristics for
>         a quorum device in corosync.conf-equivalent files and able to emit
>         respective configuration commands sing pcs tool, provided that the
>         target versions of relevant packages already support that
>       . {cib,pcs}2pcscmd* commands are now aware of configured meta
>         attributes for top-level "bundle" encapsulations in CIB and able
>         to emit respective configuration commands using pcs tool, provided
>         that the target versions of relevant packages already support that
>     - internal enhancements:
>       . as pacemaker crossed the bridge and CIB schema versioning is no
>         longer compatible with string- and float-based comparisons, some
>         adaptation was required both code and release automation wise,
>         along with the update of higher-bound (merged-form) schema for CIB
>         up to this 2.10 version, which actually initiated those changes
>         [http://oss.clusterlabs.org/pipermail/developers/2017-July/000478.html]
> 
> v0.77.1         
> -------
> bug fix and slight fine-tuning release
>     
>     - bug fixes:
>       . runnning [cp]cs2pccmd commands in the absence of "defaults"
>         submodule (repository checkout, broken installation) no longer
>         fails
>       . corosync configuration parser, employed e.g. with pcs2pcscmd-needle
>         command, no longer mistreats commented-out lines with spaces or tabs
>         in front of the respective delimiter
>       . some options that were mechanically introduced recently for the
>         purpose of converting existing configuration into the procedural
>         steps leading there with the help pcs tool (i.e. {cib,pcs}2pcscmd
>         belong to affected commands) turned to be not actually suppported
>         by current pcs versions under closer examination, specifically
>         "nodelist.node.name" setting in corosync.conf (added into embedded
>         corosync configuration schema, regardless, in accord with having
>         this value legalized in corosync proper, just as with
>         "resources.watchdog_device"), and "quorum.device.votes" with
>         "quorum.device.net.tls" herein as well
>         [resolves/related: rhbz#1517834]
>         [resolves: rhbz#1552666]
>     - feature extensions:
>       . beyond, Linux top-level system selection (via --sys) of the target
>         the command output is to be taylored to (standing for a whole bunch
>         of parameters that would be too unwieldy to work with individually),
>         there's now also (rather preliminary) support for BSD family,
>         currently comprising only FreeBSD, for a simple fact that some
>         cluster packages are downstreamed there the usual way
>     - internal enhancements:
>       . handling of ambiguous specification of the target distribution plus
>         its version has been improved, also for the input values coming
>         from run-time auto-detection, leading to more widely normalized
>         values internally, hence bringing more reliable outputs
>         (now also with Fedora Rawhide, for instance)


Summarization of the most important items, taking [bug 1509381 comment 4]
and [RHEL7.4relnotes] into account:

---

When producing `pcs` commands, the `clufter` tool now supports
a preferred ability to generate `pcs` commands that will update only
the modifications made to a configuration by means of a differential
update rather than a pushing a wholesale update of the entire
configuration. Likewise when applicable, the `clufter` tool now
supports instructing the `pcs` tool to configure user permissions
(ACLs). For this to work across the instances of various major versions
of the document schemas, `clufter` gained the notion of internal
on-demand format upgrades, mirroring the internal mechanics of
`pacemaker`.  Similarly, `clufter` is now capable of configuring
the `bundle` feature. (BZ#1381531)

In any script-like output sequence such as that produced by the
`ccs2pcscmd` and `pcs2pcscmd` families of `clufter` commands,
the intended shell interpreter is now emitted in a valid form,
so that the respective commented line can finally be honored
by the operating system. (BZ#1381531)

When using `clufter` to translate existing configuration with
`pcs2pcscmd-needle` command whereby `corosync.conf` equivalent
omits `cluster_name` option (not the case with standard `pcs`
initiated configurations), the contained `pcs cluster setup`
invocation no longer causes cluster misconfiguration with the
name of the first given node interpreted as the required cluster
name specification.  The same invocation will now be enriched
with "--encryption 0|1" switch when available, in order to
reflect the original configuration truthfully.

All `clufter` commands having sequence of `pcs` commands at the
output, which are hence passed through a post-processing with the
intention to gain on human friendliness (unless disabled with
`--noop=cmd-wrap`), shall no longer suffer from some characters
with special meaning in shell language not being quoted, thus
changing their interpretation undesirably.

The `clufter` tool now also covers some other recently added
means of configuration as facilitated with `pcs` (heuristics
for a quorum device, meta attributes for top-level `bundle`
encapsulations) when producing sequence of configuring `pcs`
commands to reflect existing configurations when applicable.
On the `corosync` configuration interfacing side, the format
parser no longer mistreats commented-out lines with spaces
or tabulators in front of the respective delimiter, and
support for some mechanically introduced options was
reconsidered under closer examination of what `pcs` actually
handles. (BZ#1517834, BZ#1552666)

---

[RHEL7.4relnotes] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.4_release_notes/

Comment 9 Jan Pokorný [poki] 2018-04-19 16:35:26 UTC
re [comment 8]:

"(BZ#1381531)" should be removed at the first item ("When producing ...")
but retained at the second one ("In any script-like").

Comment 15 errata-xmlrpc 2018-06-19 05:23:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1923