Bug 1447910 - bundle resources are missing meta attributes
Summary: bundle resources are missing meta attributes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pcs
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Tomas Jelinek
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On: 1447903
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-04 08:29 UTC by Tomas Jelinek
Modified: 2018-04-25 14:07 UTC (History)
15 users (show)

Fixed In Version: pcs-0.9.158-6.el7
Doc Type: Enhancement
Doc Text:
Feature: Add pcs commands for managing meta attributes of bundle resources. Reason: With no meta attributes management it is not possible to stop bundle resources or set them to unmanaged mode. Result: New pcs commands have been added to disable, enable, unmanage and manage bundle resources and change their meta attributes. It is possible to create a bundle resource in a way it will not be started automatically upon creation. Bundles' meta attributes are displayed in resources listing.
Clone Of: 1447903
Environment:
Last Closed: 2017-08-01 18:26:07 UTC
Target Upstream Version:


Attachments (Terms of Use)
proposed fix + tests (117.07 KB, patch)
2017-06-08 13:00 UTC, Tomas Jelinek
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1958 normal SHIPPED_LIVE pcs bug fix and enhancement update 2017-08-01 18:09:47 UTC

Description Tomas Jelinek 2017-05-04 08:29:38 UTC
+++ This bug was initially created as a clone of Bug #1447903 +++

It is not possible to disable a bundle resource by setting target-role=stopped because bundle resources have no meta attributes. This leads to the following issues:

a) Users cannot stop a bundle resource.

b) "crm_resource --restart --resource <bundle resource>" doesn't work:
# crm_resource --restart --resource http-bundle
Could not set target-role for http-bundle: Update does not conform to the configured schema (-203)

c) When deleting a bundle resource, the bundle cannot be stopped beforehand, leaving orphaned resources behind. When stop-orphan-resources is set to false, these resources just stay there.


Ken:
Yes, this is a known limitation. It also applies to is-managed.


Andrew:
My original thoughts were to not have a meta block and instead enumerate the specific options (as was done for masters and replicas) that we need.

Happy to leave the decision in your hands though.

Comment 2 Ken Gaillot 2017-05-26 16:24:43 UTC
Upstream pacemaker now supports setting meta-attributes on bundles, and having them inherited by the components. At least for target-role, this works as expected.

Initial tests with is-managed did not work as expected, so I'm continuing to investigate that. But at least some support, including stopping, will already be available with the current proposed 7.4 pacemaker packages.

Comment 3 Ken Gaillot 2017-05-26 20:52:28 UTC
I was wrong about target-role working ... with the current pacemaker packages, only the primitive inside the bundle (not the implicit docker, ip, or remote resources) inherits any of the meta-attributes.

However, pacemaker does at least support setting meta-attributes on the bundle, so pcs can move forward with implementing the commands for it.

Comment 4 Ken Gaillot 2017-05-31 14:02:19 UTC
Upstream pacemaker support for bundle meta-attributes is fully implemented. I hope to have a 7.4 pacemaker build with it by end of next week.

Comment 5 Tomas Jelinek 2017-06-08 13:00:12 UTC
Created attachment 1286145 [details]
proposed fix + tests

* pcs resource bundle create
  * added meta keyword
  * added --disabled flag
* pcs resource bundle update
  * added meta keyword
* pcs config show and similar
  * displays bundles' meta attributes 
* pcs resource enable | disable
  * now works with bundles
* pcs resource manage | unmanage
  * now works with bundles
* pcs resource delete
  * stops bundles before deleting them

Comment 7 Tomas Jelinek 2017-06-15 13:35:17 UTC
After fix:

> --disabled in create

[root@rh73-node1:~]# pcs resource bundle create http-bundle container docker image=pcmktest:http replicas=2 --disabled
[root@rh73-node1:~]# pcs resource
 Docker container set: http-bundle [pcmktest:http]
   http-bundle-docker-0 (ocf::heartbeat:docker):        Stopped (disabled)
   http-bundle-docker-1 (ocf::heartbeat:docker):        Stopped (disabled)

> resource enable (the bundle is stopped as there is no resource in it to run)
[root@rh73-node1:~]# pcs resource enable http-bundle
[root@rh73-node1:~]# pcs resource
 Docker container set: http-bundle [pcmktest:http]
   http-bundle-docker-0 (ocf::heartbeat:docker):        Stopped
   http-bundle-docker-1 (ocf::heartbeat:docker):        Stopped

> resource disable
[root@rh73-node1:~]# pcs resource disable http-bundle
[root@rh73-node1:~]# pcs resource
 Docker container set: http-bundle [pcmktest:http]
   http-bundle-docker-0 (ocf::heartbeat:docker):        Stopped (disabled)
   http-bundle-docker-1 (ocf::heartbeat:docker):        Stopped (disabled)

> resource unmanage, manage
[root@rh73-node1:~]# pcs resource unmanage http-bundle
[root@rh73-node1:~]# pcs resource
 Docker container set: http-bundle [pcmktest:http] (unmanaged)
   http-bundle-docker-0 (ocf::heartbeat:docker):        Stopped (disabled, unmanaged)
   http-bundle-docker-1 (ocf::heartbeat:docker):        Stopped (disabled, unmanaged)
[root@rh73-node1:~]# pcs resource manage http-bundle
[root@rh73-node1:~]# pcs resource
 Docker container set: http-bundle [pcmktest:http]
   http-bundle-docker-0 (ocf::heartbeat:docker):        Stopped (disabled)
   http-bundle-docker-1 (ocf::heartbeat:docker):        Stopped (disabled)

> create and update with meta, meta attributes displayed
[root@rh73-node1:~]# pcs resource bundle create http-bundle container docker image=pcmktest:http meta target-role=Stopped is-managed=false
[root@rh73-node1:~]# pcs resource --full
 Bundle: http-bundle
  Docker: image=pcmktest:http
  Meta Attrs: is-managed=false target-role=Stopped
root@rh73-node1:~]# pcs resource bundle update http-bundle meta is-managed=true
[root@rh73-node1:~]# pcs resource --full
 Bundle: http-bundle
  Docker: image=pcmktest:http
  Meta Attrs: is-managed=true target-role=Stopped

Comment 9 Damien Ciabrini 2017-06-21 16:07:49 UTC
Additionally, Michele Baldessari and myself are testing the new features from this build since a month now, so I can say that's it's working as expected for us.

As explained in https://bugzilla.redhat.com/show_bug.cgi?id=1447903, we have successfully validated that this new build allows us to create bundles in the desired target role, and also to update this target role once the resource is added into the cluster.

Comment 11 Damien Ciabrini 2017-06-21 21:35:49 UTC
Correction: Unfortunately, Michele and I tested a pre-release version of the fix [1], and the pcs command run by puppet to create a bundle is no longer working with pcs-0.9.158-6.el7:

For the sake of the example, the puppet-generated command:

/usr/sbin/pcs -f 1.xml resource bundle create galera-bundle container image=192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-06-19.1 replicas=3 masters=3 options="--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" run-command="/bin/bash /usr/local/bin/kolla_start" network=host storage-map id=mysql-cfg-files source-dir=/var/lib/kolla/config_files/mysql.json target-dir=/var/lib/kolla/config_files/config.json options=ro storage-map id=mysql-cfg-data source-dir=/var/lib/config-data/mysql target-dir=/var/lib/kolla/config_files/src options=ro storage-map id=mysql-hosts source-dir=/etc/hosts target-dir=/etc/hosts options=ro storage-map id=mysql-localtime source-dir=/etc/localtime target-dir=/etc/localtime options=ro storage-map id=mysql-lib source-dir=/var/lib/mysql target-dir=/var/lib/mysql options=rw storage-map id=mysql-log-mariadb source-dir=/var/log/mariadb target-dir=/var/log/mariadb options=rw storage-map id=mysql-pki-extracted source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted options=ro storage-map id=mysql-pki-ca-bundle-crt source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt options=ro storage-map id=mysql-pki-ca-bundle-trust-crt source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt options=ro storage-map id=mysql-pki-cert source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem options=ro storage-map id=mysql-dev-log source-dir=/dev/log target-dir=/dev/log options=rw network control-port=3123 --disabled

Now fails with:

Error: '' is not a valid container type value, use docker


By looking at the rpm changelog, it could just be because we're not passing an argument that is considered mandatory:

- It is now mandatory to specify container type in the "resource bundle create" command

Tomas, maybe you know the exact syntax that we should use for specifying the container type, and if by any chance we can already use it in pcs-0.9.158-5.el7? 


[1] http://people.redhat.com/mbaldess/rpms/container-repo/srpms/pcs-0.9.158-5.pidone.bundle.1.el7.centos.src.rpm

Comment 12 Tomas Jelinek 2017-06-22 08:11:36 UTC
Damien,

You are right, it is now mandatory to specify the container type. To fix your scripts simply add "docker" after "container" in the resource bundle create command.

Originally, container type was an optional parameter with "docker" being the default. My idea was as follows:
* Currently docker is the only supported container type.
* So for one-time command line usage we can make it optional. That way users will not be forced to type "docker" while there are no other options than "docker" that could be used there.
* When scripting pcs, authors of scripts will always specify the container type to make their scripts resilient to possible changes of default container type.

Based on bug 1433016 comment 13 I changed the bundle create command so the container type is mandatory. By not specifying the container type in your scripts you only proved my original idea was wrong and that it was right decision to make the container type mandatory.

Comment 13 Damien Ciabrini 2017-06-22 12:12:05 UTC
Thanks Tomas, I confirm that calling pcs with "container docker" works as expected for us.

I've updated the our setup code accordingly for OpenStack upstream [2]

I confirm that pcs-0.9.158-6.el7 works for us.



[2] https://review.openstack.org/#/c/476486/

Comment 15 errata-xmlrpc 2017-08-01 18:26:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1958

Comment 16 Jan Pokorný [poki] 2017-11-02 21:21:33 UTC
re [comment 11]:
Going off-topic now, Damien, there's actually a trick to avoid
such ineligible, hard-to-follow monolitic commands, especially
when it comes to pcs as it involves clufter :)

$ python2 $(which clufter) cmd-wrap <<EOF
/usr/sbin/pcs -f 1.xml resource bundle create galera-bundle container image=192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-06-19.1 replicas=3 masters=3 options="--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" run-command="/bin/bash /usr/local/bin/kolla_start" network=host storage-map id=mysql-cfg-files source-dir=/var/lib/kolla/config_files/mysql.json target-dir=/var/lib/kolla/config_files/config.json options=ro storage-map id=mysql-cfg-data source-dir=/var/lib/config-data/mysql target-dir=/var/lib/kolla/config_files/src options=ro storage-map id=mysql-hosts source-dir=/etc/hosts target-dir=/etc/hosts options=ro storage-map id=mysql-localtime source-dir=/etc/localtime target-dir=/etc/localtime options=ro storage-map id=mysql-lib source-dir=/var/lib/mysql target-dir=/var/lib/mysql options=rw storage-map id=mysql-log-mariadb source-dir=/var/log/mariadb target-dir=/var/log/mariadb options=rw storage-map id=mysql-pki-extracted source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted options=ro storage-map id=mysql-pki-ca-bundle-crt source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt options=ro storage-map id=mysql-pki-ca-bundle-trust-crt source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt options=ro storage-map id=mysql-pki-cert source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem options=ro storage-map id=mysql-dev-log source-dir=/dev/log target-dir=/dev/log options=rw network control-port=3123 --disabled
EOF
> /usr/sbin/pcs -f 1.xml resource bundle create galera-bundle \
>   container \
>   image=192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-06-19.1 \
>   replicas=3 masters=3 \
>   'options=--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS' \
>   'run-command=/bin/bash /usr/local/bin/kolla_start' network=host \
>   storage-map id=mysql-cfg-files \
>   source-dir=/var/lib/kolla/config_files/mysql.json \
>   target-dir=/var/lib/kolla/config_files/config.json options=ro \
>   storage-map id=mysql-cfg-data source-dir=/var/lib/config-data/mysql \
>   target-dir=/var/lib/kolla/config_files/src options=ro \
>   storage-map id=mysql-hosts source-dir=/etc/hosts \
>   target-dir=/etc/hosts options=ro \
>   storage-map id=mysql-localtime source-dir=/etc/localtime \
>   target-dir=/etc/localtime options=ro \
>   storage-map id=mysql-lib source-dir=/var/lib/mysql \
>   target-dir=/var/lib/mysql options=rw \
>   storage-map id=mysql-log-mariadb source-dir=/var/log/mariadb \
>   target-dir=/var/log/mariadb options=rw \
>   storage-map id=mysql-pki-extracted \
>   source-dir=/etc/pki/ca-trust/extracted \
>   target-dir=/etc/pki/ca-trust/extracted options=ro \
>   storage-map id=mysql-pki-ca-bundle-crt \
>   source-dir=/etc/pki/tls/certs/ca-bundle.crt \
>   target-dir=/etc/pki/tls/certs/ca-bundle.crt options=ro \
>   storage-map id=mysql-pki-ca-bundle-trust-crt \
>   source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt \
>   target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt options=ro \
>   storage-map id=mysql-pki-cert source-dir=/etc/pki/tls/cert.pem \
>   target-dir=/etc/pki/tls/cert.pem options=ro \
>   storage-map id=mysql-dev-log source-dir=/dev/log target-dir=/dev/log \
>   options=rw network control-port=3123 --disabled
> [cmd-wrap] output: <stdout>

(python2 is needed also on Fedoras as I just hit some unexpected
encoding issue with python3, which otherwise default here)

Comment 17 Jan Pokorný [poki] 2017-11-10 22:08:35 UTC
(re [comment 16]:
fixed: https://pagure.io/clufter/c/7df4b13add615e40862271caa484b3d7a4386f9b)


Note You need to log in before you can comment on or make changes to this bug.