Bug 1549688 - OC deployment fails "APIException: resources.WorkflowTasks_Step2_Execution: Invalid input for field/attribute wf_ex. Value: <long string> . No JSON object could be decoded
Summary: OC deployment fails "APIException: resources.WorkflowTasks_Step2_Execution: I...
Keywords:
Status: CLOSED DUPLICATE of bug 1549677
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: beta
: ---
Assignee: Angus Thomas
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-27 16:22 UTC by Alexander Chuzhoy
Modified: 2018-02-27 16:49 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-27 16:49:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Alexander Chuzhoy 2018-02-27 16:22:23 UTC
OC deployment fails "APIException: resources.WorkflowTasks_Step2_Execution: Invalid input for field/attribute wf_ex. Value: <long string> . No JSON object could be decoded


Environment:
ansible-role-redhat-subscription-0.0.1-0.20180209232232.00defe7.el7ost.noarch
ceph-ansible-3.0.14-1.el7cp.noarch
openstack-tripleo-heat-templates-8.0.0-0.20180215092255.el7ost.noarch
ansible-tripleo-ipsec-8.0.2-0.20180209232721.7815cc5.el7ost.noarch
instack-undercloud-8.2.1-0.20180214022816.8a7609e.el7ost.noarch
ansible-2.4.2.0-2.el7.noarch
openstack-puppet-modules-11.0.0-0.20180211142351.71ad01c.el7ost.noarch


Steps to reproduce:

Try to deploy OC with:
openstack overcloud deploy \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/network-environment.yaml \
-e /home/stack/ceph.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /home/stack/dns_wipedisk_root_password.yaml


Result:
The deploy fails:

openstack stack failures list overcloud --long
overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::Mistral::ExternalResource
  physical_resource_id: 
  status: CREATE_FAILED
  status_reason: |
    APIException: resources.WorkflowTasks_Step2_Execution: Invalid input for field/attribute wf_ex. Value: '{'input': 'Heat managed', 'workflow_name': 'tripleo.overcloud.workflow_tasks.step2', 'params': '{"evaluate_env": false, "env": {"role_merged_configs": {"ObjectStorage": {"swift::storage::all::incoming_chmod": "Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r", "swift::storage::all::account_server_workers": "auto", "tripleo::profile::base::sshd::options": {"Subsystem": "sftp  /usr/libexec/openssh/sftp-server", "UsePAM": "yes", "HostKey": ["/etc/ssh/ssh_host_rsa_key", "/etc/ssh/ssh_host_ecdsa_key", "/etc/ssh/ssh_host_ed25519_key"], "GSSAPICleanupCredentials": "no", "SyslogFacility": "AUTHPRIV", "GSSAPIAuthentication": "yes", "PasswordAuthentication": "no", "AuthorizedKeysFile": ".ssh/authorized_keys", "AcceptEnv": ["LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES", "LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT", "LC_IDENTIFICATION LC_ALL LANGUAGE", "XMODIFIERS"], "UsePrivilegeSeparation": "sandbox", "X11Forwarding": "yes", "ChallengeResponseAuthentication": "no"}, "tripleo::profile::base::swift::ringbuilder::min_part_hours": 1, "snmp::agentaddress": ["udp:161", "udp6:[::1]:161"], "tripleo::profile::base::snmp::snmpd_password": "42209a820335410010a894b873cbd8caf77ef1fd", "snmp::snmpd_options": "-LS0-5d", "swift::storage::all::account_pipeline": ["healthcheck", "account-server"], "tripleo::profile::base::snmp::snmpd_user": "ro_snmp_user", "tripleo.swift_storage.firewall_rules": {"123 swift storage": {"dport": [873, 6000, 6001, 6002]}}, "swift::swift_hash_path_suffix": "QGNYT8uZVQWyvvgZgT7nrdVxC", "tripleo::profile::base::docker::network_options": "--bip=172.31.0.1/24", "ntp::maxpoll:": 10, "tripleo::profile::base::database::mysql::client::ssl_ca": "/etc/ipa/ca.crt", "swift::storage::all::storage_local_net_ip": "storage_mgmt", "tripleo::profile::base::database::mysql::client::enable_ssl": false, "tripleo::profile::base::tuned::profile": "", "tripleo::firewall::purge_firewall_rules": false, "swift::storage::all::object_pipeline": ["healthcheck", "recon", "object-server"], "tripleo::profile::base::database::mysql::client::mysql_client_bind_address": "internal_api", "tripleo::packages::enable_install": false, "tripleo::profile::base::docker::configure_network": true, "tripleo::profile::base::swift::storage::enable_swift_storage": true, "tripleo::profile::base::swift::ringbuilder::build_ring": true, "kernel_modules": {"nf_conntrack": {}, "nf_conntrack_proto_sctp": {}}, "ntp::servers": ["clock.redhat.com"], "tripleo::profile::base::swift::ringbuilder::swift_ring_put_tempurl": "http://192.168.0.1:8080/v1/AUTH_c8ecee6246354b06ac46d8885aaf3e89/overcloud-swift-rings/swift-rings.tar.gz?temp_url_sig=f50148e9ebacf8e2d5c802f098885c85f6a3edc7&temp_url_expires=1519497313", "swift::storage::all::mount_check": false, "timezone::timezone": "UTC", "tripleo::profile::base::docker::debug": true, "tripleo.snmp.firewall_rules": {"127 snmp": {"dport": 161, "proto": "udp"}}, "tripleo::profile::base::swift::storage::use_local_dir": true, "swift::storage::all::container_server_workers": "auto", "tripleo::profile::base::sshd::motd": "", "tripleo::profile::base::docker::docker_options": "--log-driver=journald --signature-verification=false --iptables=false --live-restore", "ntp::iburst_enable": true, "tripleo::profile::base::swift::ringbuilder::replicas": 3, "ntp::minpoll:": 6, "tripleo::profile::base::sshd::bannertext": "", "tripleo.ntp.firewall_rules": {"105 ntp": {"dport": 123, "proto": "udp"}}, "tripleo::profile::base::swift::ringbuilder::raw_disk_prefix": "r1z1-", "tripleo::profile::base::swift::ringbuilder:skip_consistency_check": true, "tripleo::profile::base::swift::ringbuilder::part_power": 10, "swift::storage::all::object_server_workers": "auto", "tripleo::profile::base::swift::ringbuilder::swift_ring_get_tempurl": "http://192.168.0.1:8080/v1/AUTH_c8ecee6246354b06ac46d8885aaf3e89/overcloud-swift-rings/swift-rings.tar.gz?temp_url_sig=5399f20cd9609ae6f4582576210c85316ed5ad45&temp_url_expires=1519497252", "tripleo::trusted_cas::ca_map": {}, "swift::storage::all::outgoing_chmod": "Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r", "tripleo::profile::base::swift::ringbuilder::raw_disks": [":%PORT%/d1"], "tripleo::profile::base::docker::insecure_registries": ["192.168.0.1:8787"], "tripleo::firewall::manage_firewall": true, "swift::storage::all::container_pipeline": ["healthcheck", "container-server"], "swift::storage::disks::args": {}, "sysctl_settings": {"net.ipv4.conf.all.arp_accept": {"value": 1}, "net.ipv6.conf.default.autoconf": {"value": 0}, "net.ipv6.conf.default.accept_redirects": {"value": 0}, "net.nf_conntrack_max": {"value": 500000}, "fs.inotify.max_user_instances": {"value": 1024}, "net.ipv4.conf.default.log_martians": {"value": 1}, "net.ipv4.conf.all.send_redirects": {"value": 0}, "net.core.netdev_max_backlog": {"value": 10000}, "net.ipv4.neigh.default.gc_thresh1": {"value": 1024}, "net.ipv6.conf.all.autoconf": {"value": 0}, "net.ipv4.tcp_keepalive_probes": {"value": 5}, "kernel.pid_max": {"value": 1048576}, "net.ipv4.conf.all.log_martians": {"value": 1}, "fs.suid_dumpable": {"value": 0}, "net.ipv4.conf.default.accept_redirects": {"value": 0}, "net.ipv6.conf.all.accept_ra": {"value": 0}, "net.ipv4.conf.default.secure_redirects": {"value": 0}, "net.ipv4.tcp_keepalive_intvl": {"value": 1}, "net.ipv4.tcp_keepalive_time": {"value": 5}, "net.ipv6.conf.default.accept_ra": {"value": 0}, "kernel.dmesg_restrict": {"value": 1}, "net.ipv6.conf.all.accept_redirects": {"value": 0}, "net.ipv6.conf.all.disable_ipv6": {"value": 0}, "net.ipv4.conf.all.secure_redirects": {"value": 0}, "net.netfilter.nf_conntrack_max": {"value": 500000}, "net.ipv4.neigh.default.gc_thresh2": {"value": 2048}, "net.ipv4.neigh.default.gc_thresh3": {"value": 4096}, "net.ipv4.conf.default.send_redirects": {"value": 0}, "net.ipv6.conf.default.disable_ipv6": {"value": 0}}}, "Controller": {"cinder::keystone::authtoken::project_domain_name": "Default", "nova::rabbit_port": 5672, "nova::wsgi::apache_placement::bind_host": "internal_api", "aodh::db::mysql::dbname": "aodh", "heat::enable_proxy_headers_parsing": true, "nova::api::sync_db_api": true, "horizon::django_session_engine": "django.contrib.sessions.backends.cache", "panko::keystone::authtoken::password": "eTq8CzpDKqnaaZj39JJ2DePjY", "tripleo::profile::base::gnocchi::api::incoming_storage_driver": "redis", "horizon::secret_key": "qvqHVuqkRj", "aodh::keystone::auth::public_url": "http://10.19.184.188:8042", "snmp::snmpd_options": "-LS0-5d", "heat::keystone::auth::region": "regionOne", "ceilometer::keystone::authtoken::user_domain_name": "Default", "tripleo::profile::pacemaker::cinder::volume_bundle::cinder_volume_docker_image": "192.168.0.1:8787/rhosp13/openstack-cinder-volume:pcmklatest", "tripleo::profile::base::glance::api::glance_nfs_enabled": false, "nova::db::mysql::host": "10.19.94.19", "cinder::cron::db_purge::user": "cinder", "aodh::db::mysql::allowed_hosts": ["%", "%{hiera(\'mysql_bind_host\')}"], "tripleo.gnocchi_api.firewall_rules": {"129 gnocchi-api": {"dport": [8041, 13041]}}, "nova::policy::policies": {}, "swift::keystone::auth::internal_url": "http://10.19.95.15:8080/v1/AUTH_%(tenant_id)s", "tripleo::profile::base::rabbitmq::enable_internal_tls": false, "tripleo::haproxy::crl_file": null, "rabbitmq::default_user": "guest", "horizon::cache_backend": "django.core.cache.backends.memcached.MemcachedCache", "tripleo::profile::base::cinder::volume::rbd::cinder_rbd_user_name": "openstack", "ceilometer::agent::auth::auth_project_domain_name": "Default", "tripleo::profile::base::cinder::volume::iscsi::cinder_iscsi_helper": "lioadm", "neutron::keystone::auth::password": "bbBBdzFxPQn4vhPszx8Dsr7wu", "swift::keystone::auth::password": "pBZCAAH6Jb7bx6gTnrdKAwkcU", "gnocchi::db::mysql::allowed_hosts": ["%", "%{hiera(\'mysql_bind_host\')}"], "nova::wsgi::apache_api::servername": "%{hiera(\'fqdn_internal_api\')}", "cinder::cron::db_purge::hour": "0", "ceilometer::snmpd_readonly_username": "ro_snmp_user", "rabbit_ipv6": false, "glance::db::mysql::allowed_hosts": ["%", "%{hiera(\'mysql_bind_host\')}"], "glance::api::enable_proxy_headers_parsing": true, "neutron::agents::dhcp::dnsmasq_dns_servers": [], "ceilometer::debug": true, "redis::sentinel_auth_pass": "8RK4rR7Ddg4448tY2nh2jmuTt", "keystone::wsgi::apache::threads": 1, "heat::keystone::auth_cfn::tenant": "service", "cinder::wsgi::apache::ssl": false, "heat::cron::purge_deleted::minute": "1", "gnocchi::debug": true, "cinder::volume::manage_service": false, "nova::rabbit_password": "hghAXsFHK6N4dzTARqawKxprr", "neutron::db::mysql::allowed_hosts": ["%", "%{hiera(\'mysql_bind_host\')}"], "gnocchi::wsgi::apache::bind_host": "internal_api", "aodh::auth::auth_url": "http://10.19.94.19:5000", "neutron::agents::dhcp::interface_driver": "neutron.agent.linux.interface.OVSInterfaceDriver", "neutron::agents::metadata::metadata_host": "%{hiera(\'cloud_name_internal_api\')}", "tripleo.neutron_dhcp.firewall_rules": {"115 neutron dhcp input": {"dport": 67, "proto": "udp"}, "116 neutron dhcp output": {"dport": 68, "chain": "OUTPUT", "proto": "udp"}}, "nova::placement::project_name": "service", "aodh::api::gnocchi_external_project_owner": "service", "neutron::agents::l3::agent_mode": "legacy", "nova::placement_database_connection": "mysql+pymysql://nova_placement:g3s4dRaTQufcWENRcvvsUg9UY.94.19/nova_placement?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf", "keystone::endpoint::version": "", "neutron::service_plugins": ["router", "qos", "trunk"], "gnocchi::keystone::auth::tenant": "service", "panko::auth::auth_tenant_name": "service", "panko::keystone::auth::internal_url": "http://10.19.94.19:8977", "redis_ipv6": false, "gnocchi::db::mysql::dbname": "gnocchi", "tripleo.gnocchi_statsd.firewall_rules": {"140 gnocchi-statsd": {"dport": 8125, "proto": "udp"}}, "heat::api_cfn::bind_host": "internal_api", "redis::bind": "internal_api", "heat::api::bind_host": "internal_api", "ceilometer::telemetry_secret": "z2rThdXRMmK7a2XcpsmFE7RCG", "keystone::cron::token_flush::minute": ["1"], "tripleo::profile::pacemaker::haproxy_bundle::internal_certs_directory": "/etc/pki/tls/certs/haproxy", "aodh::keystone::auth::tenant": "service", "tripleo::profile::base::cinder::volume::nfs::cinder_nfs_servers": [], "keystone::rabbit_port": 5672, "aodh::keystone::auth::admin_url": "http://10.19.94.19:8042", "glance::api::image_member_quota": 128, "mysql::server::manage_config_file": true, "swift::keystone::auth::configure_s3_endpoint": false, "swift::storage::all::storage_local_net_ip": "storage_mgmt", "redis::sentinel::redis_host": "%{hiera(\'bootstrap_nodeid_ip\')}", "redis::service_manage": false, "gnocchi::api::service_name": "httpd", "cinder::rabbit_heartbeat_timeout_threshold": 60, "neutron::plugins::ml2::firewall_driver": "iptables_hybrid", "tripleo::profile::pacemaker::database::redis_bundle::redis_docker_image": "192.168.0.1:8787/rhosp13/openstack-redis:pcmklatest", "aodh::wsgi::apache::ssl": false, "heat::keystone::auth::admin_url": "http://10.19.94.19:8004/v1/%(tenant_id)s", "tripleo::haproxy::haproxy_service_manage": false, "panko::expirer::hour": "0", "tripleo::profile::base::haproxy::certificates_specs": {}, "heat::wsgi::apache_api_cfn::servername": "%{hiera(\'fqdn_internal_api\')}", "keystone::wsgi::apache::servername_admin": "%{hiera(\'fqdn_ctlplane\')}", "tripleo::profile::base::swift::proxy::tls_proxy_port": "8080", "neutron::db::sync::db_sync_timeout": 300, "horizon::enforce_password_check": true, "corosync_ipv6": false, "heat::cron::purge_deleted::hour": "0", "keystone::rabbit_password": "hghAXsFHK6N4dzTARqawKxprr", "heat::rpc_response_timeout": 600, "nova::network::neutron::neutron_ovs_bridge": "br-int", "swift::proxy::versioned_writes::allow_versioned_writes": true, "nova::network::neutron::neutron_project_name": "service", "tripleo::profile::base::database::mysql::client_bind_address": "internal_api", "heat::keystone::auth_cfn::password": "WCY3Yub998HE94s3cM98jzJe6", "panko::policy::policies": {}, "keystone::wsgi::apache::servername": "%{hiera(\'fqdn_internal_api\')}", "cinder::db::mysql::allowed_hosts": ["%", "%{hiera(\'mysql_bind_host\')}"], "aodh::auth::auth_tenant_name": "service", "nova::network::neutron::dhcp_domain": "", "tripleo.cinder_volume.firewall_rules": {"120 iscsi initiator": {"dport": 3260}}, "tripleo::profile::base::glance::api::tls_proxy_port": "9292", "neutron::agents::metadata::shared_secret": "FvCJ3PXettHrap76dXhGXaee6", "nova::cell0_database_connection": "mysql+pymysql://nova:g3s4dRaTQufcWENRcvvsUg9UY.94.19/nova_cell0?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf", "neutron::plugins::ml2::network_vlan_ranges": ["datacentre:1:1000"], "tripleo::profile::base::snmp::snmpd_password": "42209a820335410010a894b873cbd8caf77ef1fd", "neutron::agents::metadata::metadata_ip": "%{hiera(\'nova_metadata_vip\')}", "panko::expirer::minute": "1", "tripleo::profile::base::snmp::snmpd_user": "ro_snmp_user", "nova::scheduler::discover_hosts_in_cells_interval": -1, "ceilometer::notification_driver": "messagingv2", "tripleo::profile::base::docker::configure_network": true, "cinder::scheduler::scheduler_driver": "cinder.scheduler.filter_scheduler.FilterScheduler", "tripleo::haproxy::haproxy_log_address": "/dev/log", "cinder::rabbit_password": "hghAXsFHK6N4dzTARqawKxprr", "cinder::keystone::auth::public_url": "http://10.19.184.188:8776/v1/%(tenant_id)s", "tripleo::profile::base::cinder::volume::rbd::cinder_rbd_pool_name": "volumes", "tripleo::profile::pacemaker::database::mysql_bundle::bind_address": "%{hiera(\'fqdn_internal_api\')}", "nova::api::neutron_metadata_proxy_shared_secret": "FvCJ3PXettHrap76dXhGXaee6", "ceilometer::dispatcher::gnocchi::archive_policy": "low", "cinder::api::nova_catalog_info": "compute:nova:internalURL", "tripleo::profile::base::swift::proxy::rabbit_port": 5672, "swift::proxy::authtoken::project_name": "service", "nova::wsgi::apache_api::bind_host": "internal_api", "heat::cron::purge_deleted::destination": "/dev/null", "mysql_ipv6": false, "aodh::keystone::authtoken::project_domain_name": "Default", 
...

truncated output (longer than 65K chars)


. No JSON object could be decoded





Also, checking mistral for error:
 ZaqarAction.claim_messages failed: Error response from Zaqar. Code: 503. Title: Service temporarily unavailable. Description: Claim could not be created. Please try again in a few seconds.."

Comment 2 Alexander Chuzhoy 2018-02-27 16:49:16 UTC

*** This bug has been marked as a duplicate of bug 1549677 ***


Note You need to log in before you can comment on or make changes to this bug.