Bug 2227724

The overcloud upgrade prepare failed. It seems that the error is due to the CephRgwKey key. Searching for the value of the key seemed that the $STACK-0-passwords.yaml is exported with passwords as '*****'.

parameter_defaults:
  AdminPassword: '******'
  AdminToken: '******'
  AodhPassword: 1IIz0JYxupsXvFy0xCxI7EcpIWXxxxx
  BarbicanPassword: WLeCZExFx7xNxOWtKnwneBJI5owRq
  BarbicanSimpleCryptoKek: diqBMF8gxHxUxXxu_8XBFQo1DRoqTWD0c2kO2R7RLspBQTw=
  CeilometerMeteringSecret: Nu5zThmr1jx0uxoPxxXkg7t0yfq3x
  CeilometerPassword: m1BxCx5xwqrxFxdGQzog1jKhlZScIv
  CephClientKey: '******'
  CephClusterFSID: cdf24729-bd06-46c2-ba47-7018ae220197
  CephDashboardAdminPassword: '******'
  CephGrafanaAdminPassword: '******'
  CephManilaClientKey: '******'
  CephRgwKey: '******'
  CinderPassword: '******'

Error log
############################### 
2023-07-29 00:36:32 | Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 215, in validate_constraints
    constraint.validate(value, self, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 305, in validate
    raise ValueError(err_msg)
', 'ValueError: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 91, in _validate_default
    self.validate_constraints(default_value, context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 217, in validate_constraints
    raise exception.StackValidationFailed(message=str(ex))
', 'heat.common.exception.StackValidationFailed: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 235, in validate
    self.schema.validate(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 145, in validate
    self._validate_default(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 95, in _validate_default
    raise exception.InvalidSchemaError(
', 'heat.common.exception.InvalidSchemaError: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 915, in validate
    self.parameters.validate(context=self.context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 533, in validate
    param.validate(validate_value, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 251, in validate
    raise exception.InvalidSchemaError(message=msg)
', 'heat.common.exception.InvalidSchemaError: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/heat/resource_chain.py", line 107, in validate_nested_stack
    super(ResourceChain, self).validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 416, in wrapped
    return func(self, ctx, *args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 849, in create_stack
    stack = self._parse_template_and_validate_stack(
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 757, in _parse_template_and_validate_stack
    stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ControllerServiceChain<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/common/services/controller-role.yaml>.resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
'].
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare Traceback (most recent call last):
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 381, in _try_overcloud_deploy_with_compat_yaml
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     deployment_options=deployment_options)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 159, in _heat_deploy
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     self.log, self.working_dir)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 62, in deploy_without_plan
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     orchestration_client.stacks.create(**stack_args)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     data=kwargs, headers=headers)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 292, in post
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     return self.client_request("POST", url, **kwargs)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 282, in client_request
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     resp, body = self.json_request(method, url, **kwargs)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 271, in json_request
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     resp = self._http_request(url, method, **kwargs)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 234, in _http_request
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     raise exc.from_response(resp)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare heatclient.exc.HTTPInternalServerError: ERROR: Internal Error
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 215, in validate_constraints
    constraint.validate(value, self, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 305, in validate
    raise ValueError(err_msg)
', 'ValueError: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 91, in _validate_default
    self.validate_constraints(default_value, context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 217, in validate_constraints
    raise exception.StackValidationFailed(message=str(ex))
', 'heat.common.exception.StackValidationFailed: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 235, in validate
    self.schema.validate(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 145, in validate
    self._validate_default(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 95, in _validate_default
    raise exception.InvalidSchemaError(
', 'heat.common.exception.InvalidSchemaError: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 915, in validate
    self.parameters.validate(context=self.context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 533, in validate
    param.validate(validate_value, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 251, in validate
    raise exception.InvalidSchemaError(message=msg)
', 'heat.common.exception.InvalidSchemaError: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/heat/resource_chain.py", line 107, in validate_nested_stack
    super(ResourceChain, self).validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 416, in wrapped
    return func(self, ctx, *args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 849, in create_stack
    stack = self._parse_template_and_validate_stack(
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 757, in _parse_template_and_validate_stack
    stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ControllerServiceChain<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/common/services/controller-role.yaml>.resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
'].
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare 
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare During handling of the above exception, another exception occurred:
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare 
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare Traceback (most recent call last):
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     super(Command, self).run(parsed_args)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     return super(Command, self).run(parsed_args)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     return_code = self.take_action(parsed_args) or 0
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 88, in take_action
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     super(UpgradePrepare, self).take_action(parsed_args)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1271, in take_action
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     working_dir=self.working_dir)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     self.force_reraise()
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     raise self.value
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1208, in take_action
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     user_tht_root, created_env_files)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 346, in deploy_tripleo_heat_templates
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     deployment_options=deployment_options)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 384, in _try_overcloud_deploy_with_compat_yaml
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare     raise ValueError(messages)
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare ValueError: Failed to deploy: ERROR: Internal Error
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 215, in validate_constraints
    constraint.validate(value, self, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 305, in validate
    raise ValueError(err_msg)
', 'ValueError: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 91, in _validate_default
    self.validate_constraints(default_value, context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 217, in validate_constraints
    raise exception.StackValidationFailed(message=str(ex))
', 'heat.common.exception.StackValidationFailed: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 235, in validate
    self.schema.validate(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 145, in validate
    self._validate_default(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 95, in _validate_default
    raise exception.InvalidSchemaError(
', 'heat.common.exception.InvalidSchemaError: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 915, in validate
    self.parameters.validate(context=self.context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 533, in validate
    param.validate(validate_value, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 251, in validate
    raise exception.InvalidSchemaError(message=msg)
', 'heat.common.exception.InvalidSchemaError: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/heat/resource_chain.py", line 107, in validate_nested_stack
    super(ResourceChain, self).validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 416, in wrapped
    return func(self, ctx, *args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 849, in create_stack
    stack = self._parse_template_and_validate_stack(
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 757, in _parse_template_and_validate_stack
    stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ControllerServiceChain<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/common/services/controller-role.yaml>.resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
'].
2023-07-29 00:36:32 | 2023-07-29 00:36:32.041 808873 ERROR tripleoclient.v1.overcloud_upgrade.UpgradePrepare [00m
2023-07-29 00:36:32 | 2023-07-29 00:36:32.045 808873 ERROR openstack [-] Failed to deploy: ERROR: Internal Error
2023-07-29 00:36:32 | Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 215, in validate_constraints
    constraint.validate(value, self, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 305, in validate
    raise ValueError(err_msg)
', 'ValueError: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 91, in _validate_default
    self.validate_constraints(default_value, context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 217, in validate_constraints
    raise exception.StackValidationFailed(message=str(ex))
', 'heat.common.exception.StackValidationFailed: "******" does not match pattern "^[a-zA-Z0-9+/]{38}==$"
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 235, in validate
    self.schema.validate(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/constraints.py", line 145, in validate
    self._validate_default(context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 95, in _validate_default
    raise exception.InvalidSchemaError(
', 'heat.common.exception.InvalidSchemaError: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 915, in validate
    self.parameters.validate(context=self.context,
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 533, in validate
    param.validate(validate_value, context)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/parameters.py", line 251, in validate
    raise exception.InvalidSchemaError(message=msg)
', 'heat.common.exception.InvalidSchemaError: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
    nested_stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/heat/resource_chain.py", line 107, in validate_nested_stack
    super(ResourceChain, self).validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
', '
During handling of the above exception, another exception occurred:

', 'Traceback (most recent call last):
', '  File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 416, in wrapped
    return func(self, ctx, *args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 849, in create_stack
    stack = self._parse_template_and_validate_stack(
', '  File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 757, in _parse_template_and_validate_stack
    stack.validate()
', '  File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
    result = f(*args, **kwargs)
', '  File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
    result = res.validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
    result = super(TemplateResource, self).validate()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
    self.validate_nested_stack()
', '  File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
    raise exception.StackValidationFailed(
', 'heat.common.exception.StackValidationFailed: InvalidSchemaError: resources.ControllerServiceChain<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/common/services/controller-role.yaml>.resources.ServiceChain<nested_stack>.resources.2<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-client.yaml>.resources.CephBase<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployment/cephadm/ceph-base.yaml>: Parameter \'CephRgwKey\' is invalid: Invalid default ****** ("******" does not match pattern "^[a-zA-Z0-9+/]{38}==$")
'].[00m
2023-07-29 00:36:32 | 2023-07-29 00:36:32.047 808873 INFO osc_lib.shell [-] END return value: 1[00m

#################