Bug 1421228
Summary: | using yaml to register to satellite is yeilding: Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Steve Reichard <sreichar> |
Component: | rhosp-director | Assignee: | Angus Thomas <athomas> |
Status: | CLOSED NEXTRELEASE | QA Contact: | Amit Ugol <augol> |
Severity: | low | Docs Contact: | |
Priority: | low | ||
Version: | 10.0 (Newton) | CC: | aschultz, dbecker, dcritch, dgurtner, djuran, mburns, mcornea, morazi, rhel-osp-director-maint, sreichar |
Target Milestone: | --- | Keywords: | Reopened, Triaged, ZStream |
Target Release: | 10.0 (Newton) | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-07-23 14:50:22 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Steve Reichard
2017-02-10 16:26:16 UTC
Just attempted to use the openstack overcloud deploy command line options with similar results: openstack overcloud deploy \$ --templates \$ --environment-directory /home/stack/mytemplates/env \$ --stack rhosp \$ --debug \$ --log-file overcloudDeploy.log \$ --ceph-storage-flavor ceph-storage \$ --ceph-storage-scale 0 \$ --control-flavor control \$ --control-scale 3 \$ --compute-flavor compute \$ --compute-scale 2 \$ --block-storage-scale 0 \$ --swift-storage-scale 0 \$ --rhel-reg --reg-method satellite \$ --reg-org "Systems_Engineering" \$ --reg-sat-url http://se-sat6.cloud.lab.eng.bos.redhat.com \$ --reg-activation-key se-rhel7-osp10 \$ --ntp-server 10.16.255.1$ ... "GET /v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/events?marker=fa15e023-4313-44f0-a91b-db8f77f8add4&nested_depth=2&sort_dir=asc HTTP/1.1" 200 6106 RESP: [200] Content-Type: application/json; charset=UTF-8 Content-Length: 6106 X-Openstack-Request-Id: req-888907da-aa61-4366-bef5-fb87e1ddbe70 Date: Wed, 15 Feb 2017 22:54:21 GMT RESP BODY: {"events": [{"resource_name": "Controller", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/Controller/events/a45299fd-16ce-4a12-ad35-9921b7d337e9", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/Controller", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "Controller", "resource_status": "CREATE_FAILED", "resource_status_reason": "resources[1]: resources.Controller.Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned", "physical_resource_id": "32db9554-0ba0-49ef-a654-cdf75582ced1", "id": "a45299fd-16ce-4a12-ad35-9921b7d337e9"}, {"resource_name": "ObjectStorage", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/ObjectStorage/events/8cc79dc0-30b3-4025-9e73-6e7085b95aff", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/ObjectStorage", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "ObjectStorage", "resource_status": "CREATE_FAILED", "resource_status_reason": "CREATE aborted", "physical_resource_id": "6dd74e3c-7e85-4332-bd7d-d58acdc903d2", "id": "8cc79dc0-30b3-4025-9e73-6e7085b95aff"}, {"resource_name": "Compute", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/Compute/events/0bea06a6-29b5-4492-a5b2-7a137d9c97cf", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/Compute", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "Compute", "resource_status": "CREATE_FAILED", "resource_status_reason": "CREATE aborted", "physical_resource_id": "ed80d96f-ddac-45d6-b4a5-a51151774e2e", "id": "0bea06a6-29b5-4492-a5b2-7a137d9c97cf"}, {"resource_name": "BlockStorage", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/BlockStorage/events/8e8a68f8-bf8b-43da-8223-9ee4599613d8", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/BlockStorage", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "BlockStorage", "resource_status": "CREATE_FAILED", "resource_status_reason": "CREATE aborted", "physical_resource_id": "c5599ff6-a5ee-4742-9400-8f3e2b7d3f63", "id": "8e8a68f8-bf8b-43da-8223-9ee4599613d8"}, {"resource_name": "CephStorage", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/CephStorage/events/887b1663-f321-470c-91d5-be98397b3e9f", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/CephStorage", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "CephStorage", "resource_status": "CREATE_FAILED", "resource_status_reason": "CREATE aborted", "physical_resource_id": "ec3f4dce-63fc-4085-a6bd-4652d2973659", "id": "887b1663-f321-470c-91d5-be98397b3e9f"}, {"resource_name": "rhosp", "event_time": "2017-02-15T22:54:16Z", "links": [{"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/rhosp/events/5b25e8c6-6951-4385-a646-cf9d079d64df", "rel": "self"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a/resources/rhosp", "rel": "resource"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "stack"}, {"href": "https://192.168.213.2:13004/v1/49be6a4fac3b4a95b3a786a9e2cc6e86/stacks/rhosp/033362e4-73ab-4670-b862-3e19e73a3f9a", "rel": "root_stack"}], "logical_resource_id": "rhosp", "resource_status": "CREATE_FAILED", "resource_status_reason": "Resource CREATE failed: resources[1]: resources.Controller.Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned", "physical_resource_id": "033362e4-73ab-4670-b862-3e19e73a3f9a", "id": "5b25e8c6-6951-4385-a646-cf9d079d64df"}]} Heat Stack create failed. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1183, in take_action self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 403, in _deploy_tripleo_heat_templates_tmpdir new_tht_root, tht_root) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 474, in _deploy_tripleo_heat_templates parsed_args.timeout, env, parsed_args.update_plan_only) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 484, in _try_overcloud_deploy_with_compat_yaml tht_root, env, update_plan_only) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 262, in _heat_deploy self.app_args.verbose_level, timeout) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 80, in deploy_and_wait raise exceptions.DeploymentError("Heat Stack create failed.") DeploymentError: Heat Stack create failed. clean_up DeployOvercloud: Heat Stack create failed. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1183, in take_action self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 403, in _deploy_tripleo_heat_templates_tmpdir new_tht_root, tht_root) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 474, in _deploy_tripleo_heat_templates parsed_args.timeout, env, parsed_args.update_plan_only) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 484, in _try_overcloud_deploy_with_compat_yaml tht_root, env, update_plan_only) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 262, in _heat_deploy self.app_args.verbose_level, timeout) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 80, in deploy_and_wait raise exceptions.DeploymentError("Heat Stack create failed.") DeploymentError: Heat Stack create failed. END return value: 1 -02-15 22:54:14Z [rhosp.Controller]: CREATE_FAILED Resource CREATE failed: resources[1]: Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned 2017-02-15 22:54:14Z [rhosp.ObjectStorage]: CREATE_IN_PROGRESS state changed 2017-02-15 22:54:14Z [rhosp.Compute]: CREATE_IN_PROGRESS Stack CREATE started 2017-02-15 22:54:14Z [rhosp.Compute.1]: CREATE_IN_PROGRESS state changed 2017-02-15 22:54:14Z [rhosp.BlockStorage]: CREATE_IN_PROGRESS state changed 2017-02-15 22:54:15Z [rhosp.Compute.1]: CREATE_FAILED resources[1]: Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned 2017-02-15 22:54:15Z [rhosp.Compute]: CREATE_FAILED Resource CREATE failed: resources[1]: Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned 2017-02-15 22:54:16Z [rhosp.Controller]: CREATE_FAILED resources[1]: resources.Controller.Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned 2017-02-15 22:54:16Z [rhosp.ObjectStorage]: CREATE_FAILED CREATE aborted 2017-02-15 22:54:16Z [rhosp.Compute]: CREATE_FAILED CREATE aborted 2017-02-15 22:54:16Z [rhosp.BlockStorage]: CREATE_FAILED CREATE aborted 2017-02-15 22:54:16Z [rhosp.CephStorage]: CREATE_FAILED CREATE aborted 2017-02-15 22:54:16Z [rhosp]: CREATE_FAILED Resource CREATE failed: resources[1]: resources.Controller.Property error: resources.NodeExtraConfig.properties: Property rhel_reg_auto_attach not assigned Stack rhosp CREATE_FAILED I've been testing this out too and have gotten a little further, but there are issues. Here's my env: parameter_defaults: rhel_reg_activation_key: "OSP10KEY" rhel_reg_auto_attach: "" rhel_reg_base_url: "" rhel_reg_environment: "" rhel_reg_force: "true" rhel_reg_machine_name: "" rhel_reg_org: "ORG" rhel_reg_password: "" rhel_reg_pool_id: "" rhel_reg_release: "" rhel_reg_repos: "" rhel_reg_sat_url: "http://<SATURL>" rhel_reg_server_url: "" rhel_reg_service_level: "" rhel_reg_user: "" rhel_reg_type: "" rhel_reg_method: "satellite" rhel_reg_sat_repo: "rhel-7-server-satellite-tools-6.2-rpms" I found if I commented things out, it failed with the same error Steve saw, e.g. rhel_reg_auto_attach. Setting it to 'false' resulted in an error about not having auto_attach set to true in conjunction with an activation key. So it seems like anything other than an empty string evaluates as true? With this env file, nodes attempted to register, but they all seem to try and register as 'localhost.' This causes some nodes to fail due to a locked task in Satellite to register the duplicate 'localhost's. A couple of others will get in, but when the last node registers as localhost, it 'wins' and is the only node left registered. Stack create failure: ============ 2017-02-17 15:08:05Z [ospte.Controller.2.ControllerExtraConfigPre]: CREATE_IN_PROGRESS state changed 2017-02-17 15:08:07Z [ospte.Controller.2.ControllerExtraConfigPre]: CREATE_COMPLETE state changed 2017-02-17 15:08:07Z [ospte.Controller.2.NodeExtraConfig]: CREATE_IN_PROGRESS state changed 2017-02-17 15:08:16Z [ospte.Compute.1.NodeExtraConfig]: CREATE_FAILED Error: resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:16Z [ospte.Compute.1]: CREATE_FAILED Resource CREATE failed: Error: resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:17Z [ospte.Compute.1]: CREATE_FAILED Error: resources[1].resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:17Z [ospte.Compute.0]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:17Z [ospte.Compute]: CREATE_FAILED Resource CREATE failed: Error: resources[1].resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:18Z [ospte.Compute]: CREATE_FAILED Error: resources.Compute.resources[1].resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:18Z [ospte.Compute.0.NodeExtraConfig]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:19Z [ospte.Compute.0]: CREATE_FAILED Resource CREATE failed: Operation cancelled 2017-02-17 15:08:19Z [ospte.Controller]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:19Z [ospte.CephStorage]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:19Z [ospte]: CREATE_FAILED Resource CREATE failed: Error: resources.Compute.resources[1].resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 70 2017-02-17 15:08:19Z [ospte.CephStorage.0.NodeExtraConfig]: CREATE_FAILED Error: resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 69 2017-02-17 15:08:19Z [ospte.CephStorage.0]: CREATE_FAILED Resource CREATE failed: Error: resources.NodeExtraConfig.resources.RHELRegistrationDeployment: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 69 2017-02-17 15:08:19Z [ospte.CephStorage.0]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.CephStorage]: CREATE_FAILED Resource CREATE failed: Operation cancelled 2017-02-17 15:08:20Z [ospte.Controller.1]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.Controller.0]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.Controller.1.NodeExtraConfig]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.Controller.1]: CREATE_FAILED Resource CREATE failed: Operation cancelled 2017-02-17 15:08:20Z [ospte.Controller.2]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.Controller]: CREATE_FAILED Resource CREATE failed: Operation cancelled 2017-02-17 15:08:20Z [ospte.Controller.0.NodeExtraConfig]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:20Z [ospte.Controller.0]: CREATE_FAILED Resource CREATE failed: Operation cancelled 2017-02-17 15:08:21Z [ospte.Controller.2.NodeExtraConfig]: CREATE_FAILED CREATE aborted 2017-02-17 15:08:21Z [ospte.Controller.2]: CREATE_FAILED Resource CREATE failed: Operation cancelled Stack ospte CREATE_FAILED Heat Stack create failed. ============ Here's the locks popping up: ============ $ run-on-overcloud "hostname; sudo grep 'Required lock is already' /var/log/messages" ospte-cephstorage-0.localdomain ospte-cmpt0.localdomain ospte-cmpt1.localdomain Feb 17 10:08:11 host-192-168-211-211 os-collect-config: [2017-02-17 10:08:11,228] (heat-config) [INFO] {"deploy_stdout": "HTTP/1.1 200 OK\r\nStatus: 200 OK\r\nSatellite 6 detected at http://se-sat6.cloud.lab.eng.bos.redhat.com\nPreparing... ########################################\nUpdating / installing...\nkatello-ca-consumer-se-sat6.cloud.lab.########################################\n", "deploy_stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 6768 100 6768 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k\nRequired lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: read\nConflicts with tasks:\n- https://se-sat6.cloud.lab.eng.bos.redhat.com/foreman_tasks/tasks/32badf15-a75c-4918-8542-318116ef1cf4\n", "deploy_status_code": 70} Feb 17 10:08:11 host-192-168-211-211 os-collect-config: Required lock is already taken by other running tasks. ospte-ctrl0.localdomain ospte-ctrl1.localdomain ospte-ctrl2.localdomain Feb 17 10:09:23 host-192-168-211-205 os-collect-config: [2017-02-17 10:09:23,147] (heat-config) [INFO] {"deploy_stdout": "HTTP/1.1 200 OK\r\nStatus: 200 OK\r\nSatellite 6 detected at http://se-sat6.cloud.lab.eng.bos.redhat.com\nPreparing... ########################################\nUpdating / installing...\nkatello-ca-consumer-se-sat6.cloud.lab.########################################\n", "deploy_stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 6768 100 6768 0 0 106k 0 --:--:-- --:--:-- --:--:-- 106k\nRequired lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: read\nConflicts with tasks:\n- https://se-sat6.cloud.lab.eng.bos.redhat.com/foreman_tasks/tasks/d374b9a6-7b5f-4a67-8e9a-3c031c43d971\n", "deploy_status_code": 70} Feb 17 10:09:23 host-192-168-211-205 os-collect-config: Required lock is already taken by other running tasks. ============ And the end state with just one node registered: ============ $ run-on-overcloud "hostname; sudo subscription-manager status" ospte-cephstorage-0.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Unknown ospte-cmpt0.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Unknown ospte-cmpt1.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Unknown ospte-ctrl0.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Unknown ospte-ctrl1.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Current ospte-ctrl2.localdomain +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Unknown ============ I'm going to try and see if I can map rhel_reg_machine_name to the actual hostname and whether that helps. On further investigation, this might be an issue w/ Satellite. I re-ran my deployment pointing to a different, slightly older version of Satellite and the nodes all registered fine. Working Satellite is v6.2.2. The one w/ errors is 6.2.7. I updated the working Satellite from v6.2.2 to 6.2.7 and the problem has appeared there as well. All nodes try and register as 'localhost' leading to blocked tasks and only one node registered at the end. Attaching output of foreman-tail during the time of failure. There is a lot in there, but the 10.19.139.* hosts are the overcloud nodes trying to register. I updated to Satellite 6.2.8 released this week and the problem persists. Digging deeper, I found the following BZ and related foreman bug: https://bugzilla.redhat.com/show_bug.cgi?id=1129463 http://projects.theforeman.org/issues/7384 I was able to workaround it by creating a host entry in /etc/hosts in the rhel-registration script, prior to the registration and it works. Not sure why this is only an issue on the newer versions of Satellite. Later in the OSP deployment, /etc/hosts gets populated with all the overcloud entries, but it does not happen earlier enough to fix this issue. Should we pursue this from the Satellite side, or tweak the rhel-registration script to permanently work around it? I think this might have been an issue with the documentation that was updated as part of BZ#1433421. Can you check to see if that fixes your problem? If you look at the updates by David, in Comment #2 he provided the other parameteres and still had an error, but it corresponds with a Sat6 update. In comment #6 he identify the BZ and a workaround. I will set up to retest just using the full set of parameters. Stand corrected, Redeployed and all 5 overcloud nodes were registered closing - updated doc and retry worked for me. But shouldn't https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml take care of setting reasonable (empty) defaults for all needed parameters? If some are missing, I feel we should add them. Or why not just set default values on the parameters in the first place? The issue was previously the example file didn't have all the proper variables defined. Unfortunately from a product perspective we can't fill in any of those details because they are customer specific. I think we could improve the documentation within the file itself to ensure that if people are looking at that, they know which ones should be filled out. Empty strings are perfectly good defaults. I mean, there is no reason why every end user should need to specify empty values for e.g. all parameters related to Satellite 5 if he is using satellite 6. So in my opinion, we should provide empty values for all defaults. (In reply to David Critch from comment #6) > I updated to Satellite 6.2.8 released this week and the problem persists. > > Digging deeper, I found the following BZ and related foreman bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1129463 > http://projects.theforeman.org/issues/7384 > > I was able to workaround it by creating a host entry in /etc/hosts in the > rhel-registration script, prior to the registration and it works. > > Not sure why this is only an issue on the newer versions of Satellite. > > Later in the OSP deployment, /etc/hosts gets populated with all the > overcloud entries, but it does not happen earlier enough to fix this issue. > > Should we pursue this from the Satellite side, or tweak the > rhel-registration script to permanently work around it? David, I'm facing the same issue and came to the same conclusion of updating /etc/hosts, but the issue persists. Can you share your working approach to updating /etc/hosts? Thanks! Hi David, I gave this another whirl last week and was successful. I did not mess w/ /etc/hosts this time. The issue for me seemed to the domain name. Satellite maybe doesn't like the .localdomain? I set the overcloud domain name to match what is configured in Satellite and everything is working w/o any modifications to the script. I followed the steps listed here to set the domain name: https://access.redhat.com/solutions/786153 Hope that helps! (In reply to David Critch from comment #15) > Hi David, > > I gave this another whirl last week and was successful. I did not mess w/ > /etc/hosts this time. The issue for me seemed to the domain name. Satellite > maybe doesn't like the .localdomain? > > I set the overcloud domain name to match what is configured in Satellite and > everything is working w/o any modifications to the script. I followed the > steps listed here to set the domain name: > https://access.redhat.com/solutions/786153 > > Hope that helps! Thanks for your reply! The issue I was facing was that the nodes would show up in Satellite as localhost.localdomain, independent of the actual hostnames. This lead to an issue where all nodes registered under the same name, so I would only get a single entry in Satellite and the registration would thus fail. I since managed to overwrite /etc/hosts, so I'm fine for now, but the bug is on Satellite side and not in OpenStack as far as my case is concerned. Yup, that is the exact same thing I saw... all trying to register as localhost.localdomain and stomping on each other trying to get to Satellite. Once I set a domain name, they started to register fine under their FQDN w/ nothing local* at all anymore.... its weird. We have reworked the subscription process to leverage a new ansible role in the next release. Feel free to reopen if it still occurs in the next OSP release when using the new configurations. |