1987325 – [FFU OSP13 to 16.1] Ceph mon upgrade failed

Bug 1987325 - [FFU OSP13 to 16.1] Ceph mon upgrade failed

Summary: [FFU OSP13 to 16.1] Ceph mon upgrade failed

Keywords:
Status:	CLOSED DUPLICATE of bug 1964889
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	ceph-ansible
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Guillaume Abrioux
QA Contact:	Yogev Rabl
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-07-29 13:54 UTC by Paras Babbar
Modified:	2022-08-10 16:06 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-08-03 09:42:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-6599	0	None	None	None	2022-08-10 16:06:39 UTC

Description Paras Babbar 2021-07-29 13:54:47 UTC

Description of problem:

"fatal: [controller-0]: FAILED! => {\"msg\": \"the field 'args' has an invalid value ({'monitor_keyring': \\\"{{ (initial_mon_key.stdout | from_json)[0]['key'] if initial_mon_key is not skipped else monitor_keyring.stdout }}\\\"}), and could not be converted to an dict.The error was: Expecting value: line 1 column 1 (char 0)\\n\\nThe error appears to be in '/usr/share/ceph-ansible/roles/ceph-mon/tasks/deploy_monitors.yml': line 30, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n    - name: get initial keyring when it already exists\\n      ^ here\\n\"}",

2021-07-28 03:03:25 |         "fatal: [controller-0]: FAILED! => {\"changed\": false, \"msg\": \"There was an error during monitor upgrade. Please, check the previous task results.\"}",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : mons handler] **********************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:7",
2021-07-28 03:03:25 |         "Wednesday 28 July 2021  03:03:23 +0000 (0:00:00.023)       0:52:29.086 ******** ",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : osds handler] **********************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:12",
2021-07-28 03:03:25 |         "Wednesday 28 July 2021  03:03:23 +0000 (0:00:00.000)       0:52:29.087 ******** ",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : mdss handler] **********************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:17",
2021-07-28 03:03:25 |         "Wednesday 28 July 2021  03:03:23 +0000 (0:00:00.000)       0:52:29.088 ******** ",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : rgws handler] **********************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:22",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : rbdmirrors handler] ****************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:32",
2021-07-28 03:03:25 |         "Wednesday 28 July 2021  03:03:23 +0000 (0:00:00.000)       0:52:29.089 ******** ",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : mgrs handler] **********************************",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:37",
2021-07-28 03:03:25 |         "RUNNING HANDLER [ceph-handler : rbd-target-api and rbd-target-gw handler] ******",
2021-07-28 03:03:25 |         "task path: /usr/share/ceph-ansible/roles/ceph-handler/handlers/main.yml:47",
2021-07-28 03:03:25 |         "PLAY RECAP *********************************************************************",
2021-07-28 03:03:25 |         "ceph-0                     : ok=44   changed=2    unreachable=0    failed=0    skipped=93   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "ceph-1                     : ok=44   changed=2    unreachable=0    failed=0    skipped=93   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "ceph-2                     : ok=44   changed=2    unreachable=0    failed=0    skipped=93   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "compute-0                  : ok=33   changed=2    unreachable=0    failed=0    skipped=92   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "compute-1                  : ok=33   changed=2    unreachable=0    failed=0    skipped=92   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "controller-0               : ok=116  changed=11   unreachable=0    failed=1    skipped=232  rescued=1    ignored=0   ",
2021-07-28 03:03:25 |         "controller-1               : ok=36   changed=2    unreachable=0    failed=0    skipped=90   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "controller-2               : ok=36   changed=2    unreachable=0    failed=0    skipped=90   rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   ",
2021-07-28 03:03:25 |         "Wednesday 28 July 2021  03:03:23 +0000 (0:00:00.002)       0:52:29.092 ******** ",
2021-07-28 03:03:25 |         "=============================================================================== ",
2021-07-28 03:03:25 |         "ceph-mon : check if monitor initial keyring already exists ----------- 3001.67s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-mon/tasks/deploy_monitors.yml:5 ------------",
2021-07-28 03:03:25 |         "create potentially missing keys (rbd and rbd-mirror) ------------------- 10.98s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:179 -------",
2021-07-28 03:03:25 |         "gather and delegate facts ----------------------------------------------- 8.77s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:73 --------",
2021-07-28 03:03:25 |         "ceph-config : create ceph initial directories --------------------------- 5.25s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-config/tasks/create_ceph_initial_dirs.yml:2 ",
2021-07-28 03:03:25 |         "ceph-infra : install chrony --------------------------------------------- 4.37s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-infra/tasks/setup_ntp.yml:15 ---------------",
2021-07-28 03:03:25 |         "ceph-validate : get devices information --------------------------------- 3.83s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-validate/tasks/check_devices.yml:80 --------",
2021-07-28 03:03:25 |         "Gathering Facts --------------------------------------------------------- 2.87s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:17 --------",
2021-07-28 03:03:25 |         "Gathering Facts --------------------------------------------------------- 2.37s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:112 -------",
2021-07-28 03:03:25 |         "ceph-validate : read information about the devices ---------------------- 2.09s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-validate/tasks/check_devices.yml:59 --------",
2021-07-28 03:03:25 |         "ceph-facts : find a running mon container ------------------------------- 2.03s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-facts/tasks/facts.yml:72 -------------------",
2021-07-28 03:03:25 |         "ceph-facts : resolve device link(s) ------------------------------------- 1.94s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-facts/tasks/facts.yml:202 ------------------",
2021-07-28 03:03:25 |         "ceph-infra : enable chronyd --------------------------------------------- 1.91s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-infra/tasks/setup_ntp.yml:46 ---------------",
2021-07-28 03:03:25 |         "ceph-facts : find a running mon container ------------------------------- 1.86s",
2021-07-28 03:03:25 |         "stop ceph mon - shortname ----------------------------------------------- 1.69s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:199 -------",
2021-07-28 03:03:25 |         "ceph-validate : resolve devices in devices ------------------------------ 1.67s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-validate/tasks/check_devices.yml:41 --------",
2021-07-28 03:03:25 |         "unmask the mon service -------------------------------------------------- 1.59s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:270 -------",
2021-07-28 03:03:25 |         "gather facts ------------------------------------------------------------ 1.56s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:65 --------",
2021-07-28 03:03:25 |         "ensure /var/lib/ceph/bootstrap-rbd-mirror is present -------------------- 1.52s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:166 -------",
2021-07-28 03:03:25 |         "ceph-facts : get current fsid if cluster is already running ------------- 1.50s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/roles/ceph-facts/tasks/facts.yml:135 ------------------",
2021-07-28 03:03:25 |         "get ceph cluster status ------------------------------------------------- 1.43s",
2021-07-28 03:03:25 |         "/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml:150 -------"
2021-07-28 03:03:25 |     ],
2021-07-28 03:03:25 |     "failed_when_result": true
2021-07-28 03:03:25 | }
2021-07-28 03:03:25 | 

Version-Release number of selected component (if applicable):
13 to 16.1

Steps to Reproduce:
1. deploy ffu 13 to 16.1
2. failing at ceph upgrade
3.

Actual results:
ceph upgrade failed

Expected results:
ceph upgrade pass
Additional info:

Note You need to log in before you can comment on or make changes to this bug.