Description of problem: In 4.1 (ceph-ansible-4.0.23-1.el8cp.noarch) we added a check to fail when either dashboard_admin_password or grafana_admin_password was not explicitly set. In 4.0 (ceph-ansible-4.0.14-1.el8cp.noarch) these defaults were set: #dashboard_admin_password: p@ssw0rd #grafana_admin_password: admin When upgrading from RHCS 4.0 where the above defaults were used the check causes a failure to fully upgrade (see Bug 1844852). If you uncomment those settings and use the defaults everything proceeds fine. If you set dashboard_admin_password to a new password and just uncomment grafana_admin_password and use the default for grafana, everything proceeds fine. If you set grafana_admin_password to a new password, the following failure is seen: - When I set "grafana_admin_password: redhat" ``` fatal: [ceph4osd1 -> ceph4osd1]: FAILED! => changed=false cmd: - docker - exec - ceph-mon-ceph4osd1 - ceph - --cluster - ceph - dashboard - grafana - dashboards - update delta: '0:00:01.177480' end: '2020-06-07 11:46:52.813923' msg: non-zero return code rc: 22 start: '2020-06-07 11:46:51.636443' stderr: |- Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 974, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/dashboard/module.py", line 386, in handle_command push_local_dashboards() File "/usr/share/ceph/mgr/dashboard/grafana.py", line 133, in push_local_dashboards retry() File "/usr/share/ceph/mgr/dashboard/grafana.py", line 90, in __call__ result = self.func(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/grafana.py", line 128, in push grafana.push_dashboard(body) File "/usr/share/ceph/mgr/dashboard/grafana.py", line 55, in push_dashboard response.raise_for_status() File "/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://7.7.7.101:3000/api/dashboards/db - when I set "grafana_admin_password: admin" it is fine. Additionally, after upgrade if you try to change grafan_admin_password, it fails with the same issue as above. See the following original reports: This is an outgrowth of https://bugzilla.redhat.com/show_bug.cgi?id=1844852#c2 https://bugzilla.redhat.com/show_bug.cgi?id=1844852#c8 Version-Release number of selected component (if applicable): RHCS 4.0 cluster upgrade to RHCS 4.1 ceph-ansible-4.0.14-1.el7cp.noarch ceph-ansible-4.0.23-1.el8cp.noarch How reproducible: Only tested on Mustafa's test cluster Steps to Reproduce: 1. see above 2. 3. Actual results: Expected results: Additional info: