Bug 1620699
| Summary: | [docs] Major upgrade fails while checking OSD health and timing out | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | PURANDHAR SAIRAM MANNIDI <pmannidi> |
| Component: | documentation | Assignee: | Laura Marsh <lmarsh> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | RHOS Documentation Team <rhos-docs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | cminkema, gfidente, jbiao, johfulto, lmarsh, mburns, slinaber, srevivo, tshefi |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | Flags: | tshefi:
automate_bug-
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-26 16:04:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
PURANDHAR SAIRAM MANNIDI
2018-08-23 10:39:56 UTC
This does not look like a bug to me. Depending on the infra the timings should be adjusted. We might need to include this in the documentation. Aren't those values be configured via some variables rather than changing the default ansible scripts? You can run ansible with an extra var like -e health_mon_check_retries=200 and this will work without editing the playbook file. changing component from ceph-ansible to tripleo-common for the same changes to be done in mistral workbook if it should be set there. Please update chapter 4.6 and 4.7 of the overcloud upgrade document [1] to include the following note.
"""
During the migration of Ceph to containers, each Ceph monitor and OSD is brought down sequentially and then the migration does not continue until the same service service that was brought down, is successfully brought back up. Ansible will wait 15 seconds (the delay) and recheck 5 times (the retries) for the service to come back and if the service does not come back the migration will stop so that the operator may intervene. Depending on the size of your Ceph cluster, the retry or delay values may need to be increased. The exact names of these parameters and their defaults are as follows:
health_mon_check_retries: 5
health_mon_check_delay: 15
health_osd_check_retries: 5
health_osd_check_delay: 15
For example, to have the cluster recheck 30 times and wait 40 seconds between each check, pass the following parameters in a yaml file with a -e to the 'openstack overcloud deploy' command.
parameter_defaults:
CephAnsibleExtraConfig:
health_osd_check_delay: 40
health_osd_check_retries: 30
"""
[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html/upgrading_red_hat_openstack_platform/assembly-preparing_for_overcloud_upgrade#preparing_for_ceph_storage_node_upgrades
Doc bug, Nothing for QE to test/automate with regards to close loop process. |