Bug 1576744
| Summary: | embedded ansible starts only after restart first time | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat CloudForms Management Engine | Reporter: | Ievgen Zapolskyi <izapolsk> | ||||
| Component: | cfme-openshift-embedded-ansible | Assignee: | Nick Carboni <ncarboni> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Ievgen Zapolskyi <izapolsk> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 5.9.0 | CC: | cpelland, izapolsk, jrafanie, lavenel, simaishi | ||||
| Target Milestone: | GA | Flags: | izapolsk:
automate_bug+
|
||||
| Target Release: | 5.10.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | 5.10.0.4 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-02-07 22:45:46 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | Bug | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Ievgen Zapolskyi
2018-05-10 10:12:30 UTC
I'm not actually sure this is an issue. At startup the embedded ansible pod has to migrate its database which can take a long time. The timing of this is heavily dependent on environment and latency to the database pod so I could see this behavior happening if the database were deployed on a separate node from the embedded ansible pod. How long is it before the initial pod is rescheduled? Maybe we can change that timeout value. Nick, I use empty openshift env. So, there is no heavy load which may influence pod deployment. DB is deployed on the same node with the rest of appliance pods. And I anyway have pod deployment restarted. I do agree that the issue is in readiness/liveness probs if there is no timeout check from app pod. So, I believe we have to correct timeout values. As a fix for this I'm going to remove the health checks from the embedded ansible deployment config and introduce a timeout into the setup time that we can increase. We already do health checking in the EmbeddedAnsibleWorker so adding the checks in OpenShift was really overkill in this case. Unfortunately, this won't really be something that we can backport to 5.9 because we are actually relying on the OpenShift health checks in that version. If this "fix" needs to be backported we'll have to investigate if we can increase the readiness/liveness initial timeouts. the issue is still present in 5.10.0.1. ansible deploy pod fails because ansible pod doesn't get ready on time. As a result I have to redeploy pod manually. New commits detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/2a766b2999141a7ce4a4972d7ea2fefb754bbb07 commit 2a766b2999141a7ce4a4972d7ea2fefb754bbb07 Author: Nick Carboni <ncarboni> AuthorDate: Mon Jun 18 15:12:46 2018 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Mon Jun 18 15:12:46 2018 -0400 Add a timeout to docker and container Embedded Ansible startup Previously we would wait forever. That's never good. https://bugzilla.redhat.com/show_bug.cgi?id=1576744 config/settings.yml | 2 + lib/embedded_ansible/container_embedded_ansible.rb | 2 +- lib/embedded_ansible/docker_embedded_ansible.rb | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) https://github.com/ManageIQ/manageiq/commit/572f295173cbd69d6c2a6a4d062ebc0a08197d12 commit 572f295173cbd69d6c2a6a4d062ebc0a08197d12 Author: Nick Carboni <ncarboni> AuthorDate: Mon Jun 18 15:13:31 2018 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Mon Jun 18 15:13:31 2018 -0400 Remove the health checks from ContainerEmbeddedAnsible We already do health checks from the EmbeddedAnsibleWorker so we don't really need OpenShift to do more, especially if they have small built-in timeouts. https://bugzilla.redhat.com/show_bug.cgi?id=1576744 lib/embedded_ansible/container_embedded_ansible.rb | 27 +- 1 file changed, 5 insertions(+), 22 deletions(-) Verified in 5.10.0.4 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0213 |