Bug 1608262
| Summary: | guest-host synchronization does not work | ||
|---|---|---|---|
| Product: | [Retired] Restraint | Reporter: | Jan Tluka <jtluka> |
| Component: | general | Assignee: | Martin Styk <mastyk> |
| Status: | CLOSED NEXTRELEASE | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 0.1.32 | CC: | aloughla, asavkov, bpeck, ddumas, jburke, mastyk, olichtne, osabart, pholica, rkhan, rpotts, sukulkar |
| Target Milestone: | 0.1.37 | Keywords: | TestBlocker |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-01-11 08:05:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Jan Tluka
2018-07-25 08:23:43 UTC
If I'm not mistaken, restraint uses staf for synchronization. Unfortunately staf is not its dependency and is not even in any RHEL compose. (In reply to Pavel Holica from comment #1) > If I'm not mistaken, restraint uses staf for synchronization. Unfortunately > staf is not its dependency and is not even in any RHEL compose. Restraint does not use staf anymore. Quote from an email on beaker-user-list: "The latest versions of restraint (0.1.25 and newer) have dropped staf in favor of internal syncing methods: https://git.beaker-project.org/cgit/restraint/commit/?id=702f32a7d160" (In reply to Jan Tluka from comment #2) > Restraint does not use staf anymore. > > Quote from an email on beaker-user-list: > > "The latest versions of restraint (0.1.25 and newer) have dropped staf in > favor of internal syncing methods: > https://git.beaker-project.org/cgit/restraint/commit/?id=702f32a7d160" Oh, I didn't know that, thanks! Hi Roman, could you please justify what future_maint means and when can I expect this is fixed? It is currently a test blocker for my team. Thanks, Jan Dear Jan, I'm sorry for the non-explanatory switch of fields there. The field 'future_maint' means, that it goes right into our column of next scheduled work items. Furthermore, it is on top of the column, so whoever is picking up the next item will pick this one. Let me know if this is helpful or further explanation is needed. (In reply to Roman Joost from comment #5) > Dear Jan, > > I'm sorry for the non-explanatory switch of fields there. > > The field 'future_maint' means, that it goes right into our column of next > scheduled work items. Furthermore, it is on top of the column, so whoever is > picking up the next item will pick this one. > > Let me know if this is helpful or further explanation is needed. Thanks! No need to explain more. Looking forward to see it in next update. Hi Roman, Would you be able to provide your plans to address this one? Unfortunately, it blocks critical testing that we have to do in our RHEL8 LNST CI. Thanks! Sushil Hi Sushil, I'm currently working on reproducing this bug in our test environment in order to make it simpler to replicate and isolate the issue, but also to incorporate it into our self tests. Once I've isolated the problem in the restraint harness I'll update the bug with further information. Regards, Matt. If you have an example of this test working using beah, could you please link it? Cheers, Matt. Thanks Matt for the update..Unfortunately, Jan is out this week on PTO. Ondrej, Aniss, Are you able to help out? Thanks! Sushil Looking at our job history, this one looks similar: https://beaker.engineering.redhat.com/jobs/2696379 It's running RHEL7 and beah and has the same configuration of guests on hosts, also using the /kernel/distribution/lnst/prepare-testbed task for synchronization. I don't fully understand the mechanism and the issue here but I'll try to help out if needed. -Ondrej Thanks very much. I've replicated this in my test environment. I'll update when I know more. (In reply to Matt Tyson from comment #12) > Thanks very much. I've replicated this in my test environment. I'll update > when I know more. Hi Matt, any updates? Automated testing completed blocked for NST SST This is basically the same as Beah bug 887283 which I fixed a long time ago. The issue with guest recipes is that the hostname of each guest is not known in advance, until they are finished installing and they check in with Beaker to report which IP address (and thus FQDN) they have been assigned. This is in contrast to normal ("host") recipes where the scheduler assigns a machine from Beaker's inventory in advance so the FQDN is always known from the very beginning. The solution in Beah is that we needed some extra code to refresh the set of hostnames in each role at the beginning of each task: https://git.beaker-project.org/cgit/beah/commit/?id=cba2c2e8e29d40b27b96ac478cdac1d81957b7aa So the expected behaviour is that the initial tasks in the host recipe will be run with "None" as the FQDN for the guests (this is what you get today) but once the guests are finished installing any tasks from that point will get the correct FQDNs populated in RECIPE_MEMBERS and other role environment variables. Currently Restraint just populates those variables based on the original recipe XML it fetched or was given, just like Beah did. So we will need a similar patch to add an extra call at the start of each task. Hi Dan, thanks for looking into this. I just wanted to let you know that we have just created a workaround for this case. Though I'd like to see this fixed I don't want to take you away from other important stuff. Our workaround is to simply fetch the job details from the Beaker server and parse out the roles and fqdn's from it and use this data to override the environment variables for the roles. Details at: http://pkgs.devel.redhat.com/cgit/tests/kernel/commit/?h=kernel-distribution-lnst-prepare-testbed-1_0-7&id=a505c719b4b6071ab6c8c061e74d8a891bc98fa1 http://pkgs.devel.redhat.com/cgit/tests/kernel/commit/distribution/lnst/prepare-testbed?id=226893f39b5be4d6d2226f7ca71505dde04aaea4 I am working on a patch for this but it requires a bit of refactoring (the recipe XML fetch + parse code is currently tied to the global app_data state which makes it difficult to re-use). Sadly this stuff is not a one-liner like it would be in Python... https://gerrit.beaker-project.org/#/c/restraint/+/6304 refactor XML parsing to separate it from global recipe state https://gerrit.beaker-project.org/#/c/restraint/+/6305 refresh roles at the start of every task I am using this as my reproducer: https://beaker-devel.app.eng.bos.redhat.com/jobs/15335 The job is structured as: RHEL7 host SERVERS /distribution/install /distribution/virt/install /distribution/virt/start /distribution/beaker/Sanity/sync-set_block-tests RHEL7 guest CLIENTONE /distribution/install /distribution/dummy /distribution/dummy /distribution/beaker/Sanity/sync-set_block-tests RHEL7 guest CLIENTTWO /distribution/install /distribution/dummy /distribution/dummy /distribution/beaker/Sanity/sync-set_block-tests You can see the bug in the local watchdog timeout on the host recipe. It is trying to sync with "None" which is not the real guest FQDN. From taskoutput.log: :: [ 04:08:44 ] :: [ BEGIN ] :: Running 'rhts-sync-block -s READY None None' Tried the patch in my dev environment and it works. Yay! This bug fix is included in restraint-0.1.36-1.git.8.d88f913 which is currently available here: https://beaker-project.org/nightlies/harness/ Restraint 0.1.37 has been released. |