Problem Description: Define tests, including gating tests, for swtpm. Additional Information: In RHEL-8 and in RHEL AV, the virt module contains the definition of tests to be run whenever any component in the module is changed. In RHEL-9 there is no virt module so the tests must now be defined for each of the individual components, of which swtpm is one. The definition must include both gating and non-gating tests. The suggested approach to resolving this BZ is to copy the test setup for the virt module in RHEL AV-8.5, namely the file named 'gating.yaml' and the directory named 'tests'. Then modify the files to exclude tests and gating tests that are not relevant for swtpm. Please work closely with QE! The test definitions in gating.yaml describe test jobs that run in QE's testing environment. The test jobs and/or the testing environment may need to be modified to work in RHEL-9, or perhaps brand new test jobs may need to be created.
Delegating for further assignment. I don't expect maintainers to do all the work, but to help move things along. Please work with the following: * Yash - he knows how to create the test and gating files that OSCI reads to run tests. He also knows how the QE test systems work. * lead engineer for the component (I'll help if you don't know who) - for reviewing what tests make sense to run for the component. * QE assignee - for verifying that the jobs we run which execute on QE's infrastructure will run on RHEL-9 in a non-modular environment. I expect that some tests may need modification.
Reassigning to the package maintainer. In case of doubt, as mentioned by Jeff, you can also discuss the process with Beraldo as he has been doing that for other components as well.
What's gating.yaml & tests directory doing? From packaging point of view, we have the regular "make check" tests only. QA has integration tests, I suppose using autotest. I don't know where/how they are gating the build. Beraldo, can you help figure this out?
Hi Marc-Andre, sure. OSCI is listening for UMB messages that come from build (ie: brew, koji, copr) and CI systems. Greenwave listens for some events on UMB as well and based on those events and the gating criteria (gating.yaml file inside dist-git) it will trigger Pass or Fail. By default OSCI adds some basic tests as gating, even when you don't have tests defined. But on top of that, your gating.yaml file should have some tests defined as "gating". Something like qemu-kvm is doing: https://gitlab.com/redhat/centos-stream/rpms/qemu-kvm/-/blob/c9s/gating.yaml Ideally your test systems should be publishing the results to a topic on UMB, so you could use those topic references on your gating.yaml. You can see the qemu-kvm example above. Please, feel free to ping me if you have any questions here. -- Beraldo
The qemu gating file: # recipients: kvmqe-ci, yfu --- !Policy product_versions: - rhel-9 decision_context: osci_compose_gate subject_type: brew-build rules: - !PassingTestCaseRule {test_case_name: kvm-ci.qemu-kvm.x86_64-intel.brew-build.gating.tier1.functional} - !PassingTestCaseRule {test_case_name: kvm-ci.qemu-kvm.x86_64-amd.brew-build.gating.tier1.functional} It seems like I could that as template / boilerplate. - What to put for recipients? - What test_case_name? The documentation I could find on gating is scarce. (https://docs.fedoraproject.org/en-US/rawhide-gating/optin/#_configure_gating) Is there a more complete doc elsewhere? Beraldo, do you know?
(In reply to Marc-Andre Lureau from comment #5) > The qemu gating file: > > # recipients: kvmqe-ci, yfu > --- !Policy > product_versions: > - rhel-9 > decision_context: osci_compose_gate > subject_type: brew-build > rules: > - !PassingTestCaseRule {test_case_name: > kvm-ci.qemu-kvm.x86_64-intel.brew-build.gating.tier1.functional} > - !PassingTestCaseRule {test_case_name: > kvm-ci.qemu-kvm.x86_64-amd.brew-build.gating.tier1.functional} > > It seems like I could that as template / boilerplate. > > - What to put for recipients? By default the gating notifications are send to the build issuer. If you want to extend this notification list provide additional recipients, in form of comma delimited kerberos user names (or mailing lists), without the @redhat suffix. > - What test_case_name? AFAICT, you should add here a list of "namespace" messages (sent to the internal message bus, UMB) that your build/CI system generates. > The documentation I could find on gating is scarce. > (https://docs.fedoraproject.org/en-US/rawhide-gating/optin/ > #_configure_gating) Is there a more complete doc elsewhere? > > Beraldo, do you know? Yes, I do have a few links, but since they are talking about internal RH infrastructure, they are private. I will send in pvt. -- Beraldo
https://gitlab.com/redhat/centos-stream/rpms/swtpm/-/merge_requests/3
(In reply to Marc-Andre Lureau from comment #8) > https://gitlab.com/redhat/centos-stream/rpms/swtpm/-/merge_requests/3 Hi Marc-Andre, thank you for this. However, do you have the list of tests those functional tier1 are running? I'm asking this because gave me the impression that any broken test there (let's say with qemu) will impact this component. Unless I'm missing something, ideally we should have specific component's tests here. -- Beraldo
I agree with Beraldo; I would drop the pull request mentioned in comment 8. Running the qemu-kvm tests for swtpm is more of a last-resort approach, and it is not at all clear if those tests would actually exercise swtpm anyway. This [1] gives a reasonable overview of how testing works. It's written for Fedora but it applies to RHEL too. I would look at the sections that follow as they describe test automation [2], and gating tests [3], respectively. Next, I would review Chapter 12 of the RHEL Developer Guide, specifically section 12.5. The steps in section 12.5.3 describe in detail how to define tests, enable them, and then select which ones are to be used as gating tests. >For vTPM packages: >- What should be used for "test_case_name"? I think you'll have to write something specific for swtpm. >- How to test it? Start simple. Follow the RHEL Developer Guide. For example, section 12.5.1 says, "To learn how to invoke, create, or change tests, refer to the guides in Fedora or contact the OSCI team to help you get started. A minimal example can be found in this email thread." I would copy that simple example, or I would use one of the examples from the Fedora guide [1]. Once you have the simple test defined, you can then create the gating.yaml file to enable it as a gating test. When the basic environment is set up and you know it works, you can take the next step: work with QE to define additional tests that should be run and add them to the test description. Select which ones (if any) should be considered gating. [1] https://docs.fedoraproject.org/en-US/ci/tests/ [2] https://docs.fedoraproject.org/en-US/ci/pipeline/ [3] https://docs.fedoraproject.org/en-US/ci/gating/
(In reply to Marc-Andre Lureau from comment #8) > https://gitlab.com/redhat/centos-stream/rpms/swtpm/-/merge_requests/3 Hi, The test_case_name is: libvirt-ci.swtpm.brew-build.gating.x86_64.tier1.functional Please help update the patch accordingly, thanks! Test results: https://datagrepper.engineering.redhat.com/id?id=ID:messaging-devops-broker01.web.prod.ext.phx2.redhat.com-35024-1621926610945-8:136639:-1:1:1&is_raw=true&size=extra-large https://resultsdb-frontend.host.prod.eng.bos.redhat.com/results?&testcases=libvirt-ci.swtpm.brew-build.gating.x86_64.tier1.functional (Just for reference, cases failed by other issue, under fixing.)
(In reply to yanqzhan from comment #19) > The test_case_name is: > libvirt-ci.swtpm.brew-build.gating.x86_64.tier1.functional > Please help update the patch accordingly, thanks! done, thanks
The MR was merged. And job can pass now. https://datagrepper.engineering.redhat.com/id?id=ID:messaging-devops-broker01.web.prod.ext.phx2.redhat.com-35024-1621926610945-8:273924:-1:1:1&is_raw=true&size=extra-large https://resultsdb-frontend.host.prod.eng.bos.redhat.com/results?&testcases=libvirt-ci.swtpm.brew-build.gating.x86_64.tier1.functional Mark as verified.
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.