Bug 923637 - beaker-wizard: Update skeletons with parametrization
Summary: beaker-wizard: Update skeletons with parametrization
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: command line
Version: 0.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 22.2
Assignee: Iveta Wiedermann
QA Contact:
URL:
Whiteboard: BeakerWizard
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-20 09:16 UTC by Petr Šplíchal
Modified: 2018-02-06 00:41 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-15 01:51:13 UTC
Embargoed:


Attachments (Terms of Use)
proposed patch (1.91 KB, patch)
2016-01-06 13:24 UTC, Iveta Wiedermann
no flags Details | Diff
proposed patch (2.49 KB, patch)
2016-01-15 16:07 UTC, Iveta Wiedermann
no flags Details | Diff

Description Petr Šplíchal 2013-03-20 09:16:33 UTC
Description of problem:

Currently the test skeleton created by beaker-wizard does not
expect the task to be parametrized. For example to be able to run
the same test against different versions of python:

 * system python (python)
 * parallel install (python3)
 * software collection (python27-python)

The default template should be modified to allow asserting not
only the fixed package name provided in the PACKAGE variable. This
is now easy as BeakerLib has a fresh support for rlAssertRpm --all.

I suggest to provide the following default template:

    # Include Beaker environment
    . /usr/bin/rhts-environment.sh || exit 1
    . /usr/share/beakerlib/beakerlib.sh || exit 1

    # Packages to be tested
    PACKAGES=${PACKAGES:-<run-for>}
    # Other required packages
    REQUIRES=${REQUIRES:-<additional-requires>}

    rlJournalStart
        rlPhaseStartSetup
            rlAssertRpm --all
            rlRun "TMP=\$(mktemp -d)" 0 "Creating tmp directory"
            rlRun "pushd $TMP"
        rlPhaseEnd

        rlPhaseStartTest
            rlRun "touch foo" 0 "Creating the foo test file"
            rlAssertExists "foo"
            rlRun "ls -l foo" 0 "Listing the foo test file"
        rlPhaseEnd

        rlPhaseStartCleanup
            rlRun "popd"
            rlRun "rm -r $TMP" 0 "Removing tmp directory"
        rlPhaseEnd
    rlJournalPrintText
    rlJournalEnd

Where <run-for> will be substituted with the space-separated list
of packages set in the RunFor and <additional-requires> with the
rest of required packages (not duplicating those in RunFor).

Let's also provide a new skeleton called 'parametrized' which will
add other convenience features useful for testing components with
non-fixed binaries. The new skeleton should look like this when
'python' is provided as the RunFor component:

    # Include Beaker environment
    . /usr/bin/rhts-environment.sh || exit 1
    . /usr/share/beakerlib/beakerlib.sh || exit 1

    # Packages to be tested
    PACKAGES=${PACKAGES:-python}
    # Other required packages
    REQUIRES=${REQUIRES:-}
    # Binary name parametrized
    PYTHON=${PYTHON:-python}
    
    rlJournalStart
        rlPhaseStartSetup
            rlAssertRpm --all
            rlAssertBinaryOrigin $PYTHON
            rlRun "TMP=\$(mktemp -d)" 0 "Creating tmp directory"
            rlRun "pushd $TMP"
        rlPhaseEnd
    
        rlPhaseStartTest
            rlRun -l "$PYTHON --version" 0 "Query binary version"
        rlPhaseEnd
    
        rlPhaseStartCleanup
            rlRun "popd"
            rlRun "rm -r $TMP" 0 "Removing tmp directory"
        rlPhaseEnd
    rlJournalPrintText
    rlJournalEnd

Note that rlAssertBinaryOrigin is used to make sure that the
binary under test is owned by one of rpms listed in PACKAGES.
This is especially useful when testing software collections.

Comment 2 Roman Joost 2015-11-03 05:25:14 UTC
Dear Iveta,

I've moved this item back to NEW since it is in ASSIGNED since 2013. Are you still working on it? Do you have problems? Can we help?

Comment 3 Iveta Wiedermann 2015-11-03 07:45:48 UTC
Hello Roman,

I'm sorry. When I started to work on this there has been raised some questions regarding of dynamic test requires and this was huge topic for several months which led to unclear solution for this bug. I will bring this up for discussion and then decide next steps.

Comment 4 Petr Šplíchal 2015-11-25 10:20:42 UTC
Update from today's Test Life Cycle meeting: Instead of modifying
the default skeleton let's use the one below as a new skeleton
called "parametrized" and leave the default skeleton unmodified.

    # Include Beaker environment
    . /usr/bin/rhts-environment.sh || exit 1
    . /usr/share/beakerlib/beakerlib.sh || exit 1

    # Packages to be tested
    PACKAGES=${PACKAGES:-<run-for>}
    # Other required packages
    REQUIRES=${REQUIRES:-<additional-requires>}

    rlJournalStart
        rlPhaseStartSetup
            rlAssertRpm --all
            rlRun "TMP=\$(mktemp -d)" 0 "Creating tmp directory"
            rlRun "pushd $TMP"
        rlPhaseEnd

        rlPhaseStartTest
            rlRun "touch foo" 0 "Creating the foo test file"
            rlAssertExists "foo"
            rlRun "ls -l foo" 0 "Listing the foo test file"
        rlPhaseEnd

        rlPhaseStartCleanup
            rlRun "popd"
            rlRun "rm -r $TMP" 0 "Removing tmp directory"
        rlPhaseEnd
    rlJournalPrintText
    rlJournalEnd

Comment 5 Iveta Wiedermann 2016-01-06 13:24:04 UTC
Created attachment 1112145 [details]
proposed patch

Proposed patch - adding parametrized skeleton and mentioning it in doc as available skeleton type

Comment 6 Roman Joost 2016-01-07 00:13:20 UTC
Patch has been moved to gerrit and is open for review:

https://gerrit.beaker-project.org/#/c/4570/1

Comment 7 Roman Joost 2016-01-11 01:14:49 UTC
There are questions with this patch:

* beaker-wizard currently does not accept multiple packages to be passed (e.g. -p argument). Therefore <runfor /> would always be a single package. Is that the desired behaviour?

* There is currently no substitution for additional-requires. Is it perhaps a mistake and `requires` should be used or an additional substitution added?

Updated patch can be found here:

https://gerrit.beaker-project.org/#/c/4570/

Comment 8 Iveta Wiedermann 2016-01-11 13:52:48 UTC
Hello Roman,

 * yes, <runfor /> is one default package. The variable PACKAGES is filled in during job scheduling or if there is need to have there more packages, it's modified in a test.

 * hm, I left REQUIRES blank intentionally as this is mainly used in software collections and is also filled in during job scheduling. What does "additional-requires" contain? (or should contain?)

Comment 9 Roman Joost 2016-01-14 03:57:10 UTC
Dear Iveta,

I thought it would contain additional required packages as outlined in comment 4 by Petr. If it's not needed I can remove it again?

Comment 10 Iveta Wiedermann 2016-01-14 07:53:18 UTC
Yes, I can see that. You know what, I'll discuss it with Petr to be really sure. Hopefully he will be recovered from illness soon.

Comment 11 Petr Šplíchal 2016-01-15 15:06:41 UTC
Yes, both PACKAGES and REQUIRES should be pre-filled. Using
<runfor/> and <requires/> seems to be the best option here.

By the way, it is possible to provide several package for both
required and runfor packages:

$ cd /tmp
$ beaker-wizard -yf -o a,b,c -q x,y,z

/CoreOS/tmp/Sanity/a-few-descriptive-words

...

Directory Sanity/a-few-descriptive-words created
File Sanity/a-few-descriptive-words/PURPOSE written
File Sanity/a-few-descriptive-words/runtest.sh written
File Sanity/a-few-descriptive-words/Makefile written

$ egrep 'RunFor|Requires' Sanity/a-few-descriptive-words/Makefile
    @echo "RunFor:          a b c tmp" >> $(METADATA)
    @echo "Requires:        tmp x y z" >> $(METADATA)

Comment 13 Iveta Wiedermann 2016-01-15 16:07:51 UTC
Created attachment 1115186 [details]
proposed patch

I've added <requires/> to fill in REQUIRES variable and adjusted runfor and requires variables so it's space separated in the skeleton.

Comment 14 Dan Callaghan 2016-01-18 06:52:09 UTC
(In reply to Petr Šplíchal from comment #11)
> $ beaker-wizard -yf -o a,b,c -q x,y,z

Okay, let's make sure the beaker-wizard test cases cover this functionality (passing multiple package names separated by commas) because I think none of us noticed that this exists. :-)

Comment 15 Iveta Wiedermann 2016-03-02 10:35:15 UTC
Dan, is there anything else needed from our side?

Thanks.

Comment 16 Dan Callaghan 2016-03-02 23:39:43 UTC
(In reply to Iveta Wiedermann from comment #15)
> Dan, is there anything else needed from our side?

No nothing else is needed I think. Roman was working on this before he went on PTO and we were still just hammering out the right test cases on Gerrit before we merge the patch. I should have picked this up while he was gone, but I forgot, sorry about that.

Comment 17 Dan Callaghan 2016-03-08 03:54:34 UTC
This bug fix is included in beaker-client-22.2-0.git.19.07eb8fd which is currently available for download here:

https://beaker-project.org/nightlies/release-22/

Comment 19 Petr Šplíchal 2016-03-11 07:52:45 UTC
Note to verification: The main point of this bug was to introduce
a new skeleton named "parametrized" with the template containing
parametrization stuff. Comment #18 seems to verify only support
for multiple packages provided in RunFor and Requires.

Comment 20 Dan Callaghan 2016-03-14 06:37:51 UTC
Yeah I guess a better verification would be:

1. Run: beaker-wizard -yf -o a,b,c -q x,y,z -s parametrized
2. Check Sanity/a-few-descriptive-words/runtest.sh

# Packages to be tested
PACKAGES=${PACKAGES:-a asdf b c}
# Other required packages
REQUIRES=${REQUIRES:-asdf x y z}

Comment 21 Dan Callaghan 2016-03-15 01:51:13 UTC
Beaker 22.2 has been released.


Note You need to log in before you can comment on or make changes to this bug.