Bug 2027563 - e2e/add-flow-ci.feature fix accessibility violations
Summary: e2e/add-flow-ci.feature fix accessibility violations
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Dev Console
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.10.0
Assignee: Jan Richter
QA Contact: spathak@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-30 04:06 UTC by David Taylor
Modified: 2022-03-10 16:31 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:31:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift console pull 10583 0 None open Bug 2027563: Wait for developer catalog to load in e2e-console cypress tests 2021-12-01 12:08:03 UTC
Github openshift console pull 10654 0 None open Bug 2027563: e2e tests wait for add forms to load 2021-12-13 13:49:28 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:31:22 UTC

Description David Taylor 2021-11-30 04:06:00 UTC
The following accessibility violations need to be addressed:

https://search.ci.openshift.org/?search=accessibility+violation+was+detected&maxAge=336h&context=1&type=junit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job


  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    8.5.0                                                                              │
  │ Browser:    Chrome 94 (headless)                                                               │
  │ Specs:      1 found (e2e/add-flow-ci.feature)                                                  │
  │ Searched:   features/e2e/add-flow-ci.feature                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  e2e/add-flow-ci.feature                                                         (1 of 1)
Couldn't determine Mocha version


  Logging in as kubeadmin
  Create the different workloads from Add page
    ✓ Getting started resources on Developer perspective (16626ms)
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    1) Deploy Application using Catalog Template "CI/CD": A-01-TC02 (example #1)
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    2) Deploy Application using Catalog Template "Databases": A-01-TC02 (example #2)
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    3) Deploy Application using Catalog Template "Languages": A-01-TC02 (example #3)
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    4) Deploy Application using Catalog Template "Middleware": A-01-TC02 (example #4)
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    5) Deploy Application using Catalog Template "Other": A-01-TC02 (example #5)
1 accessibility violation was detected for Deploy Page
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    6) Deploy secure image with Runtime icon from external registry: A-02-TC02 (example #1)
1 accessibility violation was detected for Deploy Page
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    7) Deploy image with Runtime icon from internal registry: A-02-TC03 (example #1)
1 accessibility violation was detected for Deploy Page
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    8) Edit Runtime Icon while Editing Image: A-02-TC05
1 accessibility violation was detected for Developer Catalog
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    9) Create the Database from Add page: A-03-TC01
    10) Deploy git workload with devfile from topology page: A-04-TC01
1 accessibility violation was detected for Import from Docker file
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    11) Create a workload from Docker file with "Deployment" as resource type: A-05-TC02 (example #1)
    12) Create a workload from YAML file: A-07-TC01
1 accessibility violation was detected for Upload JAR file
┌─────────┬────────┬───────────┬───────────────────────────────────────────────┬───────┐
│ (index) │   id   │  impact   │                  description                  │ nodes │
├─────────┼────────┼───────────┼───────────────────────────────────────────────┼───────┤
│    0    │ 'list' │ 'serious' │ 'Ensures that lists are structured correctly' │   1   │
└─────────┴────────┴───────────┴───────────────────────────────────────────────┴───────┘
    13) Upload Jar file page details: A-10-TC01


  1 passing (3m)
  13 failing

Comment 2 Jan Richter 2021-11-30 12:44:25 UTC
From the cypress recordings, it all seems to go south when developer catalog takes some time to load. Let me see if adding a simple wait solves this.

Comment 4 David Taylor 2021-11-30 14:11:57 UTC
Hi jrichter, I understand the desire to add a simple wait, but it is typically better to strive for a deterministic solution:  https://docs.cypress.io/guides/references/best-practices#Unnecessary-Waiting.  thanks

Comment 5 Jan Richter 2021-11-30 14:23:39 UTC
(In reply to David Taylor from comment #4)
> Hi jrichter, I understand the desire to add a simple wait, but it
> is typically better to strive for a deterministic solution: 
> https://docs.cypress.io/guides/references/best-practices#Unnecessary-Waiting.
> thanks

The simple part is just adding a conditional wait that we already use to wait for pages to load. The wait itself is not that 'simple', sorry for the confusion :)

Comment 7 David Taylor 2021-12-10 14:27:38 UTC
Hi, still seeing this flake, CI success rate is currently at 0% partially due to this issue!
Indeterminate waiting is not reliable to squash flakes like this.  Best to wait on an assertion such as url should equal, or a 'cy.contains'
Thanks,
Dave

# Create the different workloads from Add page.Create the different workloads from Add page Deploy image with Runtime icon from internal registry: A-02-TC03 (example #1)
AssertionError: 1 accessibility violation was detected: expected 1 to equal 0
    at Context.eval (https://console-openshift-console.apps.ci-op-lb7gpwtl-75d12.**********************/__cypress/tests?p=support/commands/index.ts:10178:20)

Comment 8 Jan Richter 2021-12-10 15:26:50 UTC
(In reply to David Taylor from comment #7)
> Hi, still seeing this flake, CI success rate is currently at 0% partially
> due to this issue!
> Indeterminate waiting is not reliable to squash flakes like this.  Best to
> wait on an assertion such as url should equal, or a 'cy.contains'
> Thanks,
> Dave
> 
> # Create the different workloads from Add page.Create the different
> workloads from Add page Deploy image with Runtime icon from internal
> registry: A-02-TC03 (example #1)
> AssertionError: 1 accessibility violation was detected: expected 1 to equal 0
>     at Context.eval
> (https://console-openshift-console.apps.ci-op-lb7gpwtl-75d12.
> **********************/__cypress/tests?p=support/commands/index.ts:10178:20)

AFAIK this is a different issue to what was happening before. The developer catalog cases are still nice and green. And the wait is actually not indeterminate:
https://github.com/openshift/console/blob/d482d27a0dd8ec5c42521d99f9c9e33a7b7402df/frontend/packages/dev-console/integration-tests/support/pages/app.ts#L15

Now I just need to figure out what's wrong in this picture:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_console/10645/pull-ci-openshift-console-master-e2e-gcp-console/1469230738531422208/artifacts/e2e-gcp-console/test/artifacts/gui_test_screenshots/cypress/screenshots/e2e/add-flow-ci.feature/Create%20the%20different%20workloads%20from%20Add%20page%20--%20Edit%20Runtime%20Icon%20while%20Editing%20Image%20A-02-TC05%20(failed).png

Comment 12 Christoph Jerolimov 2022-01-26 11:24:41 UTC
Searched for the reported CI issues with the link in the first comment:

https://search.ci.openshift.org/?search=accessibility+violation+was+detected&maxAge=336h&context=1&type=junit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

620 runs, 43% failed, 0% of failures match = 0% impact

Only tests are changed, so it can not affect any user. Mark this as verified.

Comment 15 errata-xmlrpc 2022-03-10 16:31:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.