Bug 1471033

Summary: Sometime get "Failed to list templates/v1(undefined)" error in catalog console
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: Management ConsoleAssignee: Samuel Padgett <spadgett>
Status: CLOSED ERRATA QA Contact: XiaochuanWang <xiaocwan>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, chali, dma, jforrest, jokerman, mmccomas, wjiang, xtian, xxia, yapei
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, some requests for templates in the web console could timeout or take a long time to complete over high latency network connections. This could cause an error when loading the "Add to Project" page. The web console can now load templates using much less data, which fixes the problem.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:01:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot.png
none
98items.png
none
template fail to load none

Description DeShuai Ma 2017-07-14 09:49:47 UTC
Created attachment 1298226 [details]
screenshot.png

Description of problem:
Enalbe service catalog in openshift-ansible then login to catalog console. Sometime can get error "Failed to list templates/v1(undefined)"

Version-Release number of selected component (if applicable):
openshift v3.6.143
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
Sometime

Steps to Reproduce:
1. Login catalog console and refresh the home page

Actual results:
1. Met error "Failed to list templates/v1(undefined)"

Expected results:
1. No errors

Additional info:
screenshot is attached

Comment 1 DeShuai Ma 2017-07-14 09:54:27 UTC
Created attachment 1298229 [details]
98items.png

Also met something strange things. Sometime total=98, Sometime total=176. But actually there is 176 app. When total=98, there is no any error dialog.

Comment 2 Jessica Forrester 2017-07-14 12:08:30 UTC
The second issue is not an issue, the difference you are seeing is when you are viewing the top level "All" category versus the Languages -> All subcategory.  Verified this on your server.

Comment 3 Jessica Forrester 2017-07-14 12:14:05 UTC
For the issue with templates failing to load.  I'm not able to reproduce this against your server.  In the cases where you see it can you please check the Network tab and see what the failure was on the request.  I suspect you may be some hitting some kind of timeout. We have an existing bug tracking the fact that our OOTB templates for OCP results in a very large template request. See https://bugzilla.redhat.com/show_bug.cgi?id=1435174

Comment 4 DeShuai Ma 2017-07-14 16:34:54 UTC
Yes, I have read the slow bug before report this. catalog console really very sloooow every time when loading. But for this error, can't always reproduce, I just met about four times one day. Maybe they are same, When this happen again I'll check the Network tab and see what the failure was on the request.

Comment 5 Jessica Forrester 2017-07-14 16:43:27 UTC
I would also double check the javascript console for any errors.

Comment 6 weiwei jiang 2017-08-01 05:42:12 UTC
Created attachment 1307366 [details]
template fail to load

checked again, and found that the response json for templates has additional content "This request caused apiserver to panic. Look in log for details."

Comment 8 weiwei jiang 2017-08-01 06:15:33 UTC
The latest version I can reproduce this issue is:

# openshift version 
openshift v3.6.172.0.1
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

Comment 9 Jordan Liggitt 2017-08-04 03:27:53 UTC
that is a panic caused by a timeout. how many templates do you have loaded and how long do they take to list?

Comment 10 weiwei jiang 2017-08-04 03:34:40 UTC
I remember 176 totally.

Comment 11 Jordan Liggitt 2017-08-04 03:37:31 UTC
what is the average time to list, and size of listed data?

time oc get templates
oc get templates -o json > templates.json
ls -la templates.json

time oc get templates -n openshift
oc get templates -o json -n openshift > templates.json
ls -la templates.json

Comment 13 weiwei jiang 2017-08-04 05:14:20 UTC
# oc get templates -n openshift -o json > templates.json

# ls -al templates.json 
-rw-r--r--. 1 root root 4584324 Aug  4 13:12 templates.json

# ls -alh templates.json 
-rw-r--r--. 1 root root 4.4M Aug  4 13:12 templates.json

Comment 15 Jordan Liggitt 2017-08-04 14:02:43 UTC
The server timeout on list requests is 1 minute.

Listing multiple megabytes of templates over a slow connection can hit that timeout. When that happens, an error in the server log and in the client are expected.

We should fix the display of the error in the client (should not include "(undefined)")

Possible improvements:
- have the client remember template data instead of reloading it on every page change
- have the client only request template metadata ("Accept: application/json;as=Table;v=v1alpha1;g=meta.k8s.io" header and  "?includeObject=Metadata" query parameter), available in 3.7

Comment 16 Jordan Liggitt 2017-08-04 14:06:02 UTC
looks like the includeObject param has a bug (https://github.com/kubernetes/kubernetes/issues/50149), but it defaults to Metadata, which works for our purposes

Comment 17 Samuel Padgett 2017-08-04 14:19:38 UTC
The "(undefined)" message should be fixed by

https://github.com/openshift/origin-web-common/pull/126

We'll look at updating the web console to request template metadata only.

Comment 18 Jordan Liggitt 2017-08-04 14:26:23 UTC
actually, PartialObjectMetadataList looks like exactly what the UI would want

Accept: application/json;as=PartialObjectMetadataList;v=v1alpha1;g=meta.k8s.io

Comment 20 openshift-github-bot 2017-08-15 22:54:00 UTC
Commit pushed to master at https://github.com/openshift/origin-web-console

https://github.com/openshift/origin-web-console/commit/946fe5b30967ba4e7c63f8ca45bf998fd21a3101
Bug 1471033 - Only request template metadata

Bumps origin-web-common to 0.0.46
Bumps origin-web-catalog to 0.0.36

Only request template metadata when showing the catalog.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1471033
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1435174

Comment 21 Samuel Padgett 2017-08-15 23:10:08 UTC
*** Bug 1435174 has been marked as a duplicate of this bug. ***

Comment 23 Yadan Pei 2017-09-26 09:40:45 UTC
This bug is checked on OpenStack 3.7 env, error "Failed to list tempaltes/v1(undefined)" no longer shows up in web console

I would also double check this bug on OCP 3.7 envs built on other IaaS

Comment 24 Xingxing Xia 2017-09-26 09:48:38 UTC
Please also check the loading time for bug 1435174, which has comments exposing the underlying cause OOTB (https://bugzilla.redhat.com/show_bug.cgi?id=1435174#c7).
This bug depends on bug 1486623 for real service catalog env to check, IMO

Comment 25 weiwei jiang 2017-09-26 11:06:28 UTC
Checked with 
# openshift version 
openshift v3.7.0-0.127.0
kubernetes v1.7.0+80709908fd
etcd 3.2.1

template is loading via PartialObjectMetadataList now.

I extend the loading duration of templates to 9 min via network throttler to verify this bug.

Comment 26 Yadan Pei 2017-09-27 09:38:36 UTC
Average loading time during testing 

real	0m2.458s
user	0m0.037s
sys	0m0.089s

real	0m2.496s
user	0m0.041s
sys	0m0.078s

real	0m3.063s
user	0m0.045s
sys	0m0.074s

real	0m2.456s
user	0m0.035s
sys	0m0.086s

real	0m2.459s
user	0m0.042s
sys	0m0.076s

real	0m2.411s
user	0m0.029s
sys	0m0.057s

real	0m2.461s
user	0m0.041s
sys	0m0.068s

real	0m3.149s
user	0m0.051s
sys	0m0.067s

real	0m2.762s
user	0m0.049s
sys	0m0.062s

real	0m2.429s
user	0m0.034s
sys	0m0.081s


Move to VERIFIED as we verified templates get loaded via PartialObjectMetadataList and no errors found

Comment 29 errata-xmlrpc 2017-11-28 22:01:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188