Red Hat Bugzilla – Bug 1386054
Repeated "unable to write JSON response" + "error instantiating deployment config" while creating DC
Last modified: 2017-03-08 13:43 EST
Created attachment 1211585 [details] Deployment config definition Description of problem: We have a cluster loader script which repeatedly creates DCs with RCs and services in new namespaces from a JSON file. The DC, pod and associated artifact creation is working fine, but each time it runs the system logs are flooded with messages about being unable to write a JSON response and error instantiating deployment config: Oct 17 22:23:26 192 atomic-openshift-master-api: I1017 22:23:26.021613 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:26 192 atomic-openshift-master-controllers: I1017 22:23:26.022008 5540 controller.go:55] Error instantiating deployment config mff/deploymentconfig1: empty data Oct 17 22:23:26 192 atomic-openshift-master-api: E1017 22:23:26.072751 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:26 192 atomic-openshift-master-api: E1017 22:23:26.072816 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Oct 17 22:23:26 192 atomic-openshift-master-controllers: I1017 22:23:26.073250 5540 controller.go:55] Error instantiating deployment config mff/deploymentconfig1: empty data Oct 17 22:23:26 192 atomic-openshift-master-api: I1017 22:23:26.072874 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:26 192 atomic-openshift-master-api: E1017 22:23:26.164786 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:26 192 atomic-openshift-master-api: E1017 22:23:26.164842 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Oct 17 22:23:26 192 atomic-openshift-master-api: I1017 22:23:26.164878 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:26 192 atomic-openshift-master-controllers: E1017 22:23:26.165345 5540 controller.go:60] empty data Oct 17 22:23:27 192 atomic-openshift-master-controllers: I1017 22:23:27.416759 5540 replication_controller.go:323] Observed updated replication controller deploymentconfig1-1. Desired pod count change: 1->1 Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.442443 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.442498 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: I1017 22:23:27.442553 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:27 192 atomic-openshift-master-controllers: I1017 22:23:27.442967 5540 controller.go:55] Error instantiating deployment config mff/deploymentconfig1: empty data Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.457189 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.457244 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: I1017 22:23:27.457279 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:27 192 atomic-openshift-master-controllers: I1017 22:23:27.457735 5540 controller.go:55] Error instantiating deployment config mff/deploymentconfig1: empty data Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.477489 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.477550 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: I1017 22:23:27.477585 5499 logs.go:41] http: multiple response.WriteHeader calls Oct 17 22:23:27 192 atomic-openshift-master-controllers: I1017 22:23:27.478003 5540 controller.go:55] Error instantiating deployment config mff/deploymentconfig1: empty data Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.507602 5499 apiserver.go:516] apiserver was unable to write a JSON response: http: request method or response status code does not allow body Oct 17 22:23:27 192 atomic-openshift-master-api: E1017 22:23:27.507654 5499 errors.go:63] apiserver received an error that is not an unversioned.Status: http: request method or response status code does not allow body Version-Release number of selected component (if applicable): 3.4.0.11 How reproducible: always Steps to Reproduce: 1. Install 3.4.0.11 2. Create a new project 3. Download the attached file (dc.json) 4. oc create -f ./dc.json Actual results: DC and pods are created ok, but the system logs get flooded with the above messages. Expected results: DCs created without errors
This is an HA env with 3 masters and 3 separate etcds. The errors are in the syslog of the master controller, but that might be coincidence since they seem to be api server messages
Did we change deployment configs in some way in 3.4?
The API objects haven't changed, but we pulled in serialization changes that make protobuf a possibility. That error comes from protobuf. The path that serializes a DC to JSON for an annotation may need to be updated.
Can you guys dump a representative decoloyment config and replication controller?
Created attachment 1212135 [details] Replication controller created from attached template
Created attachment 1212136 [details] Deployment config created from attached template Attached dumps of DC and RC. Let me know what else you need.
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/3ab8c313b2ce862abf44d9ca62b7873760b2f4cf Bug 1386054: enqueue in the trigger controller only when really needed
Hi Mike Fiedler Any update for this issue?
Testing the fix today. Will report results.
While creating a single DC (oc create -f ./dc.json): 3.4.0.13: The error message (apiserver was unable to write a JSON response: http: request method or response status code does not allow body) is written to the log 29 times 3.4.0.14: The message fires 4 times 3.4.0.15: with the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1386018 the message appears only once. @mkargaki - is it ok for this message to fire once on every DC creation?
It is fine but I have another fix that should eliminate that single log. Let's keep this open until that PR is merged: https://github.com/openshift/origin/pull/11501 Dropping priority...
https://github.com/openshift/origin/pull/11501 is included in 3.4.0.16
Verified on 3.4.0.16. Seeing a couple of these messages very rarely while scaling up a cluster. Nothing on the order of "n per deployment created"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066