Bug 968991

Summary:	error while create large number of new topics through contentspec
Product:	[Community] PressGang CCMS	Reporter:	Aakanksha <aasingh>
Component:	CSProcessor	Assignee:	Lee Newson <lnewson>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Aakanksha <aasingh>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	1.x	CC:	lnewson
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	0.33.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-10-09 05:23:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aakanksha 2013-05-30 12:29:15 UTC

Description of problem:
While creating a large number of new topics through the
contentspec, using the command: csprocessor push <name of the
contentspec> it throws an error "ERROR: An error occurred when inserting
into the database please try again.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Create 30-50 new topics entries in the contentspec
2.Run csprocessor push <contentspec ID>
3.

Actual results:
ERROR: An error occurred when inserting
into the database please try again.

Expected results:


Additional info:
I happened to run the csprocessor push command multiple times to
be able to create topics, each times with a different no of new topics,
like i began with creating 50 new topics, reduced to 40, 30, 20,15,10 .
But each time i got the above error, Finally with a lesser set of new
topics the command worked fine. But as I happened to run the command so
many times, it has infact created duplicate topics in CCMS. though it
was giving the error.

Comment 1 Lee Newson 2013-06-07 12:25:25 UTC

I've found the main cause of this. The problem is the way the server handles new URLs. If a URL is passed without a title then the server will load the html page on the server and extract the <title> from the <head> of the html. So having to do this multiple times causes the HTTP request to timeout before it finishes.

So there are three things that could be done to fix this (that I can think of aat this stage):

1. Remove the server processing to not open the url and read the html to get the title. (Optionally add it back in, however it would have to be done as a post commit step, which would also mean it would need it's own management).

2. Change the csprocessor to not set the URL's in the initial request and add them one by one in a latter step.

3. Change the csprocessor to process each URL and set the title for the URL before sending it to the server.

As an additional fourth option that is not very wise, we could just increase the HTTP timeout on the server.

I'll have to look into these and test which option is better, however the best to keep the current functionality the same is a combination Option 1 and Option 3.

As an example using the content Aakanksha gave me via Email, the create request took ~1 min 40 secs, while removing the URL's lowered the request time to ~5 secs.

Comment 2 Lee Newson 2013-06-07 12:26:21 UTC

Workaround:

Remove all URL's when creating new Topics and add them in manually via the UI.

Comment 3 Lee Newson 2013-06-12 09:52:59 UTC

After discussing this with the team, we've determined that Option 1 definitely needs to be done and I've also implemented Option 3 for csprocessor 0.33.0.

I'll update this bug again when the server fix goes live and when csprocessor 0.33.0 is released.

Comment 4 Lee Newson 2013-06-28 03:12:36 UTC

CSProcessor 0.33.0 has now been released. The server has been fixed, but has not been release yet.

Comment 5 Lee Newson 2013-10-09 05:23:59 UTC

Moving this bug to CLOSED CURRENT_RELEASE to clean up old bugs that were QA'd
by the PressGang team but not by the original reporter. If the bug is still present then please re-open the bug.