Red Hat Bugzilla – Bug 968991
error while create large number of new topics through contentspec
Last modified: 2013-10-09 01:23:59 EDT
Description of problem: While creating a large number of new topics through the contentspec, using the command: csprocessor push <name of the contentspec> it throws an error "ERROR: An error occurred when inserting into the database please try again. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Create 30-50 new topics entries in the contentspec 2.Run csprocessor push <contentspec ID> 3. Actual results: ERROR: An error occurred when inserting into the database please try again. Expected results: Additional info: I happened to run the csprocessor push command multiple times to be able to create topics, each times with a different no of new topics, like i began with creating 50 new topics, reduced to 40, 30, 20,15,10 . But each time i got the above error, Finally with a lesser set of new topics the command worked fine. But as I happened to run the command so many times, it has infact created duplicate topics in CCMS. though it was giving the error.
I've found the main cause of this. The problem is the way the server handles new URLs. If a URL is passed without a title then the server will load the html page on the server and extract the <title> from the <head> of the html. So having to do this multiple times causes the HTTP request to timeout before it finishes. So there are three things that could be done to fix this (that I can think of aat this stage): 1. Remove the server processing to not open the url and read the html to get the title. (Optionally add it back in, however it would have to be done as a post commit step, which would also mean it would need it's own management). 2. Change the csprocessor to not set the URL's in the initial request and add them one by one in a latter step. 3. Change the csprocessor to process each URL and set the title for the URL before sending it to the server. As an additional fourth option that is not very wise, we could just increase the HTTP timeout on the server. I'll have to look into these and test which option is better, however the best to keep the current functionality the same is a combination Option 1 and Option 3. As an example using the content Aakanksha gave me via Email, the create request took ~1 min 40 secs, while removing the URL's lowered the request time to ~5 secs.
Workaround: Remove all URL's when creating new Topics and add them in manually via the UI.
After discussing this with the team, we've determined that Option 1 definitely needs to be done and I've also implemented Option 3 for csprocessor 0.33.0. I'll update this bug again when the server fix goes live and when csprocessor 0.33.0 is released.
CSProcessor 0.33.0 has now been released. The server has been fixed, but has not been release yet.
Moving this bug to CLOSED CURRENT_RELEASE to clean up old bugs that were QA'd by the PressGang team but not by the original reporter. If the bug is still present then please re-open the bug.