Bug 1070040

With this, I wonder if we should actually be storing this on the Content Specification node elements. As a fixed URL only needs to be unique in the context of the book and not on a global scale.

This would also help when having to generate the Fixed URLs, since there would be less elements we'd have to check against and also we in theory could calculate the fixed url values without hitting the server at all (expect for the final update).

Added in 1.9-SNAPSHOT build 201407220854

Fixed URL's are now stored inside of a content specification against it's nodes (see the example below). This means that Fixed URL's only need to be unique in the context of a content specification instead of the whole database.

Additionally to be able to do this we've had to modify the workflow a little. Now the Fixed URL's won't be set when doing a local build and instead they will be set when you freeze the content specification. This also has the benefit of not setting the Fixed URL to early when the topic title may change before GA and also gives the user a little more control about the actual Fixed URL used. This means that the workflow will be

1. Edit spec as per usual.
  - Note: During this time temporary fixed urls will be generated when doing local builds (the way docbuilder currently does it).

2. When it's time to publish a user freezes the book to make sure no changes affect the book. This will also apply the Fixed URL's to the content spec.

3. Build the book and publish using whatever means are available.

To help maintain readability due to specs now containing the fixed urls (see part two above), we've also implemented an auto code folding mechanism that will automatically hide all the fixed urls in the spec and when clicking the <-> symbol it will expand to show the Fixed URL.

Example of a frozen spec:

Title = Test
Product = Test
Version = 1.0
Copyright Holder = Red Hat, Inc

Chapter: Test [Fixed URL = chap-Test]
  My Topic [33, rev: 12345, Fixed URL = My_Topic]
  Another Topic [32, rev: 12456, Fixed URL = Another_Topic]

The way Fixed URL's are generated is to first check if a topic is associated with a node and if a topic exists then check if it has the Fixed URL property tag. If it does then pull the fixed url from there, make sure it's still unique and then set it on the node. If a topic or property tag doesn't exist then it will generate the fixed url from the nodes title.

Note: The following nodes won't have a Fixed URL set, because it cannot be used: Common Content, Initial Text and Initial Text Topics. In the event that a user tries to set a fixed url on one of these nodes, they will get the following error message:

Line 10: The node has a Fixed URL specified, however this type of node cannot contain Fixed URL's, as such it has been removed.

The possible validation error messages are as follows:

Invalid Fixed URL:

The Fixed URL specified is not a valid URL. Please ensure that it starts with a letter, has no spaces and doesn't contain special characters.

Non-Unique Fixed URL:

"<FIXED_URL>" is not a unique URL within the Content Specification.

Below is an example that shows that the output when temporary fixed urls are used (as well as the new build order):

Starting to build...
Doing en-US Populate Database Pass
Doing en-US First topic pass
	Processing 2 Topics
	First topic Pass 50% Done
	First topic Pass 100% Done
Doing Fixed URL Pass
	Using 1 temporary Fixed URLs
Doing en-US Topic Link Pass
Doing en-US Spec Topic Pass
	Processing 2 Spec Topics
	Processing Pass 50% Done
	...

Created attachment 920359 [details]
Errors from server.log when saving a spec

The attached error log shows an error that can be replicated by doing the following.

Create and save a new spec with this text:

Title = Test
Product = PressGang
Version = 1
Copyright Holder = Red Hat
Locale = en-US

Chapter: Test Chapter [Fixed URL = chap-Test_Chapter]
  Initial Text:
    New Topic [41072, rev: 683033]
    Common_Content/Feedback.xml [Common Content]
  New Topic [41069, rev: 683021, Fixed URL = テスト]
  New Topic [41071, rev: 683021, Fixed URL = hi]
  New Topic [41068, rev: 683021, Fixed URL = New_Topic4]
  New Topic [41066, rev: 683021, Fixed URL = New_Topic3]
  New Topic [41067, rev: 683021, Fixed URL = New_Topic1]
  New Topic [41065, rev: 683021, Fixed URL = New_Topic]
  New Topic [41070, rev: 683021, Fixed URL = New_Topic2]
  Section: Blah [Fixed URL = sect-Blah]
    New Topic [41076, rev: 683038, Fixed URL = New_Topic6]

Process: Test [Fixed URL = proc-Test]
  New Topic [41075, rev: 683038, Fixed URL = New_Topic5]

Part: Test [Fixed URL = part-Test]
  Chapter: Test [Fixed URL = chap-Test]
    New Topic [41073, rev: 683038, Fixed URL = New_Topic8]

Appendix: Test [Fixed URL = appe-Test]
  New Topic [41074, rev: 683038, Fixed URL = New_Topic7]

Then change the line "Common_Content/Feedback.xml [Common Content]" to "Common_Content/Feedback.xml [Common Content, Fixed URL = テスト]"

Fixed urls like "[hi]", "{hi}", "$hi" save ok but can't be built by Publican. You get the error:

DTD Validation failed for '/tmp/Test/tmp/en-US/xml/Test.xml': 
%24hi.xml:16: validity error : Syntax of value for attribute id of chapter is not valid

Tested using non english characters as fixed urls ok.
Tested freezing a spec with topics that have duplicated titles ok.
Tested adding duplicated fixed urls ok.
Tested freezing a content spec with initial topics ok.
Tested adding an fixed url to an initial topic container and its children ok.
Tested chapters, sections, parts, process, appendix with fixed urls ok.
Tested collapsing content in the web ui ok (with things like copying and pasting).
Tested freeing spec with topics that have existing fixed urls ok.
Tested freeing spec with topics that have existing duplicated fixed urls ok.

Tested syncing book with fixed urls to zanata ok.

(In reply to Matthew Casperson from comment #7)
> The attached error log shows an error that can be replicated by doing the
> following.
> 
> Create and save a new spec with this text:
> 
> Title = Test
> Product = PressGang
> Version = 1
> Copyright Holder = Red Hat
> Locale = en-US
> 
> Chapter: Test Chapter [Fixed URL = chap-Test_Chapter]
>   Initial Text:
>     New Topic [41072, rev: 683033]
>     Common_Content/Feedback.xml [Common Content]
>   New Topic [41069, rev: 683021, Fixed URL = テスト]
>   New Topic [41071, rev: 683021, Fixed URL = hi]
>   New Topic [41068, rev: 683021, Fixed URL = New_Topic4]
>   New Topic [41066, rev: 683021, Fixed URL = New_Topic3]
>   New Topic [41067, rev: 683021, Fixed URL = New_Topic1]
>   New Topic [41065, rev: 683021, Fixed URL = New_Topic]
>   New Topic [41070, rev: 683021, Fixed URL = New_Topic2]
>   Section: Blah [Fixed URL = sect-Blah]
>     New Topic [41076, rev: 683038, Fixed URL = New_Topic6]
> 
> Process: Test [Fixed URL = proc-Test]
>   New Topic [41075, rev: 683038, Fixed URL = New_Topic5]
> 
> Part: Test [Fixed URL = part-Test]
>   Chapter: Test [Fixed URL = chap-Test]
>     New Topic [41073, rev: 683038, Fixed URL = New_Topic8]
> 
> Appendix: Test [Fixed URL = appe-Test]
>   New Topic [41074, rev: 683038, Fixed URL = New_Topic7]
> 
> Then change the line "Common_Content/Feedback.xml [Common Content]" to
> "Common_Content/Feedback.xml [Common Content, Fixed URL = テスト]"

This is actually another bug altogether at a first glance. The problem is that the node type is saved as an "Initial Text Topic", however it should be saved as  "Common Content".

(In reply to Matthew Casperson from comment #8)
> Fixed urls like "[hi]", "{hi}", "$hi" save ok but can't be built by
> Publican. You get the error:
> 
> DTD Validation failed for '/tmp/Test/tmp/en-US/xml/Test.xml': 
> %24hi.xml:16: validity error : Syntax of value for attribute id of chapter
> is not valid

Fixed. The regex that checks this has been updated to be stricter. It will now make sure the first character is a letter, underscore or colon and then anything that follows has to be a letter, digit, underscore, hypen, colon or decimal. (see http://stackoverflow.com/a/1077111/1330640 for more details)

Fixed in 1.9-SNAPSHOT build 201407241122

See Comment #12 for the symbol issue and as for the other problem mentioned in Comment #7 it was a bug in the parser. It was not setting the right node type when parsing in some cases and additionally it was quietly dropping any additional attributes (ie fixed url, source url, description).

Note: This version has been deployed to the test/development server.

Additionally I also fixed an issue where Fixed URL's were being updated when the "Updated Revisions" option was set.

> Comment 11
> This is actually another bug altogether at a first glance. The problem is that the node type is saved as an "Initial Text Topic", however it should be saved as  "Common Content".

Verified that this no longer results in an error.

> Comment 12
Fixed. The regex that checks this has been updated to be stricter. It will now make sure the first character is a letter, underscore or colon and then anything that follows has to be a letter, digit, underscore, hypen, colon or decimal. (see http://stackoverflow.com/a/1077111/1330640 for more details)

Verified that the fixed urls are now more strict, and every combination I tried built a valid book with Publican.

Verified that a spec created using csprocessor is saved to the server correctly.
Verified the a snapshot created with csprocessor has the fixed URLs populated.

Found an issue when testing the docbuilder changes. Sometimes when making changes you can get an error like the following:

Caused by: java.sql.BatchUpdateException: Duplicate entry '6895-About_JBoss_Enterprise_Application_Platform_6-2' for key 'ContentSpecURL'
	at org.mariadb.jdbc.MySQLPreparedStatement.executeBatch(MySQLPreparedStatement.java:232)
	at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:708)
	at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077)
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:110) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
	... 68 more
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '6895-About_JBoss_Enterprise_Application_Platform_6-2' for key 'ContentSpecURL'
	at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:132)
	at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
	at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:263)
	at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:287)
	at org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141)
	at org.mariadb.jdbc.MySQLPreparedStatement.executeBatch(MySQLPreparedStatement.java:222)
	... 71 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Duplicate entry '6895-About_JBoss_Enterprise_Application_Platform_6-2' for key 'ContentSpecURL'
	at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:934)
	at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:983)
	at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:280)

I believe the above is caused by deleting a node and then recreating it, however because hibernate does delete queries after insert queries you get the above duplicate entry problem.

The above error was indeed because of the execution order for my case, however after fixing that you could still get the error by swapping fixed urls between nodes. As such we've removed the unique index, as it cannot be used reliably without support for deferred constraint checks, which means this will be enforced at the application level via the Content Spec validator (which was already the case anyways).

If you have already deployed the previous version locally you'll need to run the following queries to remove the index:

ALTER TABLE  `ContentSpecNodeURL` DROP FOREIGN KEY  `FK_CSNodeURL_TO_ContentSpec` ;
ALTER TABLE  `ContentSpecNodeURL` DROP INDEX ContentSpecURL;
ALTER TABLE  `ContentSpecNodeURL` ADD FOREIGN KEY (  `ContentSpecID` ) REFERENCES  `Skynet`.`ContentSpec` (`ContentSpecID`) ON DELETE RESTRICT ON UPDATE RESTRICT ;
DELETE FROM `Skynet`.`DATABASECHANGELOG` WHERE `DATABASECHANGELOG`.`ID` = '1-other' AND `DATABASECHANGELOG`.`AUTHOR` = 'lnewson' AND `DATABASECHANGELOG`.`FILENAME` = 'db/changelog/db.changelog-1.8.xml';
DELETE FROM `Skynet`.`DATABASECHANGELOG` WHERE `DATABASECHANGELOG`.`ID` = '1-mysql' AND `DATABASECHANGELOG`.`AUTHOR` = 'lnewson' AND `DATABASECHANGELOG`.`FILENAME` = 'db/changelog/db.changelog-1.8.xml';

Note: The fixed version and sql queries have been run on the dev server.

Tried adding, deleting, commenting out and moving nodes with fixed urls. Also tried swapping the fixed urls between nodes. All worked ok.