RFE: Include database validation as part of the upgrade workflow. This includes pre-emptive validation, such as sanity checking the size of 
nova db (https://bugzilla.redhat.com/show_bug.cgi?id=1526117#c25) before even beginning the main upgrade tasks (e.g. step0 validations). Other validations can be post-upgrade to verify the integrity of data (after package updates and dbsync have completed). One example is to create test data before the upgrade and then verify it after the upgrade (in one of the later upgrade_tasks for example). We should file individual BZ to track subtasks and use this for tracking if useful.

This BZ was filed based on the discussion in BZ 1534204 BZ 1526117 and the RDO Newton->Ocata upgrade retrospective on Jan 12 2018@ https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting - including here verbatim:

        RDO cloud:
        the good:
        [dmanchad] commitment from people and different sme were successfully engaged (heat, nova, telemetry, cinder, etc.)
        the bad
        [lbezdick] - state of RDO cloud - the API change for SSL
        [apevec] - post update state unclear https://etherpad.openstack.org/p/RDO-Cloud < list of bugs raised by ops team(already on CIX board https://trello.com/c/thvgg3aH/417-cixlp1742827nova-nova-scheduler-reports-dead-compute-nodes-but-nova-compute-is-enabled-and-up and https://trello.com/c/Hd2AS49Y/419-cixlp1742826novanova-reports-wrong-quota-usage-on-rdo-cloud-tenants )
        [dmanchad] not sure all the upgrade steps/tasks are 100% idempotent
        Action Items:
        Post update validations (with tripleo-validation ?)
        Copy cleanup also to upgrade procedure, dont rely on cron job
        Validate DB structures
        validate datas (running only a test db_sync in ooo-validation)
        validation example - create tenant before update with ~2vms, do validations before & after upgrade, delete tenant
        checklist for operator ? (to make sure the environment is in a healthy status an avoid issues during the upgrade)
        Adding some test jobs at the tripleo/rdo level that will perform big validation such as databases validation.
        Thinking off running upgrade in the promotion pipeline on a backuped env  (pike backup by the pike promotion pipeline, pop up and run upgrade there)