Description of problem:

Upgrading to Satellite 6.11 fails on db:migrate stage with error "null value in column "created_at" violates not-null constraint"


Version-Release number of selected component (if applicable):

Satellite 6.11 [ RHEL 7 ]


How reproducible:

In customer environments 


Steps to Reproduce:
1. Install Satellite 6.10 
2. Upgrade to 6.11 as per documentation
3.

Actual results:

Upgrade fails:
~~
\ Executing installer2022-07-19 10:48:00 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-07-19 10:48:00 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
| Executing installer2022-07-19 10:48:42 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-07-19 10:48:42 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
~~

Error from logs:
~~
2022-07-19 10:48:00 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:
2022-07-19 10:48:00 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Caused by:
2022-07-19 10:48:00 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::NotNullViolation: ERROR:  null value in column "created_at" violates not-null constraint
2022-07-19 10:48:00 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: DETAIL:  Failing row contains (321220, 4, 1, ForemanPuppet::Environment, null, null).
~~


"foreman-rake db:migrate" output:
~~~
== 20220208135305 MigrateEnvironmentIgnoreType: migrating =====================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::NotNullViolation: ERROR:  null value in column "created_at" violates not-null constraint
DETAIL:  Failing row contains (321227, 4, 1, ForemanPuppet::Environment, null, null).
~~~


Expected results:

The upgrade should be successful. 


Additional info:

The issue happens as The schema of taxable_taxonomies table somehow is not correct.

For a satellite having this issue:

~~~
[root@rhsat ~]$ echo "\d+ taxable_taxonomies" | su - postgres -c "psql foreman"
                                                             Table "public.taxable_taxonomies"
    Column    |            Type             | Collation | Nullable |                    Default                     | Storage  | Stats target | Description
--------------+-----------------------------+-----------+----------+------------------------------------------------+----------+--------------+-------------
 id           | integer                     |           | not null | nextval('taxable_taxonomies_id_seq'::regclass) | plain    |              |
 taxonomy_id  | integer                     |           |          |                                                | plain    |              |
 taxable_id   | integer                     |           |          |                                                | plain    |              |
 taxable_type | character varying(255)      |           |          |                                                | extended |              |
 created_at   | timestamp without time zone |           | not null |                                                | plain    |              |
 updated_at   | timestamp without time zone |           | not null |                                                | plain    |              |
Indexes:
    "taxable_taxonomies_pkey" PRIMARY KEY, btree (id)
    "taxable_index" UNIQUE, btree (taxable_type, taxable_id, taxonomy_id)
    "index_taxable_taxonomies_on_taxonomy_id_and_taxable_type" btree (taxonomy_id, taxable_type)
Foreign-key constraints:
    "taxable_taxonomies_taxonomy_id_fk" FOREIGN KEY (taxonomy_id) REFERENCES taxonomies(id)
Access method: heap


[root@rhsat ~]$ echo "select * from taxable_taxonomies where created_at is null;" | su - postgres -c "psql foreman"
 id | taxonomy_id | taxable_id | taxable_type | created_at | updated_at
----+-------------+------------+--------------+------------+------------
(0 rows)
~~~

Expected:

~~~
# echo "\d+ taxable_taxonomies" | su - postgres -c "psql foreman"
                                                             Table "public.taxable_taxonomies"
    Column    |            Type             | Collation | Nullable |                    Default                     | Storage  | Stats target | Description 
--------------+-----------------------------+-----------+----------+------------------------------------------------+----------+--------------+-------------
 id           | integer                     |           | not null | nextval('taxable_taxonomies_id_seq'::regclass) | plain    |              | 
 taxonomy_id  | integer                     |           |          |                                                | plain    |              | 
 taxable_id   | integer                     |           |          |                                                | plain    |              | 
 taxable_type | character varying(255)      |           |          |                                                | extended |              | 
 created_at   | timestamp without time zone |           |          |                                                | plain    |              | 
 updated_at   | timestamp without time zone |           |          |                                                | plain    |              | 
Indexes:
    "taxable_taxonomies_pkey" PRIMARY KEY, btree (id)
    "taxable_index" UNIQUE, btree (taxable_type, taxable_id, taxonomy_id)
    "index_taxable_taxonomies_on_taxonomy_id_and_taxable_type" btree (taxonomy_id, taxable_type)
Foreign-key constraints:
    "taxable_taxonomies_taxonomy_id_fk" FOREIGN KEY (taxonomy_id) REFERENCES taxonomies(id)
Access method: heap
~~~


Workaround:

# su - postgres -c "psql foreman"
ALTER TABLE taxable_taxonomies ALTER COLUMN created_at DROP NOT NULL;
ALTER TABLE taxable_taxonomies ALTER COLUMN updated_at DROP NOT NULL;
exit

# foreman-rake db:migrate --trace


If no issues are reported this time, proceed with re-running the upgrade.