Bug 1513143

Summary: Enabling replication after upgrading from 5.5 to 5.9 fails on db schema
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: ApplianceAssignee: Nick Carboni <ncarboni>
Status: CLOSED CURRENTRELEASE QA Contact: luke couzens <lcouzens>
Severity: high Docs Contact:
Priority: high    
Version: 5.9.0CC: abellott, jhardy, lcouzens, obarenbo, smallamp
Target Milestone: GAKeywords: Regression, TestOnly
Target Release: 5.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: black:replication:upgrade:migration
Fixed In Version: 5.9.0.10 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1508537
: 1517386 (view as bug list) Environment:
Last Closed: 2018-03-06 15:08:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:
Bug Depends On: 1508537    
Bug Blocks: 1517386    

Comment 2 luke couzens 2017-11-17 11:14:55 UTC
Rake task has been added to latest build but its still not working as expected. We can now run the db:check_schema but the rails r tools/fix_column_ordering.rb task still fails. see output below.

[root@vmdb]# rake db:check_schema
rake aborted!
Schema validation failed for host localhost:

Columns for table computer_systems in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "managed_entity_type", "managed_entity_id", "created_at", "updated_at"]

got:
["id", "managed_entity_id", "managed_entity_type", "created_at", "updated_at"]

Schema validation failed for host localhost:

Columns for table event_streams in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_type", "target_id", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"]

got:
["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_id", "target_type", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"]

Schema validation failed for host localhost:

Columns for table security_contexts in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml

expected:
["id", "resource_type", "resource_id", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]

got:
["id", "resource_id", "resource_type", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]
/opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/lib/tasks/schema_tasks.rake:5:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:check_schema
(See full trace by running task with --trace)


[root@vmdb]# rails r tools/fix_column_ordering.rb computer_systems
Correcting column ordering for table computer_systems
/opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `initialize': No such file or directory @ rb_sysopen - /var/www/miq/vmdb/db/schema.yml (Errno::ENOENT)
	from /opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `open'
	from /opt/rh/rh-ruby23/root/usr/share/gems/gems/psych-2.0.17/lib/psych.rb:471:in `load_file'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:18:in `expected_columns'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:213:in `assert_column_list_sizes_match!'
	from /var/www/miq/vmdb/tools/column_ordering/column_ordering.rb:30:in `fix_column_ordering'
	from tools/fix_column_ordering.rb:22:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:138:in `require'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:138:in `require_command!'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:104:in `runner'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rh/cfme-gemset/gems/railties-5.0.6/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

Comment 3 Nick Carboni 2017-11-17 13:33:03 UTC
I actually just removed this tool because the new version of pglogical we use doesn't have the same issue with column ordering.

https://github.com/ManageIQ/manageiq/pull/16488

When that gets backported, I'll just close this bug.

Comment 4 Nick Carboni 2017-11-17 18:20:36 UTC
Moving this to POST.

Luke, as a test for this can you just skip the column reordering steps and ensure that replication works?

The new version of pglogical will handle out of order columns.

Comment 5 luke couzens 2017-11-17 19:32:12 UTC
Is the new version of pglogical on 5.9.0.9 then? I still see the same ordering issues when trying to setup replication on the global region.

Error during replication configuration save: Failed to save subscription to : Schema validation failed for host : Columns for table computer_systems in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "managed_entity_type", "managed_entity_id", "created_at", "updated_at"] got: ["id", "managed_entity_id", "managed_entity_type", "created_at", "updated_at"] Schema validation failed for host : Columns for table event_streams in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_type", "target_id", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"] got: ["id", "event_type", "message", "timestamp", "host_name", "host_id", "vm_name", "vm_location", "vm_or_template_id", "dest_host_name", "dest_host_id", "dest_vm_name", "dest_vm_location", "dest_vm_or_template_id", "source", "chain_id", "ems_id", "is_task", "full_data", "created_on", "username", "ems_cluster_id", "ems_cluster_name", "ems_cluster_uid", "dest_ems_cluster_id", "dest_ems_cluster_name", "dest_ems_cluster_uid", "availability_zone_id", "container_node_id", "container_node_name", "container_group_id", "container_group_name", "container_namespace", "type", "target_id", "target_type", "container_id", "container_name", "container_replicator_id", "container_replicator_name", "middleware_server_id", "middleware_server_name", "middleware_deployment_id", "middleware_deployment_name", "generating_ems_id", "physical_server_id", "ems_ref", "middleware_domain_id", "middleware_domain_name", "user_id", "group_id", "tenant_id"] Schema validation failed for host : Columns for table security_contexts in the current schema do not match the columns listed in /opt/rh/cfme-gemset/bundler/gems/manageiq-schema-828d50c74f4b/db/schema.yml expected: ["id", "resource_type", "resource_id", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"] got: ["id", "resource_id", "resource_type", "se_linux_user", "se_linux_role", "se_linux_type", "se_linux_level"]

Comment 6 Nick Carboni 2017-11-17 20:24:37 UTC
Ah, nope. Let me unset the fixed-in version.

https://github.com/ManageIQ/manageiq/pull/16488 was just backported today, so it should be in the next build.

Comment 7 luke couzens 2017-11-23 11:37:38 UTC
Verified in 5.9.0.10

Comment 8 CFME Bot 2017-11-29 20:47:50 UTC
New commit detected on ManageIQ/manageiq-schema/gaprindashvili:
https://github.com/ManageIQ/manageiq-schema/commit/828d50c74f4b772d7ed57e78d9a94f14244c866a

commit 828d50c74f4b772d7ed57e78d9a94f14244c866a
Author:     Jason Frey <fryguy9>
AuthorDate: Mon Nov 13 15:52:26 2017 -0500
Commit:     Satoe Imaishi <simaishi>
CommitDate: Tue Nov 14 20:15:54 2017 -0500

    Merge pull request #124 from carbonin/make_schema_check_and_write_public
    
    Move the db:check_schema task to the public tasks.rake file
    (cherry picked from commit ad336bd67a970b12d29a74295881fcf7a4b5895f)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1513143

 lib/tasks/schema_tasks.rake         | 8 ++++++++
 lib/tasks_private/schema_tasks.rake | 7 -------
 2 files changed, 8 insertions(+), 7 deletions(-)
 create mode 100644 lib/tasks/schema_tasks.rake