Hide Forgot
Description of problem: A vmdb which had a schema that was downlevel from the 5.5.2.4 code level expected schema due to upgrade issues was allowed to operate until a 5.5.2.4 OVF based appliance tried to register into the VMDB which then failed. Error being reported is that the three already existing appliances that had been running 5.5.0.13 and that were upgraded to 5.5.2.4 did not fail to startup since they were already registered in the VMDB. It was not until a virgin 5.5.2.4 appliance tried to register that the issue was discovered. Version-Release number of selected component (if applicable):5.5.2.4 How reproducible: Steps to Reproduce: 1. install a 5.5.0.13 appliance and fully initialize 2. perform an upgrade of the appliance to 5.5.2.4 but do not run the db migrate so that the VMDB schema is backlevel. 3. restart the evmserverd and do a case insensitive grep of the evnm.log for the string 'Database Schema Version". there should be a log line indicating that the VMDB is not at the expected schema level. Actual results: the evmserverd continues running Expected results: evmserverd should emit a clear error message indicating a schema mismatch with the need to correct the mismatch and a reference to the need to run the appropriate rake db:migrate to bring the two into agreement. Additional info: Error from the virgin 5.5.2.4 is provided below which started the investigation into this issue. Since this appliance was trying to register into the database, the registration failed. ===== [----] E, [2016-03-11T08:30:23.921706 #24230:537988] ERROR -- : MIQ(MiqDbConfig#verify_config) Error: The requested database is not empty and has 2 migrations to apply. =====
This is not a bug. This expected behavior. When performing an upgrade the user is expected to review the logs for issues and to correct any reported issues before continuing. I am changing this to an RFE and re-assigning to Gregg Tanzillo for priority evaluation.
I'll work this as a bug. After talking with Josh I realized this issues creates support complications. Thanks Josh! JoeV
We don't have DB schema change migrations on a z-stream, however we *do* allow optional migrations in z-streams. These optional migrations would be things like adding an index or perhaps a data migration that purges data from tables that is not necessary. In these cases, the system can still run without them. During a z-stream update we do not want to force the customer to have to shut down their entire complex, which is the reasoning behind not allowing DB schema changes. However, for the optional migrations, the customer has the choice to do the db:migrate, to not do it at all, or to upgrade and then wait until a convenient time to do the db:migrate. Another reason we allow these optional migrations is that new customers who install fresh will automatically have these migrations in place. So, overall, there isn't a problem here to fix as this is by design. I think the only real problem here is that the log line in question should not be an ERROR, but instead should be a WARN. When it appears as an error, then the support team rightly assumes that this is part of the problem, but in reality it isn't.
Josh, > As this is a change in behavior with start up of the service I consider this a bug as that the service should not start with the wrong schema for the version of the codebase that is installed on the appliance. This is how it has been forever, so there is no change in behavior. > This is a pressing issue for support since a large number of customers do not follow the migration guide or miss the guide completely which results in a lot of high severity support case and escalations to support & engineering. The upgrade guide for a zstream update should not have the customer doing a db:migrate, so I see no way this could cause support issues (outside of the false positive analysis I mentioned above). Do you have specific BZs (particularly high priority ones) where the customer not doing a db:migrate on a zstream causes issues?
Josh, Ok, that makes sense, and we should discuss potential ystream update problems. This *particular* BZ was specifically about a zstream update, hence my response. Do you agree that this BZ should then be closed since it's not really an issue as stated for zstreams? I'm thinking let's create a new BZ that specifically addresses your concern about ystream updates. > Customers do not always find our migration documentation and alot of times they only perform a "yum update" on the appliance after changing the repo names. We may be able to prevent the system from coming up if we can detect a y-stream update. I may have to talk to Satoe on how best to implement a check like this, perhaps at the RPM spec level.