Description of problem: Enable featuregate to trigger upgradeable=false condition. Y version upgrade from v4.3 to v4.4 was not blocked. Version-Release number of the following components: v4.3.15-4.4.0-rc.11 How reproducible: always Steps to Reproduce: 1. Install v4.3.15 2. Enable featuregate to trigger upgradeable=false condition { "lastTransitionTime": "2020-04-23T10:51:07Z", "message": "Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: \"TechPreviewNoUpgrade\" does not allow updates", "reason": "FeatureGates_RestrictedFeatureGates_TechPreviewNoUpgrade", "status": "False", "type": "Upgradeable" } 3. Patch cv to fast-4.4 channel and do upgrade # ./oc adm upgrade --to 4.4.0-rc.11 Updating to 4.4.0-rc.11 ... # ./oc adm upgrade info: An upgrade is in progress. Working towards 4.4.0-rc.11: downloading update ... # ./oc adm upgrade info: An upgrade is in progress. Unable to apply 4.4.0-rc.11: the update could not be applied ... # ./oc adm upgrade info: An upgrade is in progress. Working towards 4.4.0-rc.11: 83% complete At last, upgrade successfully. # ./oc get clusterversion -o json|jq -r '.items[0].status.history[]' { "completionTime": "2020-04-23T11:44:30Z", "image": "quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07", "startedTime": "2020-04-23T10:54:08Z", "state": "Completed", "verified": true, "version": "4.4.0-rc.11" } { "completionTime": "2020-04-23T10:32:08Z", "image": "quay.io/openshift-release-dev/ocp-release@sha256:0e9642d28c12f5f54c1ab0fffbfd866daa6179a900e745a935f17f8e6e1e28fc", "startedTime": "2020-04-23T10:15:06Z", "state": "Completed", "verified": false, "version": "4.3.15" } Actual results: Upgrade can bump successfully. Expected results: Upgrade should be blocked. Additional info: Please attach logs from ansible-playbook with the -vvv flag
Created attachment 1681092 [details] cvo.log
Moved to 4.5. My hunch is that this is a race where the CVO is delayed in noticing the ClusterOperator setting Upgradeable=False (from comment 0, the delay was from the 10:51:07 ClusterOperator change to the 10:54:08 update initialization, so ~3m). If that's the issue, this is probably low or medium priority. If the CVO continues to allow this sort of thing even with a longer delay (10m?), then it's probably high/urgent and... not sure how likely it is to get named a 4.4.0 blocker. But it would be close. Will test with a longer delay...
Sadly, seem like "not a race". $ oc get -o json clusterversion version | jq -r .status.desired.version 4.3.15 $ oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "candidate-4.4"}]' $ oc patch featuregate cluster --type json -p '[{"op": "add", "path": "/spec/featureSet", "value": "TechPreviewNoUpgrade"}]' $ sleep 600 Meanwhile, in a second terminal: $ oc -n openshift-cluster-version get pods NAME READY STATUS RESTARTS AGE cluster-version-operator-5fb76fc5cc-bwngd 1/1 Running 0 18m $ oc -n openshift-cluster-version logs -f cluster-version-operator-5fb76fc5cc-bwngd ... Then, post-sleep: $ oc adm upgrade --to 4.4.0-rc.11 Updating to 4.4.0-rc.11 $ oc adm upgrade info: An upgrade is in progress. Working towards 4.4.0-rc.11: 17% complete ... Back to the streamed logs from the second terminal: I0424 17:34:31.777559 1 cvo.go:471] Desired version from operator is v1.Update{Version:"4.3.15", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:0e9642d28c12f5f54c1ab0fffbfd866daa6179a900e745a935f17f8e6e1e28fc", Force:false} I0424 17:34:31.778304 1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (832.691µs) I0424 17:34:38.952861 1 cvo.go:526] Started syncing upgradeable "openshift-cluster-version/version" (2020-04-24 17:34:38.952853217 +0000 UTC m=+2140.667929080) I0424 17:34:38.952926 1 upgradeable.go:28] Upgradeable conditions were recently checked, will try later. ... I0424 17:34:38.953036 1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-rc.11", Image:"quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07", Force:false} I0424 17:34:38.953144 1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (178.62µs) I0424 17:34:38.953190 1 sync_worker.go:471] Running sync 4.4.0-rc.11 (force=false) on generation 3 in state Updating at attempt 0 I0424 17:34:38.953203 1 sync_worker.go:477] Loading payload I0424 17:34:38.953229 1 store.go:74] use cached most recent signature config maps I0424 17:34:38.959158 1 store.go:65] remember most recent signature config maps: signatures-managed I0424 17:34:38.959187 1 store.go:116] searching for sha256-6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 in signature config map signatures-managed I0424 17:34:46.777483 1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-24 17:34:46.777469065 +0000 UTC m=+2148.492544927) I0424 17:34:46.777549 1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-rc.11", Image:"quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400d\ d8df98cd2d6f07", Force:false} I0424 17:34:46.786577 1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (9.099581ms) I0424 17:34:46.786775 1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-24 17:34:46.786769485 +0000 UTC m=+2148.501845310) I0424 17:34:46.786818 1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-rc.11", Image:"quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400d\ d8df98cd2d6f07", Force:false} I0424 17:34:46.786930 1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (157.544µs) I0424 17:34:46.786583 1 cvo.go:526] Started syncing upgradeable "openshift-cluster-version/version" (2020-04-24 17:34:46.786573984 +0000 UTC m=+2148.501649839) I0424 17:34:46.786962 1 upgradeable.go:28] Upgradeable conditions were recently checked, will try later. I0424 17:34:46.786968 1 cvo.go:528] Finished syncing upgradeable "openshift-cluster-version/version" (391.505µs) I0424 17:34:46.786589 1 cvo.go:503] Started syncing available updates "openshift-cluster-version/version" (2020-04-24 17:34:46.786583501 +0000 UTC m=+2148.501659341) I0424 17:34:46.786988 1 availableupdates.go:39] Available updates were recently retrieved, will try later. I0424 17:34:46.786993 1 cvo.go:505] Finished syncing available updates "openshift-cluster-version/version" (407.771µs) I0424 17:34:48.148622 1 payload.go:210] Loading updatepayload from "/etc/cvo/updatepayloads/3dV_-8XN1JB7Qodu_msYrg" I0424 17:34:48.595128 1 sync_worker.go:539] Payload loaded from quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 with hash KYpQ0Ba7y-Y= ... I0424 17:34:48.657875 1 sync_worker.go:621] Running sync for deployment "openshift-cluster-version/cluster-version-operator" (5 of 573) I0424 17:34:48.674142 1 apps.go:115] Deployment cluster-version-operator is not ready. status: (replicas: 1, updated: 1, ready: 1, unavailable: 0) I0424 17:34:48.736146 1 start.go:140] Shutting down due to terminated I0424 17:34:48.736355 1 start.go:188] Stepping down as leader I0424 17:34:48.736790 1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-24 17:34:48.736780502 +0000 UTC m=+2150.451856328) I0424 17:34:48.736841 1 task_graph.go:583] Canceled worker 12 E0424 17:34:48.736877 1 task.go:77] error running apply for deployment "openshift-cluster-version/cluster-version-operator" (5 of 573): timed out waiting for the condition ... I0424 17:34:48.740667 1 task_graph.go:603] Workers finished I0424 17:34:48.740681 1 task_graph.go:611] Result of work: [Could not update deployment "openshift-cluster-version/cluster-version-operator" (5 of 573)] I0424 17:34:48.740701 1 sync_worker.go:783] Summarizing 1 errors I0424 17:34:48.740709 1 sync_worker.go:787] Update error 5 of 573: UpdatePayloadFailed Could not update deployment "openshift-cluster-version/cluster-version-operator" (5 of 573) (*errors.errorString: timed out waiting for the condition) E0424 17:34:48.740738 1 sync_worker.go:329] unable to synchronize image (waiting 1m26.262851224s): Could not update deployment "openshift-cluster-version/cluster-version-operator" (5 of 573) I0424 17:34:48.740240 1 task_graph.go:531] No more reachable nodes in graph, continue I0424 17:34:48.740761 1 task_graph.go:567] No more work I0424 17:34:48.740912 1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-rc.11", Image:"quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07", Force:false} I0424 17:34:48.816852 1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (80.060854ms) I0424 17:34:48.816980 1 cvo.go:366] Shutting down ClusterVersionOperator I0424 17:34:48.863476 1 start.go:199] Finished shutdown I see the signature check happening, but do not see any evidence of the upgradeable precondition check happening. On the other hand, our only precondition logging is the negative "check failed" [1]. I've filed [2] to get the CVO to explain itself. [1]: https://github.com/openshift/cluster-version-operator/blob/86b9bdba55a85e2e071603916db4c43b481e7588/pkg/payload/precondition/precondition.go#L74 [2]: https://github.com/openshift/cluster-version-operator/pull/360
Trying to test now with [1] -> 4.4. [1]: https://github.com/openshift/cluster-version-operator/pull/362
Running from cvo#362: $ oc get -o json clusterversion version | jq -r .status.desired.version 0.0.1-2020-04-24-185833 $ oc patch featuregate cluster --type json -p '[{"op": "add", "path": "/spec/featureSet", "value": "TechPreviewNoUpgrade"}]' In another terminal: $ oc -n openshift-cluster-version get pods NAME READY STATUS RESTARTS AGE cluster-version-operator-57d9c446df-2s24g 1/1 Running 0 24m $ oc -n openshift-cluster-version logs -f cluster-version-operator-57d9c446df-2s24g | tee cvo.log Then trigger the update: $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.4.0-rc.11-x86_64 | grep Pull Pull From: quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 $ oc get -o json clusterversion version | jq -r '.status.conditions[] | .lastTransitionTime + " " + .type + " " + .status + " " + .message' | sort 2020-04-24T19:06:24Z RetrievedUpdates False Unable to retrieve available updates: currently installed version 0.0.1-2020-04-24-185833 not found in the "stable-4.3" channel 2020-04-24T19:23:07Z Available True Done applying 0.0.1-2020-04-24-185833 2020-04-24T19:32:20Z Upgradeable False Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates 2020-04-24T19:32:22Z Failing True The update cannot be verified: unable to locate a valid signature for one or more sources 2020-04-24T19:32:22Z Progressing True Unable to apply quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07: the image may not be safe to use Grr. I guess I'll force: $ oc adm upgrade --clear $ oc adm upgrade --force --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 Checking the gathered logs: I0424 19:36:50.605797 1 verify.go:298] Unable to verify sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 against keyring verifier-public-key-openshift-ci W0424 19:36:50.606081 1 updatepayload.go:100] An image was retrieved from "quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07" that failed verification: The update cannot be verified: unable to locate a valid signature for one or more sources I0424 19:36:50.606690 1 payload.go:210] Loading updatepayload from "/etc/cvo/updatepayloads/3dV_-8XN1JB7Qodu_msYrg" E0424 19:36:51.604090 1 precondition.go:59] Precondition "ClusterVersionUpgradeable" failed: Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates I0424 19:36:51.604316 1 sync_worker.go:530] Forcing past precondition failures: Precondition "ClusterVersionUpgradeable" failed because of "FeatureGates_RestrictedFeatureGates_TechPreviewNoUpgrade": Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates I0424 19:36:51.604404 1 sync_worker.go:547] Payload loaded from quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 with hash KYpQ0Ba7y-Y= So that looks right. Not sure why that precondition wasn't failing when I was starting from 4.3.15. Should eventually get CI logs for this cluster up at [1], for folks who want to poke around. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/538
Coming in again, with a new run from cvo#362: $ oc get -o json clusterversion version | jq -r .status.desired.version 0.0.1-2020-04-25-041030 $ oc patch featuregate cluster --type json -p '[{"op": "add", "path": "/spec/featureSet", "value": "TechPreviewNoUpgrade"}]' $ oc get -o json clusteroperator kube-apiserver | jq -r '.status.conditions[] | select(.type == "Upgradeable") | .lastTransitionTime + " " + .type + " " + .status + " " + .message' 2020-04-25T04:43:21Z Upgradeable False FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates $ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "Upgradeable") | .lastTransitionTime + " " + .type + " " + .status + " " + .message' ...no output; CVO hasn't noticed yet... $ oc get -o json clusterversion version | jq -r '.status.conditions[] | select(.type == "Upgradeable") | .lastTransitionTime + " " + .type + " " + .status + " " + .message' 2020-04-25T04:44:58Z Upgradeable False Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates In another terminal: $ oc -n openshift-cluster-version get pods NAME READY STATUS RESTARTS AGE cluster-version-operator-687b7d7f89-655zb 1/1 Running 0 20m $ oc -n openshift-cluster-version logs -f --tail=1 cluster-version-operator-687b7d7f89-655zb | tee cvo.log Looks like we don't sign CI builds at all at the moment, although I've opened [1] to allow CI -> offical updates. Anyhow, forcing again: $ oc adm upgrade --force --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 Checking the gathered logs: I0425 04:46:44.858369 1 store.go:116] searching for sha256-6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 in signature config map signatures-managed I0425 04:46:45.002408 1 verify.go:298] Unable to verify sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 against keyring verifier-public-key-openshift-ci W0425 04:46:45.002447 1 updatepayload.go:100] An image was retrieved from "quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07" that failed verification: The update cannot be verified: unable to locate a valid signature for one or more sources ... I0425 04:47:03.025422 1 payload.go:210] Loading updatepayload from "/etc/cvo/updatepayloads/3dV_-8XN1JB7Qodu_msYrg" E0425 04:47:03.913561 1 precondition.go:59] Precondition "ClusterVersionUpgradeable" failed: Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates I0425 04:47:03.913609 1 sync_worker.go:530] Forcing past precondition failures: Precondition "ClusterVersionUpgradeable" failed because of "FeatureGates_RestrictedFeatureGates_TechPreviewNoUpgrade": Cluster operator kube-apiserver cannot be upgraded: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates I0425 04:47:03.913622 1 sync_worker.go:547] Payload loaded from quay.io/openshift-release-dev/ocp-release@sha256:6f09986c2c878f9675afcf9ee5d4720cf8ec0b9b832ab9400dd8df98cd2d6f07 with hash KYpQ0Ba7y-Y= I dunno, still looks fine to me. I guess we'll see what happens if we land these debugging logs and then try to take the 4.5-nightly back down to an official 4.4 release. [1]: https://github.com/openshift/cluster-update-keys/pull/16
Logging alone is not a fix.
Do we know of the upgradeable = false is not blocking the upgrade only when we enable featuregate or for any reason upgradeable = false is set?
This bug is important and we want to fix it, but we will probably not have time to close it out this sprint. I'm adding UpcomingSprint now, and we'll revisit next sprint.
Haven't had time to get to this yet; adding UpcomingSprint
During debug of https://bugzilla.redhat.com/show_bug.cgi?id=1822513, which is also caused by a race condition, I found it very easy to bypass the Preconditions by: $ oc patch featuregate cluster --type json -p '[{"op": "add", "path": "/spec/featureSet", "value": "TechPreviewNoUpgrade"}]' Initially cluster_operator_conditions{name=kube-apiserver} is Upgradeable false but it can take ~minutes before cluster_operator_conditions{name=version} is set Upgradeable false during which time Precondition ClusterVersionUpgradeable passes with "no Upgradeable condition on ClusterVersion".
During 4.4.10 to 4.5.0-rc.5 upgrade with default upgradeable condition triggered by marketplace. We still hit it when do upgrade with --to command(no this issue when run upgrade with --to-image) # ./oc version Client Version: 4.5.0-rc.5 Server Version: 4.4.10
If you use --to-image you may be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1822513 which is being caused by the CVO thread that updates version history adds a new cluster version history entry before the thread loading the upgrade version has set the version field resulting in an empty field. The y-version upgrade check then fails and allows the upgrade.
I'm still not aware of any CVO logs (since we grew the additional logging) from the outgoing pod when we trip whatever the Upgradeable=False bug was. Punting to UpcomingSprint.
I'm going to close this INSUFFICIENT_DATA. If we can get a reproducer with outgoing-CVO logs, we can re-open.
Hi W. Trevor King, I'm not sure why this bus is closed with INSUFFICIENT_DATA. It's reported in v4.4 for the 1st time with cvo log attached in comment1. And it's not fixed yet so we hit it again during v4.5. Anything missed here?
[1] landed some additional logging I need to diagnose the issue. That went out in 4.5, but I'm not aware of any 4.5-to-whatever updates where the outgoing CVO included the new logging and someone captured the outgoing CVO's logs. If you can get logs with the logging (they will contain 'Precondition ... passed'), please attach them here and reopen the bug. [1]: https://github.com/openshift/cluster-version-operator/pull/360
sure, i will give a cvo log for 4.5 later.
Still hit it on v4.5.5-v4.6 nightly upgrade with upgradeable=false. I post both pre and post cvo logs in attachment.
cvo_pre is a 4.5.5 install that is later asked to move to a 4.6 nightly: $ grep 'Running sync.*in state' cvo_pre | head -n1 I0807 07:00:06.965053 1 sync_worker.go:471] Running sync 4.5.5 (force=false) on generation 1 in state Initializing at attempt 0 $ grep 'Running sync.*in state' cvo_pre | tail -n1 I0807 08:24:58.441976 1 sync_worker.go:471] Running sync 4.6.0-0.nightly-2020-08-07-034746 (force=false) on generation 11 in state Updating at attempt 0 Here's the transition: $ grep 'Desired version\|Precondition\|Payload loaded' cvo_pre | tail -n7 I0807 08:24:52.439911 1 cvo.go:469] Desired version from operator is v1.Update{Version:"4.5.5", Image:"quay.io/openshift-release-dev/ocp-release@sha256:a58573e1c92f5258219022ec104ec254ded0a70370ee8ed2aceea52525639bd4", Force:false} I0807 08:24:58.441689 1 cvo.go:466] Desired version from spec is v1.Update{Version:"4.6.0-0.nightly-2020-08-07-034746", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:53fa74a6d426822340be6bf86a659cbae92de021ade6d2048fa117713e3ec379", Force:false} I0807 08:25:07.439708 1 cvo.go:466] Desired version from spec is v1.Update{Version:"4.6.0-0.nightly-2020-08-07-034746", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:53fa74a6d426822340be6bf86a659cbae92de021ade6d2048fa117713e3ec379", Force:false} I0807 08:25:07.454705 1 cvo.go:466] Desired version from spec is v1.Update{Version:"4.6.0-0.nightly-2020-08-07-034746", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:53fa74a6d426822340be6bf86a659cbae92de021ade6d2048fa117713e3ec379", Force:false} I0807 08:25:08.087714 1 upgradeable.go:70] Precondition "ClusterVersionUpgradeable" passed: minor from the current 4.6.0-0.nightly-2020-08-07-034746 matches minor from the target 4.6.0-0.nightly-2020-08-07-034746 (both 6). I0807 08:25:08.087745 1 sync_worker.go:547] Payload loaded from registry.svc.ci.openshift.org/ocp/release@sha256:53fa74a6d426822340be6bf86a659cbae92de021ade6d2048fa117713e3ec379 with hash Gtm_sq0K8MQ= I0807 08:25:08.531547 1 cvo.go:466] Desired version from spec is v1.Update{Version:"4.6.0-0.nightly-2020-08-07-034746", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:53fa74a6d426822340be6bf86a659cbae92de021ade6d2048fa117713e3ec379", Force:false} So the problem seems to be that the "current" version being fed into the precondition is actually the new target version. I think that makes this a dup of bug 1826115, but we can re-open this one if we can still reproduce in updates out of future versions which fix 1826115. *** This bug has been marked as a duplicate of bug 1826115 ***