Description of problem: Provider migration script failed to edit corrdct tag filed in volumes attached to old provider Version-Release number of selected component (if applicable): $ oc get csv -n fusion-storage NAME DISPLAY VERSION REPLACES PHASE managed-fusion-agent.v2.0.11 Managed Fusion Agent 2.0.11 Succeeded observability-operator.v0.0.20 Observability Operator 0.0.20 observability-operator.v0.0.19 Succeeded ocs-operator.v4.13.0-168.stable OpenShift Container Storage 4.13.0-168.stable Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 Succeeded route-monitor-operator.v0.1.500-6152b76 Route Monitor Operator 0.1.500-6152b76 route-monitor-operator.v0.1.498-e33e391 Pending clusterVersion: NAME VERSION version 4.12.13 How reproducible: 2/2 Steps to Reproduce: 1. Install Appliance mode cluster 2. install FasS agent provider 3. Use migrate.sh script to migrate cluster. ./migrate.sh -provider <oldClusterID> <newClusterID> -d -dev repo checked out after PR#33 merged 4. wait for provider migration completion 5. Keep a watch on all cluste staus using rosa list cluster command Actual results: Provider migration script failed to edit correct tag key in volumes attached to old provider . ex: mon and osd volumes-> details page Tag should having entry of like "key kubernetes.io/cluster/sgatfane-p1425-svjlq" with value "owned" Expected results: Provider migration script should edit the correct tag key in volumes attached to the old provider mon and osd volumes-> details page Tag should not have entry of key with old cluster name. should have only one entry of new cluster with "owned" value. Additional info: Found below error in script of migration process for provider: ---------------------------------------------- ... deployment.apps/ocs-operator scaled deployment.apps/ocs-provider-server scaled Storage Provider endpoint: a3c12482a85cd4ec7abcca054045dc99-1658346607.us-east-2.elb.amazonaws.com:50051 Migration of provider is completed! Cluster ID found, getting kubeconfig Update EBS volume tags started Updating tags and storageClass for volume Id vol-02c134c1f2a17f43a parse error: Invalid numeric literal at line 1, column 43 parse error: Invalid numeric literal at line 1, column 43 VOLUMEMODIFICATION modifying 150 False 50 gp2 0 2023-04-25T04:58:26.000Z 3000 False 50 125 gp3 vol-02c134c1f2a17f43a Updating tags and storageClass for volume Id vol-0036e3478ce7dc63d parse error: Invalid numeric literal at line 1, column 5 parse error: Invalid numeric literal at line 1, column 5 VOLUMEMODIFICATION modifying 150 False 50 gp2 0 2023-04-25T04:58:31.000Z 3000 False 50 125 gp3 vol-0036e3478ce7dc63d Updating tags and storageClass for volume Id vol-054b0d3e24f5cde93 parse error: Invalid numeric literal at line 1, column 43 parse error: Invalid numeric literal at line 1, column 43 VOLUMEMODIFICATION modifying 12288 False 4096 gp2 0 2023-04-25T04:58:37.000Z 12000 False 4096 250 gp3 vol-054b0d3e24f5cde93 Updating tags and storageClass for volume Id vol-0340b99770e483fc2 parse error: Invalid numeric literal at line 1, column 34 parse error: Invalid numeric literal at line 1, column 34 VOLUMEMODIFICATION modifying 12288 False 4096 gp2 0 2023-04-25T04:58:43.000Z 12000 False 4096 250 gp3 vol-0340b99770e483fc2 Updating tags and storageClass for volume Id vol-0f335fe2462de68a6 parse error: Invalid numeric literal at line 1, column 40 parse error: Invalid numeric literal at line 1, column 40 VOLUMEMODIFICATION modifying 12288 False 4096 gp2 0 2023-04-25T04:58:49.000Z 12000 False 4096 250 gp3 vol-0f335fe2462de68a6 Updating tags and storageClass for volume Id vol-07cee571d61185aac parse error: Invalid numeric literal at line 1, column 43 parse error: Invalid numeric literal at line 1, column 43 VOLUMEMODIFICATION modifying 150 False 50 gp2 0 2023-04-25T04:58:54.000Z 3000 False 50 125 gp3 vol-07cee571d61185aac Finished Updating EBS volume tags Deleting the old/backup cluster Deletion of Service is started I: Service "2OtmXbK6rf058KWghkjO7lEhsX4" will start uninstalling now Cluster ID found, getting kubeconfig waiting for service to be deleted current state is deleting service storageconsumer.ocs.openshift.io/storageconsumer-2a4ec76d-d72d-4faf-a443-d5a1f6abdc40 patched (no change) storageconsumer.ocs.openshift.io/storageconsumer-6981ffc5-edf1-4170-b519-f0ed4beb3baf patched (no change) storageconsumer.ocs.openshift.io "storageconsumer-2a4ec76d-d72d-4faf-a443-d5a1f6abdc40" deleted storageconsumer.ocs.openshift.io "storageconsumer-6981ffc5-edf1-4170-b519-f0ed4beb3baf" deleted storagesystem.odf.openshift.io/ocs-storagecluster-storagesystem patched storagecluster.ocs.openshift.io/ocs-storagecluster patched Deletion of Old Provider Service cluster is started, the service will be deleted soon. Run the following commands in new terminal, sequentially/parellel to migrate the consumers. ... -------------------------------------------------------------------------------------
The script fetches the aws EBS volumes key, using aws ec2 describe-volumes --volume-id $volumeID --filters Name=tag:kubernetes.io/created-for/pvc/namespace,Values=openshift-storage --region $region --query "Volumes[*].Tags" | jq .[] | jq -r '.[]| select (.Value == "owned")|.Key', if the default output is not set to json in aws configure, it will fail. We added the --output flag to each commands where we fetch the tags. That should solve the issue. PR: https://github.com/rchikatw/odf-managed-service-migration/pull/34
Suchita, PR: https://github.com/rchikatw/odf-managed-service-migration/pull/34 is merged please take the latest changes and verify the migration.
In migration console logs no error was found related to the tag updated. Also, mon and osd volumes are attached to migrated cluster-> details page-> Tags 1. No entry of key with old cluster name. 2. Only one entry of the new cluster with an "owned" value. more outputs captured Here in doc:https://docs.google.com/document/d/13FeKH1zImP8pMqDDlpapNDkpk3iVmBg2FcRIHLPN7Xc/edit?usp=sharing Hence Marking this BZ as verified. verified with ROSA4.12 and ODF4.12.3-12 Fass Provider migration setup.