Description of problem: Executing a migration with a hook attached fails at the prebackupHooks stage. Version-Release number of selected component (if applicable): MTC 1.7.6 How reproducible: Always Steps to Reproduce: 1. Create a namespace, deploy an app in it 2. Create a new hook 3. Create a migplan and set the hook to run in prebackup stage Actual results: The migplan fails, because of the hook failing on prebackup stage Expected results: The migplan should be executed successfully without errors Additional Info: From the logs openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"dpanic","ts":1669892862.7693396,"logger":"migration","msg":"odd number of arguments passed as key-value pairs for logging","migMigration":"migration-766bc","phase":"PreBackupHooks","ignored key":"list-pods-hook-hz2xs","stacktrace":"github.com/konveyor/controller/pkg/logging.(*Logger).Info\n\t/remote-source/mig-controller/app/vendor/github.com/konveyor/controller/pkg/logging/logger.go:68\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*Task).runHooks\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/hooks.go:79\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*Task).Run\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/task.go:516\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*ReconcileMigMigration).migrate\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/migrate.go:70\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*ReconcileMigMigration).Reconcile\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/migmigration_controller.go:264\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"} openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"info","ts":1669892862.7693353,"logger":"migration","msg":"Creating Job for MigHook","migMigration":"migration-766bc","phase":"PreBackupHooks","job":"robot-source","migHook":"openshift-migration"} openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"info","ts":1669892862.7774823,"logger":"migration","msg":"Found abnormal event for Hook Job","migMigration":"migration-766bc","phase":"PreBackupHooks","Job":"robot-source/test-hook-prebackup-9284w","eventType":"Warning","eventReason":"BackoffLimitExceeded","eventMessage":"Job has reached the specified backoff limit","eventFirstTimestamp":"2022-12-01 11:07:41 +0000 UTC"} openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"info","ts":1669892862.7775545,"logger":"migration","msg":"Marking migration as FAILED. See Status.Errors","migMigration":"migration-766bc","phase":"PreBackupHooks","migrationErrors":["Hook job test-hook-prebackup-9284w failed."]} openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"info","ts":1669892862.7775774,"logger":"migration","msg":"Phase execution failed.","migMigration":"migration-766bc","phase":"PreBackupHooksFailed","phaseDescription":"Migration failed while running user-defined pre-backup hooks.","error":"Hook job test-hook-prebackup-9284w failed."} openshift-migration migration-controller-85786b644d-6xtdp mtc {"level":"info","ts":1669892862.777595,"logger":"migration","msg":"","migMigration":"migration-766bc","error":"Hook job test-hook-prebackup-9284w failed.","stacktrace":"\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*Task).runHooks()\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/hooks.go:84\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*Task).Run()\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/task.go:516\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*ReconcileMigMigration).migrate()\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/migrate.go:70\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*ReconcileMigMigration).Reconcile()\n\t/remote-source/mig-controller/app/pkg/controller/migmigration/migmigration_controller.go:264\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler()\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem()\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n\t/remote-source/mig-controller/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1581"}
Sachin, Was this working in the last MTC 1.7.6 build?
No, its failing in 1.7.6
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'kubernetes' fatal: [localhost]: FAILED! => {"changed": false, "error": "No module named 'kubernetes'", "msg": "Failed to import the required Python library (kubernetes) on test-5465b59747-qgssk's Python /usr/bin/python3.9. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"} We have a dnf-update in our Dockerfile that's probably picking up a newer ansible. On the base image ansible is using python 3.8 and all the python modules, etc. are installed against python 3.8. Our downstream Dockerfile needs to be corrected to prevent upgrading ansible to a version that won't work with the runner base layer.
Verified using latest MTC stage build
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.7.6 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:9047