Bug 2073488
| Summary: | Errors in rsync pod creation are not printed in the controller logs | |||
|---|---|---|---|---|
| Product: | Migration Toolkit for Containers | Reporter: | Pranav Gaikwad <pgaikwad> | |
| Component: | Controller | Assignee: | Pranav Gaikwad <pgaikwad> | |
| Status: | CLOSED ERRATA | QA Contact: | Xin jiang <xjiang> | |
| Severity: | medium | Docs Contact: | Richard Hoch <rhoch> | |
| Priority: | medium | |||
| Version: | 1.6.0 | CC: | prajoshi, rjohnson | |
| Target Milestone: | --- | |||
| Target Release: | 1.6.4 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2073494 2073496 (view as bug list) | Environment: | ||
| Last Closed: | 2022-04-19 08:08:11 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2073496 | |||
Verified with MTC 1.6.4 (Stage)
Image: registry.redhat.io/rhmtc/openshift-migration-controller-rhel8@sha256:7c703c6b173ea68ac235e61b093e2e355e1bcf9969774bd1131f2318ddc762e4
I can see the rsync creation error logs in controller pod.
$ oc logs migration-controller-6986458657-4smvm -n openshift-migration mtc
{"level":"info","ts":1649858724.3218942,"logger":"directvolume","msg":"Getting Rsync Password from Secret on host MigCluster","dvm":"migration-692a9-lph9s","migMigration":"migration-692a9","phase":"RunRsyncOperations","secret":""}
{"level":"error","ts":1649858724.4183102,"logger":"directvolume","msg":"failed creating rsync client","dvm":"migration-692a9-lph9s","migMigration":"migration-692a9","phase":"RunRsyncOperations","pvc":"bug/test-data","error":"pods \"rsync-52g8t\" is forbidden: exceeded quota: zero-pod-quota, requested: pods=1, used: pods=0, limited: pods=0","errorCauses":[{"error":"pods \"rsync-52g8t\" is forbidden: exceeded quota: zero-pod-quota, requested: pods=1, used: pods=0, limited: pods=0","errorCauses":[{"error":"pods \"rsync-52g8t\" is forbidden: exceeded quota: zero-pod-quota, requested: pods=1, used: pods=0, limited: pods=0"}]}],"stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/app/vendor/github.com/go-logr/zapr/zapr.go:132\ngithub.com/konveyor/controller/pkg/logging.(*Logger).Error\n\t/remote-source/app/vendor/github.com/konveyor/controller/pkg/logging/logger.go:109\ngithub.com/konveyor/controller/pkg/logging.(*Logger).Error\n\t/remote-source/app/vendor/github.com/konveyor/controller/pkg/logging/logger.go:109\ngithub.com/konveyor/mig-controller/pkg/controller/directvolumemigration.(*Task).createRsyncTransferClients\n\t/remote-source/app/pkg/controller/directvolumemigration/rsync.go:496\ngithub.com/konveyor/mig-controller/pkg/controller/directvolumemigration.(*Task).runRsyncOperations\n\t/remote-source/app/pkg/controller/directvolumemigration/rsync.go:1542\ngithub.com/konveyor/mig-controller/pkg/controller/directvolumemigration.(*Task).Run\n\t/remote-source/app/pkg/controller/directvolumemigration/task.go:383\ngithub.com/konveyor/mig-controller/pkg/controller/directvolumemigration.(*ReconcileDirectVolumeMigration).migrate\n\t/remote-source/app/pkg/controller/directvolumemigration/migrate.go:39\ngithub.com/konveyor/mig-controller/pkg/controller/directvolumemigration.(*ReconcileDirectVolumeMigration).Reconcile\n\t/remote-source/app/pkg/controller/directvolumemigration/directvolumemigration_controller.go:148\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99"}
{"level":"info","ts":1649858724.4184334,"logger":"directvolume","msg":"encountered repeated errors attempting to create Rsync Pods","dvm":"migration-692a9-lph9s","migMigration":"migration-692a9","phase":"RunRsyncOperations"}
Moving this to verified status.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Migration Toolkit for Containers (MTC) 1.6.4 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1393 |
Description of problem: Any error happening during creation of Rsync pods does not get printed in the controller logs Version-Release number of selected component (if applicable): 1.6.0 How reproducible: Always Steps to Reproduce: 1. Deploy any sample application in source namespace 2. Create a ResourceQuota (after creating application) so that no new Pods can be created in the source namespace: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: zero-pod-quota namespace: rocket-chat spec: hard: pods: "0" ``` 3. Create a migration plan for above namespace and select DVM 4. Run a Stage/Cutover migration, and observe that it will get stuck at DirectVolume step with message "Running Rsync Pods to migrate Persistent Volume data" Actual results: The migration gets stuck, there are no logs in the controller pertaining to creation of Pods. Expected results: Migration getting stuck is expected, but there should be some logs explaining the actual error in Pod creation. Additional info: