Bug 1749906

Created attachment 1612501 [details]
nodejs/mongodb workload script

Description of problem:
mongodb pod fails to connect to server after migration

Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.0-0.nightly-2019-09-04-090826   True        False         2d3h    Cluster version is 4.2.0-0.nightly-2019-09-04-090826

# oc describe pod/migration-operator-bfdbbd946-vkpkl | grep Image
    Image:         quay.io/ocpmigrate/mig-operator:stable
    Image ID:      quay.io/ocpmigrate/mig-operator@sha256:89ee280659a198a106243b1a0494cb8d98b6b2dbc3f072118b902b989d3ea491
    Image:          quay.io/ocpmigrate/mig-operator:stable
    Image ID:       quay.io/ocpmigrate/mig-operator@sha256:89ee280659a198a106243b1a0494cb8d98b6b2dbc3f072118b902b989d3ea491
 # oc describe pod/migration-ui-99578967c-gt8kc | grep Image
    Image:          quay.io/ocpmigrate/mig-ui:stable
    Image ID:       quay.io/ocpmigrate/mig-ui@sha256:bffe00e5c6a812a14018c0d0916e626e00342b5254546de47d0ce5d6052776d8
# oc describe pod/velero-7474bd69fd-56l8j | grep Image
    Image:          quay.io/ocpmigrate/migration-plugin:stable
    Image ID:       quay.io/ocpmigrate/migration-plugin@sha256:b4493d826260eb1e3e02ba935aaedfd5310fefefb461ca7dcd9a5d55d4aa8f35
    Image:         quay.io/ocpmigrate/velero:stable
    Image ID:      quay.io/ocpmigrate/velero@sha256:957725dec5f0fb6a46dee78bd49de9ec4ab66903eabb4561b62ad8f4ad9e6f05

How reproducible:
always

Steps to Reproduce:
1. Create projects with the attached workload on OCP 3.11 cluster.
2. Create migration operator, velero and restic on the OCP 3.11 cluster.
3. Create migration operator and controller on an OCP 4.2 cluster.
4. configure migration plan to include the namespace to be migrated.
5. Configure source/destination clusters, and the storage in the migration CR yamls.
6. Start migration.

Actual results:
Migration is successful but mongodb pod fails when trying to connect to the server.

# oc logs mongodb-1-4zbxf
=> sourcing /usr/share/container-scripts/mongodb/pre-init//10-check-env-vars.sh ...
=> sourcing /usr/share/container-scripts/mongodb/pre-init//20-setup-wiredtiger-cache.sh ...
=> [Fri Sep 06 18:01:09] wiredTiger cache_size set to 256M
=> sourcing /usr/share/container-scripts/mongodb/pre-init//30-set-config-file.sh ...
=> sourcing /usr/share/container-scripts/mongodb/pre-init//35-setup-default-datadir.sh ...
=> sourcing /usr/share/container-scripts/mongodb/pre-init//40-setup-keyfile.sh ...
=> sourcing /usr/share/container-scripts/mongodb/pre-init//41-setup-ssl.sh ...
=>  Waiting for MongoDB daemon up
2019-09-06T18:01:09.909+0000 I STORAGE  [main] Engine custom option: cache_size=256M
MongoDB shell version v3.6.3
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] MongoDB starting : pid=26 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-1-4zbxf
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] db version v3.6.3
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] git version: nogitversion
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] modules: none
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] build environment:
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-09-06T18:01:09.914+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "localhost", port: 27017 }, replication: { oplogSizeMB: 64 }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb/data", wiredTiger: { engineConfig: { configString: "cache_size=256M" } } }, systemLog: { quiet: true } }
2019-09-06T18:01:09.914+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb/data created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
2019-09-06T18:01:09.919+0000 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/data/journal
2019-09-06T18:01:09.919+0000 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2019-09-06T18:01:09.919+0000 I JOURNAL  [initandlisten] 
2019-09-06T18:01:09.919+0000 E JOURNAL  [initandlisten] Insufficient free space for journal files
2019-09-06T18:01:09.919+0000 I JOURNAL  [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/data/journal or use --smallfiles
2019-09-06T18:01:09.919+0000 I JOURNAL  [initandlisten] 
2019-09-06T18:01:09.919+0000 I STORAGE  [initandlisten] exception in initAndListen: Location15926: Insufficient free space for journals, terminating
2019-09-06T18:01:09.919+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2019-09-06T18:01:09.919+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-09-06T18:01:09.919+0000 I REPL     [initandlisten] shutdown: removing all drop-pending collections...
2019-09-06T18:01:09.919+0000 I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
2019-09-06T18:01:09.919+0000 I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...
2019-09-06T18:01:09.919+0000 I STORAGE  [initandlisten] shutdown: final commit...
2019-09-06T18:01:09.919+0000 I STORAGE  [initandlisten] shutdown: closing all files...
2019-09-06T18:01:09.919+0000 I STORAGE  [initandlisten] closeAllFiles() finished
2019-09-06T18:01:09.919+0000 I CONTROL  [initandlisten] now exiting
2019-09-06T18:01:09.920+0000 I CONTROL  [initandlisten] shutting down with code:100
connecting to: mongodb://127.0.0.1:27017/localhost
2019-09-06T18:01:09.943+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2019-09-06T18:01:09.944+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
=>  Waiting for MongoDB daemon up
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017/localhost
2019-09-06T18:01:11.005+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2019-09-06T18:01:11.005+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
=>  Waiting for MongoDB daemon up
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017/localhost
2019-09-06T18:01:12.055+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2019-09-06T18:01:12.055+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
=>  Waiting for MongoDB daemon up
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017/localhost
2019-09-06T18:01:13.104+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2019-09-06T18:01:13.104+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
=>  Waiting for MongoDB daemon up


Expected results:
pods should be functional after migration.

Additional info: