Bug 963071 - Resolve references to env vars during migration to raw values
Resolve references to env vars during migration to raw values
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-15 01:49 EDT by Paul Morie
Modified: 2015-05-14 19:11 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-10 23:58:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paul Morie 2013-05-15 01:49:27 EDT
In V1 it is valid to have an env var reference another env var as follows:

export SOME_VAR=$SOME_OTHER_VAR

We don't support this in V2.  The migration should resolve these references.
Comment 1 Jhon Honce 2013-05-15 14:12:14 EDT
What context were these used?  Any ruby code reading these environment variables would have failed.
Comment 2 Paul Morie 2013-05-16 10:52:46 EDT
After talking about this with Jhon, we decided that we should delete the OPENSHIFT_LOG_DIR var (which is the variable which is currently a reference to another var), since we won't update the value of this old var if the new one changes.
Comment 4 Meng Bo 2013-05-20 06:29:12 EDT
This issue not fixed on devenv_3247, with following steps:

1. Create php app on old instance
2. Write OPENSHIFT_LOG_DIR with following content to the app .env dir
export OPENSHIFT_LOG_DIR=$OPENSHIFT_PHP_LOG_DIR
3. Change the context of this file to match with others
4. Do upgrade and migrate

During the migration:
Migrated PATH v1 value [export PATH=/usr/libexec/openshift/cartridges/php-5.3/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin] to raw value [/usr/libexec/openshift/cartridges/php-5.3/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin]
Migrated OPENSHIFT_LOG_DIR v1 value [export OPENSHIFT_LOG_DIR=$OPENSHIFT_PHP_LOG_DIR] to raw value [$OPENSHIFT_PHP_LOG_DIR]

After migration:
[app1-bmeng111.dev.rhcloud.com 5199ec78ac3424090e000001]\> env|grep LOG
OPENSHIFT_PHP_LOG_DIR=/var/lib/openshift/5199ec78ac3424090e000001/php//logs/
OPENSHIFT_LOG_DIR=$OPENSHIFT_PHP_LOG_DIR
LOGNAME=5199ec78ac3424090e000001

Move bug back.
Comment 5 Paul Morie 2013-05-20 09:40:32 EDT
We've only coded to remove the OPENSHIFT_LOG_DIR var from TYPELESS_TRANSLATED_VARS.  There should not be any discrete OPENSHIFT_LOG_DIR variables in existing V1 apps.
Comment 6 Meng Bo 2013-05-21 05:49:10 EDT
According comment#5,

Add TYPELESS_TRANSLATED_VARS file under /var/lib/openshift/$uuid/.env, with following contents:

export OPENSHIFT_RUNTIME_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/runtime/"
export OPENSHIFT_RUN_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/run/"
export OPENSHIFT_GEAR_TYPE="diy-0.1"
export OPENSHIFT_LOG_DIR="$OPENSHIFT_DIY_LOG_DIR"
export OPENSHIFT_GEAR_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/"
export OPENSHIFT_GEAR_CTL_SCRIPT="/usr/libexec/openshift/cartridges/diy-0.1/info/bin/app_ctl.sh"

After migration,
OPENSHIFT_LOG_DIR will not be translated to variables anymore.

Move bug to verified.
Comment 7 Meng Bo 2013-05-21 06:16:51 EDT
Hi Paul,

Should the file TYPELESS_TRANSLATED_VARS be deleted after migration?
If not, the TYPELESS_TRANSLATED_VARS and it's value will be sourced to the gear bash. And can be shown via command 'env' in rhcsh.
Comment 8 Paul Morie 2013-05-21 11:35:49 EDT
Thanks for pointing this out - the TYPELESS_TRANSLATED_VARS file should be deleted after migrating - I've made a PR for the fix.
Comment 9 openshift-github-bot 2013-05-21 15:50:05 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/2792851986400cb5352eff8909ac94edbfb143c5
Fix bug 963071: remove TYPELESS_TRANSLATED_VARS after migrating
Comment 10 Jianwei Hou 2013-05-22 05:24:15 EDT
Verified on devenv_3258

The TYPELESS_TRANSLATED_VARS is now removed after migration
Comment 11 Jianwei Hou 2013-05-22 05:51:11 EDT
@Paul I discovered there are still some environment variables referring to another environment variable in TYPELESS_TRANSLATED_VARS in prod, they are like:
export OPENSHIFT_NOSQL_DB_PORT="$OPENSHIFT_MONGODB_DB_PORT"
export OPENSHIFT_NOSQL_DB_HOST="$OPENSHIFT_MONGODB_DB_HOST"
export OPENSHIFT_NOSQL_DB_PASSWORD="$OPENSHIFT_MONGODB_DB_PASSWORD"
export OPENSHIFT_NOSQL_DB_URL="$OPENSHIFT_MONGODB_DB_URL"
export OPENSHIFT_NOSQL_DB_USERNAME="$OPENSHIFT_MONGODB_DB_USERNAME"
export OPENSHIFT_DB_URL="$OPENSHIFT_MYSQL_DB_URL"
export OPENSHIFT_DB_SOCKET="$OPENSHIFT_MYSQL_DB_SOCKET"
export OPENSHIFT_DB_PORT="$OPENSHIFT_MYSQL_DB_PORT"
export OPENSHIFT_DB_HOST="$OPENSHIFT_MYSQL_DB_HOST"
export OPENSHIFT_DB_PASSWORD="$OPENSHIFT_MYSQL_DB_PASSWORD"
export OPENSHIFT_DB_USERNAME="$OPENSHIFT_MYSQL_DB_USERNAME"

Above env vars will be migrated and sourced. After migration, they will be like:
OPENSHIFT_NOSQL_DB_HOST=$OPENSHIFT_MONGODB_DB_HOST
OPENSHIFT_NOSQL_DB_PASSWORD=$OPENSHIFT_MONGODB_DB_PASSWORD
OPENSHIFT_NOSQL_DB_PORT=$OPENSHIFT_MONGODB_DB_PORT
OPENSHIFT_NOSQL_DB_URL=$OPENSHIFT_MONGODB_DB_URL
OPENSHIFT_NOSQL_DB_USERNAME=$OPENSHIFT_MONGODB_DB_USERNAME
OPENSHIFT_DB_HOST=$OPENSHIFT_MYSQL_DB_HOST
OPENSHIFT_DB_PASSWORD=$OPENSHIFT_MYSQL_DB_PASSWORD
OPENSHIFT_DB_PORT=$OPENSHIFT_MYSQL_DB_PORT
OPENSHIFT_DB_SOCKET=$OPENSHIFT_MYSQL_DB_SOCKET
OPENSHIFT_DB_URL=$OPENSHIFT_MYSQL_DB_URL
OPENSHIFT_DB_USERNAME=$OPENSHIFT_MYSQL_DB_USERNAME

Maybe they also need to be processed?
Comment 13 Paul Morie 2013-05-23 08:13:44 EDT
QE-

We had a mismatch on this issue in the development team, so what you should look for has changed.  The latest PR implements the following behavior:

1. If an entry in TYPELESS_TRANSLATED_VARS is a reference to another env var, attempt to resolve it and write the discrete var with the _content_ of the referenced var.
2. If you cannot resolve the referenced var, do not create a discrete file for the _referencing_ var.
Comment 14 Meng Bo 2013-05-24 04:50:16 EDT
Test this again on devenv-3268, with the following data:

export OPENSHIFT_NOSQL_DB_PORT="$OPENSHIFT_MONGODB_DB_PORT"
export OPENSHIFT_NOSQL_DB_HOST="$OPENSHIFT_MONGODB_DB_HOST"
export OPENSHIFT_NOSQL_DB_PASSWORD="$OPENSHIFT_MONGODB_DB_PASSWORD"
export OPENSHIFT_NOSQL_DB_URL="$OPENSHIFT_MONGODB_DB_URL"
export OPENSHIFT_NOSQL_DB_USERNAME="$OPENSHIFT_MONGODB_DB_USERNAME"
export OPENSHIFT_DB_URL=$OPENSHIFT_MYSQL_DB_URL
export OPENSHIFT_DB_SOCKET=$OPENSHIFT_MYSQL_DB_SOCKET
export OPENSHIFT_DB_PORT=$OPENSHIFT_MYSQL_DB_PORT
export OPENSHIFT_DB_HOST=$OPENSHIFT_MYSQL_DB_HOST
export OPENSHIFT_DB_PASSWORD=$OPENSHIFT_MYSQL_DB_PASSWORD
export OPENSHIFT_DB_USERNAME=$OPENSHIFT_MYSQL_DB_USERNAME
export OPENSHIFT_RUNTIME_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/runtime/"
export OPENSHIFT_RUN_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/run/"
export OPENSHIFT_GEAR_TYPE="diy-0.1"
export OPENSHIFT_LOG_DIR="$OPENSHIFT_DIY_LOG_DIR"
export OPENSHIFT_GEAR_DIR="/var/lib/openshift/e419a130bb9011e2b6f522000a8dbcca/diy-0.1/"
export OPENSHIFT_GEAR_CTL_SCRIPT="/usr/libexec/openshift/cartridges/diy-0.1/info/bin/app_ctl.sh"
export TEST_VAR_1='foo'
export TEST_VAR_2='bar'
export TEST_VAR_3="baz"


After migration, all the vars which referenced to other ones are all removed from the user env_vars list. 
But the previous removed two, OPENSHIFT_GEAR_TYPE OPENSHIFT_GEAR_CTL_SCRIPT are present.

From the code, found the blacklist has been deleted. Not sure if this the expected result.

Please help confirm, and then I will mark the bug as verified.
Comment 15 Jianwei Hou 2013-05-24 07:46:59 EDT
This can be migrated successfully on devenv_3269, the reason for the result in comment 14 is because the environment variables being referred to are null themselves(in comment 14, the TYPELESS_TRANSLATED_VARS is create manually for test purpose, but no db cartridges are actually added). 

Re-test and after v2 migration, the result is as follows:
OPENSHIFT_NOSQL_DB_HOST=127.0.250.129
OPENSHIFT_NOSQL_DB_PASSWORD=ZFJaQyv1QEsu
OPENSHIFT_NOSQL_DB_PORT=27017
OPENSHIFT_NOSQL_DB_URL=mongodb://admin:ZFJaQyv1QEsu@127.0.250.129:27017/
OPENSHIFT_NOSQL_DB_USERNAME=admin
OPENSHIFT_DB_HOST=127.0.250.129
OPENSHIFT_DB_PASSWORD=kL9CllAll9SG
OPENSHIFT_DB_PORT=3306
OPENSHIFT_DB_SOCKET=/var/lib/openshift/50cb1304c46511e2992b22000a974b0e//mysql-5.1/socket/mysql.sock
OPENSHIFT_DB_URL=mysql://adminRMmgKsZ:kL9CllAll9SG@127.0.250.129:3306/
OPENSHIFT_DB_USERNAME=adminRMmgKsZ


Just need to confirm, the OPENSHIFT_LOG_DIR and OPENSHIFT_GEAR_CTL_SCRIPT were once in blacklist and will be abandoned after v2 migration. With this fix, the are sourced again. Should the be there are removed?

After v2 migration:
OPENSHIFT_GEAR_CTL_SCRIPT=/usr/libexec/openshift/cartridges/diy-0.1/info/bin/app_ctl.sh
OPENSHIFT_LOG_DIR=/var/lib/openshift/50cb1304c46511e2992b22000a974b0e/diy-0.1/logs/
Comment 16 Paul Morie 2013-05-26 11:15:32 EDT
PR submitted.  The (hopefully!) final behavior to test is as follows:

The following transformations are applied to each variable in TYPELESS_TRANSLATED_VARS:

1. OPENSHIFT_GEAR_CTL_SCRIPT is blacklisted and will not be broken out into a discrete variable.
2. Other variables are broken out into discrete variables as follows:
  a. If the variable is assigned to a reference to another variable, and the referenced variable can be resolved, a discrete env variable is created with the value of the referenced variable.
  b. If the variable is assigned to a reference to another variable, and the referenced variable cannot be resolved, no discrete env variable is created.
  c. If the variable is assigned to a raw value, a discrete env variable is created with the literal value.
Comment 17 openshift-github-bot 2013-05-26 12:29:53 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/ac7c92c7b5b01fd7c38728119cb06da02c63e7ff
Fix bug 963071: re-add blacklist for TYPELESS_TRANSLATED_VARS
Comment 18 Paul Morie 2013-05-26 16:47:59 EDT
Fix will be in devenv ami 3278
Comment 19 Meng Bo 2013-05-27 02:16:08 EDT
Test this issue with following contents in TYPELESS_TRANSLATED_VARS, app with all 3 dbs added.
#cat TYPELESS_TRANSLATED_VARS
export OPENSHIFT_NOSQL_DB_PORT="$OPENSHIFT_MONGODB_DB_PORT"
export OPENSHIFT_NOSQL_DB_HOST="$OPENSHIFT_MONGODB_DB_HOST"
export OPENSHIFT_NOSQL_DB_PASSWORD="$OPENSHIFT_MONGODB_DB_PASSWORD"
export OPENSHIFT_NOSQL_DB_URL="$OPENSHIFT_MONGODB_DB_URL"
export OPENSHIFT_NOSQL_DB_USERNAME="$OPENSHIFT_MONGODB_DB_USERNAME"
export OPENSHIFT_DB_URL=$OPENSHIFT_MYSQL_DB_URL
export OPENSHIFT_DB_SOCKET=$OPENSHIFT_MYSQL_DB_SOCKET
export OPENSHIFT_DB_PORT=$OPENSHIFT_MYSQL_DB_PORT
export OPENSHIFT_DB_HOST=$OPENSHIFT_MYSQL_DB_HOST
export OPENSHIFT_DB_PASSWORD=$OPENSHIFT_MYSQL_DB_PASSWORD
export OPENSHIFT_DB_USERNAME=$OPENSHIFT_MYSQL_DB_USERNAME
export OPENSHIFT_RUNTIME_DIR="/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/runtime/"
export OPENSHIFT_RUN_DIR="/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/run/"
export OPENSHIFT_GEAR_TYPE="diy-0.1"
export OPENSHIFT_LOG_DIR="$OPENSHIFT_DIY_LOG_DIR"
export OPENSHIFT_GEAR_DIR="/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/"
export OPENSHIFT_GEAR_CTL_SCRIPT="/usr/libexec/openshift/cartridges/diy-0.1/info/bin/app_ctl.sh"
export TEST_VAR_1='foo'
export TEST_VAR_2='bar'
export TEST_VAR_3="baz"
export TEST_VAR=$TEST_VAR_NO_EXIST


After migration, 
OPENSHIFT_DB_HOST=127.1.2.1
OPENSHIFT_DB_PASSWORD=is1k8qxG_WPX
OPENSHIFT_DB_PORT=3306
OPENSHIFT_DB_SOCKET=/var/lib/openshift/51a2ce44aef0570700000001//mysql-5.1/socket/mysql.sock
OPENSHIFT_DB_URL=mysql://adminHlzvZlg:is1k8qxG_WPX@127.1.2.1:3306/
OPENSHIFT_DB_USERNAME=adminHlzvZlg
OPENSHIFT_NOSQL_DB_HOST=127.1.2.1
OPENSHIFT_NOSQL_DB_PASSWORD=_a7t5IlsqWRD
OPENSHIFT_NOSQL_DB_PORT=27017
OPENSHIFT_NOSQL_DB_URL=mongodb://admin:_a7t5IlsqWRD@127.1.2.1:27017/
OPENSHIFT_NOSQL_DB_USERNAME=admin
TEST_VAR_1=foo
TEST_VAR_2=bar
TEST_VAR_3=baz
OPENSHIFT_GEAR_TYPE=diy-0.1
OPENSHIFT_GEAR_DIR=/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/
OPENSHIFT_LOG_DIR=/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/logs/
OPENSHIFT_RUN_DIR=/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/run/
OPENSHIFT_RUNTIME_DIR=/var/lib/openshift/51a2ce44aef0570700000001/diy-0.1/runtime/

Var which in blacklist is not created.  OPENSHIFT_GEAR_CTL_SCRIPT
Var which cannot be resolved is not created.    TEST_VAR

Move bug to verified.
Comment 20 Meng Bo 2013-05-27 02:27:55 EDT
Oh, missed the build info.

Verified on devenv_3277.

Note You need to log in before you can comment on or make changes to this bug.