Bug 869236 - Some environment variables are incorrect after migration
Some environment variables are incorrect after migration
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Paul Morie
libra bugs
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-23 07:15 EDT by jizhao
Modified: 2015-05-14 19:01 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-06 13:50:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description jizhao 2012-10-23 07:15:26 EDT
Description of problem:
Migrate from devenv-stage_239 to devenv_2364, some environment variables of existing apps are incorrect, see detail in actual results below


Version-Release number of selected component (if applicable):
from devenv-stage_239 to devenv_2364


How reproducible:
Always


Steps to Reproduce:
1.Launch devenv-stage_239
2.Create test data on it
3.Run the zend_fix_script provided in Bug 864775
4.Modify the li.repo to use the candidate mirror
5.Copy the jbosseap/jbossews related repos and key files from latest devenv
6.Fix the selinux by following the rmillner's comment
7.Run "yum update -y --enablerepo li-local" to upgrade to fork_ami_US2953_245
8.Restart instance.
9.Do the following dir operations.
#mv /var/lib/stickshift /var/lib/stickshift.old
#rm -rf /tmp/rhc/migration /var/lib/openshift/* /var/lib/openshift/.httpd.d/*
#ln -sf /var/lib/openshift /var/lib/stickshift
#cp -rP --preserve=all /var/lib/stickshift.old/* /var/lib/openshift/
#cp -rP --preserve=all /var/lib/stickshift.old/.httpd.d/* /var/lib/openshift/.httpd.d/
10.Run migrate-mongo-2.0.18
11.Run hascaledb-migrate-2.0.18
12.Run rhc-admin-migrate-2.0.19
13.Try to ssh login to the apps which migrated without error.
14.ssh login apps to check environment variables
  

Actual results:
global:
The following env vars, which should be removed, still exist.
OPENSHIFT_GEAR_NAME, OPENSHIFT_GEAR_DIR, OPENSHIFT_GEAR_CTL_SCRIPT, OPENSHIFT_GEAR_TYPE, OPENSHIFT_RUNTIME_DIR, OPENSHIFT_LOG_DIR

mysql-5.1:
(1)In main web&haproxy gear of scalable apps, the following env vars do not  have values:
OPENSHIFT_MYSQL_DB_PORT=
OPENSHIFT_MYSQL_DB_PASSWORD=
OPENSHIFT_MYSQL_DB_USERNAME=
OPENSHIFT_MYSQL_DB_SOCKET=
(2)In mysql gear of scalable apps, or in non-scalable apps, mysql env vars are correct.

postgresql-8.4:
(1)In main web&haproxy gear of scalable apps, the following env vars do not have  values:
OPENSHIFT_POSTGRESQL_DB_SOCKET=
OPENSHIFT_POSTGRESQL_DB_PASSWORD=
OPENSHIFT_POSTGRESQL_DB_USERNAME=
OPENSHIFT_POSTGRESQL_DB_PORT=
(2)In postgresql gear of both scalable apps and  non-scalable apps, no namespaced postgresql env vars are generated after migration. It only has the old environment variables.

mongodb-2.2:
(1)In main web&haproxy gear of scalable apps, the following env vars have no values:
OPENSHIFT_MONGODB_DB_USERNAME=
OPENSHIFT_MONGODB_DB_PASSWORD=
OPENSHIFT_MONGODB_DB_PORT=
(2)In mongodb gear of both scalable apps and non-scalable apps, no namespaced mongodb env vars are generated after migration. It only has the old environment variables.

cron-1.4:
The following env vars, which should be removed, still exist.
OPENSHIFT_BATCH_CTL_SCRIPT
OPENSHIFT_BATCH_CRON_14_EMBEDDED_TYPE
OPENSHIFT_BATCH_TYPE


Expected results:
All the env vars should be correct


Additional info:
Comment 1 Mrunal Patel 2012-10-23 13:01:24 EDT
OPENSHIFT_GEAR_DIR, OPENSHIFT_GEAR_CTL_SCRIPT, OPENSHIFT_GEAR_TYPE, OPENSHIFT_RUNTIME_DIR, OPENSHIFT_LOG_DIR are translated for
existing apps so that existing application code that uses those variables does
not break. 

OPENSHIFT_GEAR_NAME is not getting removed.
Comment 2 jizhao 2012-10-23 22:31:38 EDT
(In reply to comment #1)
> OPENSHIFT_GEAR_DIR, OPENSHIFT_GEAR_CTL_SCRIPT, OPENSHIFT_GEAR_TYPE,
> OPENSHIFT_RUNTIME_DIR, OPENSHIFT_LOG_DIR are translated for
> existing apps so that existing application code that uses those variables
> does
> not break. 
> 
> OPENSHIFT_GEAR_NAME is not getting removed.

Well, it seems the attachment of US2105 in rally is outdated. I'll update corresponding test cases.
Comment 3 Jianwei Hou 2012-10-24 01:13:38 EDT
This may also caused some problem on INT
After upgrade of INT, env vars for mongodb did not got updated, and I was unable to connect to mongo shell via rhcsh

[perl1-jhou.int.rhcloud.com ~]\> set -x
[perl1-jhou.int.rhcloud.com ~]\> mongo
+ mongo
+ test 0 -gt 0
+ uopt='--username '\''admin'\'''
+ '[' -n '' ']'
+ echo
+ egrep '\-\-host|'
+ hopt=
+ unset LD_LIBRARY_PATH
+ /usr/bin/mongo --username ''\''admin'\'''
MongoDB shell version: 2.2.0
connecting to: test
Wed Oct 24 00:35:20 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91

[perl1-jhou.int.rhcloud.com ~]\> env|grep DB
OPENSHIFT_NOSQL_DB_PASSWORD=UYKK4ub99cne
OPENSHIFT_NOSQL_DB_PORT=27017
OPENSHIFT_NOSQL_DB_HOST=127.1.255.129
OPENSHIFT_NOSQL_DB_USERNAME=admin

[perl1-jhou.int.rhcloud.com ~]\> /usr/bin/mongo 127.1.255.129:27017 -uadmin -pUYKK4ub99cne
+ /usr/bin/mongo 127.1.255.129:27017 -u admin -pUYKK4ub99cne
MongoDB shell version: 2.2.0
connecting to: 127.1.255.129:27017/test
Error: { errmsg: "auth fails", ok: 0.0 }
Wed Oct 24 00:36:46 uncaught exception: login failed
exception: login failed
Comment 4 Jianwei Hou 2012-10-24 01:34:14 EDT
Also failed for postgresql connection via rhcsh

[python1-jhou.int.rhcloud.com ~]\> psql
+ psql
+ PGDATABASE=python1
+ PGHOST=
+ PGPORT=5432
+ PGUSER=''\''admin'\'''
+ PGPASSWORD=
+ /usr/bin/psql --set 'HISTFILE=~/app-root/data/.psql_history'
psql: FATAL:  Ident authentication failed for user "'admin'"

[python1-jhou.int.rhcloud.com ~]\> env|grep DB
+ env
+ grep DB
OPENSHIFT_DB_HOST=127.2.5.129
OPENSHIFT_DB_CTL_SCRIPT=/var/lib/openshift/a912d3fc0a5c44acaa532796309010d0//postgresql-8.4/python1_postgresql_ctl.sh
OPENSHIFT_DB_PASSWORD=RiCRhGD2qiTu
OPENSHIFT_DB_USERNAME=admin
OPENSHIFT_DB_POSTGRESQL_84_RESTORE=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_restore.sh
OPENSHIFT_DB_POSTGRESQL_84_DUMP_CLEANUP=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_cleanup.sh
OPENSHIFT_DB_POSTGRESQL_84_EMBEDDED_TYPE=postgresql-8.4
OPENSHIFT_DB_URL=postgresql://admin:RiCRhGD2qiTu@127.2.5.129:5432/
OPENSHIFT_DB_SOCKET=/var/lib/openshift/a912d3fc0a5c44acaa532796309010d0//postgresql-8.4/socket/postgres.sock
OPENSHIFT_DB_POSTGRESQL_84_DUMP=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_dump.sh
OPENSHIFT_DB_TYPE=postgresql
OPENSHIFT_DB_PORT=5432
Comment 5 Paul Morie 2012-10-24 15:30:39 EDT
Recreated issue with postgres shell via rhcsh, will investigate.  Was not able to reproduce problem for mongo shell.
Comment 6 Paul Morie 2012-10-24 16:15:51 EDT
Reproduced issues both with mongo and postgres.  The root cause here was incorrect values in a ruby case statement which was causing the mongo and postgres carts not to be migrated for the typeless gears changes (I introduced this when refactoring the migration script to include rename changes).
Comment 7 Paul Morie 2012-10-24 18:10:00 EDT
Fixed in devenv AMI 2376.
Comment 8 Jianwei Hou 2012-10-25 03:06:16 EDT
Tested on INT(same as devenv_2376)

Seems postgresql and mongo for non-scaling apps are not migrated successfully, they are still using the old fashioned environment variables, and can't connect to them via rhcsh

mongodb:
OPENSHIFT_NOSQL_DB_CTL_SCRIPT=/var/lib/openshift/a186701afaee4224997685ea6b7409a1//mongodb-2.2/perl1_mongodb_ctl.sh
OPENSHIFT_NOSQL_DB_HOST=127.1.255.129
OPENSHIFT_NOSQL_DB_MONGODB_22_DUMP_CLEANUP=/usr/libexec/openshift/cartridges/embedded/mongodb-2.2/info/bin/mongodb_cleanup.sh
OPENSHIFT_NOSQL_DB_MONGODB_22_DUMP=/usr/libexec/openshift/cartridges/embedded/mongodb-2.2/info/bin/mongodb_dump.sh
OPENSHIFT_NOSQL_DB_MONGODB_22_EMBEDDED_TYPE=mongodb-2.2
OPENSHIFT_NOSQL_DB_MONGODB_22_RESTORE=/usr/libexec/openshift/cartridges/embedded/mongodb-2.2/info/bin/mongodb_restore.sh
OPENSHIFT_NOSQL_DB_PASSWORD=UYKK4ub99cne
OPENSHIFT_NOSQL_DB_PORT=27017
OPENSHIFT_NOSQL_DB_TYPE=mongodb
OPENSHIFT_NOSQL_DB_URL=mongodb://admin:UYKK4ub99cne@127.1.255.129:27017/
OPENSHIFT_NOSQL_DB_USERNAME=admin

postgresql:
OPENSHIFT_DB_CTL_SCRIPT=/var/lib/openshift/a912d3fc0a5c44acaa532796309010d0//postgresql-8.4/python1_postgresql_ctl.sh
OPENSHIFT_DB_HOST=127.2.5.129
OPENSHIFT_DB_PASSWORD=RiCRhGD2qiTu
OPENSHIFT_DB_PORT=5432
OPENSHIFT_DB_POSTGRESQL_84_DUMP_CLEANUP=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_cleanup.sh
OPENSHIFT_DB_POSTGRESQL_84_DUMP=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_dump.sh
OPENSHIFT_DB_POSTGRESQL_84_EMBEDDED_TYPE=postgresql-8.4
OPENSHIFT_DB_POSTGRESQL_84_RESTORE=/usr/libexec/openshift/cartridges/embedded/postgresql-8.4/info/bin/postgresql_restore.sh
OPENSHIFT_DB_SOCKET=/var/lib/openshift/a912d3fc0a5c44acaa532796309010d0//postgresql-8.4/socket/postgres.sock
OPENSHIFT_DB_TYPE=postgresql
OPENSHIFT_DB_URL=postgresql://admin:RiCRhGD2qiTu@127.2.5.129:5432/
OPENSHIFT_DB_USERNAME=admin

For scaling apps:
the env vars are updated properly, but can't connect to the db shell
[python1s-jhou.int.rhcloud.com ~]\> mongo
MongoDB shell version: 2.2.0
connecting to: 06496d5f6b-jhou.int.rhcloud.com:27017/admin
Thu Oct 25 03:01:41 Error: couldn't connect to server 06496d5f6b-jhou.int.rhcloud.com:27017 src/mongo/shell/mongo.js:93
exception: connect failed
Comment 9 jizhao 2012-10-25 04:17:21 EDT
I've done migration testing from devenv-stage_239 to devenv_2377. There's re still some bugs:

Version:
from devenv-stage_239 to devenv_2377

Actual results:
1.OPENSHIFT_DB_PROXY_PORT, OPENSHIFT_NOSQL_DB_PROXY_PORT are missing.
2.For scalable apps: 
OPENSHIFT_MONGODB_DB_SOCKET is missing; OPENSHIFT_NOSQL_DB_SOCKET has no value.
For normal app: 
OPENSHIFT_NOSQL_DB_SOCKET has no value(jbossas-7) or missing(nodejs-0.6), OPENSHIFT_MONGODB_DB_SOCKET is missing.
Comment 10 jizhao 2012-10-25 04:23:09 EDT
Besides, for scalable apps, the old database environment variables only exist in the database gear after migration. Will this cause any problem?
Comment 11 Paul Morie 2012-10-25 10:19:33 EDT
Retesting using these permutations:

['php-5.3', 'mysql-5.1'], 
['php-5.3', 'postgresql-8.4'], 
['php-5.3', 'mongodb-2.0'],
['nodejs-0.6', 'mysql-5.1'], 
['nodejs-0.6', 'postgresql-8.4'], 
['nodejs-0.6', 'mongodb-2.0'],
['jbossas-7', 'mysql-5.1'], 
['jbossas-7', 'postgresql-8.4'], 
['jbossas-7', 'mongodb-2.0'],
Comment 12 Mrunal Patel 2012-10-25 10:48:02 EDT
Running the hascaledb migration script re-executes the db connectors for scalable
apps, so the old db environment variables aren't required on the non-db gears
in a scalable app after migration.
Comment 13 Mrunal Patel 2012-10-25 11:07:59 EDT
Actually, you are right, we need to translate the DB variables on the other gears
of a scalable app.
Comment 14 Paul Morie 2012-10-26 17:49:06 EDT
Pull request made.  The migratomatic tool has been changed to test this use-case by default.

Please re-test using devenv AMI 2389 using migratomatic:

https://engineering.redhat.com/trac/Libra/wiki/Migratomatic

The migratomatic tool will create scalable apps with DBs by default.
Comment 15 jizhao 2012-10-29 05:12:28 EDT
Finished migration testing from devenv-stage_239 to devenv_2397. Still some bugs.

Version: from devenv-stage_239 to devenv_2397

Actual results:
1.global: OPENSHIFT_APP_DIR, OPENSHIFT_APP_TYPE are missing after migration. According to the latest env var spreadsheet, they should be translated instead of removed.
2.mongodb:
- scalable apps: In the haproxy gear, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_MONGODB_DB_PROXY_PORT are missing; In mongodb gear, OPENSHIFT_NOSQL_DB_SOCKET has no value, OPENSHIFT_NOSQL_DB_PROXY_PORT and OPENSHIFT_NOSQL_DB_PROXY_PORT are missing. 
- non-scalable apps: OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET are missing
- both scalable and non-scalable apps: OPENSHIFT_NOSQL_DB_GEAR_DNS, OPENSHIFT_NOSQL_DB_GEAR_UUID, OPENSHIFT_NOSQL_DB_HOST, OPENSHIFT_NOSQL_DB_PASSWORD, OPENSHIFT_NOSQL_DB_PORT, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_NOSQL_DB_URL, OPENSHIFT_NOSQL_DB_USERNAME
3.mysql: For scalable apps, in haproxy gear, OPENSHIFT_MYSQL_DB_PROXY_PORT, OPENSHIFT_DB_PROXY_PORT are missing; in mysql gear, OPENSHIFT_DB_PROXY_PORT is missing.
4.postgresql: For scalable apps, in haproxy gear, OPENSHIFT_POSTGRESQL_DB_PROXY_PORT, OPENSHIFT_DB_PROXY_PORT are missing; In postgresql gear, OPENSHIFT_DB_PROXY_PORT is missing.
5.jbossas/jbosseap: OPENSHIFT_JBOSS_CLUSTER, OPENSHIFT_JBOSS_CLUSTER_PORT, OPENSHIFT_JBOSS_CLUSTER_PROXY_PORT, OPENSHIFT_JBOSS_CLUSTER_REMOTING, OPENSHIFT_JBOSS_MESSAGING_PORT, OPENSHIFT_JBOSS_MESSAGING_THROUGHPUT_PORT, OPENSHIFT_JBOSS_REMOTING_PORT are missing.
6.10gen-mms-agent: OPENSHIFT_10GEN_MMS_AGENT_CTL_SCRIPT, OPENSHIFT_10GEN_MMS_AGENT_GEAR_DIR still exist.
Comment 16 Mrunal Patel 2012-10-29 13:15:40 EDT
OPENSHIFT_APP_DIR, OPENSHIFT_APP_TYPE were translated as part of earlier typeless
gear migrations. The current migration does not touch these variables.
Comment 17 Mrunal Patel 2012-10-29 13:38:59 EDT
---
3.mysql: For scalable apps, in haproxy gear, OPENSHIFT_MYSQL_DB_PROXY_PORT, OPENSHIFT_DB_PROXY_PORT are missing; in mysql gear, OPENSHIFT_DB_PROXY_PORT is missing.
4.postgresql: For scalable apps, in haproxy gear, OPENSHIFT_POSTGRESQL_DB_PROXY_PORT, OPENSHIFT_DB_PROXY_PORT are missing; In postgresql gear, OPENSHIFT_DB_PROXY_PORT is missing.
---

OPENSHIFT_DB_PROXY_PORT variable only exists on db gear for a scalable app and
it is moved (not translated). So, the behavior that we see above is correct.
(I am updating the spreadsheet to reflect this).
Comment 18 Mrunal Patel 2012-10-29 20:52:12 EDT
----
2.mongodb:
- scalable apps: In the haproxy gear, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_MONGODB_DB_PROXY_PORT are missing; In mongodb gear, OPENSHIFT_NOSQL_DB_SOCKET has no value, OPENSHIFT_NOSQL_DB_PROXY_PORT and OPENSHIFT_NOSQL_DB_PROXY_PORT are missing. 
- non-scalable apps: OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET are missing
- both scalable and non-scalable apps: OPENSHIFT_NOSQL_DB_GEAR_DNS, OPENSHIFT_NOSQL_DB_GEAR_UUID, OPENSHIFT_NOSQL_DB_HOST, OPENSHIFT_NOSQL_DB_PASSWORD, OPENSHIFT_NOSQL_DB_PORT, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_NOSQL_DB_URL, OPENSHIFT_NOSQL_DB_USERNAME
----
There is no db socket for mongodb. Will fix the other issue.
Comment 19 Mrunal Patel 2012-10-29 21:36:33 EDT
------
2.mongodb:
- scalable apps: In the haproxy gear, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_MONGODB_DB_PROXY_PORT are missing;

mrunal: Expected behavior. 

In mongodb gear, OPENSHIFT_NOSQL_DB_SOCKET has no value,

mrunal: Fixed, it won't be created.
 
OPENSHIFT_NOSQL_DB_PROXY_PORT and OPENSHIFT_NOSQL_DB_PROXY_PORT are missing.

mrunal: Expected behavior.
 
- non-scalable apps: OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_MONGODB_DB_SOCKET are missing

mrunal: Expected behavior.

- both scalable and non-scalable apps: OPENSHIFT_NOSQL_DB_GEAR_DNS, OPENSHIFT_NOSQL_DB_GEAR_UUID, OPENSHIFT_NOSQL_DB_HOST, OPENSHIFT_NOSQL_DB_PASSWORD, OPENSHIFT_NOSQL_DB_PORT, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_NOSQL_DB_URL, OPENSHIFT_NOSQL_DB_USERNAME

mrunal: Fixed. 
----

----
5.jbossas/jbosseap: OPENSHIFT_JBOSS_CLUSTER, OPENSHIFT_JBOSS_CLUSTER_PORT, OPENSHIFT_JBOSS_CLUSTER_PROXY_PORT, OPENSHIFT_JBOSS_CLUSTER_REMOTING, OPENSHIFT_JBOSS_MESSAGING_PORT, OPENSHIFT_JBOSS_MESSAGING_THROUGHPUT_PORT, OPENSHIFT_JBOSS_REMOTING_PORT are missing.

mrunal: Fixed.

----
6.10gen-mms-agent: OPENSHIFT_10GEN_MMS_AGENT_CTL_SCRIPT, OPENSHIFT_10GEN_MMS_AGENT_GEAR_DIR still exist.

mrunal: Fixed.
-----

Pull request for remaining issues - https://github.com/openshift/li/pull/549
Comment 20 Mrunal Patel 2012-10-30 00:47:33 EDT
Submitted new pull request because of merge conflict - https://github.com/openshift/li/pull/551
Comment 21 jizhao 2012-10-30 07:39:51 EDT
The code has been merged. Change this bug to ON_QA.
Comment 22 jizhao 2012-10-30 07:40:14 EDT
Verify this bug.

Version: from devenv-stage_239 to devenv_2404.

Actual results:
1.OPENSHIFT_NOSQL_DB_SOCKET isn't created
2.The following env vars still exist after migration: OPENSHIFT_NOSQL_DB_GEAR_DNS, OPENSHIFT_NOSQL_DB_GEAR_UUID, OPENSHIFT_NOSQL_DB_HOST, OPENSHIFT_NOSQL_DB_PASSWORD, OPENSHIFT_NOSQL_DB_PORT, OPENSHIFT_NOSQL_DB_PROXY_PORT, OPENSHIFT_NOSQL_DB_SOCKET, OPENSHIFT_NOSQL_DB_URL, OPENSHIFT_NOSQL_DB_USERNAME
3.The following env vars still exist after migration:
OPENSHIFT_JBOSS_CLUSTER, OPENSHIFT_JBOSS_CLUSTER_PORT, OPENSHIFT_JBOSS_CLUSTER_PROXY_PORT, OPENSHIFT_JBOSS_CLUSTER_REMOTING, OPENSHIFT_JBOSS_MESSAGING_PORT, OPENSHIFT_JBOSS_MESSAGING_THROUGHPUT_PORT, OPENSHIFT_JBOSS_REMOTING_PORT
4.OPENSHIFT_10GEN_MMS_AGENT_CTL_SCRIPT, OPENSHIFT_10GEN_MMS_AGENT_GEAR_DIR no longer exist.

Note You need to log in before you can comment on or make changes to this bug.