Bug 1650986 - cinder incremental backup failed when backup backend is in different availability zone
Summary: cinder incremental backup failed when backup backend is in different availabi...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Cinder Bugs List
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks: 1554013
TreeView+ depends on / blocked
 
Reported: 2018-11-18 13:39 UTC by Avi Avraham
Modified: 2018-11-21 15:08 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-21 15:08:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Avi Avraham 2018-11-18 13:39:21 UTC
Description of problem:
While trying to create an incremental backup the following error returned from Cinder CLI : 
ERROR: Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-207069a5-8699-4b65-b8b5-1d72364a0f83)

Version-Release number of selected component (if applicable):
python-cinder-13.0.1-0.20181013185427.31ff628.el7ost.noarch
python2-cinderclient-4.0.1-0.20180809133302.460229c.el7ost.noarch
puppet-cinder-13.3.1-0.20181013114719.25b1ba3.el7ost.noarch
openstack-cinder-13.0.1-0.20181013185427.31ff628.el7ost.noarch


How reproducible:


Steps to Reproduce:
1.Install setup with backup and backend in different availability zone
2.Create volume from image while using micro version 3.51 in the CLI
3.Create volume backup
3.Mount volume and add file to volume
4.Unmount the volume 
5.create incremental backup by running the following command:
cinder backup-create --name vol-cirros-backup-inc --availability-zone nova --incremental vol-cirros

Actual results:
The following error is returned from the CLI 
ERROR: Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-207069a5-8699-4b65-b8b5-1d72364a0f83)

Expected results:
Incremental backup will created successfully

Additional info:
(overcloud) [stack@undercloud-0 ~]$ cinder --debug backup-create --name vol-cirros-backup-inc --availability-zone nova --incremental vol-cirros
DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://10.0.0.120:5000/v3 -H "Accept: application/json" -H "User-Agent: cinder keystoneauth1/3.10.0 python-requests/2.19.1 CPython/2.7.5"
DEBUG:keystoneauth.session:RESP: [200] Content-Encoding: gzip Content-Length: 192 Content-Type: application/json Date: Sun, 18 Nov 2018 13:29:01 GMT Server: Apache Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-07901c9f-39af-4869-a90f-dcdeea5f676a
DEBUG:keystoneauth.session:RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.0.0.120:5000/v3/", "rel": "self"}]}}
DEBUG:keystoneauth.session:GET call to http://10.0.0.120:5000/v3 used request id req-07901c9f-39af-4869-a90f-dcdeea5f676a
DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://10.0.0.120:5000/v3/auth/tokens
DEBUG:keystoneauth.identity.v3.base:{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9914a7ee6d68421ca2a6753fac7055c2", "name": "admin"}, {"id": "ace017c68cc040c8bca658760249dccd", "name": "member"}, {"id": "6a13911276e844c9ba2a4e7e89112385", "name": "reader"}], "expires_at": "2018-11-18T14:29:01.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "11db1f9f92cd4d008768cbd8be6b401a", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://10.0.0.120:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "1747ce3a6aa44fca966f74c9c8cffcbc"}, {"url": "http://172.17.1.29:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "42b3a24e147046548800c9a4c13a2469"}, {"url": "http://172.17.1.29:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f44fb36d5b8e4e91b2471533fa79ef50"}], "type": "network", "id": "0d022049362d4ea5808d4ab494760ddb", "name": "neutron"}, {"endpoints": [{"url": "http://10.0.0.120:8080/v1/AUTH_11db1f9f92cd4d008768cbd8be6b401a", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "4b323269141c4bdfad78ecbd45510011"}, {"url": "http://172.17.3.21:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "957a651edafd4ba8b27f12025bfcb39f"}, {"url": "http://172.17.3.21:8080/v1/AUTH_11db1f9f92cd4d008768cbd8be6b401a", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "b62cffb6132046d3b1c9c2313b06285f"}], "type": "object-store", "id": "19b219164e11442c9ea92a1c891cb45e", "name": "swift"}, {"endpoints": [{"url": "http://172.17.1.29:8042", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "671b028aacb14d4c88e12d6775504a59"}, {"url": "http://10.0.0.120:8042", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ac8ad7e1dfff4dc0a1c0abf209f621dd"}, {"url": "http://172.17.1.29:8042", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e7784f6170bd43bd88c444f78670e0e4"}], "type": "alarming", "id": "2cf6eec333554fb79bc1666effae93bb", "name": "aodh"}, {"endpoints": [], "type": "metering", "id": "4023e4573636473aa381c8dfc23f9f5f", "name": "ceilometer"}, {"endpoints": [{"url": "http://10.0.0.120:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "11c4fbeeb46b4ff18fc20a4a539f764e"}, {"url": "http://172.17.1.29:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1ab5e2d25d944102869cb4294867ec13"}, {"url": "http://172.17.1.29:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "3a6c401bdd5240f1a4b796de2aa3fd4b"}], "type": "placement", "id": "4a5680cfddf4418e93fadee50e2bedcf", "name": "placement"}, {"endpoints": [{"url": "http://172.17.1.29:8776/v3/11db1f9f92cd4d008768cbd8be6b401a", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "24f865013c8d4ed1a4ff3901f55f8b1f"}, {"url": "http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3bdd331b7fb346649ef82fd92c6e9c52"}, {"url": "http://172.17.1.29:8776/v3/11db1f9f92cd4d008768cbd8be6b401a", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "977faf4a078b4398b83d4edce986e369"}], "type": "volumev3", "id": "4e6bab7279424e21abc2dc0baefc810a", "name": "cinderv3"}, {"endpoints": [{"url": "http://10.0.0.120:8000/v1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2df749a8079540e484eb5f69ed2cd1e6"}, {"url": "http://172.17.1.29:8000/v1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "8e6f965d6b6a458c9553c7fe3631377f"}, {"url": "http://172.17.1.29:8000/v1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d198d581e1974be5911849545cc1468e"}], "type": "cloudformation", "id": "4f45833e3d3644518da6c63705ddcf17", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.17.1.29:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "3268964d090c4eedbb0322c57456f405"}, {"url": "http://10.0.0.120:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b8181f989e18446e9f923277600e66d4"}, {"url": "http://172.17.1.29:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "fc708e8623c14f0988f10db857b9af0f"}], "type": "image", "id": "5f7bb8ab4be14676abe359af8ea4a411", "name": "glance"}, {"endpoints": [{"url": "http://10.0.0.120:8776/v2/11db1f9f92cd4d008768cbd8be6b401a", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "8f48db8170424a56b0cfd8650e8ad45e"}, {"url": "http://172.17.1.29:8776/v2/11db1f9f92cd4d008768cbd8be6b401a", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "b71e1c0f291a43e89446ede6ae579ec0"}, {"url": "http://172.17.1.29:8776/v2/11db1f9f92cd4d008768cbd8be6b401a", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e8d875caad5b47d89ca4afbca01357d1"}], "type": "volumev2", "id": "7330e889b4a54e61ac2895f4bd10495a", "name": "cinderv2"}, {"endpoints": [{"url": "http://10.0.0.120:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "453ad9ae6d9248058406cd82fcd0301d"}, {"url": "http://172.17.1.29:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "c97a8dc0763d49cc9c675b542d84e377"}, {"url": "http://172.17.1.29:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ef7bf5fcef6645a2951609e7464d2d38"}], "type": "compute", "id": "7e0c8864962d4bb8a4003c491ccf306c", "name": "nova"}, {"endpoints": [{"url": "http://172.17.1.29:8977", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "34ecd544d23a4f0bbb209237f50a71d3"}, {"url": "http://172.17.1.29:8977", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "727d776fadf543e68daaf8de70ac1577"}, {"url": "http://10.0.0.120:8977", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "86acccf4ff284cc7baf038b55019b3c1"}], "type": "event", "id": "83435330b5b84beea1dbed589589d263", "name": "panko"}, {"endpoints": [{"url": "http://10.0.0.120:8776/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "294ee9160ac84a69868d64323be97a78"}, {"url": "http://172.17.1.29:8776/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6fd12eeca446400f8308f957f12aaf10"}, {"url": "http://172.17.1.29:8776/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "74763432dc464c40934fd372719a5a9b"}], "type": "volume", "id": "8c7838f3bb6f4e2badbe97cb611231b1", "name": "cinder"}, {"endpoints": [{"url": "http://10.0.0.120:8041", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "260ba3e0d3134683ac9abb9b91fabb43"}, {"url": "http://172.17.1.29:8041", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e79cf4eb7d724ee6adb0946b4e04fe6f"}, {"url": "http://172.17.1.29:8041", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "fd0e800b9cad4dbc88f49c1a9281866c"}], "type": "metric", "id": "90992b93cd2f41fb9de2088b10e0ccea", "name": "gnocchi"}, {"endpoints": [{"url": "http://172.17.1.29:9311", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11e4eaa97f834048ba94de2fcf1faee6"}, {"url": "http://172.17.1.29:9311", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1934c7ce66054e5fb77902678e207e5d"}, {"url": "http://10.0.0.120:9311", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d68fd75985a84eaeaa8ddf4ada82158c"}], "type": "key-manager", "id": "a95d9f79e1dc4e4bad5474df8a1120d0", "name": "barbican"}, {"endpoints": [{"url": "http://10.0.0.120:8004/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37cf41b3ba97435a97fed9d2afd2868a"}, {"url": "http://172.17.1.29:8004/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "72fe062ed8b04884af5ab27b1e49c949"}, {"url": "http://172.17.1.29:8004/v1/11db1f9f92cd4d008768cbd8be6b401a", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d53ce550f49e46b98799ee97de7f4059"}], "type": "orchestration", "id": "bd1e629930524ee5a2932070127003dd", "name": "heat"}, {"endpoints": [{"url": "http://172.17.1.29:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "710aa453e5ac445d95bb308200237674"}, {"url": "http://192.168.24.8:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "8121696e9f1b4696bfffe30b0e114f04"}, {"url": "http://10.0.0.120:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "edf3295b2cc94177a07b2151bc2c0320"}], "type": "identity", "id": "f1ed5125fad14361aa0857bf335f4033", "name": "keystone"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "327285d828884f3f84a886f9e6a70631"}, "audit_ids": ["LFYlRpbWTsKJlHRQ6fzikA"], "issued_at": "2018-11-18T13:29:01.000000Z"}}
DEBUG:keystoneauth:REQ: curl -g -i -X GET http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a/volumes/detail?all_tenants=1&name=vol-cirros -H "Accept: application/json" -H "OpenStack-API-Version: volume 3.51" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}a27443c0fb1d999fc1b626d0098115cd042c347c"
DEBUG:keystoneauth:RESP: [200] Content-Encoding: gzip Content-Length: 780 Content-Type: application/json Date: Sun, 18 Nov 2018 13:29:02 GMT OpenStack-API-Version: volume 3.51 Server: Apache Vary: OpenStack-API-Version,Accept-Encoding x-compute-request-id: req-0beb128b-184f-4c01-bbca-2f98ed425348 x-openstack-request-id: req-0beb128b-184f-4c01-bbca-2f98ed425348
DEBUG:keystoneauth:RESP BODY: {"volumes": [{"migration_status": null, "provider_id": null, "attachments": [], "links": [{"href": "http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a/volumes/c3ede874-a327-4205-8a4f-cf340f1930e1", "rel": "self"}, {"href": "http://10.0.0.120:8776/11db1f9f92cd4d008768cbd8be6b401a/volumes/c3ede874-a327-4205-8a4f-cf340f1930e1", "rel": "bookmark"}], "availability_zone": "alfa", "os-vol-host-attr:host": "hostgroup@tripleo_netapp#rhos_cinder", "encrypted": false, "updated_at": "2018-11-18T12:20:13.000000", "replication_status": null, "snapshot_id": null, "id": "c3ede874-a327-4205-8a4f-cf340f1930e1", "size": 1, "user_id": "327285d828884f3f84a886f9e6a70631", "os-vol-tenant-attr:tenant_id": "11db1f9f92cd4d008768cbd8be6b401a", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "volume_image_metadata": {"checksum": "7316af7358dd32ca1956d72ac2c9e147", "min_ram": "0", "disk_format": "qcow2", "image_name": "cirros", "owner_specified.shade.sha256": "53ef7941846012d031db80d2936b23de65c912ae45a1a68b21adb661f080a21b", "image_id": "0ed6295e-aa77-4431-be0a-24b8849758ae", "signature_verified": "False", "owner_specified.shade.object": "images/cirros", "container_format": "bare", "min_disk": "0", "owner_specified.shade.md5": "7316af7358dd32ca1956d72ac2c9e147", "size": "12528640"}, "description": null, "multiattach": false, "service_uuid": "9353317d-52b1-492b-87b0-5ef5817f5b7d", "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "vol-cirros", "bootable": "true", "created_at": "2018-11-14T06:53:45.000000", "volume_type": "tripleo_netapp", "group_id": null, "shared_targets": true}]}
DEBUG:keystoneauth:GET call to volumev3 for http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a/volumes/detail?all_tenants=1&name=vol-cirros used request id req-0beb128b-184f-4c01-bbca-2f98ed425348
DEBUG:keystoneauth:REQ: curl -g -i -X POST http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a/backups -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: volume 3.51" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}a27443c0fb1d999fc1b626d0098115cd042c347c" -H "X-OpenStack-Request-ID: req-0beb128b-184f-4c01-bbca-2f98ed425348" -d '{"backup": {"force": false, "description": null, "availability_zone": "nova", "snapshot_id": null, "incremental": true, "volume_id": "c3ede874-a327-4205-8a4f-cf340f1930e1", "container": null, "name": "vol-cirros-backup-inc"}}'
DEBUG:keystoneauth:RESP: [400] Content-Length: 119 Content-Type: application/json Date: Sun, 18 Nov 2018 13:29:03 GMT OpenStack-API-Version: volume 3.51 Server: Apache Vary: OpenStack-API-Version x-compute-request-id: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074 x-openstack-request-id: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074
DEBUG:keystoneauth:RESP BODY: {"badRequest": {"message": "Invalid backup: The parent backup must be available for incremental backup.", "code": 400}}
DEBUG:keystoneauth:POST call to volumev3 for http://10.0.0.120:8776/v3/11db1f9f92cd4d008768cbd8be6b401a/backups used request id req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074
DEBUG (shell:914) Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cinderclient/shell.py", line 909, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/site-packages/cinderclient/shell.py", line 729, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/shell.py", line 2396, in do_backup_create
    **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/api_versions.py", line 407, in substitution
    return method.func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 93, in create
    availability_zone)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 126, in _create_backup
    return self._create('/backups', body, 'backup')
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 330, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 203, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
BadRequest: Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074)
DEBUG:cinderclient.shell:Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cinderclient/shell.py", line 909, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/site-packages/cinderclient/shell.py", line 729, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/shell.py", line 2396, in do_backup_create
    **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/api_versions.py", line 407, in substitution
    return method.func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 93, in create
    availability_zone)
  File "/usr/lib/python2.7/site-packages/cinderclient/v3/volume_backups.py", line 126, in _create_backup
    return self._create('/backups', body, 'backup')
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 330, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 203, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
BadRequest: Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074)
ERROR: Invalid backup: The parent backup must be available for incremental backup. (HTTP 400) (Request-ID: req-92bff459-6289-4fb0-a4f7-7ea5f6ac3074)

Comment 1 Alan Bishop 2018-11-18 21:54:56 UTC
Questions:
- Where are the logs?
- What is the status of the initial backup?

Comment 2 Avi Avraham 2018-11-19 06:35:23 UTC
I will add the log files to bugzilla 
The parent backup is in available state

Comment 7 Alan Bishop 2018-11-21 15:08:47 UTC
The problem was caused by a known bug in the NFS backend. The problem was not observed when testing with a different backend.


Note You need to log in before you can comment on or make changes to this bug.