Bug 1976053 - reinstalling openjdk packages removes alternatives links [rhel-8, openjdk-8]
Summary: reinstalling openjdk packages removes alternatives links [rhel-8, openjdk-8]
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: java-1.8.0-openjdk
Version: 8.4
Hardware: All
OS: Linux
unspecified
high
Target Milestone: beta
: ---
Assignee: jiri vanek
QA Contact: OpenJDK QA
URL:
Whiteboard:
Depends On: 1200302
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-25 04:55 UTC by Peter Ajamian
Modified: 2021-10-26 09:12 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Peter Ajamian 2021-06-25 04:55:58 UTC
Description of problem:

If you reinstall one of the following packages it causes the alternatives links to be removed from the system:
java-1.8.0-openjdk-headless
java-1.8.0-openjdk-devel
java-11-openjdk-headless
java-11-openjdk-devel

How reproducible:

Always

Steps to Reproduce:
1. dnf install java-1.8.0-openjdk-headless
2. ls -lh /usr/bin/java
3. dnf reinstall java-1.8.0-openjdk-headless
4. ls -lh /usr/bin/java

Actual results:

[root@rhel8 ~]# dnf install java-1.8.0-openjdk-headless
...
Installed:
  copy-jdk-configs-3.7-4.el8.noarch     java-1.8.0-openjdk-headless-1:1.8.0.292.b10-1.el8_4.x86_64     javapackages-filesystem-5.3.0-1.module+el8+2447+6f56d9a6.noarch     libjpeg-turbo-1.5.3-10.el8.x86_64    
  lksctp-tools-1.0.18-3.el8.x86_64      lua-5.3.4-11.el8.x86_64                                        tzdata-java-2021a-1.el8.noarch                                     

Complete!
[root@rhel8 ~]# ls -lh /usr/bin/java
lrwxrwxrwx. 1 root root 22 Jun 25 15:41 /usr/bin/java -> /etc/alternatives/java
[root@rhel8 ~]# dnf reinstall java-1.8.0-openjdk-headless
...
Reinstalled:
  java-1.8.0-openjdk-headless-1:1.8.0.292.b10-1.el8_4.x86_64                                                                                                                                                       

Complete!
[root@rhel8 ~]# ls -lh /usr/bin/java
ls: cannot access '/usr/bin/java': No such file or directory

Expected results:

The link should still be there after the reinstall.

Additional info:

This bug seems to affect all the openjdk packages that install alternatives links.  I have verified it with the packages listed above.

The alternatives --removes commands in the postinstall scriptlets should be wrapped in `if [ "$1" = 0 ]; then ... fi` which will result in the removal only happening on an actual uninstall, and not an update or reinstall.

Comment 1 Peter Ajamian 2021-06-25 05:14:43 UTC
The following one-liner is a temporary workaround to re-add the alternatives links after an update or reinstall:

rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh

Comment 2 jiri vanek 2021-06-25 08:00:38 UTC
(In reply to Peter Ajamian from comment #1)
> The following one-liner is a temporary workaround to re-add the alternatives
> links after an update or reinstall:
> 
> rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n
> '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh

The real workaround is to do the reinstall second times. It will restore all what it needs. It is old issue, but wery hard to fix: https://bugzilla.redhat.com/show_bug.cgi?id=1200302

Comment 3 Peter Ajamian 2021-06-25 10:28:57 UTC
(In reply to jiri vanek from comment #2)
> (In reply to Peter Ajamian from comment #1)
> > The following one-liner is a temporary workaround to re-add the alternatives
> > links after an update or reinstall:
> > 
> > rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n
> > '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh
> 
> The real workaround is to do the reinstall second times. It will restore all
> what it needs.

This is incorrect, if you run it twice, three times, twenty times it will never restore the links.  What happens is it actually installs the alternatives links and then removes them again in the same transaction, every time.

> It is old issue, but wery hard to fix:
> https://bugzilla.redhat.com/show_bug.cgi?id=1200302

Not hard to fix at all, and after reading through that entire thread I find it very disturbing that no one there proposed the very solution that is actually in the Fedora Packaging Guidelines:

https://docs.fedoraproject.org/en-US/packaging-guidelines/Alternatives/

Note the example postun section:

%postun
if [ $1 -eq 0 ] ; then
  %{_sbindir}/update-alternatives --remove %{name} %{_bindir}/antlr-java
fi

That's it, just wrap the alternatives --remove commands in if [ $1 -eq 0 ] ; then ... fi and they will only be run on an uninstall, not on a reinstall or update.  This works in rpm, dnf, yum, you name it.

Really, I find it amazing that this has come up and no one has proposed this very simple solution that was pretty much designed just to solve this exact problem!

Comment 4 Peter Ajamian 2021-06-25 11:20:00 UTC
just found this, it's been fixed in upstream Fedora, can we pull the fix into RHEL, please?

https://src.fedoraproject.org/fork/jgu/rpms/java-1.8.0-openjdk/c/276631960eff444b36862d024f126942accd095e

Comment 5 jiri vanek 2021-06-25 12:09:21 UTC
(In reply to Peter Ajamian from comment #3)
> (In reply to jiri vanek from comment #2)
> > (In reply to Peter Ajamian from comment #1)
> > > The following one-liner is a temporary workaround to re-add the alternatives
> > > links after an update or reinstall:
> > > 
> > > rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n
> > > '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh
> > 
> > The real workaround is to do the reinstall second times. It will restore all
> > what it needs.
> 
> This is incorrect, if you run it twice, three times, twenty times it will
> never restore the links.  What happens is it actually installs the
> alternatives links and then removes them again in the same transaction,
> every time.

HAd just tried, and seesm you are correct. sorry.
> 
> > It is old issue, but wery hard to fix:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1200302
> 
> Not hard to fix at all, and after reading through that entire thread I find
> it very disturbing that no one there proposed the very solution that is
> actually in the Fedora Packaging Guidelines:
> 
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Alternatives/
> 
> Note the example postun section:
> 
> %postun
> if [ $1 -eq 0 ] ; then
>   %{_sbindir}/update-alternatives --remove %{name} %{_bindir}/antlr-java
> fi
> 
> That's it, just wrap the alternatives --remove commands in if [ $1 -eq 0 ] ;
> then ... fi and they will only be run on an uninstall, not on a reinstall or
> update.  This works in rpm, dnf, yum, you name it.
> 
> Really, I find it amazing that this has come up and no one has proposed this
> very simple solution that was pretty much designed just to solve this exact
> problem!

Unluckily, we have intentionally moved from the snippet you quoted, to the current state of things, to support parallel install-ability.

Comment 6 jiri vanek 2021-06-25 12:11:38 UTC
(In reply to Peter Ajamian from comment #4)
> just found this, it's been fixed in upstream Fedora, can we pull the fix
> into RHEL, please?
> 
> https://src.fedoraproject.org/fork/jgu/rpms/java-1.8.0-openjdk/c/
> 276631960eff444b36862d024f126942accd095e

That is not upstream. That is jgu personal fork, which will cancel multiple packages to coexists. Fedora have same alternatives setup as have rhel. https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/blob/rawhide/f/java-1.8.0-openjdk.spec#_501

Comment 7 Peter Ajamian 2021-06-25 12:23:21 UTC
(In reply to jiri vanek from comment #6)
> (In reply to Peter Ajamian from comment #4)
> > just found this, it's been fixed in upstream Fedora, can we pull the fix
> > into RHEL, please?
> > 
> > https://src.fedoraproject.org/fork/jgu/rpms/java-1.8.0-openjdk/c/
> > 276631960eff444b36862d024f126942accd095e
> 
> That is not upstream. That is jgu personal fork, which will cancel multiple
> packages to coexists. Fedora have same alternatives setup as have rhel.
> https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/blob/rawhide/f/java-1.
> 8.0-openjdk.spec#_501

Yeah, I saw that in the spec, but it doesn't seem to contradict the advice or ability to wrap the removals in a conditional, and I fail to see how only running the removals if the package itself is actually being removed would affect parallel installs.  Do you have a link that explains that or that discusses new guidelines for this?

Is there any reason why this won't work:

%define postun_headless() %{expand:
  if [ "$1" -eq 0 ]; then
    alternatives --remove java %{jrebindir -- %{?1}}/java
    alternatives --remove jre_%{origin} %{_jvmdir}/%{jredir -- %{?1}}
    alternatives --remove jre_%{javaver} %{_jvmdir}/%{jredir -- %{?1}}
    alternatives --remove jre_%{javaver}_%{origin} %{_jvmdir}/%{jrelnk -- %{?1}}
  fi
}

Comment 8 jiri vanek 2021-06-25 12:46:33 UTC
(In reply to Peter Ajamian from comment #7)
> (In reply to jiri vanek from comment #6)
> > (In reply to Peter Ajamian from comment #4)
> > > just found this, it's been fixed in upstream Fedora, can we pull the fix
> > > into RHEL, please?
> > > 
> > > https://src.fedoraproject.org/fork/jgu/rpms/java-1.8.0-openjdk/c/
> > > 276631960eff444b36862d024f126942accd095e
> > 
> > That is not upstream. That is jgu personal fork, which will cancel multiple
> > packages to coexists. Fedora have same alternatives setup as have rhel.
> > https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/blob/rawhide/f/java-1.
> > 8.0-openjdk.spec#_501
> 
> Yeah, I saw that in the spec, but it doesn't seem to contradict the advice
> or ability to wrap the removals in a conditional, and I fail to see how only
> running the removals if the package itself is actually being removed would
> affect parallel installs.  Do you have a link that explains that or that
> discusses new guidelines for this?
> 
> Is there any reason why this won't work:
> 
> %define postun_headless() %{expand:
>   if [ "$1" -eq 0 ]; then
>     alternatives --remove java %{jrebindir -- %{?1}}/java
>     alternatives --remove jre_%{origin} %{_jvmdir}/%{jredir -- %{?1}}
>     alternatives --remove jre_%{javaver} %{_jvmdir}/%{jredir -- %{?1}}
>     alternatives --remove jre_%{javaver}_%{origin} %{_jvmdir}/%{jrelnk --
> %{?1}}
>   fi
> }


It is the reason I already hit - we support full version in install dir, and you can install unlimited number of openjdk rpms in parallel. It includes also fastdeug and slowdebug variants and any version, not only majoir (8,11,16) but also major (eg 8u292 and u302). Together with it it supports fluent upgrade and downgrade, and it was not easy to achieve with support from side of dnf prohibited. 
The paycost that the postunst is as it is and implication is broken reinstall. Here we know the fix, (see the https://bugzilla.redhat.com/show_bug.cgi?id=1200302 atachements), but that woudl be terrible maintenance burden.


Tbh, untill now  was pretty sure that second reisntall really is fixing the issue. If it is no  longer true, the issue would be worthy of revisiting. Thank you for report and sorry for troubles.

Comment 9 Peter Ajamian 2021-06-25 15:13:03 UTC
Looking into this more closely...

Okay, I see where you're going now.  A couple of suggestions:

1.  Use %{origin} and %{javaver} instead of %{version} and %{release}.  Especially don't use release, because this can work for updates as well, unless you are absolutely certain that updates will always run %postun of the old package before %post of the new (I wouldn't bet on that).

2.  Remove the temporary state file in a %posttrans scriptlet.  Once the dnf transaction is complete it isn't needed anymore and we should clean up our mess.

I realize that this solution isn't pretty, but I honestly think that breaking alternatives is worse, if a better solution comes along we can always switch to that, but this will do for now.

Comment 10 Peter Ajamian 2021-06-25 15:18:12 UTC
Oh, another idea to *really* simplify things.  Run the alternatives --install command in %posttrans instead of %post.  That way you are absolutely guaranteed that it will run after the remove in %postun.  then we don't have to jump through all these hoops.

Comment 11 jiri vanek 2021-06-25 15:18:41 UTC
(In reply to Peter Ajamian from comment #9)
> Looking into this more closely...
> 
> Okay, I see where you're going now.  A couple of suggestions:
> 
> 1.  Use %{origin} and %{javaver} instead of %{version} and %{release}. 
> Especially don't use release, because this can work for updates as well,

That would kill the paralel installability we have now.

> unless you are absolutely certain that updates will always run %postun of
> the old package before %post of the new (I wouldn't bet on that).
> 
> 2.  Remove the temporary state file in a %posttrans scriptlet.  Once the dnf
> transaction is complete it isn't needed anymore and we should clean up our
> mess.
> 
Yes. That is very similar to what the fix is doing in 1200302.

> I realize that this solution isn't pretty, but I honestly think that
> breaking alternatives is worse, if a better solution comes along we can

When second reinstall is no longer option, then yes, I definitely agree.

> always switch to that, but this will do for now.

Comment 12 jiri vanek 2021-06-25 15:20:57 UTC
(In reply to Peter Ajamian from comment #10)
> Oh, another idea to *really* simplify things.  Run the alternatives
> --install command in %posttrans instead of %post.  That way you are
> absolutely guaranteed that it will run after the remove in %postun.  then we
> don't have to jump through all these hoops.

That is quite interesting idea!

Comment 13 Peter Ajamian 2021-06-25 15:23:29 UTC
(In reply to jiri vanek from comment #11)
> (In reply to Peter Ajamian from comment #9)
> > Looking into this more closely...
> > 
> > Okay, I see where you're going now.  A couple of suggestions:
> > 
> > 1.  Use %{origin} and %{javaver} instead of %{version} and %{release}. 
> > Especially don't use release, because this can work for updates as well,
> 
> That would kill the paralel installability we have now.

I was under the impression reading the spec that %{javaver} needs to be unique for each install as it ends up becoming the unique part of the path to the parallel install, was I mistaken?

Comment 14 jiri vanek 2021-06-25 15:34:06 UTC
(In reply to Peter Ajamian from comment #13)
> (In reply to jiri vanek from comment #11)
> > (In reply to Peter Ajamian from comment #9)
> > > Looking into this more closely...
> > > 
> > > Okay, I see where you're going now.  A couple of suggestions:
> > > 
> > > 1.  Use %{origin} and %{javaver} instead of %{version} and %{release}. 
> > > Especially don't use release, because this can work for updates as well,
> > 
> > That would kill the paralel installability we have now.
> 
> I was under the impression reading the spec that %{javaver} needs to be
> unique for each install as it ends up becoming the unique part of the path
> to the parallel install, was I mistaken?

Javaver is 8,11,16... We suport parallel installs also of minors.

Comment 15 Severin Gehwolf 2021-06-25 15:58:27 UTC
(In reply to jiri vanek from comment #12)
> (In reply to Peter Ajamian from comment #10)
> > Oh, another idea to *really* simplify things.  Run the alternatives
> > --install command in %posttrans instead of %post.  That way you are
> > absolutely guaranteed that it will run after the remove in %postun.  then we
> > don't have to jump through all these hoops.
> 
> That is quite interesting idea!

How would you know *what* to do in %posttrans? Posttrans is run once after the transaction. It's not package specific. Potentially you are removing 3 versions of a JDK and upgrading one, perhaps even installing one in addition. What alternatives do you install, which alternatives do you remove?

Comment 16 Peter Ajamian 2021-06-25 16:50:10 UTC
(In reply to Severin Gehwolf from comment #15)
> How would you know *what* to do in %posttrans? Posttrans is run once after
> the transaction. It's not package specific. Potentially you are removing 3
> versions of a JDK and upgrading one, perhaps even installing one in
> addition. What alternatives do you install, which alternatives do you remove?

I'm pretty certain that's incorrect.  %posttrans is run once for each package that defines it, but at the end of the transaction.  How could it be defined in the spec if it's not package-specific?  Also the docs suggest that it's run more than once in the suggestion here that if you want to ensure that it only runs once then you should set a flag file to make sure of that:

https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/

"If we only want the program in %posttrans to do its work once per-transaction, we may need to write out a flag file so that the %posttrans knows whether to perform an action."

...this suggests that if we don't set a flag file that the program could be run multiple times, hence it's not run just once per transaction.

%posttrans is never run for removal (only install/upgrade/reinstall) and the appropriate alternatives command will get hard-coded at the rpmbuild stage, basically put (how I understand it is that) each new package installed dumps it's %posttrans script into the queue to be run at the end of the transaction, so the %posttrans scriptlet for each individual rpm that is installed will contain the correct alternatives command to (re-)install the alternatives links for that particular rpm.

Comment 17 Severin Gehwolf 2021-06-28 10:11:32 UTC
(In reply to Peter Ajamian from comment #16)
> (In reply to Severin Gehwolf from comment #15)
> > How would you know *what* to do in %posttrans? Posttrans is run once after
> > the transaction. It's not package specific. Potentially you are removing 3
> > versions of a JDK and upgrading one, perhaps even installing one in
> > addition. What alternatives do you install, which alternatives do you remove?
> 
> I'm pretty certain that's incorrect.  %posttrans is run once for each
> package that defines it, but at the end of the transaction.  How could it be
> defined in the spec if it's not package-specific?  Also the docs suggest
> that it's run more than once in the suggestion here that if you want to
> ensure that it only runs once then you should set a flag file to make sure
> of that:
> 
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
> 
> "If we only want the program in %posttrans to do its work once
> per-transaction, we may need to write out a flag file so that the %posttrans
> knows whether to perform an action."
> 
> ...this suggests that if we don't set a flag file that the program could be
> run multiple times, hence it's not run just once per transaction.

I agree. Thanks for correcting me. It's been a while...

> %posttrans is never run for removal (only install/upgrade/reinstall) and the
> appropriate alternatives command will get hard-coded at the rpmbuild stage,
> basically put (how I understand it is that) each new package installed dumps
> it's %posttrans script into the queue to be run at the end of the
> transaction, so the %posttrans scriptlet for each individual rpm that is
> installed will contain the correct alternatives command to (re-)install the
> alternatives links for that particular rpm.

OK. The idea is to use %posttrans for installing alternatives and %postun to remove them. Do we know how %posttrans behaves if there are multiple versions of the same package in the transaction? Are they properly ordered? I guess this boils down to thorough testing (including the upgrade path) like was done for a prototype in bug 1200302 and then devise a plan for implementation. The devil is in the details. Thanks for your input!

Comment 18 jiri vanek 2021-06-28 10:13:01 UTC
Right. My understanding is that posttrans of each package  reinstalled/isntalled/updated/downgraded during transaction is run (at least) once after the trasnaction. I was not aware of possible additional runs, but that can be workaroudned by chekcing the slave is already here. If it is not package specific,then I would like to know how that is possible.

Comment 19 jiri vanek 2021-06-28 10:14:46 UTC
> OK. The idea is to use %posttrans for installing alternatives and %postun to
> remove them. Do we know how %posttrans behaves if there are multiple
> versions of the same package in the transaction? Are they properly ordered?

Does it meter?  In our case they will deal with one exact NVR anyway. And all posttrans will run after all postuns.

> I guess this boils down to thorough testing (including the upgrade path)

Yup. We shodl have pretty good coverage on those.

> like was done for a prototype in bug 1200302 and then devise a plan for
> implementation. The devil is in the details. Thanks for your input!

Indeed!

Comment 20 Severin Gehwolf 2021-06-28 10:19:53 UTC
(In reply to jiri vanek from comment #19)
> > OK. The idea is to use %posttrans for installing alternatives and %postun to
> > remove them. Do we know how %posttrans behaves if there are multiple
> > versions of the same package in the transaction? Are they properly ordered?
> 
> Does it meter?  In our case they will deal with one exact NVR anyway. And
> all posttrans will run after all postuns.

It depends which alternative you want installed. Consider (provided proper pkg config):

$ yum install java-11-openjdk-11.0.11.0.9 java-11-openjdk-11.0.10.0.9

Do you want the corresponding JDK 11 alternative to be pointing to 11.0.11 or 11.0.10? I'm guessing they are properly order, but those are corner cases nobody really exercises much ;-)

Comment 21 jiri vanek 2021-06-28 11:56:23 UTC
(In reply to Severin Gehwolf from comment #20)
> (In reply to jiri vanek from comment #19)
> > > OK. The idea is to use %posttrans for installing alternatives and %postun to
> > > remove them. Do we know how %posttrans behaves if there are multiple
> > > versions of the same package in the transaction? Are they properly ordered?
> > 
> > Does it meter?  In our case they will deal with one exact NVR anyway. And
> > all posttrans will run after all postuns.
> 
> It depends which alternative you want installed. Consider (provided proper
> pkg config):
> 
> $ yum install java-11-openjdk-11.0.11.0.9 java-11-openjdk-11.0.10.0.9
> 
> Do you want the corresponding JDK 11 alternative to be pointing to 11.0.11
> or 11.0.10? I'm guessing they are properly order, but those are corner cases
> nobody really exercises much ;-)

that will be deterministically determined by priority on auto mode - always 11.0.10 will be there (if we count priority right), and left intact in manual mode.

Comment 22 jiri vanek 2021-06-28 11:56:45 UTC
(In reply to Severin Gehwolf from comment #20)
> (In reply to jiri vanek from comment #19)
> > > OK. The idea is to use %posttrans for installing alternatives and %postun to
> > > remove them. Do we know how %posttrans behaves if there are multiple
> > > versions of the same package in the transaction? Are they properly ordered?
> > 
> > Does it meter?  In our case they will deal with one exact NVR anyway. And
> > all posttrans will run after all postuns.
> 
> It depends which alternative you want installed. Consider (provided proper
> pkg config):
> 
> $ yum install java-11-openjdk-11.0.11.0.9 java-11-openjdk-11.0.10.0.9
> 
> Do you want the corresponding JDK 11 alternative to be pointing to 11.0.11
> or 11.0.10? I'm guessing they are properly order, but those are corner cases
> nobody really exercises much ;-)

that will be deterministically determined by priority on auto mode - always 11.0.11 will be there (if we count priority right), and left intact in manual mode.

Comment 23 jiri vanek 2021-06-28 11:57:30 UTC
I hit reload in very bad moment. - 11.0.11 will be there always, while we ocunt priorities right.

Comment 24 Peter Ajamian 2021-06-28 12:02:30 UTC
Assuming that each version has a different path to the linked files and we set the priority to the version number (or something akin to the version number) then the latest version should take precedence for the links, unless the user manually sets another one.  Also as long as we remove the same versioned links that we install then the scriptlets should just do the right thing.

Absolutely we should test this thoroughly.

Comment 25 jiri vanek 2021-06-28 14:17:00 UTC
(In reply to Peter Ajamian from comment #24)
> Assuming that each version has a different path to the linked files and we

That assumption is crrect

> set the priority to the version number (or something akin to the version
> number) then the latest version should take precedence for the links, unless

The  calcualtion f priority is pretty nasty, but autoamted and checked in specfile quite a lot.So should be ok.

> the user manually sets another one.  Also as long as we remove the same
> versioned links that we install then the scriptlets should just do the right
> thing.
>

Comment 26 jiri vanek 2021-06-29 13:21:37 UTC
As the https://bugzilla.redhat.com/show_bug.cgi?id=1200302 is against fedora, and I wills tart with PR to fedora, I will continue there.

Comment 27 jiri vanek 2021-08-30 15:55:17 UTC
This got stalled. But https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/129# rebased 124, was reviwed and is passing testing. Will be merged tomorrow, and multiplied to jdk8 and 17 and to live fedoras.
peter@pajamian.dhs.org , all the gloty to you for the amazing idea with postrans, and sorry for delay.


Note You need to log in before you can comment on or make changes to this bug.