Bug 963215

Created attachment 748245 [details]
configuration.zip

Hi Mirek,

I think this is a config issue. Your <source> Connection Factory and <target> one points to one same factory, which means the <source> and <target> are in the same server. According to your description, the <target> is at server live, and <source> is at server 'jms-bridge'. Look at the server config the CF points to localhost, which means the 'jms-bridge' server. 

Can you correct this configuration and try again?

Thanks,
Howard

Hi Howard,

sorry for late reply. It took me some time to get to this after PTO. I believe the configuration is ok. There is:

<jms-bridge name="myBridge">
                <source>
                    <connection-factory name="jms/RemoteConnectionFactory"/>
                    <destination name="jms/queue/InQueue"/>
                </source>
                <target>
                    <connection-factory name="jms/RemoteConnectionFactory"/>
                    <destination name="jms/queue/OutQueue"/>
                    <context>
                        <property key="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
                        <property key="java.naming.provider.url" value="remote://192.168.40.1:4447"/>
                    </context>
                </target>
...
</jms-bridge>

the "target CF" is looked up on live server (IP 192.168.40.1). Or is there something I missed?

Thanks,
Mirek

Hi Mirek,

I need some help setting up the test. I run the live backup and 3rd server on one machine. The live and backup started successfully, but when I start the 3rd one, which has the jms bridge deployed I got exception like :

18:42:41,846 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on 127.0.0.1:10019
18:42:41,846 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on 127.0.0.1:4467
18:42:41,851 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-6) JBAS015012: Started FileSystemDeploymentService for directory /home/howard/Downloads/EAP6/jboss-eap-6.1-standalone/standalone/deployments
18:42:42,390 INFO  [org.hornetq.core.server] (MSC service thread 1-12) HQ221020: Started Netty Acceptor version 3.6.6.Final-90e1eb2 127.0.0.1:5475 for CORE protocol
18:42:42,392 INFO  [org.hornetq.core.server] (MSC service thread 1-12) HQ221020: Started Netty Acceptor version 3.6.6.Final-90e1eb2 127.0.0.1:5465 for CORE protocol
18:42:42,395 INFO  [org.hornetq.core.server] (MSC service thread 1-12) HQ221007: Server is now live
18:42:42,395 INFO  [org.hornetq.core.server] (MSC service thread 1-12) HQ221001: HornetQ Server version 2.3.2.Snapshot (Wild Hornet, 123) [b92b01f6-ded5-11e2-a4f8-5b342e5ac970] 
18:42:42,426 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 59) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
18:42:42,428 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 61) HQ221003: trying to deploy queue jms.queue.queue/InQueue
18:42:42,442 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 61) JBAS011601: Bound messaging object to jndi name java:/queue/InQueue
18:42:42,442 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 61) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/InQueue
18:42:42,444 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 60) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
18:42:42,490 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) JBAS010406: Registered connection factory java:/JmsXA
18:42:42,529 INFO  [org.hornetq.ra] (MSC service thread 1-4) HornetQ resource adaptor started
18:42:42,529 INFO  [org.jboss.as.connector.services.resourceadapters.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-4) IJ020002: Deployed: file://RaActivatorhornetq-ra
18:42:42,532 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-12) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
18:42:46,764 WARN  [org.hornetq.jms.server] (ServerService Thread Pool -- 58) HQ122010: Failed to connect JMS Bridge: javax.naming.NamingException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4467]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:213) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:144) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:125) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:241) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83) [jboss-remote-naming-1.0.6.Final-redhat-2.jar:1.0.6.Final-redhat-2]
	at javax.naming.InitialContext.lookup(InitialContext.java:411) [rt.jar:1.7.0_06]
	at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:63) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:]
	at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1142) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.start(JMSBridgeImpl.java:353) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:]
	at org.jboss.as.messaging.jms.bridge.JMSBridgeService.startBridge(JMSBridgeService.java:92) [jboss-as-messaging-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
	at org.jboss.as.messaging.jms.bridge.JMSBridgeService$1.run(JMSBridgeService.java:78) [jboss-as-messaging-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_06]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_06]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_06]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

So the bridge couldn't start successfully. I don't know what's wrong with my config, can you help take a look at it? Here is my config:

                <jms-bridge name="myBridge">
                   <source>
                      <connection-factory name="jms/RemoteConnectionFactory"/>
                      <destination name="jms/queue/InQueue"/>
                      <context>
                          <property key="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
                          <property key="java.naming.provider.url" value="remote://127.0.0.1:4467"/>
                      </context>
                   </source>
                   <target>
                      <connection-factory name="jms/RemoteConnectionFactory"/>
                      <destination name="jms/queue/OutQueue"/>
                      <context>
                          <property key="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
                          <property key="java.naming.provider.url" value="remote://localhost:4447"/>
                      </context>
                   </target>
                   <quality-of-service>AT_MOST_ONCE</quality-of-service>
                   <failure-retry-interval>1000</failure-retry-interval>
                   <max-retries>-1</max-retries>
                   <max-batch-size>10</max-batch-size>
                   <max-batch-time>100</max-batch-time>
                   <add-messageID-in-header>true</add-messageID-in-header>
                </jms-bridge>

You can see from the log the server is listening to port 4467, but yet it failed. I'll attach the full configuration file.

Thanks
Howard

Created attachment 766056 [details]
the configuration of my bridge server

Hi Howard,

I can see that there is:
                   <source>
                      <connection-factory name="jms/RemoteConnectionFactory"/>
                      <destination name="jms/queue/InQueue"/>
                      <context>
                          <property key="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
                          <property key="java.naming.provider.url" value="remote://127.0.0.1:4467"/>
                      </context>
                   </source>

could you try to remove <context>...</context> section, please?

Source queue and connection factory is on 3rd bridge server so it should make in-vm jndi lookup.

Also I guess you're not using port-offset for 3rd/bridge server. There is bad remoting port 4467 in <source><context>...part. It should be 4447 but as I mentioned, look up should work without <context> for <source>.

Thanks,

Mirek

Thanks Mirek, like you pointed out, the problem is gone after I removed <context> from it. 

Now I have some idea about what's happening here. When live is shutdown the bridge connection actually failed over. but still the exception listener in the bridge is called, which triggered the retry operation of the bridge. Each retry will start with looking up the connection factories and destinations whose JNDI context points to the live (which is dead now).

I'm trying to come up with a solution to this. My idea is that when the exception listener gets called, the bridge tries to wait for the failover to happen and base on whether failover happened or not, or failover is successful or failed, to initiate retry or not.

Howard

BTW, Howard is the correct owner and he is still working on it.

I've been working with Howard on this. I believe we have found a solution now. we will perform some final tests on monday.

PR sent and merged on hornetq:

https://github.com/hornetq/hornetq/pull/1200


this will be PRed through BZ994214 (https://bugzilla.redhat.com/show_bug.cgi?id=994214)

Reopening. Failover of JMS bridges is still not working in EAP 6.1.1.ER6.

Created attachment 787229 [details]
server_with_bridge.log

There is a reproducer (attached):
1. Download and unzip reprodurcer - next steps run in extracted directory
2. run - sh prepare.sh
3. Start live/backup pair
	- sh start-server1.sh 127.0.0.1
	- sh start-server2.sh <backup_ip>
4. Start third server with deployed bridge
	- sh start-server3.sh <bridge_ip>
5. Start producer
	-  sh start-producer.sh <bridge_ip> jms/queue/InQueue 100000
6. Kill live server
	- kill -9 ... (jps -l to get pid of server3 with deployed bridge)
7. Start consumer on backup server
	- sh start-consumer.sh <backup_ip> jms/queue/OutQueue

Created attachment 787274 [details]
reproducer.zip

What is not working? messaging not received? or the actual failover?

You need recovery from the TM to recover any messages and there is not such thing as recovery from the TM. You will need to solve issues manually on the XIDs or restart the previous node back to recover messages.

I see what happened. Howard didn't fix one of the cases on the bridge and I missed the verification on AT_MOST_ONCE


I have a fix here and we will make some tests.

Is Monday morning good for a release? is there a rush over the weekend? if there is I can do it probably by Sunday Morning.. otherwise it would be monday morning since I will get Howard to help me on this one as well.

Hi Mirek,

Where can I download the EAP 6.1.1.ER6? Can this test run on one machine?


Thanks
Howard

Thanks Carlo.

Hi Howard,

reproducer will prepare everything for you (it also downloads EAP 6.1.1.ER6) and it works on one machine. It's necessary to be on vpn. 

As Clebert mentioned there is not working quality-of-service AT_MOST_ONCE and DUPLICATES_OK. Quality-of-service ONCE_AND_ONLY_ONCE appears to be ok.

Thanks,
Mirek

Thanks Mirek. I'll try myself. However I didn't find relevant exception or error messages regarding the issue in the uploaded log file, can you point me at which step the test fails?

Howard

Step 6. in reproducer shows the problem. I see I did a typo in parenthesis. It should be:
6. Kill live server
	- kill -9 ... (jps -l to get pid of server1)

The problem is that after kill of live server the JMS bridge is not able to failover to backup and is trying to reconnect indefinitely. Server with deployed bridge repeatedly logs:

10:37:06,798 INFO  [org.jboss.messaging] (ServerService Thread Pool -- 64) JBAS011610: Started JMS Bridge myBridge
10:37:06,798 WARN  [org.hornetq.jms.server] (pool-4-thread-1) HQ122004: JMS Bridge Will retry after a pause of 1,000 ms
10:37:06,980 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://192.168.40.3:9990/management
10:37:06,981 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://192.168.40.3:9990
10:37:06,981 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-9) started in 9818ms - Started 149 of 272 services (122 services are passive or on-demand)
10:37:07,802 DEBUG [org.xnio.nio] (Remoting "config-based-naming-client-endpoint" read-1) Started channel thread 'Remoting "config-based-naming-client-endpoint" read-1', selector sun.nio.ch.EPollSelectorImpl@2376170d
10:37:07,803 DEBUG [org.xnio.nio] (Remoting "config-based-naming-client-endpoint" write-1) Started channel thread 'Remoting "config-based-naming-client-endpoint" write-1', selector sun.nio.ch.EPollSelectorImpl@3cecc1e1
10:37:12,811 WARN  [org.hornetq.jms.server] (pool-4-thread-1) HQ122010: Failed to connect JMS Bridge: javax.naming.NamingException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4447]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:213) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:144) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:125) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:241) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at javax.naming.InitialContext.lookup(InitialContext.java:392) [rt.jar:1.6.0_38]
	at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1223) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1459) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2000(JMSBridgeImpl.java:82) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:2003) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]

10:37:12,813 INFO  [org.hornetq.jms.server] (pool-4-thread-1) HQ121000: Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 1,000 ms
10:37:13,817 DEBUG [org.xnio.nio] (Remoting "config-based-naming-client-endpoint" read-1) Started channel thread 'Remoting "config-based-naming-client-endpoint" read-1', selector sun.nio.ch.EPollSelectorImpl@5810f29f
10:37:13,817 DEBUG [org.xnio.nio] (Remoting "config-based-naming-client-endpoint" write-1) Started channel thread 'Remoting "config-based-naming-client-endpoint" write-1', selector sun.nio.ch.EPollSelectorImpl@b6bf0e6
10:37:18,826 WARN  [org.hornetq.jms.server] (pool-4-thread-1) HQ122010: Failed to connect JMS Bridge: javax.naming.NamingException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4447]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:213) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:144) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:125) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:241) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83) [jboss-remote-naming-1.0.7.Final-redhat-1.jar:1.0.7.Final-redhat-1]
	at javax.naming.InitialContext.lookup(InitialContext.java:392) [rt.jar:1.6.0_38]
	at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1223) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1459) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2000(JMSBridgeImpl.java:82) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:2003) [hornetq-jms-server-2.3.4.Final-redhat-1.jar:2.3.4.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]

10:37:18,827 INFO  [org.hornetq.jms.server] (pool-4-thread-1) HQ121000: Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 1,000 ms
10:37:19,831 DEBUG [org.xnio.nio] (Remoting "config-based-naming-client-endpoint" read-1) Started channel thread 'Remoting "config-based-naming-client-endpoint" read-1', selector sun.nio.ch.EPollSelectorImpl@2d32e1b6

Thanks Mirek.

Howard

Hi Mirek,

I'm having problem running server2. When I do

sh start-server2.sh 127.0.0.2

I got the following error log:

19:49:46,220 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.network.unsecure: org.jboss.msc.service.StartException in service jboss.network.unsecure: JBAS015810: failed to resolve interface unsecure
	at org.jboss.as.server.services.net.NetworkInterfaceService.start(NetworkInterfaceService.java:97) [jboss-as-server-7.2.1.Final-redhat-9.jar:7.2.1.Final-redhat-9]
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_06]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_06]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_06]

Please help.

Thanks
Howard

I've sent mail to Howard:

the trick with 127.0.0.2 is not working for EAP 6. It's necessary to create IP aliases for this purpose:

ifconfig eth0:1 192.168.1.2 up
ifconfig eth0:2 192.168.1.3 up

eth0 is the name of my network interface when I type "ifconfig" command.

Bound the servers to those IP aliases. Server1 must be bound to 127.0.0.1 because server3 has hardcoded 127.0.0.1 for jndi lookup of target connection factory.

As Clebert mentioned there is fix ready for this issue. I'll test it as soon as it gets to me. Most likely in the next test cycle.

Setting as Verified in EAP 6.1.1.ER7(HQ 2.3.5). Bridge can failover from live->backup now. Thanks to all involved.