Cotton, Ben | 8 Oct 00:24 2013
Picon

[jgroups-users] How to manage this behaviour when using NotifyingFuture/DistributedExecutor APIs?

 

Hi JGroups team, and Bela,

 

We have an ISPN 5.1.6 data grid (that executes on top of JGroups) that includes the following topology:

 

 2 x Linux host(s)

Each with 30 x Java VM Nodes

 

TOTAL = 60 Nodes

 

On this 60 Node grid we use the

 

org.infinispan.distexec.DistributedExecutorService and

org.inifinispan.util.concurrent.NotifyingFuture

 

APIs to manage the dispatch/co-ordination/consolidation of a MapReduce TASK that originates from a dedicated  TASK SENDER Node and targets the full set of 60 TASK RECEIVER Nodes to complete the computation.

 

The exact API invoke (from the Task SENDER) – of course – looks like

 

            //build the DistributedExecutorService and Callable instance references

            List<Future<T>> futureList =  distExecSvc.submitEverywhere(ourCallableTask);

 

Now, as expected, 99+% of the time we are able to realize exactly 1 Task being distributed to all 60 RECEIVER Nodes and we see exactly 1 Future List entry being returned per Node submitted.

 

However, under very rare circumstances … and *only* when a certain subset of RECEIVER Nodes are enduring a major GC event, we are able to see undeniable evidence that the callableTask is being submitted multiple times to a certain subset of the RECEIVER Nodes.

 

Is there any ISPN/JGroups API or configuration mechanism by which we can be assured of being able to prevent the callableTask being submitted multiple times to a certain subset of the RECEIVER Nodes? 

 

Thanks for any insights,

Ben

 

 

 

 

 

Ben D. Cotton III
J.P.Morgan
Liquidity Risk Technology
277 Park Ave  Desk 08-GG64
New York, NY 10172-0003
212.622.5010
ben.cotton <at> jpmorgan.com

 

 

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
_______________________________________________
javagroups-users mailing list
javagroups-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/javagroups-users
Bela Ban | 6 Oct 16:43 2013
Picon

[jgroups-users] JGroups 3.4.0.Final has been released

FYI: http://belaban.blogspot.ch/2013/10/jgroups-340final-released.html

--

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
Aleksandar Kostadinov | 3 Oct 15:46 2013
Picon

[jgroups-users] FYI multicast on same port, different group addresses

https://bugzilla.redhat.com/show_bug.cgi?id=231899#c47

I think it would be good if jgroups is resetting IP_MULTICAST_ALL when 
possible to allow for better isolation between mcast groups without need 
to change ports or having user modify configuration.

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
noSoul | 2 Oct 18:25 2013
Picon

[jgroups-users] JGroups simualation in OMNet++

Hi, 

The main question:
-Has anyone or is anyone simulating a JGroups resorting application in
OMNet++ or its modules?

The derivative question (either if yes or no answer in the former):
-What aside from the data that is serialized and implemented in a class
should be transmitted, i.e., should I encapsulate my serialized data break
it in UDP chunks and simulate its transmission or something more?
-Encapsulation from our serialized data to UDP and then transmit it.
-What if we add a session with users from Europe and USA where some of the
participants IPs are known in advance to ensure that the channel is always
up (if at least on of the session participants is there).
-Any other pertinent questions that I can't remember and someone would like
to add?
-If there is a attempt to simulate JGroups in OMNet++ link, paper,
simulation source?
-If there is a attempt to simulate JGroups in OMNet++ are there any OMNet++
modules being used, if so which ones?

--
View this message in context: http://jgroups.1086181.n5.nabble.com/JGroups-simualation-in-OMNet-tp9914.html
Sent from the JGroups - General mailing list archive at Nabble.com.

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
Bela Ban | 30 Sep 11:54 2013
Picon

[jgroups-users] New JGroups record: 1538 nodes in a cluster

FYI: 
http://belaban.blogspot.ch/2013/09/new-record-for-large-jgroups-cluster.html

--

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
Alex Colomb | 27 Sep 16:50 2013
Picon

[jgroups-users] Fwd: failing NACK2 retransmission



---------- Forwarded message ----------
From: Alex Colomb <colombwork <at> gmail.com>
Date: Thu, Sep 26, 2013 at 11:03 AM
Subject: Re: [jgroups-users] failing NACK2 retransmission
To: Questions/problems related to using JGroups <javagroups-users <at> lists.sourceforge.net>


Inline responses to questions.


On Thu, Sep 26, 2013 at 5:06 AM, Bela Ban <belaban <at> yahoo.com> wrote:


On 9/25/13 7:18 PM, Alex Colomb wrote:
> We currently have a situation where one of our clients periodically
> refuses to receive a re-transmission. We see the xmit request from the
> client go out and see the original sender send the re-transmission
> response, but everyone once in a while the requesting client will
> continuously ask for the same sequence number over and over and never
> get it.


When this happens, is the original sender able to provide the requested
message ? You could run
probe.sh op=NAKACK2.printMessages
or
probe.sh op=NAKACK2.printDigestHistory

to get more info for diagnosis.

To get more information, we ran both the sender and receiver with trace level logging enabled. I've attached the relevant parts of the logs to this message. You can see the receiver request a re transmit that gets satisfied immediately ( more re transmits happen before that one that get satisfied immediately ), then it gets stuck on message 23802 until our NAKACK2 monitor closes the channel. Curiously ( I just noticed this while compiling the logs for this message ), I see a bunch of UNICAST3 messages from the sender to the receiver after the receiver has closed the multicast channel. Once the channel is closed on the receiver side, I wouldn't expect any communication at all between the sender and receiver. I've added a couple of those trace messages to the log as well.

 


> The affected client starts queuing up messages and eventually
> runs out of memory because it never gets the re-transmission. Once out
> of memory, the client behaves erratically and sometimes causes the
> cluster to degrade.

Can you reproduce this, or provide a description of what exactly your
program did to arrive at this ?

We can't reproduce this in our test environment. We suspect it has something to do with the workstation in question. We've seen this happen on two workstations ( out of 20 or so ) in the office. It happens about once a day. The setup is fairly straightforward. We have one "server" machine that is multicasting messages to all 20ish clients every 8 seconds or so. The messages are fairly large ( on the order of a 500k ). The clients just listen and process the messages accordingly.
 
> To work around this, we are now periodically querying the NAKACK2 digest
> of the affected client

This can also be done via JMX or probe.sh, see above.


> to see what the highest sequence number delivered
> is and how many re-transmit requests have been sent out. If the highest
> sequence number hasn't changed, but the re-transmit request increases,
> we disconnect the client from the cluster and then reconnect it.

This should not be necessary ! Again, if you can reproduce it, I'd be
very interested in looking into this, as this would be a critical bug !
Which version of JGroups is used ?


We are using JGroups 3.3.5.

As we understand it, this seems to be the correct behavior of JGroups. If a receiver missed a message, it is going to continuously request it from the sender and queue up all incoming messages until it gets it. We don't see any provision in any of the protocols for a particular message never making it a client.
 
Let me know what you think and if you'd like us to gather any additional information. Thanks for the help!

> Is there a NAKACK2 setting or other JGroups setting that would better
> address this situation? Below is our stack configuration.
>
>          UDP udpSettings = new UDP();
>          udpSettings.setMulticastPort( 45588 );
>          udpSettings.setMulticastAddress( InetAddress.getByName(
> "228.8.1.1" ) );
>          udpSettings.setLoopback( true );
>          udpSettings.setDiscardIncompatiblePackets( true );
>
>          FD_ALL heartBeatFailureDetection = new FD_ALL();
>          heartBeatFailureDetection.setTimeout( 12000 );
>          heartBeatFailureDetection.setInterval( 3000 );
>
>          NAKACK2 reliableInOrderDeliverySettings = new NAKACK2();
>          reliableInOrderDeliverySettings.setDiscardDeliveredMsgs( true );
>
>          VERIFY_SUSPECT verifySuspectUsingMulticast = new VERIFY_SUSPECT();
>          verifySuspectUsingMulticast.setValue( "use_mcast_rsps", true );
>
>          stack.addProtocol( udpSettings )
>                  .addProtocol( new PING() )
>                  .addProtocol( new MERGE3() )
>                  .addProtocol( new FD_SOCK() )
>                  .addProtocol( heartBeatFailureDetection )
>                  .addProtocol( verifySuspectUsingMulticast )
>                  .addProtocol( reliableInOrderDeliverySettings )
>                  .addProtocol( new UNICAST3() )
>                  .addProtocol( new STABLE() )
>                  .addProtocol( new GMS() )
>                  .addProtocol( new UFC() )
>                  .addProtocol( new MFC() )
>                  .addProtocol( new FRAG2() )
>                  .addProtocol( new STATE_TRANSFER() );


--
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
javagroups-users mailing list
javagroups-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/javagroups-users


------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
javagroups-users mailing list
javagroups-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/javagroups-users
Alex Colomb | 25 Sep 19:18 2013
Picon

[jgroups-users] failing NACK2 retransmission

We currently have a situation where one of our clients periodically refuses to receive a re-transmission. We see the xmit request from the client go out and see the original sender send the re-transmission response, but everyone once in a while the requesting client will continuously ask for the same sequence number over and over and never get it. The affected client starts queuing up messages and eventually runs out of memory because it never gets the re-transmission. Once out of memory, the client behaves erratically and sometimes causes the cluster to degrade.

To work around this, we are now periodically querying the NAKACK2 digest of the affected client to see what the highest sequence number delivered is and how many re-transmit requests have been sent out. If the highest sequence number hasn't changed, but the re-transmit request increases, we disconnect the client from the cluster and then reconnect it.

Is there a NAKACK2 setting or other JGroups setting that would better address this situation? Below is our stack configuration.

        UDP udpSettings = new UDP();
        udpSettings.setMulticastPort( 45588 );
        udpSettings.setMulticastAddress( InetAddress.getByName( "228.8.1.1" ) );
        udpSettings.setLoopback( true );
        udpSettings.setDiscardIncompatiblePackets( true );

        FD_ALL heartBeatFailureDetection = new FD_ALL();
        heartBeatFailureDetection.setTimeout( 12000 );
        heartBeatFailureDetection.setInterval( 3000 );

        NAKACK2 reliableInOrderDeliverySettings = new NAKACK2();
        reliableInOrderDeliverySettings.setDiscardDeliveredMsgs( true );

        VERIFY_SUSPECT verifySuspectUsingMulticast = new VERIFY_SUSPECT();
        verifySuspectUsingMulticast.setValue( "use_mcast_rsps", true );

        stack.addProtocol( udpSettings )
                .addProtocol( new PING() )
                .addProtocol( new MERGE3() )
                .addProtocol( new FD_SOCK() )
                .addProtocol( heartBeatFailureDetection )
                .addProtocol( verifySuspectUsingMulticast )
                .addProtocol( reliableInOrderDeliverySettings )
                .addProtocol( new UNICAST3() )
                .addProtocol( new STABLE() )
                .addProtocol( new GMS() )
                .addProtocol( new UFC() )
                .addProtocol( new MFC() )
                .addProtocol( new FRAG2() )
                .addProtocol( new STATE_TRANSFER() );

Thank you,

Alex
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
javagroups-users mailing list
javagroups-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/javagroups-users
Karim Ammous | 24 Sep 15:44 2013
Picon

[jgroups-users] MERGE3: some useless merge view

Hi Bela,

During large scale tests, I observed that, sometimes, MERGE3 handles "fake" MergeView.  Number of members that form the cluster still the same and The mergeView contains only one subgroup.

otherwise, when MERGE3 logging level is TRACE, views are dumped entirely. The limitation Util.MAX_LIST_PRINT_SIZE is not used (See MERGE3L360)

Best regards,
--
Karim AMMOUS
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
javagroups-users mailing list
javagroups-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/javagroups-users
Bela Ban | 18 Sep 08:01 2013
Picon

[jgroups-users] JGroups 3.4.0.Beta1

FYI,

I released 3.4.0.Beta1 yesterday. It contains delta views [1] and 
compressed merge views [2]. Useful in large clusters, delta views 
dramatically reduce the message size needed to install a new view, by 
sending only the diff to the previous view. E.g. if we have 
v5={A,B,C,D,E} and F joins, A will only send v6={v5 +F}, and everybody 
will construct a new view v6={A,B,C,D,E,F}.

The message size of a new view is therefore only a function of the 
number of members which joined or left, and not a function of the entire 
cluster size.

Full views are only sent when the first view of a coordinator is sent, 
e.g. if A crashes and B takes over, B would send the full view.

Delta views can be turned off by seting GMS.use_delta_views=false.

3.4.0.Beta1 can be downloaded from SourceForge [3] or Nexus [4].

3.4.0.Beta1 should be the last (and only) beta, I intend to release 
3.4.0.Final in early October.
Cheers,

[1] https://issues.jboss.org/browse/JGRP-1354
[2] https://issues.jboss.org/browse/JGRP-1391
[3] https://sourceforge.net/projects/javagroups/files/JGroups/3.4.0.Beta1
[4] 
http://search.maven.org/remotecontent?filepath=org/jgroups/jgroups/3.4.0.Beta1/jgroups-3.4.0.Beta1.pom

--

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
Bela Ban | 16 Sep 11:27 2013
Picon

[jgroups-users] JGroups workshop

I'm thinking of putting together a JGroups workshop. This is in its very 
early stages, and I'm still brainstorming on this, but I wanted to 
solicit feedback on the contents [1]. Please send me feedback about what 
you'd like to see covered, possible locations etc.

I'll start working on this after releasing 3.4 (Oct 3rd), in parallel 
with work on 3.5. The tentative target date is spring 2014.

I'm thinking there will be 2 parts: an introductory and an advanced 
part. The 2 parts could be booked together, or separately. I want to 
make this very hands-on, ie. attendees would work on labs, write their 
own protocol (advanced section) etc.

Another idea I have is to let attendees propose topics to look at, vote 
on them, and tackle the topic with the most votes in half a day.

A customized 'menu' could be booked onsite, perhaps followed by 1-2 days 
of consulting.

Again, very preliminary ideas, so feedback is welcome !

[1] https://github.com/belaban/workshop/blob/master/slides/toc.adoc

--

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
Bela Ban | 13 Sep 08:05 2013
Picon

Re: [jgroups-users] JGRP-1317 and large clusters


On 9/12/13 5:56 PM, Jeffrey Hoyt wrote:
> I assume you're going to have code to request a missing view or a way to
> request a full view?
>
> To clarify, in the examples in [2] if a member receives the message:
>
>   * V34=V33 + N1 + N2
>
> But the member only has V32, there will be a way to get V33 or a full View?

JGroups guarantees that a member will not receive V34 until is has 
received V33. This is because the same coordinator installs all views, 
and views are simple multicasts. So this property ensures that every 
member delivers all views in the same order (coordinator-FIFO).

If a coordinator crashes, then the new coordinator will send its first 
view as *full view*, not delta view. That's at least the design, I 
haven't implemented this yet. Full views will also be sent on a merge in 
the ensuing MergeView.

Note that this is hidden from the user; accessing a view and its methods 
will behave as it behaves now. Every member stores a full view; 
JGRP-1354 is only for sending views over the network to all cluster 
nodes. The goal is to reduce the message size for VIEWs; the views 
themselves will have the same size in memory. But every member only have 
1 view in memory anyway, so this should be ok.

> Sorry...we've been having issues where clusters fragment and the views
> are not always in sync when things start to reassemble.  I think the
> problems are related to hardware, not JGroups code.
>
> Jeff
>
>
> On 09/12/2013 10:14 AM, Bela Ban wrote:
>> FYI,
>>
>>
>> if you're running large clusters, you might be interested to know that I
>> implemented another optimization: JGRP-1317 [1].
>>
>> The effect of it is that (in decreasing order of importance):
>>
>> - Regular view messages to existing members don't carry a digest any
>> longer. This is a significant size reduction, as a digest has 2 longs
>> per member (although compressed for network transmission)
>>
>> - Where we have a view and a digest (e.g. JoinRsp, merge response,
>> MergeView), the digest's 'members' field and the view's 'members' field
>> point to the same memory location
>>
>> - When we send such an object across the network, the 'members' field is
>> only marshalled once, leading to smaller messages
>>
>>
>> Next up, I'm going to tackle [2] which will reduce the size of view
>> messages even further: instead of sending the *full view* every time, we
>> only send delta views. I believe JGRP-1317 and JGRP-1354 will have a
>> huge impact on systems with a large number of nodes.
>>
>>
>> [1]https://issues.jboss.org/browse/JGRP-1317
>> [2]https://issues.jboss.org/browse/JGRP-1354
>>
>>
>
> --
> Jeffrey Hoyt
> Innovative Information Engineering & Biometrics
> The MITRE Corporation
> 703-983-6241
>

--

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk

Gmane