Users of the Storm distributed computation system ()

Richards Peter | 24 May 2013 12:12

How do i configure maxClientCnxns in zoo.cfg for storm?

Hi,

Today we found that some of the workers on our machines were not getting initialised properly. We debugged the issue and found that it was because of the default value of maxClientCnxns in zookeeper. The default value is 10. This means that there cannot be more than 10 connections from a single client machine to a single member of zookeeper ensemble. By increasing this parameter the issue got resolved. I would like to know how to decide upon the value which has to be assigned for this parameter.

Assume that I have a storm cluster containing m machines each having n supervisor ports. I am running nimbus, supervisor and ui on one of these m machines. For safer side should I assign the value to be n+3(considering nimbus, supervisor and ui on the master machine)?

Thanks,
Richards Peter.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

kshitiz@canvien | 24 May 2013 10:58

Trident: Querying on a partitioned state.

Hello,

I have a state partitioned into 3. When I query the state I find that my stream used for querying is automatically partitioned into 3. To illustrate this point here is the sample code.

TridentState state = topology.newStream("spout",new FixedbatchSpout())

.partitionBy(new Fields("some_field"))

.partitionAggregate(new Fileds("some_fields..",""...), new TimeSlotBasedAggregator(), new Fields("slot","value"))

.partitionPersist(....)

.parallelismHint(3);

topology.newStream("tick_stream",new TickSpout())

.stateQuery(state, . . .. )

.parallelismHint(4);

My question is, in how many parts the state will be partitioned?

Regards

Kshitiz

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

suyog | 23 May 2013 23:46

Multiple topologies and worker count

Hi, in our storm (0.8.2) cluster -- we have 8 supervisor machines and each machine by default configuration has 4 slots (supervisor.slots). So our total slots are 32. We are deploying two independent topologies and each topology has topology.workers set to 8. Does this mean that 16 out of the 32 slots will be used across the clusters for both the topologies?

In our environment, I run into the issue where once the first topology is submitted (and running fine), the second topology does not start. When I explicitly make the topology.workers=6 for 1st topology and topology.workers=2 for 2nd topology, they work fine together. I was under the impression that # of slots = # of JVMs = # of total workers in the cluster? Is that assumption right? If so, I would think that both topologies should be able to start independently on the same cluster with num of workers set to 8 each

Thanks,

Suyog

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

SAG_Developer | 23 May 2013 18:59

autogen.sh: error: could not find pkg-config. pkg-config is required to run autogen.sh.

I am trying to setup a storm cluster on Ubuntu 12.04 server by following this link https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster

I have ZeroMQ installed

Running autogen.sh in the jzmq source folder is throwing an error. autogen.sh: error: could not find pkg-config. pkg-config is required to run autogen.sh.

What do I need on my server to by pass this issue?

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

andy | 23 May 2013 22:55

Combining the output of two Trident streams

If I have two Trident streams that produce the same fields as output, but at different rates, is there a way to combine their output?

So S1 has fields X, Y and Z and tuples entering it:

x1, y1, z1

x2, y2, z2

x3, y3, z3

.

And S2 has the same fields X, Y and Z, with tuples:

x10, y10, z10

x11, y11, z11

.

Can I combine them to a new stream containing:

x1, y1, z1

x2, y2, z2

x3, y3, z3

x10, y10, z10

x11, y11, z11

.

I can see there's the TridentTopology.multireduce method, but I'm not sure if that'll do the job, or what the implications are for the flow of tuples from

each input stream?

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

phlp | 23 May 2013 19:21

Ack/Fail counting when topology fails

We use BaseRichSpout and BaseRichBolt in our application.

The spout counts emitted tuples and acks/fails to derive state information when emitted-count == sum(ack-count, fail-count).

When a tuple is emitted in one topology which is then restarted before the tuple is ack'd by the bolt, will the new spout's ack routine be called when the tuple is re-emitted by storm and processed and ack'd by the bolts?

I.E. will a spout get an ack for a tuple that it did not directly emit?

I'm assuming the answer is yes so that (for instance) external state associated with the tuple is cleaned up.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

Sunil Yarram | 23 May 2013 03:22

trident batch question

Hi,

I have few basic questions on trident processing:

1. The trident document mentions only one batch will be in commit phase, does that mean there can be many parallel batches doing processing but not doing the commit?

2. If trident is processing one batch at a time then the parallelism is achevied by dividing the batch into partitions? and i am guessing the number of partitions are controlled by parallelismHint() method.

Thanks,

Sunil.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

thiswym | 22 May 2013 09:57

About setting RotatingMap ExpireTime

Hi, all fella:

I recently utilize storm for realtime computing. I required a TimePeriodMemoryCache, then I found TimeCacheMap that is annotated Deprecated. I found this link http://comments.gmane.org/gmane.comp.java.clojure.storm/4683 to recommand RotatingMap instead of TimeCacheMap. I seek its api http://nathanmarz.github.io/storm/doc/backtype/storm/utils/RotatingMap.html, however, I NOT where to set expiredTime.

Please help me.

Best Begard.

Robin.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

Stumpyfr . | 22 May 2013 21:32

storm-config, topologyLoader

Hi,

I didn't find something like that, so I created it.

Just a very first version but I wanted to share it with you and maybe get some advices from the community.

The main reason was to not need to compile my .jar each time I want try a new configuration for the parallelism of my topology.

You still need to kill/start your topology but to change the configuration, just need to change the values inside the json configuration file.

I plan to add some configuration values, open to any ideas to what will be the more useful.

./storm-0.8.2/bin/storm jar MyProject.jar com.tekforge.StormSample.App config.json

https://github.com/stumpyfr/storm-config

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

Miguel Durazo | 22 May 2013 22:03

(Newbie) Is there a very simple example to get started?

I already read the wiki, but i have no idea how to begin, i want to be able to test simple things locally first, a small code example would do

Thanks!

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

headers

art0chu | 22 May 2013 17:06

Trident ParallelismHint Set But Only One Work

Hi, all

I am using Trident topology and I am confused about the parallelism, I found https://groups.google.com/forum/?fromgroups#!topic/storm-user/QlCmDmnEghs and <at> nathanmarz explained what the parallelism use, I build my topology below:

List<HostPort> hostports = new ArrayList<HostPort>();

HostPort hostport = new HostPort(getPropertyVar("KAFKA_DOMAIN1"),

Integer.parseInt(getPropertyVar("KAFKA_PORT")));

hostports.add(hostport);

hostport = new HostPort(getPropertyVar("KAFKA_DOMAIN2"),

Integer.parseInt(getPropertyVar("KAFKA_PORT")));

hostports.add(hostport);

TridentKafkaConfig.StaticHosts hosts = new TridentKafkaConfig.StaticHosts(

Lists.newArrayList(hostport), 1);

TridentKafkaConfig config = new TridentKafkaConfig(hosts,

getPropertyVar("KAFKA_TOPIC"));

// config.scheme = new SchemeAsMultiScheme(new StringScheme());

// config.forceFromStart = true;

TransactionalTridentKafkaSpout spout = new TransactionalTridentKafkaSpout(

config);

topology

.newStream("tridentspout" + uid, spout)

.name("tridentspout0522")

.parallelismHint(4)

.shuffle()

.each(new Fields("bytes"), new Decrease(),

new Fields("ips", "type", "hour_str", "time"))

.name("decrease0522")

.each(new Fields("ips", "type", "hour_str", "time"),

new WriteRedis(), new Fields("result_id")).parallelismHint(4)

.name("writeredis0522");

But I found in the UI that although spout has four excutors, it seems only one work, others have nothing to do, just like below. the capacity is over 1, so the topology runs very slowly. Am i use it wrong?

I use zk-3.4.5, storm-0.9.0-wip16, kafka-0.7.2.

Thanks !

art0chu

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

442	May 2013
536	April 2013
647	March 2013
456	February 2013
616	January 2013
254	December 2012
428	November 2012
597	October 2012
472	September 2012
625	August 2012
592	July 2012
548	June 2012
421	May 2012
571	April 2012
478	March 2012
490	February 2012
286	January 2012
578	December 2011
286	November 2011
394	October 2011