Alexander Myltsev | 20 Aug 23:36 2015

Creating org.apache.commons.lang3.text.translate.LookupTranslator

Hi,

there is a class http://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/text/translate/LookupTranslator.html that accepts (CharSequence[]... lookup) in constructor.

In Java it is used as follows:

public static final CharSequenceTranslator ESCAPE_HTML4 = new AggregateTranslator( new LookupTranslator(EntityArrays.BASIC_ESCAPE()), new LookupTranslator(EntityArrays.ISO8859_1_ESCAPE()), new LookupTranslator(EntityArrays.HTML40_EXTENDED_ESCAPE()) );

I wonder how LookupTranslator should be created in Scala? Ugly way is as follows:

val translators = List( new LookupTranslator(EntityArrays.BASIC_UNESCAPE.map { _.map { _.asInstanceOf[CharSequence] } }: _*), new LookupTranslator(EntityArrays.ISO8859_1_UNESCAPE.map { _.map { _.asInstanceOf[CharSequence] } }: _*), new LookupTranslator(EntityArrays.HTML40_EXTENDED_UNESCAPE.map { _.map { _.asInstanceOf[CharSequence] } }: _*), new NumericEntityUnescaper )

Any prettier one?

A.

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Torsten Schmits | 16 Aug 15:42 2015
Picon

Setting positions for annotation generated trees

I use annotations on objects to generate a lot of code from very little code, so I'd like to have a way to match compiler errors from the generated code to a location in that code.
It's easy to print the expanded tree to a file using `showCode`, but the compiler doesn't know about that, of course.
Alas, I recurse into the generated tree at the very end of the expansion and update the position to use the written source file, but it's quite hard to determine the correct offset for the individual trees.
Right now, I'm doing it heuristically, treating Template and DefDef differently, which is fine, but if there is something off, the compiler doesn't use the debug printed file for determination of error context and line number, but the file where the macros annotation is, although the file name specified is correct.
It worked a few times, though with incorrect offsets.

My question is, is there an easier way to do this? Like, can I use the compiler api inside of a macro expansion to do this?

Also, is there a sensible way to set the position attributes? Right now, I use runtime reflection on the `Position.withSource` and `Tree.setPos` methods and the `SourceFile.file` field.

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jackson Davis | 11 Aug 18:27 2015

scala's StringBuilder and -XX:+OptimizeStringConcat


(Apologies in advance if this is not the right mailing list for this, it wasn't too obvious if this should go here or scala-user)

Over the last few weeks I have been doing some digging on poor performance of String concatenations. The java.lang.StringBuilder (and thus scala.collection.mutable.StringBuilder) is more or less terrible at least when it comes to initial capacity (16) and resizing logic, which creates a lot of excess garbage and array copies in regular usage.
However, it turns out there is an very important but underdocumented jvm optimization, OptimizeStringConcat, which is enabled by default. This optimization, which is pretty much only documented by its source code (http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/share/vm/opto/stringopts.cpp), is capable of turning something of approximately the form new StringBuilder().append(x).append(y)....append(z).toString() (aka about what javac would generate if you use x + y + ... + z) into a single char[] allocation of the optimal length, which is a big performance boost over what the normal java code does (2 char[] allocations at the absolute minimum if the initial size is correct, normally more)
The issue is that this same optimization doesn't work for the identical scala code. Why? well I wish I could tell you I understood enough of the C++ to give you an exact answer, but I can only approximate. To my understanding, the issue is that analysis is only done at a function-level scope of where the java.lang.StringBuilder.toString happens, which is in scala.collection.mutable.StringBuilder.toString. All of the other bits (constructor and append calls) occur elsewhere, so it more or less gives up (from looking at the debug output from -XX:+PrintOptimizeStringConcat in a debug jdk as well as adding my own debugging output in a few places in that while).
Its hard to properly quantify the performance improvement of this, particulary wrt to fewer allocations. I attempted anyways by writing a benchmark that appends a suffix to a String holding a random number millions of times with plenty of warmup in a jvm with a very small heap (32mb) to better see the effects of less GC, and the unscientific results look something like this:  (sorry for the odd formatting)
 

Optimization Enabled

Optimization Disabled

Java

5.9s

15.2s

Scala

16.5s

16.5s

Scala w/

Java SB

6.1s

16.0s

 
(Note that here Optimization Enabled means -XX:+OptimizeStringConcat, and Optimization Disabled means -XX:-OptimizeStringConcat)

I'm not quite sure what the right follow up is here. This is a sizeable performance regression from java, and enough to lead us to want to replace +s with calls to java.lang.StringBuilder in some hot parts of code. This seems potentially bug-worthy, but I'm not sure how it can ever be resolved short of scalac discontinuing the use of scala.collection.mutable.StringBuilder for +s. (The C++ more or less looks for java.lang.StringBuilder / StringBuffer, and will only attempt to optimize those 2 classes)

Thoughts?
-Jackson


--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
co.kleisli | 9 Aug 02:58 2015
Picon

Partition A Stream Error

Hi,

I'm not sure if this is possible but I want to partition a stream based on some condition that depends on the output of the stream. It will make sense with an example I think.

I will create a bunch of orders which I will stream since the actual use case is a stream of orders coming in so it is not known up front what the next order will be or even the full list of orders:

scala> case class Order(item : String, qty : Int, price : Double) defined class Order scala> val orders = List(Order("bike", 1, 23.34), Order("book", 3, 2.34), Order("lamp", 1, 9.44), Order("bike", 1, 23.34)) orders: List[Order] = List(Order(bike,1,23.34), Order(book,3,2.34), Order(lamp,1,9.44), Order(bike,1,23.34))

Now I want to partition/group these orders into one set which contain duplicate orders and another set which contains unique orders. So in the above example, when I force the stream it should create two streams: one with the two orders for a bike (Since they are the same) and another stream containing all the other orders.

I tried the following:

created the partitioning function:

scala> def matchOrders(o : Order, s : Stream[Order]) = s.contains(o) matchOrders: (o: Order, s: Stream[Order])Boolean

then tried to apply this to stream:

scala> val s : (Stream[Order], Stream[Order]) = orders.toStream.partition(matchOrders(_, s._1))

I got a null pointer exception since I guess the s._1 is empty initially?? I'm not sure. I've tried other ways but I'm not getting very far. Is there a way to achieve this partitioning?


regards,

Iftikhar

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
co.kleisli | 9 Aug 02:57 2015
Picon

Null Pointer Exception When Partitioning A Stream

Hi,


I'm not sure if this is possible but I want to partition a stream based on some condition that depends on the output of the stream. It will make sense with an example I think.


I will create a bunch of orders which I will stream since the actual use case is a stream of orders coming in so it is not known up front what the next order will be or even the full list of orders:


scala> case class Order(item : String, qty : Int, price : Double) defined class Order scala> val orders = List(Order("bike", 1, 23.34), Order("book", 3, 2.34), Order("lamp", 1, 9.44), Order("bike", 1, 23.34)) orders: List[Order] = List(Order(bike,1,23.34), Order(book,3,2.34), Order(lamp,1,9.44), Order(bike,1,23.34))


Now I want to partition/group these orders into one set which contain duplicate orders and another set which contains unique orders. So in the above example, when I force the stream it should create two streams: one with the two orders for a bike (Since they are the same) and another stream containing all the other orders.

I tried the following:


created the partitioning function:


scala> def matchOrders(o : Order, s : Stream[Order]) = s.contains(o) matchOrders: (o: Order, s: Stream[Order])Boolean


then tried to apply this to stream:


scala> val s : (Stream[Order], Stream[Order]) = orders.toStream.partition(matchOrders(_, s._1))


I got a null pointer exception since I guess the s._1 is empty initially?? I'm not sure. I've tried other ways but I'm not getting very far. Is there a way to achieve this partitioning?


Regards,

Iftikhar

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
co.kleisli | 9 Aug 02:56 2015
Picon

Null Pointer When Partitioning A Stream

Hi,


I'm not sure if this is possible but I want to partition a stream

based on some condition that depends on the current output

state of the stream. It will make sense with an example I think.


I will create a bunch of orders which I will stream since the 

actual use case is a stream of orders coming in so it is not 

known up front what the next order will be or even the full list 

of orders:


scala> case class Order(item : String, qty : Int, price : Double) defined class Order scala> val orders = List(Order("bike", 1, 23.34), Order("book", 3, 2.34), Order("lamp", 1, 9.44), Order("bike", 1, 23.34)) orders: List[Order] = List(Order(bike,1,23.34), Order(book,3,2.34), Order(lamp,1,9.44), Order(bike,1,23.34))


Now I want to partition/group these orders into one set which 

contain exactly the same order and another set which contains unique 

orders. So in the above example, when I force the stream it should 

create two streams: one with the two orders for a bike (since they are the same) 

and another stream containing all the other orders.

I tried the following:


Created the partitioning function:


scala> def matchOrders(o : Order, s : Stream[Order]) = s.contains(o) matchOrders: (o: Order, s: Stream[Order])Boolean


then tried to apply this to stream:


scala> val s : (Stream[Order], Stream[Order]) = orders.toStream.partition(matchOrders(_, s._1))


I got a null pointer exception since I guess the s._1 is empty initially?? 

I'm not sure. I've tried other ways but I'm not getting very far. Is there a 

way to achieve this partitioning?


Regards

Iftikhar

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
co.kleisli | 5 Aug 20:16 2015
Picon

StackOverFlowError When Applying Filter to a Stream

Hello,


Please can some one explain why the following code blows the stack:


val fibs: Stream[BigInt] = BigInt(0) #:: BigInt(1) #:: fibs.zip(fibs.tail).map { n => n._1 + n._2 }.takeWhile(x => x < 4000000).filter(x => x % 2 == 0) scala> fibs foreach println 0 1 java.lang.StackOverflowError

If I take out the filter and apply it to the stream in another expression as follows, it is fine:

scala> val fibs: Stream[BigInt] = BigInt(0) #:: BigInt(1) #:: fibs.zip(fibs.tail).map { n => n._1 + n._2 }.takeWhile(x => x < 4000000) fibs: Stream[scala.math.BigInt] = Stream(0, ?) scala> fibs filter ( x => x % 2 == 0) res8: scala.collection.immutable.Stream[scala.math.BigInt] = Stream(0, ?) scala> fibs filter ( x => x % 2 == 0) foreach println 0 2 8 34 144 610 2584 10946 46368 196418 832040 3524578

Why does it blow the stack with the first approach but not the second?


Thanks

Iftikhar

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Tim Preece | 5 Aug 16:17 2015

Timing window in sys.process.ProcessImpl.scala

I've noticed a failure running some Apache Spark CLI unit tests.

In class PipedProcesses we indicate the input stream has been read into a PipeInputStream on line 138 by "currentSink put None".

However if this occurs before the command has got going ( referred to in the code as second ) it can never see the input stream. The run method in class PipeSink can just fall through to the case None ( line 183 ) without any input having been read.

I believe it can be simply fixed by adding and checking a new SyncVar ( maybe called readComplete ) which is set once the runloop has completed reading data from the pipe.

Is it appropriate if I just create a Scala issue? Sorry if that's a dumb question - new to the Scala community.

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Daniel Armak | 28 Jul 22:54 2015
Picon

Toolbox typecheck throws MatchError for higher-kinded path-dependent type

Hi,

I encountered this error trying to compile code that uses the shapeless ~?> typeclass with the toolbox, and simplified it to this:

import scala.reflect.runtime.currentMirror import scala.tools.reflect.ToolBox val tb = ToolBox(currentMirror).mkToolBox() val source = """{ import scala.language.higherKinds class Outer[X] { class Inner[Y] } class Other[T[_]] new Other[Outer[Int]#Inner]() }""" val tree = tb.parse(source) tb.typecheck(tree).tpe

This fails with:

Exception in thread "main" scala.MatchError: <tree with deferred refcheck> (of class scala.tools.nsc.ast.Trees$TypeTreeWithDeferredRefCheck) at scala.reflect.internal.Importers$StandardImporter.recreateTree(Importers.scala:309)

(Full stack trace here)

I’m using scala 2.11.7.

Thanks,

Daniel Armak

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sebastian M Cheung | 24 Jul 16:35 2015

How to improve distributed app in Scala?

What are best practices in improving distributed Scala app performances?


--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sebastian M Cheung | 23 Jul 21:15 2015

Why after updating Array element I still get old array values?

Why after updating Array element I still get old array values?

var compactWay = Array("I", "like", "this", "way")>compactWay: Array[String] = Array(I, like, this, way)
compactWay(0) = "We"
println(compactWay.mkString(" "))>We like this way res3: Unit = ()
var secondList = Array("Something else")var thirdList = compactWay++secondList>thirdList: Array[String] = Array(I, like, this, way, Something else)
(expecting to see Array(We, like, this, way, Something else)

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-language+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gmane