Matthew Zimmerman | 5 Feb 03:48 2016
Picon
Gravatar

[ANN] multicorecsv - read large CSV/TSV files

I've been doing a lot of large(r) data processing recently where I have to reload the same data over and over again.  It was getting extremely annoying waiting for the processing of 10gb TSV files to be using a spiked single core while the rest of my machine went idle.

https://github.com/mzimmerman/multicorecsv

If you're data doesn't have newlines encoded in it [1], multicorecsv breaks your file up into chunks by lines, parses the CSV, and responds overall roughly ~3x faster than encoding/csv (varies by hardware and data)

I've been using it for a couple days and it's made me twiddle my thumbs a lot less!

It's almost a direct drop in replacement for encoding/csv with one additional required step of calling Close(), but you likely already do that on the underlying io.Reader.  multicorecsv requires you to pull that one more level away so it can clean up any goroutines on error. [2]

Hope it helps some others playing the waiting game on their CSV data!
- Matt

[1] - CSV accepts newlines inside each field 
[2] - FieldsPerRecord is not implemented as it would take more coordination to enforce that yielding some of the performance gains (and I never use this feature!)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Nuri Kevenoglu | 4 Feb 23:24 2016
Picon

noob: type assertion vs reflection

Hi guys,

I'm sure these questions would've been asked before but I couldn't see a way to search though the topics.

1. Is type assertion a type of reflection?

2. How does type assertion compare to reflection in performance? 

3. Is it recommended to avoid type assertion if possible, like reflection?

I have read the the laws of reflection, but as far as I can see, these questions are not answered there.

Thank you!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Eric Brown | 4 Feb 18:12 2016
Picon
Gravatar

public variables within main.go

New to goLang, and getting the whole public variable thing down; however, am having one issue... and not sure if it's possible.  In the main.go file (package main) I have a couple of public variables that are set using command-line arguments at run-time.  I want those variables accessible to other packages.  Sharing variables among packages are simple enough (after importing it, use package.Variable); however, there doesn't seem to be a way to import main (get error "...is a program, not an importable package).  How do I import and use (if possible) public variables from the main package?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
bob.ziuchkovski | 4 Feb 19:49 2016
Picon

[ANN] DeVo: A pure Go utility to decrypt TiVo files

Hi All,

I just released a pure Go utility for decrypting TiVo files.  It's called DeVo and is available at https://github.com/bobziuchkovski/devo.

This is meant as an alternative to tivodecode. I started out with the intention to patch tivodecode, fixing some mpeg-ts and mpeg-ps parsing bugs.  However, after spending a day or two refactoring and patching, I realized it would be easier to implement from scratch in Go, and would be more readable.  So I wrote DeVo!

I don't know if there's much of an audience here for the tool, but thought I'd share anyway.

Let me know if you have any questions or comments on it.

Thanks!

- Bob

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
bob.ziuchkovski | 4 Feb 19:43 2016
Picon

[ANN] turing: a pure Go implementation of the turing stream cipher

Hi All,

I released a pure Go implementation of the turing stream cipher at https://github.com/bobziuchkovski/turing.  The cipher algorithm is NOT a secure, but the implementation might be useful for anyone working with legacy applications/protocols that use it.

That was my particular case, as the turing cipher is used by TiVo devices to encrypt video files.  I'll be posting a separate release announcement for a tool to decrypt TiVo files.

Let me know if you have any questions or comments on the turing cipher.

Thanks!

- Bob


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
John McKown | 4 Feb 17:14 2016
Picon

size of executable

I just de-installed gcc-go from my Fedora system. It appears that it was causing some problems when installed along with the golang-bin package. But, when gcc-go, was installed, it was used to compile the code below. After de-installing gcc-go, I recompiled the code. The resulting executable was about 75 times as large! I _must_ be doing something wrong. I did a simple "go build" to compile.

== source code ==

package main

import "net"

func handler(c net.Conn) {
 c.Write([]byte("ok"))
 c.Close()
}

func main() {
 l, err := net.Listen("tcp", ":5000")
 if err != nil {
  panic(err)
 }
 for {
  c, err := l.Accept()
  if err != nil {
   continue
  }
  go handler(c)
 }
}

== end

-rwxr-xr-x. 1 tsh009 TSHG 2610104 Feb  4 09:57 example-simple-webserver

-rwxr-xr-x. 1 tsh009 TSHG   34368 Feb  3 06:55 example-simple-webserver.old

the ".old" was compiled with gcc-go. That's 34_368 vs 2_610_104 bytes. Just as a guess, I'd say that the golang-bin compiler does not use a "shared library" for subroutines, like gcc-go apparently does, but instead does a "static" link of the Go subroutines that are used.

Any ideas as to how to make the executable smaller, other than the strip command to remove symbols?

--
Werner Heisenberg is driving down the autobahn. A police officer pulls
him over. The officer says, "Excuse me, sir, do you know how fast you
were going?"
"No," replies Dr. Heisenberg, "but I know where I am."

Computer Science is the only discipline in which we view adding a new wing to a building as being maintenance -- Jim Horning

Schrodinger's backup: The condition of any backup is unknown until a restore is attempted.

He's about as useful as a wax frying pan.

Maranatha! <><
John McKown

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
W. David Jarvis | 4 Feb 09:24 2016
Picon
Gravatar

Idiomatic Go design pattern analog for thread-local storage case in Java

Hello, all. 

Long time listener, first time caller.

Here's the situation I'm in. I got bored and started writing a compiler from Clojure to the Go runtime. At the moment, I'm struggling with how to translate the JVM's concurrency idioms for Clojure `Vars` over to Go. In particular, Clojure Vars rely on thread-local storage (ref: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Var.java#L67). The use case for this isn't immediately obvious, but in essence it enables downstream local bindings within Clojure source code that don't clobber global references.

From what I understand, `x/net/context` exists largely to address problems very similar to this one, but going down that route feels like I could very easily end up having to pass context as an additional function argument to half the functions in the compiler. That feels ugly, though the functional programmer in me is also pleased by it.

Has anyone else run into a similar translation issue to this? What's the recommended path here?

Cheers,

 - V

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
mikespook | 4 Feb 09:14 2016
Picon
Gravatar

[ANN] goRBAC: lightweight role-based access control redesigned

Hi all,

The redesign plan was started at the end of last year and has completed now.
I tagged the original version as Version 1 and moved the redesign version to the master branch.

The major change is it's more convenient to implement self-defined Role & Permission since `Role` and `Permission` are interfaces.
Hoping your guys can give me some feedback on the new design.


Thank you & happy coding!


Xing

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Gerardo Oscar JT | 4 Feb 12:49 2016
Picon
Gravatar

[ANN]: golax, a framework to build REST APIs

Hello, this is my first intervention in this forum.

I'd like to introduce you golax (https://github.com/fulldump/golax) a framework
to build REST APIs from a different perspective: using nodes instead of full
paths, and `interceptors` instead of middlewares.

From my humble point of view it is a powerful approach to the REST APIs 
world.

Have a good day!
fulldump

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
ehlers.jens | 4 Feb 10:45 2016
Picon

Accessing individual elements with a slice

Hi,

Say that I have an array with elements {a,b,c,d,e,f,g,h}. I want to create a slice containing only elements with, say, index  0, 1, 3 and 6. The resulting slice would be {a,b,d,g}.  Is there a good way to achieve this? I tried array[0,1,3,6] to no avail...

Jens

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
mchlekpwlk | 4 Feb 15:22 2016
Picon

compiler bug

I try this:

package main

import "fmt"

func main() {
if 1 ! =2 {
fmt.Println("Hello, 世界")
}
}

and I get this:
prog.go:8: syntax error: unexpected !, expecting { prog.go:11: syntax error: unexpected }

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Gmane