beefsack | 3 Sep 04:06 2014
Picon

gar - Go application archives inspired by JAR - request for input

Hi all, I've implemented a prototype application archiver inspired by JAR called gar (https://github.com/beefsack/gar, example app https://github.com/beefsack/gar-example) which packs resources into the compiled Go binary to make the binary completely standalone.

I'd love some feedback, more from the design standpoint than the actual code, what I have in that repository is a minimal proof of concept but it's working as shown in the example app.

The main pain points which prompted me to attempt this are:
  • Some missing resources could make application unusable, such as templates for a web application.
  • Separate resources complicate deployment and distribution.
Some details of my implementation:
  • Assets are tarred up and optionally gzipped, then appended to a compiled binary.
  • A footer is appended to the binary with the size of the tar, option bits (currently only whether gzip is enabled), and an identifier so the file can be identified as a gar archive.
  • When a resource is requested, the package checks to see if the binary is a gar archive and if so, loads resources into memory.
  • Resource access functions provided through the gar package, eg. gar.Open("templates/home.tmpl") which returns an os.FileInfo and a io.ReadSeeker for the content.
  • If binary is not identified as a gar archive, it falls back to the file system by default.  Lookup methods and locations are configurable.
My concerns are:
  • Reading from the binary while executing is a bad idea and could cause issues.
  • Storing all the resources in memory is a problem if you have lots of resources.
Some alternatives I considered but not sure if they're better or feasible:
  • Creating a wrapper instead of appending resources directly to the binary.
  • Extracting the resources to the file system instead of keeping them in memory.
To build a gar archive of the example to see it in action:

$ go get github.com/beefsack/gar/gar
$ go 
get github.com/beefsack/gar-example
$ cd $GOPATH
/github.com/beefsack/gar-example
$ gar 
.
./out.gar

Looking forward to any feedback before I proceed with it as I'm not sure if I'm going in the right direction or even if it's a good idea.

Cheers,
Michael

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
akwillis | 3 Sep 03:08 2014
Picon

Re: Panic in init function

> Does the panic() in the init function get 'cancelled' by the recover in the 
> main function? 

That question gives me the impression that it is a program initialization and execution misunderstanding. There is more happening before main() than people might realize.

On Monday, September 1, 2014 8:24:53 PM UTC-4, Andrew Gerrand wrote:
The misunderstanding here is about how panic and recover interact. There's a nice blog post about that: http://blog.golang.org/defer-panic-and-recover


> Does the panic() in the init function get 'cancelled' by the recover in the
> main function?
>
> Thanks
> Josh

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts... <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Stephen Gutekanst | 2 Sep 23:29 2014
Picon

Azul3D: Mac OS X Support

Azul3D now has Mac OS X support:

http://azul3d.org/news/2014/mac-osx-support.html

After my previous announcement about Azul3D -- a 3D game engine written in Go, many of you wondered if OS X would be supported in the future. Hopefully this both answers your questions and fulfills your cross-platform Go game development desire. =)

Note: I wasn't sure if posting follow-up announcements in golang-nuts should be done in a new thread or if I should revive the old one? Does anyone know what the standard is?

Have a great day,
- Stephen

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Donovan Hide | 2 Sep 21:45 2014
Picon

Fwd: [golang-dev] Re: proposal: go generate

It is almost certainly a mistake to extend gofmt to do more sophisticated program rewrites. Write separate tools.

Russ

Or extract the rewriting out of gofmt into a standard library package that can be used to write such tools?


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Davide D'Agostino | 2 Sep 21:17 2014
Picon

Scheduler blocked with CGO / Semaphore Wait

Hi guys,

I'm porting bindings for libvips (a super fast image processor) over CGO. See: https://github.com/daddye/vips

There aren't many test on this repo since is part of a proprietary code which already have a test suite, in future I'll port more of them.

So, my problem:

I spent all the searching for a solution of a problem with CGO and Channels.

I have a quite big app, while my code works and is very fast, when I ran my benchmark suite results 1000x worse. So I'm not sure if I'm doing something wrong or is a know problem.

Here a simplified version of the code:

func main() {
        runtime
.GOMAXPROCS(0)

       
// prepare communication protocol
        urls
:= make(chan string)
        wg
:= new(sync.WaitGroup)

       
// fire a bunch of workers
       
for i := 0; i < env.MaxProcs; i++ {
                wg
.Add(1)
                go func
(i int) {
                        fmt
.Printf("starting worker %d...\n", i)
                        defer wg
.Done()
                       
for url := range urls {
                                C
.resize_image(url) // takes around 30ms
                       
}
               
}(i)
       
}

       
// db connection (native go)....

       
var url string
       
for i := 0; iter.Scan(&url) && i < 1e4; i++ { // Scan takes around 8ms
               
// with a time.Sleep(10 * time.Millisecond) results are 1000x better!
                urls
<- url
       
}
        err
= iter.Close()

       
// wait
        close
(urls)
        wg
.Wait()
}

When I run pprof:

    3512  76.4%  76.4%     3512  76.4% runtime.mach_semaphore_wait
     
968  21.1%  97.5%      968  21.1% runtime.cgocall
     
107   2.3%  99.8%      107   2.3% runtime.kevent
       
3   0.1%  99.9%        3   0.1% runtime.usleep
       
2   0.0%  99.9%        2   0.0% syscall.Syscall
       
1   0.0%  99.9%        1   0.0% math/rand.Float64
       
1   0.0% 100.0%        1   0.0% runtime.MHeap_LookupMaybe
       
1   0.0% 100.0%        1   0.0% runtime.MSpan_Sweep
       
1   0.0% 100.0%        1   0.0% runtime.memclr
       
0   0.0% 100.0%        6   0.1% GC
       
0   0.0% 100.0%     3618  78.7% System
       
0   0.0% 100.0%      968  21.1% github.com/daddye/vips.Resize
       
0   0.0% 100.0%       28   0.6% github.com/daddye/vips._Cfunc_im_close
       
0   0.0% 100.0%      278   6.0% github.com/daddye/vips._Cfunc_vips_affine_interpolator
       
0   0.0% 100.0%       18   0.4% github.com/daddye/vips._Cfunc_vips_colourspace_0
       
0   0.0% 100.0%       43   0.9% github.com/daddye/vips._Cfunc_vips_copy_0
       
0   0.0% 100.0%      258   5.6% github.com/daddye/vips._Cfunc_vips_embed_extend
       
0   0.0% 100.0%       35   0.8% github.com/daddye/vips._Cfunc_vips_image_new
       
0   0.0% 100.0%       16   0.3% github.com/daddye/vips._Cfunc_vips_jpegload_buffer_seq
       
0   0.0% 100.0%       15   0.3% github.com/daddye/vips._Cfunc_vips_jpegload_buffer_shrink

So, here things I tried:

1. Use another CGO binding: image magick - Same
2. Use runtime.Gosched() (tried to put it in different places) - Same
3. Use runtime.LockOSThread() (tried to put it in different places) - Same
4. Add a time.Sleep(10*time.Milliseconds) - the commented line - I get back good results!
5. Test on linux (ubuntu lts 12) - Same and pretty much same pprof
6. Remove CGO and use a native call - Very fast. 
7. Avoid channels just `go func()` - Very fast!

So, I know cgo ffi requires 40ns for a nop call, however since resize images can take from 20ms to 1s (in my case), I think it's worth. 
I tried all native go packages for image resizing, but unfortunately are even slower than a CGO in stuck with the semaphore. :'(

As said, in "production" I don't see this issue because between a schedule `urls <- url` and another there are few operations that takes some time, so it's "fine".

However, now I'm curious, what is the problem here? How can I avoid to be trapped in a similar scenario?

Thanks a lot!
DAddYE

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Alexey Borzenkov | 2 Sep 21:15 2014
Picon

Unlink substrings from the outer string?

Hi,

Is there a correct way to unlink substrings from the outer strings
where it's taken from? It's nice to have composable goroutines, which
e.g. parse and then do processing stages over text, and the fact that
strings and not copied when sliced makes parsing a lot faster, however
when out of very large original strings I only need tiny bits (which
are e.g. used as map keys), the extra weight of keeping those original
strings from being garbage collected adds up very fast.

Currently this seems to do the trick:

s = string([]byte(s))

However it feels wrong to do unnecessary copying and to produce
unnecessary garbage with this, because it first calls
runtime.stringtoslicebyte, and then runtime.slicebytetostring.
Unfortunately string(s) seems to be a no-op and doesn't make a copy.
Is there a better way to unlink and make a fresh copy of a string?

Thanks,
Alexey.

--

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@...
For more options, visit https://groups.google.com/d/optout.

Gerald Stan | 2 Sep 20:57 2014
Picon

Re: Matlab interface

thats what I already did...
calling a simple hello_world executable takes around 400ms...
so looking for another option...

im not sure if golang can do memory maps!?
so sockets would be the best option i guess...

Maxim Khitrov:



I would start with something simple, like using the system function to
run your Go binary and passing data via stdin/stdout or files, measure
the performance, and then move on to more complicated methods, if
needed.


2014-09-02 20:16 GMT+02:00 Maxim Khitrov <max-52VJov5/uMRBDgjK7y7TUQ@public.gmane.org>:
I would start with something simple, like using the system function to
run your Go binary and passing data via stdin/stdout or files, measure
the performance, and then move on to more complicated methods, if
needed.

On Tue, Sep 2, 2014 at 1:38 PM, Gerald Stan <gerald.stanje-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> i do some parsing with golang...matlab sends the text to be parsed to
> golang, go does the parsing and needs to send it back to matlab...
> i need an option with low latency...
>
> yeah, but i cant compile my go source with mex...
>
>
> 2014-09-02 19:32 GMT+02:00 Maxim Khitrov <max-52VJov5/uMRBDgjK7y7TUQ@public.gmane.org>:
>
>> It depends entirely on what you're trying to do and whether you mean
>> speed as in latency or throughput. You could use memory maps, as in
>> the following example:
>>
>>
>> http://www.mathworks.com/help/matlab/import_export/share-memory-between-applications.html
>>
>> Network is another option, calling an executable may also work. For
>> anything that you can't do directly in MATLAB, you can compile mex
>> functions from C or write your own Java classes.
>>
>> On Tue, Sep 2, 2014 at 1:11 PM,  <gerald.stanje-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > what IPC technique would you choose between matlab and golang if speed
>> > matters?
>> > sockets?
>> > from what i see matlab can do sockets...
>> >
>> > another way is: call the executable from matlab...i guess thats rather
>> > slow?
>> >
>> > On Tuesday, August 26, 2014 6:21:23 PM UTC+2, Maxim Khitrov wrote:
>> >>
>> >> If the developers decide to implement a way of initializing the
>> >> runtime from C, then it would be possible. I don't think there are any
>> >> plans to do that at the moment.
>> >>
>> >> On Tue, Aug 26, 2014 at 12:15 PM,  <gerald...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >> > yes...runtime does gc and other stuff...
>> >> >
>> >> > i see that matlab can call java code using JMI....could that be
>> >> > applied
>> >> > to
>> >> > golang too?
>> >> >
>> >> > On Tuesday, August 26, 2014 3:34:12 PM UTC+2, Maxim Khitrov wrote:
>> >> >>
>> >> >> Not unless something has changed recently with the runtime. You can
>> >> >> make Go -> C -> Go calls, but not C -> Go. In other words, the Go
>> >> >> runtime needs to be the first thing to run. Of course, you can
>> >> >> create
>> >> >> a separate Go executable and communicate with it through files,
>> >> >> network, or any other IPC mechanism.
>> >> >>
>> >> >> On Tue, Aug 26, 2014 at 8:40 AM, Gerald Stanje <gerald...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>> >> >> wrote:
>> >> >> > Yes, matlab supports c/c++ function calls...
>> >> >> > But you could call a golang function from c?
>> >> >> >
>> >> >> > Sent from my iPhone
>> >> >> >
>> >> >> >> On Aug 26, 2014, at 2:14 PM, Maxim Khitrov <m...-52VJov5/uMRBDgjK7y7TUQ@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >> You can call into MATLAB from Go via cgo, but not the other way
>> >> >> >> around. Here's some relevant documentation:
>> >> >> >>
>> >> >> >>
>> >> >> >> http://www.mathworks.com/help/matlab/calling-external-functions.html
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> http://www.mathworks.com/help/matlab/programming-interfaces-for-c-c-fortran-com.html
>> >> >> >>
>> >> >> >> - Max
>> >> >> >>
>> >> >> >>> On Tue, Aug 26, 2014 at 5:39 AM,  <gerald...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> is there a way to call a golang function within a Matlab
>> >> >> >>> environment.
>> >> >> >>> I know
>> >> >> >>> what you can call C/C++ functions within Matlab...
>> >> >> >>>
>> >> >> >>> Thanks,
>> >> >> >>> Gerald
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "golang-nuts" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to golang-nuts...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> > For more options, visit https://groups.google.com/d/optout.
>
>

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Anton Sekatski | 2 Sep 18:25 2014
Picon

wutrender - simple plain Go template renderer with layout and partials support

Hello to everyone,

I released a package called "wutrender" here: github.com/8protons/wutrender

No magic at all, just plain Go templates. It reads all files from a directory (default to "templates") creating templates with names in format "path/name.{format}", where "{format}" is a file content type like "html", "js", "json" (idea from Ruby on Rails). 
For example, "templates/sessions/new.html.tmpl" => "sessions/new.html".

Then we can render the templates with a variety of functions: wutrender.HTML, wutrender.JS, wutrender.RenderFormat("html", ...) - all this functions return (*bytes.Buffer, error). Also, there are functions like wutrender.WriteHTML or wutrender.WriteJS which take http.ResponseWriter as the first argument and write to it.

It also uses application environment approach (by GO_ENV=production variable) - in dev mode it recompiles all templates and in production mode it uses .Clone() function of html/template package.

Support layout with special function {{ yield }} and partials with bindings: {{ partial "path" }} (empty) or {{ partial "path" . }} (all parent binding) or {{ partial "path" "user" .User "isEdit" true }} (key-value).

Feel free to contact me if you have any questions or suggestions and please, use github issues for... issues only, all discussions go here.

Cheers.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Gary Scarr | 2 Sep 17:54 2014
Picon

How to round a Duration?

To log a time difference I initially coded:


start := time.Now()
...
fmt.Printf("...took %s\n", time.Since(start))


but that resulted in an output of 19.7996387s which is impressive in its resolution but I really only wanted a display resolution to the second.

So I thought the natural thing would be to call: 

time.Since(start).Round(time.Second) 

but there is no Round method on Duration, only on a Time.  So I resorted to:


start := time.Now().Round(time.Second)
...
end := := time.Now().Round(time.Second)
fmt.Printf("...took %s\n", end.Sub(start))


which gives me the requisite display of "20s" but isn't as clean or precise as rounding the duration itself.

Am I missing an obvious way of doing this or should there be a Round method on Duration?

Thanks,
Gary  



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Brain Wu | 2 Sep 17:24 2014
Picon

[go-nuts] struct 's field and method name can not be same? why?

I have a article struct 
type Article struct{
Tags []Tag
}
func (a *Article) Tags() {
}
it's has a error that "type Article has both field and method named Tags" when i build it.
Why is  such a design?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
slene | 2 Sep 05:05 2014
Picon

Weird request.Cookie problem

Hi,

I can not get latest cookie after req.AddCookie. Bug or expected?


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Gmane