amjad.masad | 30 Jan 10:13 2015
Picon

channel-wrapping a blocking operation

Hello,

Say I have a blocking operation, and I want to `select` on it with some other channel. The obvious thing to do, I thought, was to wrap the blocking operation with a channel wrapper. i.e. a function that starts up a go routine that would call the blocking operation and then return a channel that will eventually be passed the result. 

Now this worked, however, it's leaking goroutines. If the select statement went with some other choice, then the block operation may potentially run for ever with no way of closing.
Here is an example: (http://repl.it/90o)

package main

import "fmt"
import "time"

// A sample blocking operation, will loop and print.
func blockingFn() string {
    for i := 0; i < 10; i++ {
        time.Sleep(200 * time.Millisecond);
        fmt.Println("blocking operation running")
    }
    return "blocking operation done"
}

// Wrap the blocking operation with a channel and a goroutine
func nonBlockingWrapper() chan string {
    ch := make(chan string)
    go func() {
        ch <-blockingFn()
    }()
    return ch
}

func main() {
    select {
        case <- time.After(100 * time.Millisecond):
            fmt.Println("timeout")
        case res := <- nonBlockingWrapper():
            fmt.Println(res)
    }
    // Timer to illustrate that the blocking operation
    // from the prev select statement still runs. (see output)
    time.Sleep(1 * time.Second)
}

What's the best way to handle this?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Chai Tadada | 30 Jan 14:01 2015
Picon

Goroutline performance degrade in 1.4.1 opposed to 1.3.3

Hi,

I have an application, that intensively uses goroutines and channels. The utility is: https://github.com/chillum/httpstress-go

I've noticed, that the 1.4.1 builds demonstrate slighly less performance compared to 1.3.3.

I've read, that in Go 1.4 we have some new goroutine behavior (different buffer size), probably that's the case.

What should I discover to fill the issue? I get a performance degradation on 1.4.1 compared to 1.3.3 and want to get it fixed.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Sonia Keys | 30 Jan 13:58 2015
Picon

Must New return a pointer?

Of course technically New is not a special identifier and there are no special restrictions on it, so I'm looking for opinion here.  As a convention it was established early that it's good to name a function that has a constructor-like role New or New<Something> and I'm aware that most of the time, a New function returns a pointer.  But it's often useful to have a constructor that initializes a value and returns it directly.  Is it bad style to name such a function New?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
matt | 30 Jan 11:19 2015
Picon

Fast closure dereferencing - help needed

// This is an example profile output for some code I'm writing. I've changed
// some of the struct names (B, C, P) as those details are superfluous to my
// question.

// What I'd like to know is, how can I reduce the cost of the of the
// v.Executor(v, c) statement (line 36)? v is a struct containing a function
// pointer Executor that I'm calling with args v (itself, can't find a nicer way to
// curry that argument), and c which is a stateful struct.

// Dereferencing the Executor function from the struct v, is costing me 10% of my entire
// runtime, this is the hottest loop in the code. I'm happy for the cost of the Executor
// to remain high as that is the real work, this is just scheduling those calls.

// So in summary:
// a) is it possible to reduce the cost of v.Executor in any way?
// b) does the cost of v.Executor (the derefencing) include the cost of stacking off the
//    closure (the values that the closure can access)?
// c) is there a nice way of not having v.Executor(v i.e. some better form of currying.


         .          .     33:// Run over the functions in list, executing each in turn.
      40ms       40ms     34:func (bb *B) Run(c *C, p *P) {
     800ms      800ms     35:   for _, v := range bb.list {
     2.21s     28.87s     36:           v.Executor(v, c)
     420ms      420ms     37:           c.Increment()
         .          .     38:   }
      20ms       20ms     39:}

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
vlyamtsev | 30 Jan 09:22 2015
Picon

constructor failure ?

p, li { white-space: pre-wrap; } I defined MultiWriter type, like the following
const
(     NULL Device = 0     FILE )
type MultiWriter struct {     dev Device     file io.Writer }
then constructor:
func NewWriter(_d Device, _path string) *MultiWriter {    
    w := &MultiWriter{         dev: NULL,         file: nil}    if _d == FILE {      
        f, err := os.OpenFile(_path, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)         if err == nil {            
            w.dev = FILE             w.file = f         } else {             fmt.Println("can't create file")         }     }     return w }


but constructor always return "NULL device":
p, li { white-space: pre-wrap; } var mw *mlog.MultiWriter mw = NewWriter(1, "mylog") fmt.Print(mw) always print &{0 <nil> <nil>}
Any idea what i do wrong?
Thks


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
maxpowers477 | 30 Jan 06:52 2015
Picon

I have a new tool in my box but now everything is starting to look like a nail?

Excuse the obliqueness of the title.

Just to get this out of the way up front, I am not interested in starting a flame war, merely seeking feedback on my options.

Due to a staffing cut at work, I have recently been tasked with some sysadmin duties.
Most this involves some very, very repetitive tasks that should have been handled via cron jobs a long time ago.

As it turns out though, the previous admin spent his days being a human cron daemon. 
Outside of what the stock OS put there, I see absolutely nothing in crontab, but the bash completion history on all of the boxes is both long and rather frightening. 

There are only 8 or 9 systems in play, so something huge like puppet is probably not appropriate.
What I would like to do is to script up the 10 most repetitive tasks and turn them into cronjobs.

I used to be an admin in a bygone era. Back then I would have used a combination of bash scripting and possibly some perl to get the job done.
My job description has changed over the years from 100% pure dev ops, to 90% programming, 10% ops, and sysadmin work has for the last 7 years or so been "get it set up and let it become someone elses problem."  In otherwords, it's been years since I've needed to write a bash script and just about that long since I've touched perl.  I've been looking at perl syntax and honestly, I'm not sure I can still grok it.

My question is this...
If you were in a similar situation, would you use go (the only applicable tool I have right now that's sharp enough to write mission critical tasks in), or would you take the time and brush back up on bash, perl etc.

I'm asking in the golang forum, only because I'm not actually sure of the applicability of go outside of writing servers.
These things need to execute pretty quickly and the tasks can be rather huge, (log rotate, split files, tar up directories, rsync backups etc)

I expect a lot of application time would be spent in exec.Run for things like taring directories and rsyncing, but I can see how tasks like file splits might better be handled via slurping and piping to goroutines.
Frankly, there is no reason outside of the fact that I don't want to have to keep context switching my mind every few hours as I script these up, that I can't just try and use an admixture of all the above and in the end, that probably will be what I end up doing.  But with the move to systemd I do wonder if compiled static executables aren't just the way things are going nowdays.

The other question I have is probably completely subjective. 
But what about longterm maintainability once it's written?  Assuming I tend to comment and document as I go along and that most people who are pros in a given language can usually maintain my work without too many phonecalls and forehead slaps.

I don't expect to be doing this task in 1 year let alone 5, what are the odds that if we hired a sysadmin with at least minimal C experience, that they would be able to pickup golang quickly enough to be able to maintain the scripts?  Perl seems to be a key component of any sysadmin's toolbox and as far as shell scripting goes, well they wouldn't be a sysadmin without that, would they? 

True C competency is really rare in admins (if they had that skill, then my experience is that they would usually be employed as a developer), but knowing enough to maintain an app of a couple of hundred lines is common enough that if I wrote this stuff in C, I know I could find someone who could pick it up and run with it. 

Golang on the other hand looks to me like "C with over 3 decades hindsight", but it still has it's quirks. 
Did anyone here come from the C world and just decide that Go would be their new goto tool for systems level programming?  What were the strengths and weaknesses of go for systems programming?  Or is it really, more or less a language intended for writing servers in.

Thank you for your thoughts.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Blake Caldwell | 30 Jan 06:35 2015

What could cause all of these stuck goroutines in net/http?

My long-running service dials out to several machines. I'm making https POST requests as well as using net.Dial to then open up SSH connections. After a while, I'm seeing hundreds of stuck goroutines with this exact stack trace. I'm guessing these are from the https POSTs, since it's starting in http/transport. I'm making sure to call .Body.Close() on my http responses. 

Any ideas?

goroutine 24297553 [semacquire, 296 minutes]: sync.(*WaitGroup).Wait(0xc20819e840) /usr/src/go/src/sync/waitgroup.go:132 +0x169 net.(*singleflight).Do(0x9fba90, 0xc208999a00, 0x15, 0xc208e8a548, 0x0, 0x0, 0x0, 0x0, 0x410d38) /usr/src/go/src/net/singleflight.go:53 +0x143 net.lookupIPMerge(0xc208999a00, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/src/go/src/net/lookup.go:42 +0xae net.lookupIPDeadline(0xc208999a00, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/src/go/src/net/lookup.go:64 +0x12d net.resolveInternetAddr(0x8024d0, 0x3, 0xc208999a00, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /usr/src/go/src/net/ipsock.go:285 +0x49b net.resolveAddr(0x7f5890, 0x4, 0x8024d0, 0x3, 0xc208999a00, 0x19, 0x0, 0x0, 0x0, 0x0, ...) /usr/src/go/src/net/dial.go:110 +0x378 net.(*Dialer).Dial(0xc208d55e80, 0x8024d0, 0x3, 0xc208999a00, 0x19, 0x0, 0x0, 0x0, 0x0) /usr/src/go/src/net/dial.go:158 +0xf6 net.Dial(0x8024d0, 0x3, 0xc208999a00, 0x19, 0x0, 0x0, 0x0, 0x0) /usr/src/go/src/net/dial.go:143 +0x8a net/http.(*Transport).dial(0xc20806c090, 0x8024d0, 0x3, 0xc208999a00, 0x19, 0x0, 0x0, 0x0, 0x0) /usr/src/go/src/net/http/transport.go:481 +0xcf net/http.(*Transport).dialConn(0xc20806c090, 0x0, 0xc2087edb90, 0x5, 0xc208999a00, 0x19, 0x6d95e0, 0x0, 0x0) /usr/src/go/src/net/http/transport.go:564 +0x1678 net/http.func·019() /usr/src/go/src/net/http/transport.go:520 +0x42 created by net/http.(*Transport).getConn /usr/src/go/src/net/http/transport.go:522 +0x335


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Sergei G | 30 Jan 03:09 2015
Picon

time.Parse for time.String

I store date time using a simple call:

  time_value.String()

It works well and stores data like this:

  2011-09-04 15:54:42 -0700 PDT

Now, I need to parse that string back to time.Time struct:

  time.Parse(layout, string)

What layout should I pass?

Why do I even need to pass layout, if I used default layout to generate a string in the 1st place?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Blake Caldwell | 30 Jan 00:02 2015

PPROF weirdness with goroutine status

TLDR: PPROF shows the count of goroutines to be at 5,000, but the goroutine stack trace page only shows 20 or so - WHY???

Details:

I'm using the pprof web endpoints. It's been insanely valuable to see the stack traces for all my goroutines. I recently reported that it looks like my goroutines aren't being in some weird case here:

https://groups.google.com/forum/#!searchin/golang-nuts/blake$20caldwell/golang-nuts/uhqGZYM2LJw/8KzuW5O7IJoJ

Well, I have a lot more info now. The reason I thought the deferred statements weren't being called is because they didn't show up in the goroutine stack trace page at /debug/pprof/goroutine?debug=1. There, I only see about 20 goroutines, which is what I'd expect. However... I just noticed that they *are* showing up on the /debug/pprof/ page in the goroutine count:

/debug/pprof/

profiles:
0 block
5544 goroutine
2716 heap
12 threadcreate

Disregard the "block" count - I don't have that enabled.

So, in what state is a goroutine where it's being counted in the total count, but not showing up in the stack trace page?

Thanks!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Traun Leyden | 29 Jan 23:02 2015
Picon

Go 1.2 excessive GC seen sporadically (262 seconds)


I'm looking into a sporadic GC issue happening in our REST api server, which is compiled with Go 1.2.  Under load testing, I'm sporadically seeing 262 second GC pauses.  (normally under the same load, GC pauses are all under 0.5 seconds)

Here are the expvar memstats data during a sporadic long GC:

    https://gist.github.com/tleyden/5d50cc5d7e58c2a0ecd6#file-gistfile1-txt-L64

Also, here is the output from top while the long GC'ing was happening:

    https://gist.github.com/tleyden/6849c242a1aecb885820#file-gistfile1-txt-L14

My current plan of attack is to just do heap profiling and try to figure out why it's using so much memory.  Anything else I should be doing to get to the root of the problem?  

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
ThePiachu | 29 Jan 22:12 2015
Picon

How to make packages that work for Google Go and Google App Engine Go?

I am wondering if it is possible to develop packages for Go that work for both the vanilla Go and the GAE Go? Most notably, how to handle writing logs and handling HTTP calls?

In Google App Engine, you need to pass the appengine.Context to basically every function to allow it to write log messages and so on. The vanilla log package doesn't work when an application is deployed to the App Engine. However, relying on appengine.Context makes those functions incompatible with vanilla Go.

I am wondering whether it's possible to do something like:

- If "appengine" package is present, import it
- If "appengine" package is not present, import a dummy package that translates the appengine-specific functionality to vanilla Go

Does anyone know if something like this is possible, or will I have to keep two copies of all of my packages - one for vanilla Go, one for GAE Go?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Gmane