Tamás Gulácsi | 8 Feb 09:59 2016
Picon
Gravatar

go1.6: _cgoCheckPointer0 adds 22x overhead!

Hi,

TL;DR: _cgoCheckPointer0 adds 22x overhead - is it OK, is it unavoidable, or is it a bug?

I'm trying go make gopkg.in/rana/ora.v3 go1.6 cgocheck=1 -compatible. Now it compiles, but the tests does not finish in 10m.
I have a test case which does not finish in 10m ! 
It is a simple "INSERT 1000000 int64s into a table".
As a preparation, it calls a C function with &ints[i] for each i in the ints (1000000 time).


func (bnd *bndInt64Slice) bind(values []int64, nullInds []C.sb2, position int, stmt *Stmt) error {                                                                                                                                            
    bnd.stmt = stmt                                                                                                                                                                                                                           
    if nullInds == nil {                                                                                                                                                                                                                      
        nullInds = make([]C.sb2, len(values))                                                                                                                                                                                                 
    }                                                                                                                                                                                                                                         
    alenp := make([]C.ACTUAL_LENGTH_TYPE, len(values))                                                                                                                                                                                        
    rcodep := make([]C.ub2, len(values))                                                                                                                                                                                                      
    bnd.ociNumbers = make([]C.OCINumber, len(values))                                                                                                                                                                                         
    alen := C.ACTUAL_LENGTH_TYPE(C.sizeof_OCINumber)                                                                                                                                                                                          
    for n := range values {                                                                                                                                                                                                                   
        alenp[n] = alen                                                                                                                                                                                                                       
        r := C.OCINumberFromInt(                                                                                                                                                                                                              
            bnd.stmt.ses.srv.env.ocierr, //OCIError            *err,                                                                                                                                                                          
            unsafe.Pointer(&values[n]),  //const void          *inum,                                                                                                                                                                         
            8,                   //uword               inum_length,                                                                                                                                                                           
            C.OCI_NUMBER_SIGNED, //uword               inum_s_flag,                                                                                                                                                                           
            &bnd.ociNumbers[n])  //OCINumber           *number );                                                                                                                                                                             
        if r == C.OCI_ERROR {                                                                                                                                                                                                                 
            return bnd.stmt.ses.srv.env.ociError()                                                                                                                                                                                            
        }                                                                                                                                                                                                                                     
    }                                                     

    ...
}

Everytime it is interrupted by test timer, it has a 

:tgulacsi <at> tgulacsi-Aspire-V3-371: ~/src/gopkg.in/rana/ora.v3
$ NO_CGOCHECK_CHECK=1 go test -run=TestSession_PrepAndExe_Insert -timeout=1m
panic: test timed out after 1m0s

...

goroutine 19 [runnable]:
gopkg.in/rana/ora%2ev3._cgoCheckPointer0(0x6d4920, 0xc8200eeec0, 0xc8225bc480, 0x1, 0x1, 0x6be4a0)
        ??:0 +0x4d
gopkg.in/rana/ora%2ev3.(*bndInt64Slice).bind(0xc8200a8030, 0xc8200d4000, 0xf4240, 0xf4240, 0xc820890000, 0xf4240, 0xf4240, 0x1, 0xc8200aa000, 0x0, ...)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/bndInt64Slice.go:44 +0x3b5
gopkg.in/rana/ora%2ev3.(*Stmt).bind(0xc8200aa000, 0xc820057e78, 0x1, 0x1, 0x1, 0x0, 0x0)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/stmt.go:593 +0x8b76
gopkg.in/rana/ora%2ev3.(*Stmt).exe(0xc8200aa000, 0xc820057e78, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/stmt.go:179 +0x457
gopkg.in/rana/ora%2ev3.(*Stmt).Exe(0xc8200aa000, 0xc820057e78, 0x1, 0x1, 0x0, 0x0, 0x0)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/stmt.go:149 +0x4d
gopkg.in/rana/ora%2ev3.(*Ses).PrepAndExe(0xc8200a6000, 0xc8200a85a0, 0x22, 0xc820057e78, 0x1, 0x1, 0x0, 0x0, 0x0)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/ses.go:216 +0x265
gopkg.in/rana/ora%2ev3_test.TestSession_PrepAndExe_Insert(0xc8200d2000)
        /home/tgulacsi/src/gopkg.in/rana/ora.v3/z_session_test.go:190 +0x4b8
testing.tRunner(0xc8200d2000, 0xb52c88)
        /usr/local/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
        /usr/local/go/src/testing/testing.go:582 +0x892
exit status 2
FAIL    gopkg.in/rana/ora.v3    60.373s


With GODEBUG=cgocheck=0

:tgulacsi <at> tgulacsi-Aspire-V3-371: ~/src/gopkg.in/rana/ora.v3
$ GODEBUG=cgocheck=0 go test -run=- -bench=PrepAndExe                                                                  
Read environment variable GO_ORA_DRV_TEST_DB = '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=49161)))(CONNECT_DATA=(SERVICE_NAME=xe)))'
Read environment variable GO_ORA_DRV_TEST_USERNAME = 'test'
Read environment variable GO_ORA_DRV_TEST_PASSWORD = 'test'
Read session time zone from database...
Dropping previous tables...
Tables dropped.
PASS
BenchmarkSession_PrepAndExe_Insert_WithCGOCheck-4              0                 0 ns/op
BenchmarkSession_PrepAndExe_Insert_WithoutCGOCheck-4         100          20378562 ns/op


Benchmark with cgoCheck:

:tgulacsi <at> tgulacsi-Aspire-V3-371: ~/src/gopkg.in/rana/ora.v3
$ go test -run=- -bench=PrepAndExe
Read environment variable GO_ORA_DRV_TEST_DB = '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=49161)))(CONNECT_DATA=(SERVICE_NAME=xe)))'
Read environment variable GO_ORA_DRV_TEST_USERNAME = 'test'
Read environment variable GO_ORA_DRV_TEST_PASSWORD = 'test'
Read session time zone from database...
Dropping previous tables...
Tables dropped.
PASS
BenchmarkSession_PrepAndExe_Insert_WithCGOCheck-4              3         456911840 ns/op
BenchmarkSession_PrepAndExe_Insert_WithoutCGOCheck-4           0                 0 ns/op
ok      gopkg.in/rana/ora.v3    10.239s


The benckmark code:

func BenchmarkSession_PrepAndExe_Insert_WithCGOCheck(b *testing.B) {
        if cgocheck() == 0 {
                b.SkipNow()
        }
        benchmarkSession_PrepAndExe_Insert(b)
}
func BenchmarkSession_PrepAndExe_Insert_WithoutCGOCheck(b *testing.B) {
        if cgocheck() != 0 {
                b.SkipNow()
        }
        benchmarkSession_PrepAndExe_Insert(b)
}

func benchmarkSession_PrepAndExe_Insert(b *testing.B) {
        tableName, err := createTable(1, numberP38S0, testSes)
        testErr(err, b)
        defer dropTable(tableName, testSes, b)

        values := make([]int64, 1000000)
        for n, _ := range values {
                values[n] = int64(n)
        }
        b.ResetTimer()
        const batchLen = 100
        for i := 0; i < b.N; i++ {
                rowsAffected, err := testSes.PrepAndExe(fmt.Sprintf("INSERT INTO %v (C1) VALUES (:C1)", tableName),
                        values[i*batchLen:(i+1)*batchLen])
                if err != nil {
                        b.Error(err)
                        break
                }
                if rowsAffected != batchLen {
                        b.Fatalf("expected(%v), actual(%v)", batchLen, rowsAffected)
                }
        }
}


All code is available at  https://github.com/rana/ora/tree/go1.6

Thanks in advance!

Tamás Gulácsi

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe <at> googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ahmed Waheed | 8 Feb 04:17 2016
Picon
Gravatar

how to make this faster?

Hello,

I've worked pretty hard on the Go version of xxhash, and I got it pretty much as fast you possibly can without rewriting it in assembly.

It's almost a direct translation of the C code, the default mode for it using unsafe, but even with the direct translation and it's purely math so it really shouldn't be ~40% slower than the C version.

- https://github.com/OneOfOne/xxhash/blob/master/native/xxhash64_native.go
- https://github.com/OneOfOne/xxhash/blob/master/native/xxhash_unsafe.go

Am I missing something?
Should I open an issue as a reminder for the compiler people?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Elias Naur | 7 Feb 21:56 2016
Picon

gomobile: Go->Java callbacks on the Android UI Thread and byte buffer GC BUG()s

Hello,

For my Android/iOS app using a shared Go code base for business logic and communication with the backend, I've run into two issues:

Android UI Thread
On Android, gobind generated callbacks from Go to Java are executed in a Java thread pool. This is fine for most cases except when dealing with the Android UI thread; in my case I'd like Java event handlers to invoke Go code which in turn invoke Java methods for manipulating the UI state. However, callbacks from Go to Java are run on worker threads.

Several problems arise from that behaviour: 

 - Surprise. Go=>ObjC and Java=>Go calls, as well as Cgo in general, take care to stay on the same native thread. 
 - Android has a mechanism to run code on the UI thread, but that will have to be added to every Java method called from Go. Moreover, callbacks scheduled this way has to run after the Go caller has returned, not before as would be expected from a function call.
 - The Go caller stack is lost from backtraces.

The Gomobile design document describes this behavior; I assume this is because it is non-trivial to call into Java from arbitrary (non-Java) threads. Nevertheless, I'd like to change Gobind so that Go=>Java calls that is already running on a Java thread avoid the thread pool and executes the Java call directly. If nobody objects, I'd like to submit CLs for that change. The change shouldn't affect existing gobind apps, because apps that can tolerate callbacks on arbitrary pool threads should also work fine when reusing threads.

Byte buffers and BUG()s
Grepping for BUG in the bind directory brings up two issues related to byte buffers:

java/seq_android.go: // BUG(hyangah): the function returning a go byte slice (so fn writes a pointer into 'out') is unsafe.
objc/seq_darwin.m:   // BUG(hyangah): it is possible that *ptr is already GC'd by Go runtime.

Latent GC-triggered bugs in a nontrivial app for public use worry me, so I'd like to know if those issues still apply with Go 1.6 and the new Cgo pointer rules? If so, I'd like to help remove them.

 - elias

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
aerojoe | 7 Feb 07:33 2016
Picon

Golang on Intel Edison

Hello, my edison runs yocto and I'm trying to install and use Go. I followed the instructions to download the go binaries. The installation seems to work OK but I get exec errors whenever I try to run go commands.

I installed the files located at: https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz

I assume I can use the binaries directly without rebuilding go from source, correct? Will a yocto distribution work with go out of the  box?

thanks.
Joe

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Zippoxer | 7 Feb 14:09 2016
Picon

Refactoring: move code to new package

I was careless about project organization before and now I'm paying for it.

It's an old project and I put everything in the main package.

The names are long and confusing and (supposed to be-) isolated modules are sharing their private types and values.

I'd like to move some pieces of the code to new packages. The moved code has references and I would have to rename to include the new package name (like <moved_type> to <new_package.moved_type>.)

I tried gorename, but as soon as I move code to a new package and old references become undeclared errors, gorename refuses to refactor because of those errors.

Is there any tool that allows this kind of renaming?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Tomi Häsä | 7 Feb 15:42 2016
Picon
Gravatar

Handle Ports 80, 8080 and 443?

How should I handle multiple ports to HTTPS? Should I redirect 8080 and 80 requests to 443 if someone types http: instead of https: or uses /8080 instead of /?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
bep | 7 Feb 13:56 2016
Picon
Gravatar

Pointer detected as a literal

There is probably a good reason for this, but to me the two constructs in the main method below should be comparable, but only the second compiles ...?



package main

import "fmt"

type M
struct{}

func
(m *M) s() {
    fmt
.Println("s")
}

func main
() {
   
//&M{}.s() // ==> cannot call pointer method on M literal
   
(&M{}).s() // == >OK!
}




See: https://play.golang.org/p/Vtbqn6sZUt

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Tong Sun | 7 Feb 01:54 2016
Picon

Serialization internal data to disk

Hi, 

Microsoft is known to store all kinds of data in binary forms to disk, while Unix like to store all in plain text. I'm wondering what is the idiomatic way in Go to store internal data to disk (so as to be able to read the back next time). As the starting point, the only thing I'm storing is an array of strings, but the data structure might get complicated if I need to store more info. 

I'm thinking the YAML is the ideal format, as its format is the most concise one I can think of, and I need to store thousands of records. However, I found that its Marshal and Unmarshal format is not the same (ref https://github.com/suntong/lang/blob/master/lang/Go/src/text/Yaml-Example.go) thus I'm not able to read the back next time.

I did some "homework" myself, but as I'm in China currently, and I can't visit sites like wordpress blog, or even http://play.golang.org/, thus I haven't found my answer yet. Please help. 

Thanks

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Slawomir Pryczek | 6 Feb 23:41 2016
Picon

Named RW-Mutex? Other structure?

Hi Guys,
what you think should be appropriate structure for asynchronous key/value store implementation, basically i want to be able to allow multiple reads/single writer per KEY...

Example below
 Operation; Key; Value
GET; counter;
SET; counter; 1
SET; counter; 10
GET; counter;
GET; counter;
SET; counter_xx; 11
GET; counter_u
SET: counter_u; ABC
GET; counter;

So how it should work...
Thread 1 will read counter
Thread 2 will write counter_xx
Thread 3 will read counter_u
---- SYNCHRONIZATION POINT
Thread 1 will write counter = 1
Thread 2 will write counter_u = ABC
---- SYNCHRONIZATION POINT
Thread 1 will write counter = 10
---- SYNCHRONIZATION POINT
Thread 1 will read counter
Thread 2 will read counter
Thread 3 will read counter

... basicall reading and writing of one key cannot occur at the same time...

I have 2 ideas... 

1. Implement named RW mutex, so it'll be some pseudo-code like this

if is_writing(operation) {
 lock_for_writing(key);
 ....
 unlock_w(key)
} else {
 lock_for_reading(key);
 ....
 unlock_r(key)
}

2. Second idea is to start eg. 20 go threads, with 20 channels, and send key to one of the threads, by MODULO of CRC of name... but then reads for hot keys would become single threaded.

Any more go-like concurrency pattern that's appropriate for this case. What you think?

Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Torsten Bronger | 6 Feb 19:31 2016
X-Face
Picon
Picon
Picon
Picon
Gravatar

error handling: Which errors to ignore?

Hallöchen!

Coming from Python, I'm used to the mantra "errors should never pass
silently".  However, this seems to be a bad idea in Go if you don't
want to end up with something like this:

    if _, err := fmt.Printf("Hello!"); err != nil {
        log.Fatalln(err)
    }

I've read much online about best practices in Go error handling, but
no source answered the question which errors to ignore.  Is it
correct that there are no publications on this topic and every
Gopher must find a rule set on their own?

It still feels wrong to ignore errors in the first place, though ...

Tschö,
Torsten.

-- 
Torsten Bronger    Jabber ID: torsten.bronger@...

--

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@...
For more options, visit https://groups.google.com/d/optout.

carl.mastrangelo | 6 Feb 17:41 2016
Picon

Does Go's HTTP/2 code work in plaintext mode?

I have an HTTP server that terminates TLS connections from the Internet, and have been running with recent builds of Go.  It currently accepts HTTP/2 connections which is pretty cool, but it itself is basically just delegates the rest of the connection to another locally running Go server.  I am using the net/http/httputil reverse proxy for this.  I would prefer to use a Unix Domain Socket for the backend server to accept connection from the TLS terminator, to keep the amount of latency down.  Currently it's just using a localhost socket.

My question is: Can the httputil.ReverseProxy work with a unix socket instead of a tcp/ip one, and can the connection be in plaintext if the incoming request was HTTP/2?  Reading the source, it seems that ReverseProxy take explicit measures to use HTTP/1.1.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Gmane