Piyush Verma | 31 Oct 03:42 2014
Picon

Re: Golang S3 library

Thank you for the response.

So, I have written the program. Here's the code (snippet from the larger code)

This is the S3 walker

package main

import (
"io"
"log"

"launchpad.net/goamz/aws"
"launchpad.net/goamz/s3"
)

const LIMIT = 1000
const STARTPOS = 0

func getRegion(cfg *config) aws.Region {
return aws.Regions[cfg.S3.Region]
}

type s3crawler struct {
List       *s3.ListResp
bucket     *s3.Bucket
currentPos int
}

type s3iterator struct {
prefix  string
marker  string
cfg     *config
crawler *s3crawler
}

type LogFile struct {
Path   string
Bucket *s3.Bucket
}

func (self *LogFile) Get() (data []byte, err error) {
return self.Bucket.Get(self.Path)
}

func (self *LogFile) GetReader() (io.ReadCloser, error) {
return self.Bucket.GetReader(self.Path)
}

func (self *s3iterator) initCrawler() {
bucket := getBucket(self.cfg)
res, err := bucket.List(
self.prefix, "", self.marker, LIMIT,
)

if err != nil {
log.Fatal(err)
}

if !res.IsTruncated {
res.MaxKeys = len(res.Contents)
}

self.crawler = &s3crawler{res, bucket, 0}
}

func (self *s3iterator) Next() *LogFile {
if self.crawler.currentPos >= self.crawler.List.MaxKeys &&
self.crawler.List.MaxKeys == LIMIT {
self.initCrawler()
}

key := self.crawler.List.Contents[self.crawler.currentPos]

defer func(key s3.Key) {
self.crawler.currentPos++
self.marker = key.Key
}(key)

return &LogFile{key.Key, self.crawler.bucket}
}

func (self *s3iterator) End() bool {
if self.crawler == nil {
self.initCrawler()
}

if !self.crawler.List.IsTruncated &&
self.crawler.currentPos >= self.crawler.List.MaxKeys {
return true
} else if self.crawler.List.MaxKeys == 0 {
return true
}

return false
}

func getBucket(cfg *config) *s3.Bucket {
auth := aws.Auth{
AccessKey: cfg.S3.AccessKey,
SecretKey: cfg.S3.SecretKey,
}

region := getRegion(cfg)

connection := s3.New(auth, region)
bucket := connection.Bucket(cfg.S3.Bucket)
return bucket
}

func NewIterator(prefix string, marker string, cfg *config) *s3iterator {
return &s3iterator{
prefix, marker, cfg, nil,
}
}


 and here's the consumer:
for it := NewIterator(Settings.S3.Prefix, marker, &Settings); !it.End(); {
file := it.Next()

db.Update(Settings.S3.Prefix, file.Path)

wg.Add(1)

go func(wg *sync.WaitGroup, file *LogFile) {
defer wg.Done()
processFile(file) //Do something with it.
}(&wg, file)
}


So I am not really specifying the no. of files I am downloading simultaneously. The maxroutines would be handling that I assume.

How do I do this in groups ? You mean once a batch of 1000 or n goroutines have executed, take a breather wait for processFiles to finish and then restart ?

On Friday, October 31, 2014 8:03:05 AM UTC+5:30, Shawn Milochik wrote:
On Thu, Oct 30, 2014 at 10:27 PM, Piyush Verma <piyush...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
I have tried using github.com/crowdmob/goamz and launchpad.net/goamz/goamz and both of them seem to be misbehaving (net/http: TLS handshake timeout) when I try to download a large no. of files in gorutines.

Things are sane when I make each request sequentially.

How about writing the program to start with one, then add them one at a time until they start failing, then back off? I'm certain you can't download an arbitrarily large number of files simultaneously, and even if you could it wouldn't be faster than downloading them in groups because somewhere along the line there is going to be a bandwidth issue.

How many are you trying to download simultaneously, and why are you trying to do them all at once?

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Piyush Verma | 31 Oct 03:27 2014
Picon

Golang S3 library

Hello,

Is there a good S3 library for golang out there ?

I have tried using github.com/crowdmob/goamz and launchpad.net/goamz/goamz and both of them seem to be misbehaving (net/http: TLS handshake timeout) when I try to download a large no. of files in gorutines.

Things are sane when I make each request sequentially.

I have raised this issue https://github.com/goamz/goamz/issues/41 but looks like it might take a while for it to be solved.

Basic goal is to read data from a large no. of S3 files at once.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Gyu-Ho Lee | 30 Oct 21:10 2014
Picon

Is there any change regarding Vim support in Go 1.4 beta?

It's trivial but I just wonder there would be any change in 1.4beta1 that I need to be aware of for Vim?
I had to install back 1.3.3 because 1.4beta1 was troubling with gofmt...

I get:
Error detected while processing BufWrite Auto commands for "<buffer=2>

from my vimrc:
" Go
set rtp+=$GOPATH/src/github.com/golang/lint/misc/vim
set runtimepath+=/usr/local/go/misc/vim

let g:gofmt_command = 'goimports'
autocmd FileType go autocmd BufWritePre <buffer> Fmt 
autocmd BufWritePost,FileWritePost *.go execute 'Lint' | cwindow
" autocmd BufWritePre *.go Fmt

Gopath, and all packages were installed correctly.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Nate Finch | 30 Oct 19:16 2014
Picon

tip.golang.org giving internal server error?

I presume someone has already been alerted, but just in case not....

The internal server error seems to have gone away since I was typing this, but the "Try Go" section in the middle right currently flashes and then disappears, so I think things are not entirely fixed.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
amirtaghavi2020 | 30 Oct 18:16 2014
Picon

how i write shell in go

in php we write shell for hacking site they are Malicious code 
how we write shell in go?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
ashleydw | 30 Oct 17:02 2014

Convert a date/time into UTC

Hi

I am looking for a way to convert a date/time into UTC, is there a package for such as thing?

I have found https://github.com/bcampbell/fuzzytime which allows me to parse times, but no conversion of `Time`.

I'm new to go, so may be missing something already built in?

Thanks

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Lukáš Němec | 30 Oct 15:48 2014
Picon

May I be so bold as to ask for code-review? :)

Hello gophers!

I wanted to try a little bit of Go programming, so I wrote this little program.
It works fine, and does what I want it to do. Hoewer I'm not so sure that it is correctly idiomatic for Go as it is my first "larger" go program.

I'd very much like to hear your opinion on the code and what to improve.

Thank you!
Lukas

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
jagadeesh mn | 30 Oct 12:25 2014
Picon

How to get the Current URL of a web page

Hi All,

I have been trying to get the Current URL of web page.Please help me with this.
Any sample code would be a great help.


Thanking You

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
Haddock | 30 Oct 10:11 2014
Picon

Implicit conversion from char to int

Hello,

I just hapened to realize that this compiles and runs in Go:

myChar := 'a'
println(myChar + 2) // compiles and runs

I was surprised by this and then tried it out in Java. Same thing:

myChar := 'a'
println(myChar + 2) // compiles and runs
 
In my little conservative world this used to be wrong. I'm a bit confused now. Does anybody understand the rationale behind this?

Thanks, H.




--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
amirtaghavi2020 | 30 Oct 09:54 2014
Picon

how i set time zone for asia?

hi i need set time zoon for asia/tehran 
how do i do?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.
jagadeesh mn | 30 Oct 06:28 2014
Picon

Cannot find the template file in the path:logincontroller/login.tpl

Hi all,


I am trying a web application.Build is successful but im getting a run-time error saying

 "Cannot find the template file in the path:logincontroller/login.tpl"
Request Method: POST
Request URL: /login
RemoteAddr: [::1]:56744
Stack c:/go/src/pkg/runtime/panic.c:248 D:/Project/Go_programs/src/github.com/astaxie/beego/controller.go:253 D:/Project/Go_programs/src/github.com/astaxie/beego/controller.go:179 D:/Project/Go_programs/src/github.com/astaxie/beego/router.go:743 c:/go/src/pkg/net/http/server.go:1673 c:/go/src/pkg/net/http/server.go:1174 c:/go/src/pkg/runtime/proc.c:1445


In command prompt im getting this error 


2014/10/30 10:50:48 [C] the request url is  /login
2014/10/30 10:50:48 [C] Handler crashed with error can't find templatefile in th
e path:logincontroller/login.tpl
2014/10/30 10:50:48 [C] c:/go/src/pkg/runtime/panic.c:248
2014/10/30 10:50:48 [C] D:/Project/Go_programs/src/github.com/astaxie/beego/cont
roller.go:253
2014/10/30 10:50:48 [C] D:/Project/Go_programs/src/github.com/astaxie/beego/cont
roller.go:179
2014/10/30 10:50:48 [C] D:/Project/Go_programs/src/github.com/astaxie/beego/rout
er.go:743
2014/10/30 10:50:48 [C] c:/go/src/pkg/net/http/server.go:1673
2014/10/30 10:50:48 [C] c:/go/src/pkg/net/http/server.go:1174
2014/10/30 10:50:48 [C] c:/go/src/pkg/runtime/proc.c:1445

In the controller, i have used the following code:

func (this *LoginController) LoginView() { 
  this.TplNames = "login.html" 
 } 

Please help me out with this.

Thanking You

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Login


Gmane