headers
Rufus Pollock | 1 May 18:22
Gravatar

Re: Distributed Storage: Suggestions?

2009/4/28 Lukasz Szybalski <szybalski <at> gmail.com>:
> a follow up:
> 1. kosmosfs  www.linux-magazine.com/w3/issue/90/048-051_kosmos.pdf
> 2. sector http://sector.sourceforge.net/doc.html

Thanks for the info. This one looks interesting but not sure what it
offers over e.g. Hadoop which seems more mature and widely used.

> 3. hadoop fs

Hadoop looks very promising but seems more "cluster" oriented rather
than distributed storage oriented. I've collected some more info on
Hadoop in <http://wiki.okfn.org/projects/Distributed_Storage/Research/>.

BTW: I've now started hacking with Julian to try out what seem to be
the 2 best options at the moment: Tahoe and Hadoop. We're keeping our
code and instructions in this public mercurial repo (under dfs):

<http://knowledgeforge.net/okfn/okfncc>
<http://knowledgeforge.net/okfn/okfncc?file/tip/dfs/>

The main focus is on Tahoe as this looks the most promising option
(plus it is python based!). We've already got a basic install script
and are working on laying a more human-friendly interface over the
basic tahoe DFS structure.

Anyone wanting to get involved in hacking or wanting to volunteer
storage node space please let me know.

Rufus
(Continue reading)

Permalink | Reply | 4 comments |
headers
Mr. Puneet Kishor | 1 May 19:05

www.globalmedialaw.com

of interest...
http://www.globalmedialaw.com/blog/



-- 
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org/
Science Commons Fellow, Geospatial Data http://sciencecommons.org
Nelson Institute, UW-Madison http://www.nelson.wisc.edu/
-----------------------------------------------------------------------
collaborate, communicate, compete
=======================================================================
 

_______________________________________________
okfn-discuss mailing list
okfn-discuss@...
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 0 comments |
headers
Lukasz Szybalski | 1 May 20:22
Picon

Re: Distributed Storage: Suggestions?

On Fri, May 1, 2009 at 11:22 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
> 2009/4/28 Lukasz Szybalski <szybalski <at> gmail.com>:
>> a follow up:
>> 1. kosmosfs  www.linux-magazine.com/w3/issue/90/048-051_kosmos.pdf
>> 2. sector http://sector.sourceforge.net/doc.html
>
> Thanks for the info. This one looks interesting but not sure what it
> offers over e.g. Hadoop which seems more mature and widely used.
>
>> 3. hadoop fs
>
> Hadoop looks very promising but seems more "cluster" oriented rather
> than distributed storage oriented. I've collected some more info on
> Hadoop in <http://wiki.okfn.org/projects/Distributed_Storage/Research/>.
>
> BTW: I've now started hacking with Julian to try out what seem to be
> the 2 best options at the moment: Tahoe and Hadoop. We're keeping our
> code and instructions in this public mercurial repo (under dfs):
>
> <http://knowledgeforge.net/okfn/okfncc>
> <http://knowledgeforge.net/okfn/okfncc?file/tip/dfs/>
>
> The main focus is on Tahoe as this looks the most promising option
> (plus it is python based!). We've already got a basic install script
> and are working on laying a more human-friendly interface over the
> basic tahoe DFS structure.

1. sector is for highly "read" intensive apps, optimized for extremely
fast read.
2. kfs(kosmos) seems to be similar to google file system, where you
have main server (metaserver), and run chunk servers (how much space
you have available) , data gets divided into 64mb and send to chunk
servers. You always have 3 copies of the chunk.  The filesystem (at
metaserver) seems like one big file server. (+python bindings)

Lucas

>
> Anyone wanting to get involved in hacking or wanting to volunteer
> storage node space please let me know.

>
> Rufus
>

--

-- 
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
DataHub - create a package that gets, parses, loads, visualizes data
http://lucasmanual.com/mywiki/DataHub

_______________________________________________
okfn-discuss mailing list
okfn-discuss <at> lists.okfn.org
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 3 comments |
headers
Mr. Puneet Kishor | 3 May 05:02

Official Google Blog: Adding search power to public data


http://googleblog.blogspot.com/2009/04/adding-search-power-to-public-data.html



-- 
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org/
Science Commons Fellow, Geospatial Data http://sciencecommons.org
Nelson Institute, UW-Madison http://www.nelson.wisc.edu/
-----------------------------------------------------------------------
collaborate, communicate, compete
=======================================================================
 

_______________________________________________
okfn-discuss mailing list
okfn-discuss@...
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 0 comments |
headers
Rufus Pollock | 6 May 13:42
Gravatar

Re: Distributed Storage: Suggestions?

2009/5/1 Lukasz Szybalski <szybalski <at> gmail.com>:
> On Fri, May 1, 2009 at 11:22 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
>> 2009/4/28 Lukasz Szybalski <szybalski <at> gmail.com>:
[...]

> 1. sector is for highly "read" intensive apps, optimized for extremely
> fast read.

OK.  I feel a feature-grid coming on :)  For us, I guess that is not
ultra-high priority.

> 2. kfs(kosmos) seems to be similar to google file system, where you
> have main server (metaserver), and run chunk servers (how much space
> you have available) , data gets divided into 64mb and send to chunk
> servers. You always have 3 copies of the chunk.  The filesystem (at
> metaserver) seems like one big file server. (+python bindings)

OK. Useful info.

At the moment Allmydata-Tahoe is looking pretty promising. We've
started a basic grid with info here:

<http://grid.okfn.org/>

Next step is to create a nice filesystem like web frontend on top of
the basic web-api they provide.

We also need to grow the grid so if anyone's got a few gigs of spare
server capacity it would be great if they could contribute a node
(using the debian/ubuntu packages installing tahoe turns out to be
pretty painless ...).

Regards,

Rufus

PS: all code and content done so far is in the our hg repo at
http://knowledgeforge.net/okfn/okfncc/

_______________________________________________
okfn-discuss mailing list
okfn-discuss <at> lists.okfn.org
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 2 comments |
headers
Lukasz Szybalski | 7 May 15:45
Picon

Re: Distributed Storage: Suggestions?

On Wed, May 6, 2009 at 6:42 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
> 2009/5/1 Lukasz Szybalski <szybalski <at> gmail.com>:
>> On Fri, May 1, 2009 at 11:22 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
>>> 2009/4/28 Lukasz Szybalski <szybalski <at> gmail.com>:
> [...]
>
>> 1. sector is for highly "read" intensive apps, optimized for extremely
>> fast read.
>
> OK.  I feel a feature-grid coming on :)  For us, I guess that is not
> ultra-high priority.
>
>> 2. kfs(kosmos) seems to be similar to google file system, where you
>> have main server (metaserver), and run chunk servers (how much space
>> you have available) , data gets divided into 64mb and send to chunk
>> servers. You always have 3 copies of the chunk.  The filesystem (at
>> metaserver) seems like one big file server. (+python bindings)
>
> OK. Useful info.

What is the structure of the tahoe fs? how is it different from kfs?

Thanks,
Lucas

>
> At the moment Allmydata-Tahoe is looking pretty promising. We've
> started a basic grid with info here:
>
> <http://grid.okfn.org/>
>
> Next step is to create a nice filesystem like web frontend on top of
> the basic web-api they provide.
>
> We also need to grow the grid so if anyone's got a few gigs of spare
> server capacity it would be great if they could contribute a node
> (using the debian/ubuntu packages installing tahoe turns out to be
> pretty painless ...).

>
> Regards,
>
> Rufus
>
> PS: all code and content done so far is in the our hg repo at
> http://knowledgeforge.net/okfn/okfncc/
>

--

-- 
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
DataHub - create a package that gets, parses, loads, visualizes data
http://lucasmanual.com/mywiki/DataHub

_______________________________________________
okfn-discuss mailing list
okfn-discuss <at> lists.okfn.org
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 1 comment |
headers
Rufus Pollock | 7 May 16:56
Gravatar

CKAN package party and general online meetup: Saturday 16th May 2009

Hi,

We're planning another CKAN package party + general OKFN online meetup
for Saturday 16th of May.

It's a general meetup but with a focus on CKAN packaging so feel free
to come and chat about anything open knowledge related -- everyone and
anyone is welcome.

 * When: Saturday 16th May 2009, 1430-1730 BST (1330-1630 GMT, 0930-1230 EDT)
 * Where: #okfn irc channel on irc.oftc.net
([[http://www.mibbit.com/chat/?server=irc.oftc.net&channel=#okfn|connect
via mibbit]])

More info at (and sign yourself if you want) at:

  <http://wiki.okfn.org/CkanPackageParty/2>

Regards,

Rufus
Permalink | Reply | 2 comments |
headers
Rufus Pollock | 7 May 17:13
Gravatar

Re: Distributed Storage: Suggestions?

2009/5/7 Lukasz Szybalski <szybalski <at> gmail.com>:
> On Wed, May 6, 2009 at 6:42 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
>> 2009/5/1 Lukasz Szybalski <szybalski <at> gmail.com>:
[...]
>>> 2. kfs(kosmos) seems to be similar to google file system, where you
>>> have main server (metaserver), and run chunk servers (how much space
>>> you have available) , data gets divided into 64mb and send to chunk
>>> servers. You always have 3 copies of the chunk.  The filesystem (at
>>> metaserver) seems like one big file server. (+python bindings)
>>
>> OK. Useful info.
>
> What is the structure of the tahoe fs? how is it different from kfs?

I haven't looked at kfs in detail but:

  * Seems to be "cluster-oriented" and GFS-like (it says it is built
on ideas from GFS and can be dropped in to replaces HadoopFS).
  * It is built in C++ with some high-level bindings for python.
  * v0.3

Detailed overview of Tahoe can be found in
<http://allmydata.org/source/tahoe/trunk/docs/architecture.txt>

  * Very much about distributed *file storage*.
  * Focused storing and nodes are expected to highly distributed (so
no cluster/cloud computing here).
  * Data is encrypted (so you can put private data on it)
  * Almost entirely in python
  * v1.4

I should reiterate that what we're looking for is a storage solution
not a cluster (cloud computing) solution. That's why, at the moment,
Tahoe's looking like a much better match.

Rufus

_______________________________________________
okfn-discuss mailing list
okfn-discuss <at> lists.okfn.org
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 0 comments |
headers
Peter Murray-Rust | 7 May 17:47
Picon
Picon
Favicon
Gravatar

Re: CKAN package party and general online meetup: Saturday 16th May 2009



On Thu, May 7, 2009 at 3:56 PM, Rufus Pollock <rufus.pollock-wKZDxAJnXxE@public.gmane.org> wrote:
Hi,

We're planning another CKAN package party + general OKFN online meetup
for Saturday 16th of May.

Rufus - should we try to do a Cambridge group IRL somewhere? and maybe others can form local nuclei.

P.

 


It's a general meetup but with a focus on CKAN packaging so feel free
to come and chat about anything open knowledge related -- everyone and
anyone is welcome.

 * When: Saturday 16th May 2009, 1430-1730 BST (1330-1630 GMT, 0930-1230 EDT)
 * Where: #okfn irc channel on irc.oftc.net
([[http://www.mibbit.com/chat/?server=irc.oftc.net&channel=#okfn|connect
via mibbit]])

More info at (and sign yourself if you want) at:

 <http://wiki.okfn.org/CkanPackageParty/2>

Regards,

Rufus

_______________________________________________
okfn-discuss mailing list
okfn-discuss-6A+mB+4cr9F9rwYpqGo9+w@public.gmane.org
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss



--
Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069
_______________________________________________
okfn-discuss mailing list
okfn-discuss@...
http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
Permalink | Reply | 1 comment |
headers
Jo Walsh | 7 May 22:52
Favicon

Re: A potential econtent funding application

> This looks great! It would be interesting to see how this could fit
> with JISC application Rufus is working on with Peter Murray-Rust,
> through Cambridge University.

Right I saw Rufus' note about CKAN+ and guess there to be a lot
of overlap in potential scope. It sounds like a goer and I hope it is
a success.

I would just like to help out with something while i have a bit of time.
Oh i could hack on CKAN anyway, right? What is needed with that?

> We usually post funding related stuff on okfn-coord. Several people
> from JISC are on the okfn-discuss list. Perhaps we should rethink this
> and could start post funding stuff on okfn-help or okfn-discuss? What
> do others think?

I think it would be better to discuss fundraising in public unless there
is some matter of real confidence involved...

cheers,

jo
--
Permalink | Reply | 1 comment |

Gmane