headers
Isaac Force | 1 Mar 2011 01:33
Gravatar

Re: How many nodes can couchdb scale to?

>> Hi Niall, I think the key part is that with this topology your central
>> servers are going to need to support a sustained throughput of 20,000
>> reads/second in order to distribute the updates to all 2,000 servers.
>>  Granted, each read is repeated 2,000 times, so you'll mostly be reading
>> from page cache, but a cached read from CouchDB is not nearly as cheap as
>> reading from e.g. Varnish.
>
> Thanks Adam,
>
> Thats a good point. I suppose we could scale this by adding more nodes in
> the data centre.

An alternative would be to create peer-to-peer replication rings from
your edge nodes with a limited set of replication 'uplinks' to the
data center: draw nodes in an even ring, connect each node to its
adjacent peer to make a circle, connect each node to the node opposite
it in the ring, connect a set of nodes relative to the size of the
ring to the data center.

If a set of nodes die, each node should still have an intra-ring path
to each remaining node, and as long as there's still one uplink
remaining replication from the data center will continue. You could
even connect the rings to one another and have the data center merely
be another node on one of the rings.

This way configuration complexity and burden at the edge is traded for
egress replication burden from the data center.

-Isaac
(Continue reading)

Permalink | Reply | 1 comment |
headers
Isaac Force | 1 Mar 2011 01:59
Gravatar

Re: CouchDB to/from SQLite Replication?

On Mon, Feb 28, 2011 at 11:37 AM, Zdravko Gligic <zgligic@...> wrote:
> What existing features of CouchDB would be helpful in this case?
> Is there anything like "point-in-time" markers of some sort?
> What else would have I asked, had I known to ;?)

I would start with CouchDB's _changes API:

http://guide.couchdb.org/editions/1/en/notifications.html

It provides a sequence number which you can store in your SQLite
database for Couch => SQLite replication.

Being able to change data in a SQLite DB, have those changes replicate
to Couch, and not have the _changes replication back to SQLite not
conflict will be a different matter. If it were me I would try to keep
SQLite a read-only slave (Minus the seq ID storage) and have all
inserts/changes happen in Couch.

-Isaac
Permalink | Reply | 0 comments |
headers
Wayne Conrad | 1 Mar 2011 03:41
Favicon

Re: Replication: stalled?

On 02/21/2011 11:18 AM, Wayne Conrad wrote:
> I'm seeing replication behavior that I don't understand.  I wonder if 
> it's stalled.

It's not stalled.  It's going very, very slowly.  I think I understand why.

Some of my documents have tens of thousands of attachments.  When I 
first started storing the fat documents in couchdb, it took half an hour 
or more to add them.  To make it faster, and to prevent timeouts, I 
store the attachments inline, but in chunks of 100 attachments at a 
time.  Doing that, even my largest documents take only a minute or so to 
store.

I can store a document with 32,768 attachments of 4k each in 55 seconds 
(2.4k/sec).  But to replicate that document (using "pull" replication) 
takes 19.5 minutes.  That's 115k per second.  Storing, then, is 20 times 
faster than replicating.  When I look at the log on the source database, 
I see that the destination database is retrieving one attachment at a 
time, and (I presume) experiencing the same speed problem that caused me 
to write my "store bunches of attachments at a time" optimization.  Now 
it seems that, in order for replication to have any chance of keeping up 
with the rate at which I can store data, I'm going to need the same sort 
of optimization during replication.

I'm a couch toddler, and when it comes to Erlang, I'm not even on solid 
food yet.  What are the odds of me writing my own replication engine in, 
say, Ruby, one that can do the special optimizations I need?  How 
difficult a project is it?
(Continue reading)

Permalink | Reply | 6 comments |
headers
Zdravko Gligic | 1 Mar 2011 06:34
Picon

Re: WinCouch

Nikolai,

Apparently starting with AIR 2.0, Adobe enabled command line
integration.  However, it requires that AIR apps be installed with a
custom installer. The bad part is that apparently the updating of apps
also needs to be custom handled.

Here is a link with more info.
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/desktop/NativeProcess.html
Now, all we need is an Adobe AIR guru ;)

Thanks again,
Zdravko
Permalink | Reply | 0 comments |
headers
kowsik | 1 Mar 2011 07:42
Picon
Gravatar

EC2 ELB and continuous _changes feed

Anyone using ELB on EC2 along with _changes feed? Curious how this
works, since the persistent connection can only be made to a single
back-end instance and kinda screws up the load balancing.

Any experience and/or gotchas to share?

K.
---
http://twitter.com/pcapr
http://www.pcapr.net
http://labs.mudynamics.com
Permalink | Reply | 0 comments |
headers
Jeroen Janssen | 1 Mar 2011 09:15
Picon
Gravatar

Re: 1.0.2 RPM on epel-couchdb.repo

Hi Peter,

Did you happen to have time for updating the epel-couchdb repository
to 1.0.2 yet?

Best regards,

Jeroen Janssen

On Tue, Feb 8, 2011 at 2:57 PM, Peter Lemenkov <lemenkov@...> wrote:
> 2011/2/8 afshin afzali <a.afzali2003@...>:
>> Hi,
>>
>> Thanks to Mr. Peter Lemenkov for providing 1.0.1 RPM on RHEL-5/CentOS-5.
>> I was wondering if I could to find out that the epel-couchdb repository is
>> updated with 1.0.2 version.
>
> I'll update it in a couple of days (more likely during this weekend).
> Stay tuned!
>
> --
> With best regards, Peter Lemenkov.
>
Permalink | Reply | 0 comments |
headers
Pasi Eronen | 1 Mar 2011 09:52
Picon
Picon
Favicon

View generation stuck in "Finishing" state for long time?

Hi,

One of my databases has a design doc that seems to get stuck in
"Finishing" state for long time:

View Group Indexer: foo_local _design/bar: Finishing. (<0.3413.1>)

This is after indexing a small number (some hundreds perhaps) updates.
Usually it's stuck in this state for couple of minutes, but the longest
I've seen was over 40 minutes. (I have a cron job that logs the contents
of _active_tasks once a minute.)

If I generate the whole view from scratch, it happens relatively quickly.
The database has ~900K documents, and the view code is not especially
complex (I have a second database on the same server with 20M+ documents
and a very similar view code, and it never gets stuck in "Finishing").

What exactly is CouchDB doing in this "Finishing" state? Any hints on
how to debug this?

Best regards,
Pasi
Permalink | Reply | 0 comments |
headers
Filipe David Manana | 1 Mar 2011 11:15
Picon
Favicon
Gravatar

Re: Replication: stalled?

On Tue, Mar 1, 2011 at 2:41 AM, Wayne Conrad <wayne@...> wrote:
>
> I'm a couch toddler, and when it comes to Erlang, I'm not even on solid food
> yet.  What are the odds of me writing my own replication engine in, say,
> Ruby, one that can do the special optimizations I need?  How difficult a
> project is it?
>
>

You can also try the new replicator that landed recently into trunk
(from where 1.2 will be cut from), it adds more parallelism so you'll
likely see a better performance:

http://s.apache.org/KsY

https://github.com/apache/couchdb/commit/34eb4175f8547546cd76fbeb006421020cbf0d71

--

-- 
Filipe David Manana,
fdmanana@..., fdmanana@...

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
Permalink | Reply | 5 comments |
headers
Zdravko Gligic | 1 Mar 2011 13:46
Picon

Re: CouchDB to/from SQLite Replication?

Thanks Isaac,

Is there a way of getting full documents, instead of a stream of doc
id's - which then need to be read in separate call(s) ?

Thanks again,
Zdravko
Permalink | Reply | 1 comment |
headers
Paul Drummond | 1 Mar 2011 14:14
Picon
Gravatar

Couchdb hangs on Android 2.3 (Gingerbread/Nexus S)

Hi all,

I have an old version of couchdb that works fine on a Nexus One but
when I downloaded and installed couchdb for a Nexus S it installs fine
but hangs during start-up continually displaying "CouchDB is
loading...".  The relevant output of Logcat is below:

V/CouchDB ( 4054): =INFO REPORT==== 1-Mar-2011::13:06:07 ===
V/CouchDB ( 4054):     application: couch
V/CouchDB ( 4054):     exited: {bad_return,{{couch_app,start,
V/CouchDB ( 4054):                                     [normal,
V/CouchDB ( 4054):
["/usr/local/etc/couchdb/default.ini",
V/CouchDB ( 4054):
"/usr/local/etc/couchdb/local.ini"]]},
V/CouchDB ( 4054):                          {'EXIT',"Cannot load
library: link_image[1962]:  4062 could not load needed library
'libicudata.so' for 'couch_icu_driver.so' (load_library[1104]: Library
'libicudata.so' not found)"}}}

Can anyone help me with this please?

Thanks,
Paul Drummond
Permalink | Reply | 2 comments |

Gmane