View access during indexing

Hello,

One of my databases in CouchDB has close to 1,000,000 documents with a
size of about 200GB.
If I change one of the views, the indexing takes about 2 hours.

This time would be not a problem, but I am wondering why I cannot
access the other views, that were previously generated during the
indexing time.
Currently none of the views is available, taking my web application
down during this time.

Is there a parameter that can be adjusted to allow view access during indexing?

Thanks,
Matthias

Re: How to get the doc

On May 31, 2012, at 3:41 PM, Stephan Bardubitzki wrote:

> It sounds to me, when speed is the main concern, emitting the doc should 
> be more appropriate.

You should emit the values you need. If you need the whole doc, and don't mind the extra disk space, then emit
the whole doc. Otherwise, if you don't need the whole doc, emitting only the necessary values will save
disk space and make queries somewhat more efficient because there's less JSON to send and parse.

—Jens

Re: View access during indexing

Re: init terminating in do_boot

Fantastic, thank you. I have an updated package for JS 1.8.5. Looks like 
I'll need to create packages for ICU and cURL.

I've been avoiding working out the details, will let you know when I 
have packages that will install :)

Wendall

On 05/31/2012 03:58 PM, CGS wrote:
> Hi Wendall,
>
> Yes, of course. It's easy because I used most of the required packages form
> the repository. Still, there are few packages outdated, so, I needed to
> install them by hand from scratch.
>
> The packages I had to build because being outdated in the repository: ICU,
> JS (I used 1.8.0rc1, not so keen to try again with 1.8.5) and cURL. These
> are all of them. Take care to the following aspects:
> - ICU and cURL will be installed by default in /usr/local and
> LD_LIBRARY_PATH does not contain /usr/local/lib. To add it, there are two
> possibilities (not to attach any option to the configure script):
> a) in BASH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib (use
> setenv for TCSH);
> b) cd /usr/local/lib&&  for var in $(ls) ; do if [ ! -d ${var} ] ; then ln
> -s /usr/local/lib/${var} /usr/lib/${var} ; fi ; done&&  cd - (this will
> create a symbolic link for each library from /usr/local/lib in /usr/lib
> which is included in standard lib path - I did that after I got that error
> in do_boot, but it is better to do it from the beginning);
> - JS is even worse under x86_64 architecture as it creates /usr/local/lib64
> path in which it installs its libraries - do as above. To install it, use
> the instructions from
> http://wiki.apache.org/couchdb/Installing_SpiderMonkey(nothing more).
>
> How to build Erlang/OTP R15B01:
> - check out the prerequisite from Erlang (
> http://www.erlang.org/doc/installation_guide/INSTALL.html) and 'yum
> install' them all (at least I did that way and Erlang didn't complain),
> except for JAVA (I don't need the connection with it), fop (requires JAVA)
> and wxWidgets (I don't want to build nice windows in Erlang);
> - configure Erlang: ./configure --enable-threads --enable-smp-support
> --enable-kernel-poll --enable-sctp --enable-hipe
> --with-dynamic-trace=systemtap --without-javac (in my case, it complained
> about not finding dtrace even if I declared dynamic trace as systemtap, so,
> I had to discard it - nevertheless, it may be something which can be done,
> I hope, but I had no time to get into it);
> - build and install it: make (as simple user) and make install (as root).
> This will create (if everything goes well) /usr/local/lib/erlang and few
> symbolic links in /usr/local/bin, of course.
>
> How to build CouchDB:
>
> Check for the prerequisites on
> http://git-wip-us.apache.org/repos/asf/couchdb/repo?p=couchdb.git;a=blob;f=INSTALL.Unix;hb=master(it
> doesn't matter its outdated a bit, it provides with a list of what you
> need). In this moment you should have ICU, cURL, JS, Erlang and OpenSSL (as
> required by Erlang), so, all you need is help2man (which I yum'ed it).
>
> Furthermore, nothing simpler: ./configure&&  make (as simple user) and make
> install (as root).
>
> That's all. (Well, I omitted the steps related to download and decompress
> as I consider them as too obvious. :) )
>
> Comparing to the previous versions of CouchDB (1.1.0 was my first installed
> version :D), I really consider what the developers did was to have added a
> V6 engine to the installing scripts. ;)
>
> If you have problems, don't hesitate to post them here. I will help with
> what I can.
>
> Cheers,
> CGS
>
>
>
>
>
> On Thu, May 31, 2012 at 11:51 PM, Wendall Cada<wendallc@...>  wrote:
>
>> George,
>>
>> Can you share your build instructions? If you have a working install
>> process for Centos 5.8, I can create an rpm that could benefit others. 5.8
>> is problematic. I think the information could be very useful for
>> build-couchdb as well, as I think it isn't working currently either.
>>
>> Wendall
>>
>>
>> On 05/31/2012 02:19 PM, CGS wrote:
>>
>>> Hi Dave,
>>>
>>> Thank you for your reply. Indeed, downgrading Erlang version didn't help,
>>> but your suggestion helped me in finding the error. It was a path problem
>>> (more precisely, /usr/local/lib which wasn't in LD_LIBRARY_PATH by default
>>> under CentOS 5.8, and I had libicui18n.so.49 there). I am surprised it
>>> didn't show the problem during running the configure script or while
>>> compiling CouchDB, but I suppose it was added by hand in the script
>>> searching paths. Now, CouchDB is purring like a cat. Thanks a lot from
>>> another happy CouchDB user!
>>>
>>> Cheers,
>>> George
>>>
>>>
>>>
>>>
>>> On Thu, May 31, 2012 at 10:05 PM, Dave Cottlehuber<dave@...>
>>>   wrote:
>>>
>>>   On 31 May 2012 18:39, CGS<cgsmcmlxxv@...>   wrote:
>>>>> I haven't attached any COUCHDB_USER variable to my BASH session (unless
>>>>>
>>>> the
>>>>
>>>>> couchdb script adds it, but I see no COUCHDB_USER in that script). So,
>>>>> no
>>>>> 'su'ing away from root.
>>>>>
>>>>> Is there any known problem in between CouchDB and Erlang/OTP R15B01? Any
>>>>> idea, downgrading Erlang version would do any good?
>>>>>
>>>> No.
>>>>
>>>> You might get something useful by starting 'erl -init_debug'&   then to
>>>> start up
>>>> erlang solo (not as couch) and then progressively start up dependent
>>>> apps:
>>>>
>>>>     erl -init_debug  -env ERL_LIBS
>>>> /usr/local/Cellar/couchdb/1.2.**0/lib/couchdb/erlang/lib -couch_ini
>>>>
>>>> and then:
>>>>
>>>>     application:load(crypto), application:start(crypto).
>>>>     application:load(couch), application:start(couch).
>>>>
>>>> output should resemble this (from my mac):
>>>>
>>>> https://friendpaste.com/**6zett3aAHSafRNZbgfMBoD<https://friendpaste.com/6zett3aAHSafRNZbgfMBoD>
>>>>
>>>> A+
>>>> Dave
>>>>
>>>>

Re: init terminating in do_boot

Let us know about your progress. Who knows, maybe others are interested in
those RPM's. :)

If you need more details or if you have any problem during the
installation, let me know. I started to like CentOS and it became my
favorite after Ubuntu (which is my favorite since years). The problem with
CentOS is its outdated packages (CouchDB is in the repository with version
1.0.2 with Erlang/OTP R12B... :( ) and missing nice applications (like Yaws
which I managed to install it from source with its latest version on the
same Erlang version). But that is not a point to discuss it here.

CGS

On Fri, Jun 1, 2012 at 2:20 AM, Wendall Cada <wendallc@...> wrote:

> Fantastic, thank you. I have an updated package for JS 1.8.5. Looks like
> I'll need to create packages for ICU and cURL.
>
> I've been avoiding working out the details, will let you know when I have
> packages that will install :)
>
> Wendall
>
>
> On 05/31/2012 03:58 PM, CGS wrote:
>
>> Hi Wendall,
>>
>> Yes, of course. It's easy because I used most of the required packages
>> form
>> the repository. Still, there are few packages outdated, so, I needed to
>> install them by hand from scratch.
>>
>> The packages I had to build because being outdated in the repository: ICU,
>> JS (I used 1.8.0rc1, not so keen to try again with 1.8.5) and cURL. These
>> are all of them. Take care to the following aspects:
>> - ICU and cURL will be installed by default in /usr/local and
>> LD_LIBRARY_PATH does not contain /usr/local/lib. To add it, there are two
>> possibilities (not to attach any option to the configure script):
>> a) in BASH: export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:/usr/local/lib (use
>> setenv for TCSH);
>> b) cd /usr/local/lib&&  for var in $(ls) ; do if [ ! -d ${var} ] ; then ln
>> -s /usr/local/lib/${var} /usr/lib/${var} ; fi ; done&&  cd - (this will
>>
>> create a symbolic link for each library from /usr/local/lib in /usr/lib
>> which is included in standard lib path - I did that after I got that error
>> in do_boot, but it is better to do it from the beginning);
>> - JS is even worse under x86_64 architecture as it creates
>> /usr/local/lib64
>> path in which it installs its libraries - do as above. To install it, use
>> the instructions from
>> http://wiki.apache.org/**couchdb/Installing_**SpiderMonkey(nothing<http://wiki.apache.org/couchdb/Installing_SpiderMonkey%28nothing>more).
>>
>>
>> How to build Erlang/OTP R15B01:
>> - check out the prerequisite from Erlang (
>> http://www.erlang.org/doc/**installation_guide/INSTALL.**html<http://www.erlang.org/doc/installation_guide/INSTALL.html>)
>> and 'yum
>> install' them all (at least I did that way and Erlang didn't complain),
>> except for JAVA (I don't need the connection with it), fop (requires JAVA)
>> and wxWidgets (I don't want to build nice windows in Erlang);
>> - configure Erlang: ./configure --enable-threads --enable-smp-support
>> --enable-kernel-poll --enable-sctp --enable-hipe
>> --with-dynamic-trace=systemtap --without-javac (in my case, it complained
>> about not finding dtrace even if I declared dynamic trace as systemtap,
>> so,
>> I had to discard it - nevertheless, it may be something which can be done,
>> I hope, but I had no time to get into it);
>> - build and install it: make (as simple user) and make install (as root).
>> This will create (if everything goes well) /usr/local/lib/erlang and few
>> symbolic links in /usr/local/bin, of course.
>>
>> How to build CouchDB:
>>
>> Check for the prerequisites on
>> http://git-wip-us.apache.org/**repos/asf/couchdb/repo?p=**
>> couchdb.git;a=blob;f=INSTALL.**Unix;hb=master(it<http://git-wip-us.apache.org/repos/asf/couchdb/repo?p=couchdb.git;a=blob;f=INSTALL.Unix;hb=master%28it>
>> doesn't matter its outdated a bit, it provides with a list of what you
>> need). In this moment you should have ICU, cURL, JS, Erlang and OpenSSL
>> (as
>> required by Erlang), so, all you need is help2man (which I yum'ed it).
>>
>> Furthermore, nothing simpler: ./configure&&  make (as simple user) and
>> make
>>
>> install (as root).
>>
>> That's all. (Well, I omitted the steps related to download and decompress
>> as I consider them as too obvious. :) )
>>
>> Comparing to the previous versions of CouchDB (1.1.0 was my first
>> installed
>> version :D), I really consider what the developers did was to have added a
>> V6 engine to the installing scripts. ;)
>>
>> If you have problems, don't hesitate to post them here. I will help with
>> what I can.
>>
>> Cheers,
>> CGS
>>
>>
>>
>>
>>
>> On Thu, May 31, 2012 at 11:51 PM, Wendall Cada<wendallc@...>
>>  wrote:
>>
>>  George,
>>>
>>> Can you share your build instructions? If you have a working install
>>> process for Centos 5.8, I can create an rpm that could benefit others.
>>> 5.8
>>> is problematic. I think the information could be very useful for
>>> build-couchdb as well, as I think it isn't working currently either.
>>>
>>> Wendall
>>>
>>>
>>> On 05/31/2012 02:19 PM, CGS wrote:
>>>
>>>  Hi Dave,
>>>>
>>>> Thank you for your reply. Indeed, downgrading Erlang version didn't
>>>> help,
>>>> but your suggestion helped me in finding the error. It was a path
>>>> problem
>>>> (more precisely, /usr/local/lib which wasn't in LD_LIBRARY_PATH by
>>>> default
>>>> under CentOS 5.8, and I had libicui18n.so.49 there). I am surprised it
>>>> didn't show the problem during running the configure script or while
>>>> compiling CouchDB, but I suppose it was added by hand in the script
>>>> searching paths. Now, CouchDB is purring like a cat. Thanks a lot from
>>>> another happy CouchDB user!
>>>>
>>>> Cheers,
>>>> George
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 31, 2012 at 10:05 PM, Dave Cottlehuber<dave@...>
>>>>  wrote:
>>>>
>>>>  On 31 May 2012 18:39, CGS<cgsmcmlxxv@...>   wrote:
>>>>
>>>>> I haven't attached any COUCHDB_USER variable to my BASH session (unless
>>>>>>
>>>>>>  the
>>>>>
>>>>>  couchdb script adds it, but I see no COUCHDB_USER in that script). So,
>>>>>> no
>>>>>> 'su'ing away from root.
>>>>>>
>>>>>> Is there any known problem in between CouchDB and Erlang/OTP R15B01?
>>>>>> Any
>>>>>> idea, downgrading Erlang version would do any good?
>>>>>>
>>>>>>  No.
>>>>>
>>>>> You might get something useful by starting 'erl -init_debug'&   then to
>>>>> start up
>>>>> erlang solo (not as couch) and then progressively start up dependent
>>>>> apps:
>>>>>
>>>>>    erl -init_debug  -env ERL_LIBS
>>>>> /usr/local/Cellar/couchdb/1.2.****0/lib/couchdb/erlang/lib -couch_ini
>>>>>
>>>>>
>>>>> and then:
>>>>>
>>>>>    application:load(crypto), application:start(crypto).
>>>>>    application:load(couch), application:start(couch).
>>>>>
>>>>> output should resemble this (from my mac):
>>>>>
>>>>> https://friendpaste.com/****6zett3aAHSafRNZbgfMBoD<https://friendpaste.com/**6zett3aAHSafRNZbgfMBoD>
>>>>> <https:/**/friendpaste.com/**6zett3aAHSafRNZbgfMBoD<https://friendpaste.com/6zett3aAHSafRNZbgfMBoD>
>>>>> >
>>>>>
>>>>> A+
>>>>> Dave
>>>>>
>>>>>
>>>>>
>

local.ini settings not taking precedence

Maybe I'm missing something simple, but my understanding is that local.ini
values should take precedence over values in default.ini .  However, I'm
trying to set nodelay to true for the socket_options and that doesn't
appear to be working.

Here's what's in default.ini :
socket_options = [{keepalive, true}, {nodelay, false}]

here's what's in local.ini:
socket_options = [{nodelay, true}]

http://localhost:5984/_utils/config.html is reporting that socket_options
is equal to:
[{keepalive, true}, {nodelay, false}]

I've restarted the couchdb server multiple times since making the change,
but just noticed now that the change doesn't appear to have been made.  I'm
using v1.2.0

-Tim

Re: local.ini settings not taking precedence

On 1 June 2012 17:09, Tim Tisdall <tisdall@...> wrote:
> Maybe I'm missing something simple, but my understanding is that local.ini
> values should take precedence over values in default.ini .  However, I'm
> trying to set nodelay to true for the socket_options and that doesn't
> appear to be working.

Strictly speaking the ini file that's last in the chain takes precedence, and
its also the file that gets changes written to.

You can see which ones are being used and in what order with

$ couchdb -c
/usr/local/etc/couchdb/default.ini
/usr/local/etc/couchdb/local.ini

> Here's what's in default.ini :
> socket_options = [{keepalive, true}, {nodelay, false}]
>
> here's what's in local.ini:
> socket_options = [{nodelay, true}]
>
> http://localhost:5984/_utils/config.html is reporting that socket_options
> is equal to:
> [{keepalive, true}, {nodelay, false}]
>
>
> I've restarted the couchdb server multiple times since making the change,
> but just noticed now that the change doesn't appear to have been made.  I'm
> using v1.2.0

A restart is not required, the erlang magic picks up the changed config for you.

If your config chain above doesn't raise warning bells, can you temporarily
up the log level to debug, try this:

    COUCH=http://admin:passwd <at> localhost:5984
    curl -vXPUT $COUCH/_config/httpd/fake_key -d '"fake_value"'

You should see something like this in your logs:

[Fri, 01 Jun 2012 18:10:23 GMT] [debug] [<0.287.0>] 'PUT'
/_config/httpd/fake_key {1,1} from "127.0.0.1"
Headers: [{'Accept',"*/*"},
          {'Authorization',"Basic YWRtaW46cFGSzc3dk"},
          {'Content-Length',"12"},
          {'Content-Type',"application/x-www-form-urlencoded"},
          {'Host',"localhost:5984"},
          {'User-Agent',"curl/7.21.4 (universal-apple-darwin11.0)
libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5"}]
[Fri, 01 Jun 2012 18:10:23 GMT] [debug] [<0.287.0>] OAuth Params: []
[Fri, 01 Jun 2012 18:10:23 GMT] [info] [<0.287.0>] 127.0.0.1 - - PUT
/_config/httpd/fake_key 200

And you should be able to grep for fake_key if needed to see where
it's being written to.

In fact you can even use upping the log level as a check:

    curl -vXPUT $COUCH/_config/log/level -d '"debug"'

and back to error

    curl -vXPUT $COUCH/_config/log/level -d '"error"'

A+
Dave

Re: local.ini settings not taking precedence

I have the same config order as you.

It seems to have written the log level change to local.ini .  However, I'm
not sure how to interpret that...  does that mean it's giving local.ini the
highest precedence?

So, I looked at http://localhost:5984/_utils/config.html and it's now
reporting
socket_options  [{nodelay, true}]

This is after I changed local.ini to have the value
socket_option = [{keepalive, true}, {nodelay, true}]

since both default and local.ini say keepalive is true, it seems odd that
it's now dropped...

On Fri, Jun 1, 2012 at 2:15 PM, Dave Cottlehuber <dave@...> wrote:

> On 1 June 2012 17:09, Tim Tisdall <tisdall@...> wrote:
> > Maybe I'm missing something simple, but my understanding is that
> local.ini
> > values should take precedence over values in default.ini .  However, I'm
> > trying to set nodelay to true for the socket_options and that doesn't
> > appear to be working.
>
>
> Strictly speaking the ini file that's last in the chain takes precedence,
> and
> its also the file that gets changes written to.
>
> You can see which ones are being used and in what order with
>
> $ couchdb -c
> /usr/local/etc/couchdb/default.ini
> /usr/local/etc/couchdb/local.ini
>
> > Here's what's in default.ini :
> > socket_options = [{keepalive, true}, {nodelay, false}]
> >
> > here's what's in local.ini:
> > socket_options = [{nodelay, true}]
> >
> > http://localhost:5984/_utils/config.html is reporting that
> socket_options
> > is equal to:
> > [{keepalive, true}, {nodelay, false}]
> >
> >
> > I've restarted the couchdb server multiple times since making the change,
> > but just noticed now that the change doesn't appear to have been made.
>  I'm
> > using v1.2.0
>
> A restart is not required, the erlang magic picks up the changed config
> for you.
>
> If your config chain above doesn't raise warning bells, can you temporarily
> up the log level to debug, try this:
>
>    COUCH=http://admin:passwd <at> localhost:5984
>    curl -vXPUT $COUCH/_config/httpd/fake_key -d '"fake_value"'
>
> You should see something like this in your logs:
>
> [Fri, 01 Jun 2012 18:10:23 GMT] [debug] [<0.287.0>] 'PUT'
> /_config/httpd/fake_key {1,1} from "127.0.0.1"
> Headers: [{'Accept',"*/*"},
>          {'Authorization',"Basic YWRtaW46cFGSzc3dk"},
>          {'Content-Length',"12"},
>          {'Content-Type',"application/x-www-form-urlencoded"},
>          {'Host',"localhost:5984"},
>          {'User-Agent',"curl/7.21.4 (universal-apple-darwin11.0)
> libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5"}]
> [Fri, 01 Jun 2012 18:10:23 GMT] [debug] [<0.287.0>] OAuth Params: []
> [Fri, 01 Jun 2012 18:10:23 GMT] [info] [<0.287.0>] 127.0.0.1 - - PUT
> /_config/httpd/fake_key 200
>
> And you should be able to grep for fake_key if needed to see where
> it's being written to.
>
> In fact you can even use upping the log level as a check:
>
>    curl -vXPUT $COUCH/_config/log/level -d '"debug"'
>
> and back to error
>
>    curl -vXPUT $COUCH/_config/log/level -d '"error"'
>
> A+
> Dave
>

map-reduce question

Hi,

I have a database with documents that each include two categorical and 
one continuous value.  Something like

     {
         fruit: "apple",
         size: "small",
         quality: 1.7
     },
     {
         fruit: "banana",
         size: "large",
         quality: 3.6
     }

I'd like to be able to compute responses to views/queries that basically 
ask:
     how many documents do I have (broken down by 'fruit' and 'size') 
that have the 'quality' greater than some cutoff?

And I'm having trouble identifying the appropriate couch map/reduce view 
to implement this.  If I emit keys of the form
     [doc.quality, doc.fruit, doc.size]
then I can do the quality-based searches using
     http://couch/db/_design/whatever/_view/quality?startkey=[2]
but the default "_count" kinds of reduce functions will split things up 
by quality, which I only care about for the filtering aspects.

How do I write the reduce function to skip the top level?

     Kevin

Documents with 1000 revisions

Hello,

I noticed that a very small number of documents in my database have
1000 revisions, which somehow prevents them from being updated again.
It is unclear where that came from, but I was wondering if it is
possible to somehow compact just the document instead of compacting
the whole database i.e. delete all but the last document revision.

Thanks,
Matthias