Robert L. Read | 5 Jul 2007 00:49

db-postmodern update - Integrated!

I have just integrated Henrik Hjelte's postmodern-based backend into the official
darcs repository for elephant:

http://common-lisp.net/project/elephant/darcs/elephant/

However, this is not an official release, but presumably will be part of the next official release.
It passes all of the tests.  The Postmodern backend is not quite twice as fast on our testsuite as
the CL-SQL backend.  The CL-SQL backend still has some value, however, for non-Postgres
databases.  If someone were to write another DB-specific backend, such as a MySQL backend
or an Oracle backend, I would assume they would benefit from both the CL-SQL version and
the Postmodern version.  However, I think the Postmodern version is better-written.  I thank
Henrik very much for preparing such a well-written, easy-to-integrate backend.

I did, however, have to make three changes to get this to work for me under SBCL.
Postmodern defines "db-error" incompatibly with the BDB backend, so I renamed the BDB symbol
from db-error to bdb-db-error.  This was one issue on a "handler-case" that I assume was
acceptable under Henrik's LISP, but didn't work under SBCL, or seem to follow the CLHS.
Finally, in shared-initialize :after method that builds the cache-indices I had to explicitly
set the slot to the initform, to prevent a reading of an unbound-slot.  I still have no idea
why the slot was unbound at that point, in spite of all documentation saying that it should be.

I slightly modified Henrik's test to harmonize more with the CL-SQL version, and added an
example of using postmodern into the RUNTEST.lisp file.

Describing this in the manual remains to be done; I don't have any immediate plans to do so.

Personally, I think each backend adds to the overall usability of Elephant, and should broaden
our user base.  Particularly since one can migrate data from one backend to another, it would
seem that we are offering a solid platform.

Once this is officially released, users can expect the postmodern backend to be supported
indefinitely.  If we ever drop a backend, we will provide a way to migrate data out of it.






On Sat, 2007-05-12 at 16:46 +0200, Henrik Hjelte wrote:
I have made some changes to the postmodern postgresql backend. The darcs repo is now based on the official elephant darcs repo, so patches can be moved between them easily. The postmodern backend passes all "official" tests, but has not been extensively tested with real data yet. The db-postmodern tree can be found here: http://common-lisp.net/project/grand-prix/darcs/elephant /Henrik Hjelte _______________________________________________ elephant-devel site list elephant-devel-F1HGIaG5STRyXAeb93iumQ@public.gmane.org http://common-lisp.net/mailman/listinfo/elephant-devel
<div>
I have just integrated Henrik Hjelte's postmodern-based backend into the official<br>
darcs repository for elephant:<br><br><a href="http://common-lisp.net/project/elephant/darcs/elephant/">http://common-lisp.net/project/elephant/darcs/elephant/</a><br><br>
However, this is not an official release, but presumably will be part of the next official release.<br>
It passes all of the tests.&nbsp; The Postmodern backend is not quite twice as fast on our testsuite as<br>
the CL-SQL backend.&nbsp; The CL-SQL backend still has some value, however, for non-Postgres <br>
databases.&nbsp; If someone were to write another DB-specific backend, such as a MySQL backend<br>
or an Oracle backend, I would assume they would benefit from both the CL-SQL version and <br>
the Postmodern version.&nbsp; However, I think the Postmodern version is better-written.&nbsp; I thank<br>
Henrik very much for preparing such a well-written, easy-to-integrate backend.<br><br>
I did, however, have to make three changes to get this to work for me under SBCL.<br>
Postmodern defines "db-error" incompatibly with the BDB backend, so I renamed the BDB symbol<br>
from db-error to bdb-db-error.&nbsp; This was one issue on a "handler-case" that I assume was <br>
acceptable under Henrik's LISP, but didn't work under SBCL, or seem to follow the CLHS.<br>
Finally, in shared-initialize :after method that builds the cache-indices I had to explicitly <br>
set the slot to the initform, to prevent a reading of an unbound-slot.&nbsp; I still have no idea<br>
why the slot was unbound at that point, in spite of all documentation saying that it should be.<br><br>
I slightly modified Henrik's test to harmonize more with the CL-SQL version, and added an<br>
example of using postmodern into the RUNTEST.lisp file.<br><br>
Describing this in the manual remains to be done; I don't have any immediate plans to do so.<br><br>
Personally, I think each backend adds to the overall usability of Elephant, and should broaden<br>
our user base.&nbsp; Particularly since one can migrate data from one backend to another, it would<br>
seem that we are offering a solid platform.<br><br>
Once this is officially released, users can expect the postmodern backend to be supported <br>
indefinitely.&nbsp; If we ever drop a backend, we will provide a way to migrate data out of it.<br><br><br><br><br><br><br>
On Sat, 2007-05-12 at 16:46 +0200, Henrik Hjelte wrote:
<blockquote type="CITE">

I have made some changes to the postmodern postgresql backend. The darcs
repo is now based on the official elephant darcs repo, so patches can be
moved between them easily. The postmodern backend passes all "official"
tests, but has not been extensively tested with real data yet.

The db-postmodern tree can be found here:
<a href="http://common-lisp.net/project/grand-prix/darcs/elephant">http://common-lisp.net/project/grand-prix/darcs/elephant</a>

/Henrik Hjelte

_______________________________________________
elephant-devel site list
<a href="mailto:elephant-devel@...">elephant-devel@...</a>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a>

</blockquote>
</div>
Pierre THIERRY | 5 Jul 2007 13:58
Gravatar

Re: db-postmodern update - Integrated!

Scribit Robert L. Read dies 04/07/2007 hora 17:49:
> The Postmodern backend is not quite twice as fast on our testsuite as
> the CL-SQL backend.

Great. I found the CL-SQL backend with PostgreSQL a bit slow for my
specific use case. Do you know why and how this backend is so much
faster?

> Postmodern defines "db-error" incompatibly with the BDB backend, so I
> renamed the BDB symbol from db-error to bdb-db-error.

Was there any value in having the same condition for all backends? If
yes, will it be a goal to "fix" this divergence?

Curiously,
Pierre
-- 
nowhere.man@...
OpenPGP 0xD9D50D8A
Scribit Robert L. Read dies 04/07/2007 hora 17:49:
> The Postmodern backend is not quite twice as fast on our testsuite as
> the CL-SQL backend.

Great. I found the CL-SQL backend with PostgreSQL a bit slow for my
specific use case. Do you know why and how this backend is so much
faster?

> Postmodern defines "db-error" incompatibly with the BDB backend, so I
> renamed the BDB symbol from db-error to bdb-db-error.

Was there any value in having the same condition for all backends? If
yes, will it be a goal to "fix" this divergence?

Curiously,
Pierre
--

-- 
nowhere.man@...
OpenPGP 0xD9D50D8A
Robert L. Read | 5 Jul 2007 17:20

Re: db-postmodern update - Integrated!

On Thu, 2007-07-05 at 13:58 +0200, Pierre THIERRY wrote:
Scribit Robert L. Read dies 04/07/2007 hora 17:49: > The Postmodern backend is not quite twice as fast on our testsuite as > the CL-SQL backend. Great. I found the CL-SQL backend with PostgreSQL a bit slow for my specific use case. Do you know why and how this backend is so much faster?

I haven't studied it deeply; Henrik could answer better than I.  However, it uses tables more
effectively, and uses stored procedures which are very Postgres-specific and not portable to
a different database.
> Postmodern defines "db-error" incompatibly with the BDB backend, so I > renamed the BDB symbol from db-error to bdb-db-error. Was there any value in having the same condition for all backends? If yes, will it be a goal to "fix" this divergence?

No, this is an internal matter ---- I don't think we have attempted to provide a cross-platform
error mechanism, and I would hesitate to try to do so.

Ideally, I would like the Elephant development to be shaped by the actual needs of the users.
Although we have been responsive to bugs, I don't feel that we have been given a lot of
feedback in terms of what direction it should take.

I agree with Ian that a pure-lisp backend would be nice, because it would let someone
use Elephant without installing anything else, while still leaving open the engineering
decision of what to use later.
Curiously, Pierre _______________________________________________ elephant-devel site list elephant-devel-F1HGIaG5STRyXAeb93iumQ@public.gmane.org http://common-lisp.net/mailman/listinfo/elephant-devel
<div>
On Thu, 2007-07-05 at 13:58 +0200, Pierre THIERRY wrote:
<blockquote type="CITE">

Scribit Robert L. Read dies 04/07/2007 hora 17:49:
&gt; The Postmodern backend is not quite twice as fast on our testsuite as
&gt; the CL-SQL backend.

Great. I found the CL-SQL backend with PostgreSQL a bit slow for my
specific use case. Do you know why and how this backend is so much
faster?

</blockquote>
<br>
I haven't studied it deeply; Henrik could answer better than I.&nbsp; However, it uses tables more <br>
effectively, and uses stored procedures which are very Postgres-specific and not portable to <br>
a different database.
<blockquote type="CITE">

&gt; Postmodern defines "db-error" incompatibly with the BDB backend, so I
&gt; renamed the BDB symbol from db-error to bdb-db-error.

Was there any value in having the same condition for all backends? If
yes, will it be a goal to "fix" this divergence?

</blockquote>
<br>
No, this is an internal matter ---- I don't think we have attempted to provide a cross-platform <br>
error mechanism, and I would hesitate to try to do so.<br><br>
Ideally, I would like the Elephant development to be shaped by the actual needs of the users.<br>
Although we have been responsive to bugs, I don't feel that we have been given a lot of <br>
feedback in terms of what direction it should take.<br><br>
I agree with Ian that a pure-lisp backend would be nice, because it would let someone <br>
use Elephant without installing anything else, while still leaving open the engineering <br>
decision of what to use later.
<blockquote type="CITE">

Curiously,
Pierre
_______________________________________________
elephant-devel site list
<a href="mailto:elephant-devel@...">elephant-devel@...</a>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a>

</blockquote>
</div>
Pierre THIERRY | 7 Jul 2007 01:47
Gravatar

Re: db-postmodern update - Integrated!

Scribit Robert L. Read dies 05/07/2007 hora 10:20:
> However, it uses tables more effectively, and uses stored procedures
> which are very Postgres-specific and not portable to a different
> database.

Well, I think that constitutes an answer to my earlier question about
the performance gains of stored procedures: it's definitely worth
investigating.

I would find interesting to see how to cl-sql backend could be made
extensible, so that most of the postmodern's improvements related to the
use of the database could be ported to it.

Does the way the postmodern backend communicates with the DB could also
be a source of speedup?

> Although we have been responsive to bugs, I don't feel that we have
> been given a lot of feedback in terms of what direction it should
> take.

Well, I think my recent paper give some interesting directions:

http://lisp-ecoop07.bknr.net/submission/13418

Curiously,
Pierre
-- 
nowhere.man@...
OpenPGP 0xD9D50D8A
Scribit Robert L. Read dies 05/07/2007 hora 10:20:
> However, it uses tables more effectively, and uses stored procedures
> which are very Postgres-specific and not portable to a different
> database.

Well, I think that constitutes an answer to my earlier question about
the performance gains of stored procedures: it's definitely worth
investigating.

I would find interesting to see how to cl-sql backend could be made
extensible, so that most of the postmodern's improvements related to the
use of the database could be ported to it.

Does the way the postmodern backend communicates with the DB could also
be a source of speedup?

> Although we have been responsive to bugs, I don't feel that we have
> been given a lot of feedback in terms of what direction it should
> take.

Well, I think my recent paper give some interesting directions:

http://lisp-ecoop07.bknr.net/submission/13418

Curiously,
Pierre
--

-- 
nowhere.man@...
OpenPGP 0xD9D50D8A
Henrik Hjelte | 9 Jul 2007 10:30
Gravatar

Re: db-postmodern update - Integrated!

On 7/7/07, Pierre THIERRY <nowhere.man@...> wrote>
Scribit Robert L. Read dies 05/07/2007 hora 10:20:
> > However, it uses tables more effectively, and uses stored procedures
> > which are very Postgres-specific and not portable to a different
> > database.
>
> Well, I think that constitutes an answer to my earlier question about
> the performance gains of stored procedures: it's definitely worth
> investigating.
I tried to measure when developing the backend. Prepared queries gave
the same speed as stored procedures. Stored procedures are only used
for inserts which does a few steps in an if then clause, then I
thought it was unnecessary to send intermediate values between client
and server.

> I would find interesting to see how to cl-sql backend could be made
> extensible, so that most of the postmodern's improvements related to the
> use of the database could be ported to it.
It should be easy to factor out from clsql. Also, the postgresql
specific parts of the postmodern backend could rather easily be
factored out, they are not so many.

> Does the way the postmodern backend communicates with the DB could also
> be a source of speedup?
Yes maybe, but I don't think it is of highest importance.

>
> > Although we have been responsive to bugs, I don't feel that we have
> > been given a lot of feedback in terms of what direction it should
> > take.
>
> Well, I think my recent paper give some interesting directions:
>
> http://lisp-ecoop07.bknr.net/submission/13418
Looks interesting from a fast scan, I must read it better before
commenting on it though. But the shortcomings you address should be
fixed, definitely.

/Henrik
Robert L. Read | 10 Jul 2007 22:41

Re: db-postmodern update - Integrated!

Henrik and I have fixed some subtle bugs that occurred under certain data values (which
happened to occur in my "live" database for "konsenti.com", my for-profit business.

I have now integrated these fixes into the main darcs elephant repository.

If anybody pulls and tests the :ele-postmodern backend, please send us a note.
It works under SBCL, but has not been tested under our full matrix, AFAIK.



By the way, one might be interested in how one migrated from an exsiting CL-SQL backend
to the Postmodern-based ele-postmodern backend.  Here is approximately the lisp code that
I used to move from "konsenti", a CL-SQL based backend, to "kosenti2", a postmodern-based
backend:


(defparameter *REPOSITORY-SPEC* (list :clsql (list :postgresql
                                  (coerce-to-base "localhost.localdomain" )
                                  (coerce-to-base "konsenti" )
                                  (coerce-to-base "postgres" )
                                  (coerce-to-base "mrmtzplx"))))

(defparameter *UPGRADE-REPOSITORY-SPEC* (list :postmodern (list :postgresql
                                  (coerce-to-base "localhost.localdomain" )
                                  (coerce-to-base "konsenti2" )
                                  (coerce-to-base "postgres" )
                                  (coerce-to-base "mrmtzplx"))))

(use-package :elephant)
(setq orig (open-store *REPOSITORY-SPEC*))
(setq pm (open-store *UPGRADE-REPOSITORY-SPEC*))
(migrate pm orig)




On Wed, 2007-07-04 at 17:49 -0500, Robert L. Read wrote:
I have just integrated Henrik Hjelte's postmodern-based backend into the official
darcs repository for elephant:

http://common-lisp.net/project/elephant/darcs/elephant/

However, this is not an official release, but presumably will be part of the next official release.
It passes all of the tests.  The Postmodern backend is not quite twice as fast on our testsuite as
the CL-SQL backend.  The CL-SQL backend still has some value, however, for non-Postgres
databases.  If someone were to write another DB-specific backend, such as a MySQL backend
or an Oracle backend, I would assume they would benefit from both the CL-SQL version and
the Postmodern version.  However, I think the Postmodern version is better-written.  I thank
Henrik very much for preparing such a well-written, easy-to-integrate backend.

I did, however, have to make three changes to get this to work for me under SBCL.
Postmodern defines "db-error" incompatibly with the BDB backend, so I renamed the BDB symbol
from db-error to bdb-db-error.  This was one issue on a "handler-case" that I assume was
acceptable under Henrik's LISP, but didn't work under SBCL, or seem to follow the CLHS.
Finally, in shared-initialize :after method that builds the cache-indices I had to explicitly
set the slot to the initform, to prevent a reading of an unbound-slot.  I still have no idea
why the slot was unbound at that point, in spite of all documentation saying that it should be.

I slightly modified Henrik's test to harmonize more with the CL-SQL version, and added an
example of using postmodern into the RUNTEST.lisp file.

Describing this in the manual remains to be done; I don't have any immediate plans to do so.

Personally, I think each backend adds to the overall usability of Elephant, and should broaden
our user base.  Particularly since one can migrate data from one backend to another, it would
seem that we are offering a solid platform.

Once this is officially released, users can expect the postmodern backend to be supported
indefinitely.  If we ever drop a backend, we will provide a way to migrate data out of it.






On Sat, 2007-05-12 at 16:46 +0200, Henrik Hjelte wrote:
I have made some changes to the postmodern postgresql backend. The darcs repo is now based on the official elephant darcs repo, so patches can be moved between them easily. The postmodern backend passes all "official" tests, but has not been extensively tested with real data yet. The db-postmodern tree can be found here: http://common-lisp.net/project/grand-prix/darcs/elephant /Henrik Hjelte _______________________________________________ elephant-devel site list elephant-devel-F1HGIaG5STRyXAeb93iumQ@public.gmane.org http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________ elephant-devel site list elephant-devel-F1HGIaG5STRyXAeb93iumQ@public.gmane.org http://common-lisp.net/mailman/listinfo/elephant-devel
<div>
Henrik and I have fixed some subtle bugs that occurred under certain data values (which <br>
happened to occur in my "live" database for "konsenti.com", my for-profit business.<br><br>
I have now integrated these fixes into the main darcs elephant repository.<br><br>
If anybody pulls and tests the :ele-postmodern backend, please send us a note.<br>
It works under SBCL, but has not been tested under our full matrix, AFAIK.<br><br><br><br>
By the way, one might be interested in how one migrated from an exsiting CL-SQL backend<br>
to the Postmodern-based ele-postmodern backend.&nbsp; Here is approximately the lisp code that <br>
I used to move from "konsenti", a CL-SQL based backend, to "kosenti2", a postmodern-based<br>
backend:<br><br><br>
(defparameter *REPOSITORY-SPEC* (list :clsql (list :postgresql<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "localhost.localdomain" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "konsenti" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "postgres" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "mrmtzplx"))))<br><br>
(defparameter *UPGRADE-REPOSITORY-SPEC* (list :postmodern (list :postgresql<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "localhost.localdomain" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "konsenti2" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "postgres" )<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (coerce-to-base "mrmtzplx"))))<br><br>
(use-package :elephant)<br>
(setq orig (open-store *REPOSITORY-SPEC*))<br>
(setq pm (open-store *UPGRADE-REPOSITORY-SPEC*))<br>
(migrate pm orig)<br><br><br><br><br>
On Wed, 2007-07-04 at 17:49 -0500, Robert L. Read wrote:<br><blockquote type="CITE">
    I have just integrated Henrik Hjelte's postmodern-based backend into the official<br>darcs repository for elephant:<br><br><a href="http://common-lisp.net/project/elephant/darcs/elephant/">http://common-lisp.net/project/elephant/darcs/elephant/</a><br><br>However, this is not an official release, but presumably will be part of the next official release.<br>It passes all of the tests.&nbsp; The Postmodern backend is not quite twice as fast on our testsuite as<br>the CL-SQL backend.&nbsp; The CL-SQL backend still has some value, however, for non-Postgres <br>databases.&nbsp; If someone were to write another DB-specific backend, such as a MySQL backend<br>or an Oracle backend, I would assume they would benefit from both the CL-SQL version and <br>the Postmodern version.&nbsp; However, I think the Postmodern version is better-written.&nbsp; I thank<br>Henrik very much for preparing such a well-written, easy-to-integrate backend.<br><br>I did, however, have to make three changes to get this to work for me under SBCL.<br>Postmodern defines "db-error" incompatibly with the BDB backend, so I renamed the BDB symbol<br>from db-error to bdb-db-error.&nbsp; This was one issue on a "handler-case" that I assume was <br>acceptable under Henrik's LISP, but didn't work under SBCL, or seem to follow the CLHS.<br>Finally, in shared-initialize :after method that builds the cache-indices I had to explicitly <br>set the slot to the initform, to prevent a reading of an unbound-slot.&nbsp; I still have no idea<br>why the slot was unbound at that point, in spite of all documentation saying that it should be.<br><br>I slightly modified Henrik's test to harmonize more with the CL-SQL version, and added an<br>example of using postmodern into the RUNTEST.lisp file.<br><br>Describing this in the manual remains to be done; I don't have any immediate plans to do so.<br><br>Personally, I think each backend adds to the overall usability of Elephant, and should broaden<br>our user base.&nbsp; Particularly since one can migrate data from one backend to another, it would<br>seem that we are offering a solid platform.<br><br>Once this is officially released, users can expect the postmodern backend to be supported <br>indefinitely.&nbsp; If we ever drop a backend, we will provide a way to migrate data out of it.<br><br><br><br><br><br><br>On Sat, 2007-05-12 at 16:46 +0200, Henrik Hjelte wrote: 
    <blockquote type="CITE">

I have made some changes to the postmodern postgresql backend. The darcs
repo is now based on the official elephant darcs repo, so patches can be
moved between them easily. The postmodern backend passes all "official"
tests, but has not been extensively tested with real data yet.

The db-postmodern tree can be found here:
<a href="http://common-lisp.net/project/grand-prix/darcs/elephant">http://common-lisp.net/project/grand-prix/darcs/elephant</a>

/Henrik Hjelte

_______________________________________________
elephant-devel site list
<a href="mailto:elephant-devel@...">elephant-devel@...</a>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a>

    </blockquote>

_______________________________________________
elephant-devel site list
<a href="mailto:elephant-devel@...">elephant-devel@...</a>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a>

</blockquote>
</div>
Dmitri V. Gorbatovsky | 11 Jul 2007 01:19
Picon

garbage collection

Hello, from grateful user.
I find that size of %ELEPHANT file (BDB) only grows with 
time and transactions. No matter how many objects I delete.
Is there are any conventional way to shrink it?
Sorry if the question is stupid, Iam not really advanced
in Berkeley db usage.

Thanks, DG
Ian Eslick | 11 Jul 2007 03:32
Picon
Picon
Favicon

Re: garbage collection

Dmitri,

The latest version in CVS does export a compact function that works  
on BDB stores, but it only truncates the end of the %ELEPHANT file  
and isn't very reliable because if you haven't dropped the last  
object added then it can't truncate at all.

If you do drop objects, BDB should reuse the pages that were  
allocated to those objects so it should grow more slowly, but it  
doesn't reorganize to optimize space so fragmentation can lead to an  
inefficient use of space.

However, the best way to reclaim disk space today is to migrate from  
your current BDB store to a new one.  This will copy over only those  
objects that are globally reachable; namely anything reachable from  
the controller-root and all indexed instances.

At some point I'll finish a wrapper function for this (currently a  
sketch is in src/elephant/gc.lisp and called via elephant::stop-and- 
copy-gc) but you are better off doing the copy manually for now.

If the documentation doesn't make this process clear, please write  
back to us.

Good luck,
Ian

On Jul 10, 2007, at 7:19 PM, Dmitri V. Gorbatovsky wrote:

> Hello, from grateful user.
> I find that size of %ELEPHANT file (BDB) only grows with
> time and transactions. No matter how many objects I delete.
> Is there are any conventional way to shrink it?
> Sorry if the question is stupid, Iam not really advanced
> in Berkeley db usage.
>
> Thanks, DG
> _______________________________________________
> elephant-devel site list
> elephant-devel@...
> http://common-lisp.net/mailman/listinfo/elephant-devel

Dmitri V. Gorbatovsky | 11 Jul 2007 20:21
Picon

Re: garbage collection

On Wednesday 11 July 2007 04:32, Ian Eslick wrote:

> However, the best way to reclaim disk space today is to migrate from
> your current BDB store to a new one.  This will copy over only those
> objects that are globally reachable; namely anything reachable from
> the controller-root and all indexed instances.
...
> If the documentation doesn't make this process clear, please write
> back to us.
>
> Good luck,
> Ian

Thank you, Ian.
Yes, I am afraid, I can't get it from the documentation. 
If it possible, could you please provide me with 
simple example of whole database migration from one directory
~/bdb/db-source  to another ~/bdb/db-result.

Thanks, Dmitri
Ian Eslick | 11 Jul 2007 21:18
Picon
Picon
Favicon

Re: garbage collection

Some documentation:

http://common-lisp.net/project/elephant/doc/Repository-Migration-and- 
Upgrade.html#Repository-Migration-and-Upgrade

http://common-lisp.net/project/elephant/doc/Migration-and-Upgrading- 
API.html#Migration-and-Upgrading-API

This is the safest route:

Start lisp & load elephant:

=> (setf source-store (open-store '(:BDB "~/bdb/db-source")))
=> (setf target-store (open-store '(:BDB "~/bdb/db-result")))

=> (migrate target-store source-store)

Exit and restart lisp:

=> (open-store '(:BDB "~/bdb/db-result"))

That's it!

I recommend that you run a simple validation test:
- create a simple database
- populating it with a bunch of data, record the size
- delete 1/2 of the data randomly (or every other object)
- call (optimize-layout *store-controller*), record the compacted size
- do the migration as described above, record the resulting size

I'm sure we'd love to see the three different sizes that result from  
this simple test and I'm sure you'd feel better knowing that it works!

Regards,
Ian

On Jul 11, 2007, at 2:21 PM, Dmitri V. Gorbatovsky wrote:

> On Wednesday 11 July 2007 04:32, Ian Eslick wrote:
>
>> However, the best way to reclaim disk space today is to migrate from
>> your current BDB store to a new one.  This will copy over only those
>> objects that are globally reachable; namely anything reachable from
>> the controller-root and all indexed instances.
> ...
>> If the documentation doesn't make this process clear, please write
>> back to us.
>>
>> Good luck,
>> Ian
>
> Thank you, Ian.
> Yes, I am afraid, I can't get it from the documentation.
> If it possible, could you please provide me with
> simple example of whole database migration from one directory
> ~/bdb/db-source  to another ~/bdb/db-result.
>
> Thanks, Dmitri
> _______________________________________________
> elephant-devel site list
> elephant-devel@...
> http://common-lisp.net/mailman/listinfo/elephant-devel


Gmane