Roger Masse | 12 Jun 16:46 2013

RELEASED: DurusWorks v1.2, Dulcinea v0.22

Greetings --

(this is cross posted from qp <at> mems-exchange.org)

It's been awhile since we've done update releases of DurusWorks and Dulcinea.

These updates have some small pieces of new functionality, but largely reflect the tracking of changes in Python and the evolution of style in programing web applications… and of course fixing bugs!

This software is made available through the generosity of my employer CNRI under an open source license from the MEMS Exchange website under software.

DurusWorks, the repackaging of Durus, qp, qpy and sancho, was originally released in Sept 2011, and Dulcinea which goes back more than 10 years, has been the basis of several successful projects here at CNRI including: 

- The GRIN Exchange

Here are the details of the changes made since the last releases of  DurusWorks and Dulcinea .

For the foreseeable future, I will be the sole developer and maintainer of these packages here at CNRI.

Much of this great work was accomplished by my former colleague here at CNRI, David Binger, who for the moment anyway, is not doing software development.

My direct contact information is Roger Masse <rmasse <at> mems-exchange.org>, but please first consider using the qp list (qp <at> mems-exchange.org) to post comments or report bugs about DurusWorks or Dulcinea.

Specific questions about the durus database can be posted to the durus-users list (durus-users <at> mems-exchange.org).

Since this is my first time managing these releases, please let me know if something doesn't look right.

Thank you,

-Roger Masse - Software Developer - CNRI

_______________________________________________
Durus-users mailing list
Durus-users <at> mems-exchange.org
http://mail.mems-exchange.org/mailman/listinfo/durus-users
Jesus Cea | 4 Feb 23:33 2013
Picon

DurusWorks 1.1 is not installing "_persistence.so"


I just installed current DurusWorks (1.1) on a fresh new machine, and
I was seeing this warning in my logs: "Using Python base classes for
persistence.".

"_persistence.so" extension is being compiled correctly, but not
installed when doing "python setup.py install". I copied the shared
lib by hand, and it works correctly.

Since durus installation directory is not versioned (I think this
should be a bug), people is possibly not seeing this because they are
installing over an old Durus version, and using the old
"_persistence.so" already installed (risking suttle bugs here :).

Please, use versioned durus directories :-). Very convenient if you
want several Durus versions in the same system (for whatever reason).
I am still running 3.7 somewhere because heavily modified code :-).
And this bug would be very obvious :-P

--

-- 
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
David Binger | 16 Jan 17:38 2012

Programming Job

Anyone looking for full-time steady employment using Durus and QP?

CNRI is looking to hire for a python programmer.
We've posted it on the Python Jobs Board:
http://www.python.org/community/jobs/

Please contact me directly if you are interested,
and put "PYTHON" in the subject line to make sure
that I don't miss it.

- David Binger
MEMS Exchange
CNRI
David Hess | 14 Aug 18:44 2011

Re: Interesting BTree failure


After some more investigation - this occurred right after live packing the database and changes to this
BTree were made and committed during the pack.

Afterwards, the oids seem to be jumbled up in this BTree (at least - maybe elsewhere in the database too).
Unpickled Persistent objects are not what they should be - interior BNodes are sometimes application
classes and stored values are sometimes BNodes rather than application classes.

Dave

On Aug 14, 2011, at 10:45 AM, David Hess wrote:

> 
> We have a long running process that does a lot of work with BTrees and in the middle of doing an "in"
operation, we got this traceback:
> 
> File "/usr/local/lib/python2.6/dist-packages/durus/btree.py", line 343, in __contains__
>  return self.root.search(key) is not None
> File "/usr/local/lib/python2.6/dist-packages/durus/btree.py", line 93, in search
>  return self.nodes[position].search(key)
> File "/usr/local/lib/python2.6/dist-packages/durus/btree.py", line 93, in search
>  return self.nodes[position].search(key)
> File "/usr/local/lib/python2.6/dist-packages/durus/persistent.py", line 173, in _p_load_state
>  self._p_connection.load_state(self)
> File "/usr/local/lib/python2.6/dist-packages/durus/connection.py", line 182, in load_state
>  pickle = self.get_stored_pickle(oid)
> File "/usr/local/lib/python2.6/dist-packages/durus/connection.py", line 111, in get_stored_pickle
>  record = self.storage.load(oid)
> File "/usr/local/lib/python2.6/dist-packages/durus/file_storage.py", line 96, in load
>  raise KeyError(oid)
> KeyError: '\x00\x00\x00\x00\x00\x00T\xb4'
> 
> This is using shelf storage on Durus 3.7 and is a long running process. My best guess is a ghosted BNode that
thought it was persisted to disk but really wasn't? I think I've seen this once before a couple of years ago
(i.e. it seems to be really rare).
> 
> Is this new and/or known and fixed in Durus 3.8?
> 
> Dave
> 
David Hess | 11 Aug 16:37 2011

Durus repair

We have servers that deal with power outages (and dirty power in general) and have ended up with some
corrupted durus databases. We've used the normal "repair" feature to handle a lot of the cases but we have
another that's occurring on occasion that is not handled by repair. It is failing as this exception:

Traceback (most recent call last):
  File "/usr/local/bin/durus", line 22, in <module>
    client_main()
  File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 108, in client_main
    options.startup)
  File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 35, in interactive_client
    storage = FileStorage(file, readonly=readonly, repair=repair)
  File "/usr/local/lib/python2.6/dist-packages/durus/file_storage.py", line 73, in __init__
    self.shelf = Shelf(filename, readonly=readonly, repair=repair)
  File "/usr/local/lib/python2.6/dist-packages/durus/shelf.py", line 91, in __init__
    self.file, repair=repair)
  File "/usr/local/lib/python2.6/dist-packages/durus/shelf.py", line 296, in read_transaction_offsets
    file.seek(position + 8 + record_length)
  File "/usr/local/lib/python2.6/dist-packages/durus/file.py", line 41, in seek
    self.file.seek(n, whence)
IOError: [Errno 22] Invalid argument

We've come up with the following patch:

--- /usr/local/lib/python2.6/dist-packages/durus/shelf.py	2007-04-24 16:10:16.000000000 -0500
+++ durus/shelf.py	2011-08-11 08:18:51.169288642 -0500
 <at>  <at>  -297,18 +297,17  <at>  <at> 
         if file.tell() != transaction_end:
             raise ShortRead
         return transaction_offsets
-    except ShortRead, e:
+    except (ShortRead, IOError), e:
         position = file.tell()
         if position > transaction_start:
             if repair:
                 file.seek(transaction_start)
                 file.truncate()
             else:
-                e.args = repr(dict(
+                raise ShortRead(repr(dict(
                     transaction_start=transaction_start,
                     transaction_end = transaction_end,
-                    position=position))
-                raise
+                    position=position)))
         return None

Or test database started out as:

-rw-r--r--  1 fishfinder fishfinder 49371713 2011-08-10 14:12 db.durus

We ran with this patch and without the repair switch and got the following (expected) report:

durus.utils.ShortRead: {'position': 47501336L, 'transaction_start': 47501312L,
'transaction_end': 12061693972974895416L}

We then ran again with --repair and the database was able to be opened. The resulting database looked like this:

-rw-r--r--  1 fishfinder fishfinder 47501312 2011-08-11 08:14 db.durus

Can anybody comment on whether this patch makes the appropriate sense and is safe enough? We
(fortunately!) have a limited number of corrupted databases to try this on.

Thanks.

Dave
David Hess | 3 Jun 20:17 2011

Prevent a persistent object from being ghosted?


NB: I don't have to worry about data invalidation due to aborts and conflicts - there's only one writer to this database so by design those cannot occur.

So, my question is: if I have a long lasting reference to a PersistentObject instance, is there a safe way for that object to veto being ghosted by the cache manager? It looks like reimplementing _p_set_status_ghost as a "pass" might work but there may be some not so obvious side-effects.

Dave

_______________________________________________
Durus-users mailing list
Durus-users <at> mems-exchange.org
http://mail.mems-exchange.org/mailman/listinfo/durus-users
Jesus Cea | 24 Sep 19:57 2010
Picon

Moving cache from object count to size


Durus objects, when in RAM, could keep the object size in a (volatile,
not stored in disk) attribute. This attribute can be generated when
loading the pickle from disk (you directly have the size), or when
storing the object in the disk (you have to create the pickle, so you
have the size too). In fact, this size bookkeeping could be managed in a
separate internal dictionary, don't have to be inside the object.

I have objects of very dissimilar sizes in my storage, so current cache
control (that is, object count) is not representative of actual memory
usage. I have objects 50 bytes long, and 60Kbytes long :-(. I would
suggest to change the cache code to control cache size, instead of
object count.

Comments?.

PS: I could consider patching this myself, if durus developers are
interested but not spare time.

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
..                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Jesus Cea | 23 Sep 19:02 2010
Picon

My wishlist for Durus 3.8 (20070503)


This is a draft I wrote three years ago. Maybe can be useful.

I would like durus to be more "community driven".

"""
This is a preview version.

As ever, I'm ready to help to implement this, if you ask.

* "Factorize" the durus server socket management (in particular, socket
creation for incomming connections and socket "select") to be able to
reuse the server code in other communication media, like shared memory,
intraprocess queues or mmaped files.

* "connection" objects should provide a method to query how much
accumulated idle time was spent waiting for the storage.

* A precompiled DURUS distribution for Windows users. Please!. Durus
programmers can't do it. Any other gentle soul able to provide this
service?. Durus deserves it!.

* When getting a "late conflict", the client should get the *real* OID
list of the conflicting objects. Since an storage implementation could
choose to only report conflicts in a "late" way, this check should be
done even for "no changed objects" commits.

* Be able to raise a "read only" exception when a client requests new
OIDs or try to commit with objects changed. This would allow for
read-only connections without setting the entire storage "read-only".
Also currently, if an storage is read-only, clients will be disconnected
when trying to commit changes, with no real indication of the problem.

* New "gen_oid_record" implementation in "Storage" class breaks
encapsulation and percolates internal implementation details of FileStorage.

* Add a "connection.close()" method.

* Add a storage wrapper to be able to share any arbitrary storage
between multiple "connection()"'s. For example, you pass an storage
instance to a function and it gives you a builder object, related to
that storage. Calling that generator gives you a "mutexed" storage
instance wrapping the original storage.

 This way you can share a single filestorage between threads, for example.
"""

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

Neil Schemenauer | 15 Jul 03:47 2010

Speeding up packing, rsync

While waiting for a fairly big Durus DB to pack today, I remembered
an optimization idea I had but never had time to play with.

The layout of objects in the disk file makes a difference for
performance.  For best packing performance, the reads should be
sequential since that would be most efficient for IO scheduling.
Also, it would be nice if packing moved objects around a little as
possible in order to speed up rsyncing of DBs.  

I haven't had time to dig into it, but I suspect the relevant logic
is in file_storage.py:gen_oid_record().  It looks like references
are found using a depth first search.  Perhaps it would be better to
use breath-first (e.g. using collection.deque for efficiency).  One
way to approach this would be to have file_storage.py log file
offsets and examine how their distribution is affects by different
packing algorithms.

It's possible that optimizing the layout for packing would slow down
normal access.  Anyhow, I thought it was an interesting idea.

  Neil
Binger David | 25 Jun 22:54 2010

Re: apparently random bug in durus


On Jun 25, 2010, at 4:30 PM, Alex Hall wrote:

> On 6/25/10, Binger David <dbinger <at> mems-exchange.org> wrote:
>> Please edit c:\python26\lib\site-packages\durus\connection.py to get more
>> information.
>> In particular, insert at line 137, just before the klass = loads(data) line,
>> some lines that print or write to a file,
>> the values of sys.version, sys.path, sys.executable
>> You might also insert an explicit "import durus.persistent_dict" there
>> too, to see if that works as it should.
> I did, and got the usual error.
>> Also print out repr(data) while you are at it.
>> Perhaps some change to sys.path or a chdir occurs
>> that puts a directory containing a "durus" entry other
>> than the durus installation directory on your path
>> before the durus installation directory.
> Here is the output I got when printing the three sys attributes to a
> file. Hopefully the repr(data) one makes sense to you, because I have
> no idea what it means. BTW, thanks for continuing to pursue this.
> 
> path: c:\qwitter\src
> C:\Windows\system32\python26.zip
> c:\python26\DLLs
> c:\python26\lib
> c:\python26\lib\plat-win
> c:\python26\lib\lib-tk
> c:\python26
> c:\python26\lib\site-packages
> c:\python26\lib\site-packages\PIL
> c:\python26\lib\site-packages\pyHook
> c:\python26\lib\site-packages\win32
> c:\python26\lib\site-packages\win32\lib
> c:\python26\lib\site-packages\Pythonwin
> c:\python26\lib\site-packages\wx-2.8-msw-unicode

Do any of these directories have a surprise file or directory with a name starting
with "durus" in them?  

> 
> 
> Version: 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)]

I see you are using 2.6.5.  That seems good.

> 
> Executable: c:\python26\python.exe
> 
> Repr(data): '\x80\x02cdurus.durus.persistent_dict\nPersistentDict\nq\x01.x\x9c\xb5Z}`\\E\x11O\x8e\x92\xa6I\x9a\xb6iB\xcbw
> 
> ...

The string should start with '\x80\x02cdurus.persistent_dict\nPersistentDict\nq'
not
'\x80\x02cdurus.durus.persistent_dict\nPersistentDict\nq'.

I think this is the problem.  I'm not sure how it originates.
I noticed that qwitter imports Connection and FileStorage in an unusual way,
with "from " imports that are inside a function body.  I'm not sure if that has an
impact on this, but it is unusual.
It looks like, when the FileStorage was initialized, the root was instantiated
after an import that somehow obtained a "durus.durus.persistent_dict" module.
That's not good.

Your site-packages directory should have a durus directory that has
a 'persistent_dict.py" file directly in it.  Is that what you have?  Do you have
a durus directory inside a durus directory anywhere?

> 
>> I assume that nothing in qwitter monkey-patches durus, right?
> I am not sure what you mean by that?

Sorry to use jargon.  I mean it does not make run-time changes
to names in the durus module.  This seems like a pretty unlikely
possibility, but I thought I'd ask.

>> 
>> 
>> On Jun 21, 2010, at 2:40 PM, Alex Hall wrote:
>> 
>>> File "c:\qwitter\src\session\storage.py", line 22, in __init__
>>>   self.connection = Connection(self.store)
>>> File "c:\python26\lib\site-packages\durus\connection.py", line 61, in
>>> __init__
>>>   self.root = self.get(ROOT_OID)
>>> File "c:\python26\lib\site-packages\durus\connection.py", line 138, in
>>> get
>>>   klass = loads(data)
>>> ImportError: No module named durus.persistent_dict
>> 
>> 
> 
> 
> -- 
> Have a great day,
> Alex (msg sent from GMail website)
> mehgcap <at> gmail.com; http://www.facebook.com/mehgcap
Binger David | 22 Jun 05:13 2010

Fwd: Durus-users post from mehgcap <at> gmail.com requires approval


Begin forwarded message:

> From: Binger David <dbinger <at> mems-exchange.org>
> Date: June 21, 2010 11:11:35 PM EDT
> To: durus-users-owner <at> mems-exchange.org
> Subject: Re: Durus-users post from mehgcap <at> gmail.com requires approval
> 
> Thanks, Alex.
> 
> I was hoping to find something unusual in this, but it looks normal.
> The earlier traceback shows that a connection is opening a durus
> storage file, and it can't load the "durus.persistent_dict" class that 
> happens to be the class of the root object.  I'm having trouble 
> thinking of a reason this would happen, since durus.connection
> seems to load just fine.  
> 
> Perhaps you could remove the _persistent.pyd file to see if the
> qwitter code works without it.  I doubt if this will work, though, since
> _persistent.pyd seems to load fine when you start it by hand.
> 
> Do you have two versions of python installed, by any chance?
> Could it be that the python you run at the command line differs
> from the one used by qwitter?
> 
> Anyone else have ideas on this one?
> 
> 
> On Jun 21, 2010, at 6:58 PM, durus-users-owner <at> mems-exchange.org wrote:
> 
>> On 6/21/10, Binger David <dbinger <at> mems-exchange.org> wrote:
>>> Hi Alex,
>>> 
>>> Please start python with the -v flag,
>>> type "import durus.persistent_dict"
>>> and send me the output.
>> 
>> ...
> 
> 
>>>>> import durus.persistent_dict
>> # C:\python26\lib\encodings\cp437.pyc matches C:\python26\lib\encodings\cp437.py
>> 
>> import encodings.cp437 # precompiled from C:\python26\lib\encodings\cp437.pyc
>> import durus # directory C:\python26\lib\site-packages\durus
>> # C:\python26\lib\site-packages\durus\__init__.pyc matches C:\python26\lib\site-
>> packages\durus\__init__.py
>> import durus # precompiled from C:\python26\lib\site-packages\durus\__init__.pyc
>> 
>> # C:\python26\lib\site-packages\durus\persistent_dict.pyc matches C:\python26\li
>> b\site-packages\durus\persistent_dict.py
>> import durus.persistent_dict # precompiled from C:\python26\lib\site-packages\du
>> rus\persistent_dict.pyc
>> # C:\python26\lib\copy.pyc matches C:\python26\lib\copy.py
>> import copy # precompiled from C:\python26\lib\copy.pyc
>> # C:\python26\lib\site-packages\durus\persistent.pyc matches C:\python26\lib\sit
>> e-packages\durus\persistent.py
>> import durus.persistent # precompiled from C:\python26\lib\site-packages\durus\p
>> ersistent.pyc
>> # C:\python26\lib\site-packages\durus\utils.pyc matches C:\python26\lib\site-pac
>> kages\durus\utils.py
>> import durus.utils # precompiled from C:\python26\lib\site-packages\durus\utils.
>> pyc
>> # C:\python26\lib\struct.pyc matches C:\python26\lib\struct.py
>> import struct # precompiled from C:\python26\lib\struct.pyc
>> import _struct # builtin
>> import cStringIO # builtin
>> import cPickle # builtin
>> import durus._persistent # dynamically loaded from C:\python26\lib\site-packages
>> \durus\_persistent.pyd
>>>>> 
>> 
>> I am not sure what all this means, but there seems to be no problem
>> doing it this way. Doing the same thing without the -v switch does not
>> throw any errors either...
> 

Gmane