Re: Performance Problems using List and Pickle properties
So here is the code to reproduce the problem :
from storm.base import Storm
from storm import properties
from storm.database import create_database
from storm.store import Store
from storm.variables import MutableValueVariable
class WithList(Storm):
__storm_table__ = 'withlist'
def __init__(self, key):
self.key = key
key = properties.Int(primary = True)
v = properties.Int()
l = properties.List(
type=properties.Unicode(),
default_factory=list
)
detectCount = 0
def performanceTest():
global detectCount
# patch to see how often _detect_changes is called
original_detect_changes = MutableValueVariable._detect_changes
def patch_detect_changes(self, obj_info):
global detectCount
detectCount += 1
original_detect_changes(self, obj_info)
MutableValueVariable._detect_changes = patch_detect_changes
database = create_database('use a database supporting lists')
store = Store(database)
try:
store.execute("DROP TABLE withlist;")
except :
store.rollback()
store.execute("""
CREATE TABLE withlist(
key int primary key,
v int,
l text[]
);
""")
n = store.add(WithList(1))
del n
store.commit()
for i in range(10):
n = store.get(WithList, 1)
# access the list
n.l
# remove strong reference so the object is removed from the _alive
# WeakValueDictionary in the store
del n
store.commit()
print 'detect calls =', detectCount
detectCount = 0
The output will be :
detect calls = 3
detect calls = 4
detect calls = 6
detect calls = 8
detect calls = 10
detect calls = 12
detect calls = 14
detect calls = 16
detect calls = 18
detect calls = 20
Jürgen
On Sat, Oct 3, 2009 at 1:59 PM, Jürgen Kartnaller
<kartnaller-H84adUgip2a/3pe1ocb+swC/G2K4zDHf@public.gmane.org> wrote:
The problem exists in 0.14 and 0.15
I tried to reproduce the problem with a simple example without success :(
Jürgen
On Fri, Oct 2, 2009 at 10:16 PM, Jamu Kakar
<jkakar-kvr/BIsI4Rg@public.gmane.org> wrote:
Hi Jürgen,
On Fri, Oct 2, 2009 at 11:56 AM, Jürgen Kartnaller
<
kartnaller-H84adUgip2a/3pe1ocb+swC/G2K4zDHf@public.gmane.org> wrote:
> After a long time having problems with performance going down in our
> Zope/Storm applications I could now figure out where it is coming from. In
> my latest app using tornado and Storm I have the same problem.
>
> When using List or Pickle properties (both are derived from
> MutableValueVariable) I found that any time a property is read a flush event
> hook is added but never removed. The longer the app runs the more hooks are
> registered which result in low perfomance in store.flush.
> The flush event hook is unhooked on the 'stop-tracking-changes' event which
> is only emited if an object is removed from the database.
Are you using the latest version of Storm? I remember a problem
like this being fixed some time ago. I think this is the fix for
the problem I'm thinking of:
revno: 275 [merge]
committer: Thomas Hervé <thomas-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org>
branch nick: trunk
timestamp: Mon 2008-11-03 20:38:06 +0100
message:
Merge variable-referenceset-leak [r=niemeyer,jkakar]
Change the way MutableValueVariable hooks up to the store event system, by
overriding get and set. This works around a leak discovered when using
PickleVariable in a ReferenceSet.
Thanks,
J.
--
Jürgen Kartnaller, senior developer
Lovely Systems AG
Telefon +43 5572 908060, Fax +43 5572 908060-77
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705
Aufsichtsratsvorsitzender: Christian Lutz
Vorstand: Manfred Schwendinger
--
Jürgen Kartnaller, senior developer
Lovely Systems AG
Telefon +43 5572 908060, Fax +43 5572 908060-77
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705
Aufsichtsratsvorsitzender: Christian Lutz
Vorstand: Manfred Schwendinger
<div>
<p>So here is the code to reproduce the problem :<br><br><span>from storm.base import Storm</span><br><span>from storm import properties</span><br><span>from storm.database import create_database</span><br><span>from storm.store import Store</span><br><span>from storm.variables import MutableValueVariable</span><br><br><span>class WithList(Storm):</span><br><span> __storm_table__ = 'withlist'</span><br><br><span> def __init__(self, key):</span><br><span> self.key = key</span><br><br><span> key = properties.Int(primary = True)</span><br><br><span> v = properties.Int()</span><br><span> l = properties.List(</span><br><span> type=properties.Unicode(),</span><br><span> default_factory=list</span><br><span> )</span><br><br><span>detectCount = 0</span><br><br><span>def performanceTest():</span><br><span> global detectCount</span><br><br><span> # patch to see how often _detect_changes is called</span><br><span> original_detect_changes = MutableValueVariable._detect_changes</span><br><span> def patch_detect_changes(self, obj_info):</span><br><span> global detectCount</span><br><span> detectCount += 1</span><br><span> original_detect_changes(self, obj_info)</span><br><span> MutableValueVariable._detect_changes = patch_detect_changes</span><br><br><span> database = create_database('use a database supporting lists')</span><br><span> store = Store(database)</span><br><span> try:</span><br><span> store.execute("DROP TABLE withlist;")</span><br><span> except :</span><br><span> store.rollback()</span><br><span> store.execute("""</span><br><span> CREATE TABLE withlist(</span><br><span> key int primary key,</span><br><span> v int,</span><br><span> l text[]</span><br><span> );</span><br><span> """)</span><br><br><span> n = store.add(WithList(1))</span><br><span> del n</span><br><span> store.commit()</span><br><br><span> for i in range(10):</span><br><span> n = store.get(WithList, 1)</span><br><span> # access the list</span><br><span> n.l</span><br><span> # remove strong reference so the object is removed from the _alive</span><br><span> # WeakValueDictionary in the store</span><br><span> del n</span><br><span> store.commit()</span><br><span> print 'detect calls =', detectCount</span><br><span> detectCount = 0</span><br><br>The output will be :<br><br><span>detect calls = 3</span><br><span>detect calls = 4</span><br><span>detect calls = 6</span><br><span>detect calls = 8</span><br><span>detect calls = 10</span><br><span>detect calls = 12</span><br><span>detect calls = 14</span><br><span>detect calls = 16</span><br><span>detect calls = 18</span><br><span>detect calls = 20</span><br><br>Jürgen<br><br><br></p>
<div class="gmail_quote">On Sat, Oct 3, 2009 at 1:59 PM, Jürgen Kartnaller <span dir="ltr"><<a href="mailto:kartnaller@...">kartnaller@...</a>></span> wrote:<br><blockquote class="gmail_quote">The problem exists in 0.14 and 0.15<br><br>I tried to reproduce the problem with a simple example without success :(<br><br>Jürgen<div>
<div></div>
<div class="h5">
<br><br><div class="gmail_quote">On Fri, Oct 2, 2009 at 10:16 PM, Jamu Kakar <span dir="ltr"><<a href="mailto:jkakar@..." target="_blank">jkakar@...</a>></span> wrote:<br><blockquote class="gmail_quote">Hi Jürgen,<br><div>
<br>
On Fri, Oct 2, 2009 at 11:56 AM, Jürgen Kartnaller<br>
<<a href="mailto:kartnaller@..." target="_blank">kartnaller@...</a>> wrote:<br>
> After a long time having problems with performance going down in our<br>
> Zope/Storm applications I could now figure out where it is coming from. In<br>
> my latest app using tornado and Storm I have the same problem.<br>
><br>
> When using List or Pickle properties (both are derived from<br>
> MutableValueVariable) I found that any time a property is read a flush event<br>
> hook is added but never removed. The longer the app runs the more hooks are<br>
> registered which result in low perfomance in store.flush.<br>
> The flush event hook is unhooked on the 'stop-tracking-changes' event which<br>
> is only emited if an object is removed from the database.<br><br>
</div>Are you using the latest version of Storm? I remember a problem<br>
like this being fixed some time ago. I think this is the fix for<br>
the problem I'm thinking of:<br><br>
revno: 275 [merge]<br>
committer: Thomas Hervé <<a href="mailto:thomas@..." target="_blank">thomas@...</a>><br>
branch nick: trunk<br>
timestamp: Mon 2008-11-03 20:38:06 +0100<br>
message:<br>
Merge variable-referenceset-leak [r=niemeyer,jkakar]<br><br>
Change the way MutableValueVariable hooks up to the store event system, by<br>
overriding get and set. This works around a leak discovered when using<br>
PickleVariable in a ReferenceSet.<br><br>
Thanks,<br>J.<br>
</blockquote>
</div>
<br><br clear="all"><br>
</div>
</div>
<div class="im">-- <br>Jürgen Kartnaller, senior developer<br><br>Lovely Systems AG<br>Telefon +43 5572 908060, Fax +43 5572 908060-77<br>Schmelzhütterstraße 26a, 6850 Dornbirn, Austria<br><br>
</div>Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705<br>Aufsichtsratsvorsitzender: Christian Lutz<br>Vorstand: Manfred Schwendinger<br>
</blockquote>
</div>
<br><br clear="all"><br>-- <br>Jürgen Kartnaller, senior developer<br><br>Lovely Systems AG<br>Telefon +43 5572 908060, Fax +43 5572 908060-77<br>Schmelzhütterstraße 26a, 6850 Dornbirn, Austria<br><br>
Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705<br>Aufsichtsratsvorsitzender: Christian Lutz<br>Vorstand: Manfred Schwendinger<br>
</div>
RSS Feed