Re: [PATCH] XQueryGenerator cacheable

Hi,

> Please find attached a patch that allows XQueryGenerator to be
> cacheable by setting an optional expiration delay: cache-validity

Thanks for the patch! I just committed it to cvs.

Wolfgang

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf

Re: [PATCH] XQueryGenerator cacheable

* Jean-Baptiste Quenot:

> Please  find  attached  a   patch  that  allows  XQueryGenerator
> to  be  cacheable  by  setting  an  optional  expiration  delay:
> cache-validity

New patch attached: when  XQueryGenerator is invoked via cocoon://
the source validity could be null
-- 
Jean-Baptiste Quenot
Systèmes d'Information
ANYWARE TECHNOLOGIES
Tel : +33 (0)5 61 00 52 90
Fax : +33 (0)5 61 00 51 46
http://www.anyware-tech.com/

--- /local-home/jbq/var/repos/eXist-1.0/src/org/exist/cocoon/XQueryGenerator.java	Wed Aug 31
19:48:01 2005
+++ org/exist/cocoon/XQueryGenerator.java	Thu Sep  1 11:22:21 2005
 <at>  <at>  -18,12 +18,13  <at>  <at> 
  *  along with this program; if not, write to the Free Software
  *  Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
  *  
- *  $Id: XQueryGenerator.java,v 1.28 2005/07/12 17:16:09 wolfgang_m Exp $
+ *  $Id: XQueryGenerator.java,v 1.5 2005/08/22 10:30:01 jbquenot Exp $
  */
 package org.exist.cocoon;

 import java.io.File;

Re: Performance question

Hi,

>         I'm working with eXist since almost two years, and I'm using it in
> bioinformatics. We are starting a project where we are going to deal
> with huge volumes of XML content (my test documents have 3GB!), and we
> are going to publish little query results under request using a web
> service. Most of the XML schemas we are using were designed like a list
> of records, so content can be put in a single file or it can be
> scattered over thousands of files (10000~20000). So, my question is:
> taking into account current eXist implementation, is it better to store
> in an eXist database a single huge XML document or to store lots of
> them, but smaller? I'm mainly thinking on query speed, which is very
> important for us.

Many small documents usually have advantages if you need to update
data. Also, there's still an internal limit to the size of a single
document, but it depends on the structure of the data. For query
speed, it doesn't really make a difference if you have many small or
one large doc.

If your documents have a record-like structure, your query probably
needs to process huge node sets (maybe a few million nodes). eXist
does currently need quite a lot of memory to hold these node sets, so
it would definitely be an advantage if you can restrict your queries
to certain logical parts of the data set and organize your files and
collections accordingly. For the future, we hope to have a
query-rewriting optimizer that can optimize selections on huge node
sets automatically.

Wolfgang

Application hangs tomcat (possibly a deadlock?)

Hi, i'm back with an error who is driving us to madness.

We've a web application, running inside tomcat, based upon exist.

After some times the application is running, it starts to slow down,  
and at last hangs.

Launching a kill -3 and then stopping the tomcat service and watching  
the catalina.out we found those messages:

At the beginning there are a lot of messages like:

"Thread-6433" daemon prio=1 tid=0x8fe76280 nid=0x2fa8 runnable  
[0x8e743000..0x8e744558]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
     at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
     - locked <0x5031ee90> (a java.io.BufferedInputStream)
     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:606)
     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:554)
     at sun.net.www.protocol.http.HttpURLConnection.getInputStream 
(HttpURLConnection.java:936)
     - locked <0x50316770> (a  
sun.net.www.protocol.http.HttpURLConnection)
     at org.apache.xmlrpc.XmlRpcClient$Worker.execute 
(XmlRpcClient.java:435)
     at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)
     at org.exist.xmldb.RemoteCollection.readCollection 

Re: Application hangs tomcat (possibly a deadlock?)

Hi,

> After some times the application is running, it starts to slow down,
> and at last hangs.

If you are using one of the August snapshots: there's a known deadlock bug 
which I'm currently trying to fix. On the other hand, your code seems to 
connect to a remote eXist instance, so I'm not sure if the deadlock appears 
within eXist or within the xmldb client code? Does the eXist server still 
respond to anything?

Wolfgang

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf

Re: Trigger code refactoring issues

Hi Piotr,

> Another issue I've noticed is that any given event will only invoke one (the
> most specific) trigger.  Is this by design?

No, I think the design can be extended to support chains of triggers.
The main work would be to extend the configuration and pipe the SAX
stream through the triggers. I just started with a single trigger to
make things simpler.

> Also, what was the rationale
> for having persistent (per-database) trigger registrations rather than
> transient (per-application) ones?

I think persistent triggers are very handy, at least, if they are
general triggers like the history trigger. It's quite some time back
that I used an rdbms, but as far as I remember, I also registered my
sql triggers per database, not per application. After all, the trigger
is running within the database process, not within the application.
There are too many interfaces a client could use to talk to the
database, so how would you associate a trigger with a certain client?
But I might be missing the idea ...

> This seems like a better design since it gives the Collection control over
> the whole storage sequence (no chance of unbalanced validate/store calls)
> and eliminates the need for an externally passed memento (IndexInfo).  Take
> a look at the validateInternal and storeInternal methods in my newly
> committed code to see another application of this "parametrize what to do in
> the middle" pattern.

numeric predicates revisited

Coming back to the problem of numeric attribute values, I have revised and
rerun my earlier test, using a current CVS build.

To recap, the problem is that abbreviated-form numeric predicate values [N]
are incorrectly interpreted, whereas the unabbreviated form [position()=N]
is handled correctly.

The results indicate clearly enough what is happening, though not why.

A query using the position() function correctly traverses the tree in
document order, but the [n] version first traverses all the sibling nodes at
a given level, and only then moves on to count nodes as the next level of
the hierarchy.

So, with the Shakespeare samples in the docbase, the following query

let $results :=
for $i in (1 to count(document("/db/shakespeare/plays/hamlet.xml")//TITLE))
let $node1 :=
document("/db/shakespeare/plays/hamlet.xml")//TITLE[position()=$i]
let $node2 := document("/db/shakespeare/plays/hamlet.xml")//TITLE[$i]
return <result pos="{$i}">{$node1}{$node2}</result>
return <results>{$results}</results>

produces this
=======================
<results>
    <result pos="1">
        <TITLE>The Tragedy of Hamlet, Prince of Denmark</TITLE>
        <TITLE>The Tragedy of Hamlet, Prince of Denmark</TITLE>

Re: Application hangs tomcat (possibly a deadlock?)

On 01/set/05, at 21:21, Wolfgang Meier wrote:

> If you are using one of the August snapshots: there's a known  
> deadlock bug
> which I'm currently trying to fix.
>

We're using 20050620, now we're stress testing version 20050820 but  
we've not still encountered any problem.

> On the other hand, your code seems to
> connect to a remote eXist instance, so I'm not sure if the deadlock  
> appears
> within eXist or within the xmldb client code? Does the eXist server  
> still
> respond to anything?
>

Exist runs as a webapps inside the same tomcat instance as the  
application, and when the application hangs, tomcat stops to respond,  
so we cannot try to reach exist or any other application within  
tomcat (the tomcat manager also, i.e.)

> Wolfgang
>

thanks,

best regards,

Locks question

Hi,

During a tomcat hang, I managed to get a Java thread dump which  
contains references to EDU.oswego.cs.dl.util.concurrent classes.
I found this is an optional dependency used by cocoon.
We never use the exist web interface, so I think cocoon was never  
called during this session (but I can't be 100% sure).

Do somebody think this locks can lead to a tomcat slow-down and hang?

We use exist snapshot 20050805, tomcat 5.0.19, java 1.5.

At the end there are some exist thread in a state I can't  
understand... it looks like they wait on a lock created by themselves?

"Thread-2253" prio=1 tid=0x8e021928 nid=0x2fad in Object.wait()  
[0x8ff05000..0x8ff05758]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x4be16a20> (a java.lang.Object)
         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll(Unknown  
Source)
         - locked <0x4be16a20> (a java.lang.Object)
         at org.apache.cocoon.components.thread.ChannelWrapper.poll 
(ChannelWrapper.java:72)
         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask 
(Unknown Source)
         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run 
(Unknown Source)
         at java.lang.Thread.run(Thread.java:595)

Re: Locks question

I've done another couple of test and noticed that those thread are  
always present even on a just started instance.
So I think it's a normal situation.

BTW I can't understand the presence of  
EDU.oswego.cs.dl.util.concurrent threads.

On 5 Sep, 2005, at 17:21 , Mauro Piccini wrote:

> Hi,
>
> During a tomcat hang, I managed to get a Java thread dump which  
> contains references to EDU.oswego.cs.dl.util.concurrent classes.
> I found this is an optional dependency used by cocoon.
> We never use the exist web interface, so I think cocoon was never  
> called during this session (but I can't be 100% sure).
>
> Do somebody think this locks can lead to a tomcat slow-down and hang?
>
> We use exist snapshot 20050805, tomcat 5.0.19, java 1.5.
>
> At the end there are some exist thread in a state I can't  
> understand... it looks like they wait on a lock created by themselves?
>
>
> "Thread-2253" prio=1 tid=0x8e021928 nid=0x2fad in Object.wait()  
> [0x8ff05000..0x8ff05758]
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x4be16a20> (a java.lang.Object)
>         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll