Paul L Merchant Jr. | 24 Apr 19:59 2014

Very slow attribute

Hi everyone, I’m looking for some advice with a performance problem with what I think is a very simple
XQuery expression in eXist.  I’m having this problem with versions 2.1 and 2.2RC1.

I have narrowed the problem case down to a single XML document with a hundred records in it.  Within each
record is a list of names of people encoded like this:

<persName>
<surname>…..</surname>
<forename type=“first”>….</forename>
</persName>

I want to retrieve the names sorted in alphabetical order by last name first, so I’m using this FLOWR
expression similar to this:

for $name in doc(‘doc.xml’)//persName
order by
	$name/surname,
	$name/forename[ <at> type=‘first’]
return 
	$name

The problem I’m seeing is that the [ <at> type=‘first’] qualifier on forename dramatically slows down
the code.  With that qualifier, the expression takes 4 to 40 times as long to execute as without it.

Now this is a contrived example because obviously the type attribute isn’t needed.  The original data is
slightly (but only slightly) more complicated in that persName records have multiple forenames for
first and middle names, so the attribute is essential. 

In 2.2RC1, I have created a new style range index on the type attribute, and I see nothing in the query profile
to suggest that the index isn’t being used:
(Continue reading)

W.S. Hager | 24 Apr 17:58 2014
Picon

usesr don't stay logged in

Hi,

This problem has been driving me mad for a long time. When I create a login (exactly like the one in eXide/controller) and I login, I don't stay logged in. At some point the session will just expire. There is one minor difference, that is that I set the path to /, because I want the entire domain to be logged in. I can see that in the cookie, the path is set to /exist, but that is incorrect. It should be /.

Could this please be fixed?

Thanks.

Wouter
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
ingo.mattei | 23 Apr 16:12 2014
Picon

New Range index gone

Dear list,

I don't know why, but my index-xconfs do not work on the "official" RC2.2.

<collection xmlns="http://exist-db.org/collection-config/1.0">
   <index xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:myns="http://myns.org">
       <fulltext default="none" attributes="false"/>
       <lucene>
           <analyzer class="org.apache.lucene.analysis.standard.KeywordAnalyzer"/>
           <text match="//myns:person"/>
           <text match="//myns:company"/>
       </lucene>
       <range>
           <create qname="myns:title" type="xs:string"/>
           <create qname="myns:date" type="xs:string"/>
           <create qname=" <at> id" type="xs:string"/>
           <create qname=" <at> anAttribute" type="xs:string"/>
       </range>
   </index>
</collection>  

I can see the lucene indexes in the Admin Web Application and use them in xqueries, but not New Range indexes. 

Thanks for any help.

ingom
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Chris Tomlinson | 23 Apr 04:43 2014
Picon

sm:get-user-groups not reporting groups other than guest?!

Hello,

We had a corruption in the eXist-db and restored from the a sequence of incrementals and a full from early this morning. When the database came back up we discovered that various tools are reporting that users are not members of any groups other than guest and yet when the userName.xml in /db/system/security/exist/accounts is interrogated we find the expected group memberships.

Calling sm:get-user-groups("userName") just reports "guest" if the user is a member of group guest or nothing otherwise even if they are a member of other groups.

The system had been running for 14 days and I simply did a clean shutdown and a restart which failed leading to the restore. The failure was like:

2014-04-22 18:04:31,399 [localhost-startStop-1] ERROR (NativeBroker.java [indexInfo]:3977) - IOException while reading document data 
java.io.IOException: no data found at tid 48; page 990
at org.exist.storage.index.BFile.getAsStream(BFile.java:505)
at org.exist.storage.index.BFile.getAsStream(BFile.java:496)
at org.exist.storage.NativeBroker$DocumentCallback.indexInfo(NativeBroker.java:3963)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:402)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:1879)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1675)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1599)
...
2014-04-22 18:04:31,464 [localhost-startStop-1] ERROR (NativeBroker.java [openCollection]:957) - no data found at tid 14; page 1 
java.io.IOException: no data found at tid 14; page 1
at org.exist.storage.index.BFile.getAsStream(BFile.java:505)
at org.exist.storage.index.BFile.getAsStream(BFile.java:475)
at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:938)
at org.exist.storage.NativeBroker.openCollection(NativeBroker.java:815)
at org.exist.storage.NativeBroker.getCollection(NativeBroker.java:810)
...
2014-04-22 18:04:31,468 [localhost-startStop-1] INFO  (Journal.java [shutdown]:484) - Transaction journal cleanly shutting down with checkpoint... 
2014-04-22 18:04:31,468 [localhost-startStop-1] INFO  (FileLock.java [release]:189) - Deleting lock file: /Volumes/Data/indium/exist/apache-tomcat-7.0.32/webapps/exist/WEB-INF/data/journal.lck 
2014-04-22 18:04:31,494 [localhost-startStop-1] INFO  (FileLock.java [release]:189) - Deleting lock file: /Volumes/Data/indium/exist/apache-tomcat-7.0.32/webapps/exist/WEB-INF/data/dbx_dir.lck 
2014-04-22 18:04:31,494 [localhost-startStop-1] ERROR (BrokerPool.java [configure]:267) - Unable to initialize database instance 'exist': java.io.IOException: Collection /db/system/security/exist/groups/removed cannot be created. 
org.exist.EXistException: java.io.IOException: Collection /db/system/security/exist/groups/removed cannot be created.
at org.exist.security.AbstractRealm.initialiseRealmStorage(AbstractRealm.java:112)
at org.exist.security.AbstractRealm.start(AbstractRealm.java:237)
at org.exist.security.internal.RealmImpl.start(RealmImpl.java:137)

which persisted even after following the RECOVERY procedure.

The system information is:

Java
eXist Version: 2.1
eXist Build: 20130719
eXist Home: /Volumes/Data/indium/exist/apache-tomcat-7.0.32/webapps/exist/WEB-INF
SVN Revision: 18724
Operating System: Mac OS X 10.8.5 x86_64
File encoding: UTF-8
Vendor: Oracle Corporation
Version: 1.7.0_51
Implementation: Java HotSpot(TM) 64-Bit Server VM
Installation: /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre

I haven't tried shutting down and restarting the database since it seems to be sort of running but morning will tell how the users are affected.

Any ideas about what is happening and what remedies are appreciated.

Thanks,
Chris



------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Markus Breuer | 22 Apr 08:23 2014
Picon
Picon

exist-db 2.1 crash recovery

I am evaluating exist-db with a large set of data, about 60GB in 4-5 million xml documents. Up to now I was not able to import the whole 60GB because several issues had broken the database. There are 8gb ram assigned to the database and databaseCache is set to 2048M. A small java application sequencually inserts xml documents.
 
1. Up to 200.000 documents exist-db is really fast, when exceeding this limit the throughput significantly breaks in. Memory assignments and cache settings seems not to have any impact to this.
 
2. The db shutdown is a cruelly process. Even when using the shutdown script, the next db start will start a long running recovery. After importing 4 million documents the data directory used about 120GB and the recovery (redo) is running since 24 hours (not finished yet)
 
3. Browsing through web admin interface selection the "indexes" page blocks the whole database. Neither the web frontend nor the xmlrpc client is working for a long time. The db process does not respond to any interaction.
 
questions:
1. How to reduce crash recovery times? Probably my fingers are clumsy, but during testing the database often restarted in crash recovery mode, where it used several hours/days to finish the recovery process.
 
2. The tuning section describes some basic setting, e.g. caches, to improve performance. But when running a "wrong" xquery the whole db seems to be affected, it's try and error to kill such running job. And killing the database brings me in crash recovery issues.
 
3. The backup process looks like a flat file export. When running under windows the db accepts upper-/lower-case collections. But during export the uc/lc information are lost.
 
4. Finally the essential question: how many data can exist-db handle? It seems with 60gb there a limit for practical usage reached, isn't it?
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Paul Vanderveen | 17 Apr 22:40 2014

ACL's with ACE's for Groups that were removed

We noticed that when you remove a group that any ACE’s that had the group listed as a subject of the ACE are still active. 

This leaves orphaned ACE’s in our resource ACL’s, which sometimes looks odd. 

 

I have 2 questions:

1.       Are there any plans down the road to remove ACE’s for non-existent groups?

2.       Is there any way to run a query that will return all ACE’s where the ACE subject is a particular group or user?

 

We would like to run #2 when we remove a group from our application, then we could go through and remove the affected ACE’s.

 

Paul Vanderveen

TerraXML, Inc.

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Dan McCreary | 17 Apr 01:31 2014
Picon

Changes between eXist 2.0 and 2.1 that impact XSL-FO printing of LaTeX equations on Macs

Hi folks,

I am requesting some help debugging why XSL-FO equations are not working in some cases in eXist 2.1 on the Mac.

The problem seems to have occurred between eXist 2.0 and eXist 2.1.  On some systems (only Macs so far) the feature that allows us to render LaTeX equations within XSL-FO files seems to have stopped working.  We have a simple unit test that usually renders a LaTeX equation that no longer works.

What is odd is that it DOES work on eXist 2.1 on Windows, and it DOES work on eXist 2.0 on all systems that we are testing (mac, unix and windows), but it does NOT work on Mac systems that are running eXist 2.1.

I realize that most of you are not using eXist for typesetting equations, but it is really cool feature and if you would like to try it out you can add two jar files to your $EXIST_HOME/lib/extensions:


and run the XQuery unit test here:


You can copy this unit test file into /db/tmp/render-equation-in-pdf.xq

If you system is working you should get a equation in a PDF file.

Can anyone suggest any reason why this would stop working but only on some systems?

I have compared the Java versions and we have the equation rendering working on both 1.6 and 1.7.  I have also checked that the fop.jar file has remained unchanged.

Thanks for any help!

- Dan

--
Dan McCreary
http://danmccreary.com
Co-author: Making Sense of NoSQL
office: (952) 931-9198
cell: (612) 986-1552
skype: dmccreary47
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Guillaume Mella | 16 Apr 17:13 2014
Picon
Picon

Limit duration of http requests applying timeout

Hi,

I need to limit duration of multiples http get requests but did not find 
any solution.

Httpclient module does not seem to provide configuration API but 
http://expath.org/spec/http-client specs refers to timeout.

Here comes my code ( executed from eXide ) which does not responds 
properly :(
http:send-request(<http:request href="http://no.rep.ly/" method="get" 
timeout="1"/>)

Do you know any way to limit the duration of http request (and why not 
any other kind of process) ?

Thanks in advance,
Regards,
--
Guillaume

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
Misztur, Chris | 16 Apr 02:17 2014

Generate barcode with ZXing

Generating barcode with ZXing.
(include core.jar and javase.jar)


xquery version "3.0";

module namespace bc="http://haptixgames.com/barcode-test";

declare namespace java-zxing-BitMatrix = "java:com.google.zxing.common.BitMatrix";
declare namespace java-zxing-Code128Writer = "java:com.google.zxing.oned.Code128Writer";
declare namespace java-zxing-BarcodeFormat = "java:com.google.zxing.BarcodeFormat";
declare namespace java-zxing-MatrixToImageWriter = "java:com.google.zxing.client.j2se.MatrixToImageWriter";
declare namespace java-io-ByteArrayOutputStream = "java:java.io.ByteArrayOutputStream";
declare namespace javax-imageio-ImageIO = "java:javax.imageio.ImageIO";

declare 
    %rest:GET
    %rest:path("/bc/barcode1") 
    (:%output:method("binary")
    %output:media-type("image/png"):)
function bc:barcode1()
{
    let $m_code128Writer := java-zxing-Code128Writer:new()
    let $m_bitMatrix := java-zxing-Code128Writer:encode($m_code128Writer, 'Hello World', java-zxing-BarcodeFormat:valueOf("CODE_128") , 440, 48, ())
    let $m_bufferedImage := java-zxing-MatrixToImageWriter:toBufferedImage($m_bitMatrix)
    let $m_outputStream := java-io-ByteArrayOutputStream:new()
    let $m_imageio := javax-imageio-ImageIO:write($m_bufferedImage, 'png', $m_outputStream)
    
    let $v_flush := java-io-ByteArrayOutputStream:flush($m_outputStream)
    let $m_imageBytes := java-io-ByteArrayOutputStream:toByteArray($m_outputStream)
    let $v_close := java-io-ByteArrayOutputStream:close($m_outputStream)
    
    (:return 
    (
        <rest:response> 
            <http:response status="200"> 
                <http:header name="Content-Language" value="en"/> 
                <http:header name="Content-Type" value="image/png"/> 
            </http:response> 
        </rest:response>, 
        $m_imageBytes
    ):)
    
    let $resource := xmldb:store('/db/apps/cmisztur', util:uuid() || '.png', $m_imageBytes, 'image/png')
    
    return 
    (
        <rest:response> 
            <http:response status="200"> 
                <http:header name="Content-Type" value="text/plain"/> 
            </http:response> 
        </rest:response>, 
        'http://server:8080/exist/rest'||$resource
    )
};



The contents of this message may be privileged and confidential. Therefore, if this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author.

Please consider the environment before printing this e-mail

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Misztur, Chris | 16 Apr 02:11 2014

RESTXQ : output image/png

Hello.

I want to return 'image/png' from my restXQ function and I found this discussion [1].
I see these fixes are pre 18721, but adding the following serialization options removes the resource function from the RestXQ registry.

Am I missing something?

 %output:method("binary")
 %output:media-type("image/png")



-c


The contents of this message may be privileged and confidential. Therefore, if this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author.

Please consider the environment before printing this e-mail

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Misztur, Chris | 15 Apr 21:38 2014

Barcode4J

Is anyone using Barcode4J [1]  from eXist?

 

I tried messing around by calling Java from XQuery but I was wondering if the Saxon implementation would be the correct one to use.

 

[1] http://barcode4j.sourceforge.net/2.1/

 

-c



The contents of this message may be privileged and confidential. Therefore, if this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author.

Please consider the environment before printing this e-mail

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open

Gmane