headers
David Lee | 1 Jun 2011 02:20

GET of query files

Using REST I've been able to PUT a query file of Content-Type  application/xquery

But now I cannot get it out of the server (using GET).

Reading the docs, even when I specify _source=yes  I'm unable to get the file without editing the global descriptor.xml file to explicitly allow it.

 

But using Java admin Webstart client I CAN get the file.

What's the trick ?  What can it do that I can't using REST/GET ? and how ?

 

I understand the desire for security but if I'm a user that has the permissions to write (PUT) a file, surely I should be able to GET it back ?

 

Thanks for any suggestions.

 

 

----------------------------------------

David A. Lee

dlee <at> calldei.com

http://www.xmlsh.org

 

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Permalink | Reply | 2 comments |
headers
Adam Retter | 1 Jun 2011 02:52

Re: Stop XQuery caching

>Some sort of configuration was what I
> was looking for. The only way I could find to ensure that all the included
> queries were compiled was to change the parent query.

The 'query-pool' settings in conf.xml should allow you to disable
caching in your dev environment should you wish...

--

-- 
Adam Retter

eXist Developer
{ United Kingdom }
adam <at> exist-db.org
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 1 comment |
headers
Patrick Bosek | 1 Jun 2011 02:53

Re: When to release a broker

Hi Adam,

Thanks for the reply. This is interesting. Generally when I use a broker I'm performing a couple of operations, but not a lot. If getting the broker isn't expensive (and from looking at the code, I'd guess it isn't) this could be a good option. I actually come up with another solution a few hours that seems to be working pretty well. But first let me try to clarify my user issue...

When running this code:

let $before := util:log-system-out(concat("User before in X :: ", xmldb:get-current-user()))

let $na := java-module:do-something-as-admin( .. )

let $after := util:log-system-out(concat("User after in X :: ", xmldb:get-current-user()))


Java function "do-somethig-as-admin":

        BrokerPool pool;
        NativeBroker broker = null;
        try {
            pool = BrokerPool.getInstance();

            org.exist.security.SecurityManager security = pool.getSecurityManager();
            User asUser = security.getUser(org.exist.security.SecurityManager.DBA_USER);
           
            broker = (NativeBroker)pool.get(null);

           
            ** Stuff being done **
           
            }
            catch( .. ){ .. }
            finally{
           
            broker.getBrokerPool().release(broker);
            }
          

I end up with output that says this:

User before in X :: *me*
User after in X :: admin

So, for the rest of the execution the user is admin. I've determined this is because the BrokerPool reuses DBBrokers, and when the broker is "released" the user is still associated with it, so the next time around, unless it's specifically set back to the current user, it stays what ever it was previously (or at least I'm pretty sure this is what's happening). Oddly enough, this may actually explain another (fairly rare) issue I've run into, where one logged in user opens a document and it gets locked by another user who is also logged in.

Anyway, I've solved this issue for our module's code in the following way:

We always use our wrapper functions for getting and releasing brokers, and they are

class myUtil {

private static HashMap<NativeBroker, User> brokersInUse = new HashMap<NativeBroker, User>();

...

    public static NativeBroker getBrokerAsUser(String username)
    {
        BrokerPool pool;
        NativeBroker broker = null;
        try {
            pool = BrokerPool.getInstance();

            org.exist.security.SecurityManager security = pool.getSecurityManager();
            User asUser = security.getUser(username);
           
            broker = (NativeBroker)pool.get(null);
           
            if(brokersInUse.containsKey(broker)) existLOG.error("Broker Already in USE!!!!! :: " + broker);
           
            User pastUser = broker.getUser();

            if(pastUser != null) brokersInUse.put(broker, pastUser);
           
            broker.setUser(asUser);
           
        } catch (EXistException e) {
            System.out.println("Exception in getBrokerAsUser: " + e);
            e.printStackTrace();
           
            return null;
        }
       
        return broker;
    }

    public static void brokerReleaseWrapper(DBBroker broker)
    {
        if(broker == null)
        {
            existLOG.info("broker is null, aborting");
            return;
        }

        if(brokersInUse.containsKey(broker))
        {
            User userToRestore = brokersInUse.get(broker);
           
            broker.setUser(userToRestore);
            brokersInUse.remove(broker);
        }else
        {
            existLOG.error("Broker being released in brokerReleaseWrapper that isn't in our brokersInUser :: " + broker);
        }
       
        broker.getBrokerPool().release(broker);
    }
}


This seems to have solved my immediate issues, which just to reiterate in case anyone else has a similar situation:

1) Can't guarantee I'll have the current user when I'm getting/releasing a broker (so I can't set the user back before releasing)

2) Must be sure the user doesn't change in my XQuery when I call a java function which uses a different user in part of it's execution.


Cheers,

Patrick

On Tue, May 31, 2011 at 4:21 PM, Adam Retter <adam <at> exist-db.org> wrote:
> I've figured out why I'm getting that Error, it's because I'm creating a new
> broker by instantiating a new NativeBroker, rather than using the
> BrokerPool.get(user) function, which means the broker is never being
> registered with the BrokerPool.

Exactly. A pattern I am using for code that needs to take action on
the database looks like this -

   private interface BrokerOperation {
       public void withBroker(DBBroker broker) throws EXistException,
URISyntaxException, PermissionDeniedException;
   }

   private void executeWithBroker(BrokerOperation brokerOperation)
throws EXistException, URISyntaxException, PermissionDeniedException {
       DBBroker broker = null;
       try {
           broker = factory.getBrokerPool().get(user);
           brokerOperation.withBroker(broker);
       } finally {
           if(broker != null) {
               factory.getBrokerPool().release(broker);
           }
       }
   }

> I've been doing it this way because if I use
> BrokerPool.get(user) and the user is someone other than the current user, it
> changes for the rest of my execution. I realize I can set it back, but I
> have functions where I don't want them tied to requiring a context be passed
> in.

Erm, im not really clear on why you have this problem? or why it is a problem?
Perhaps you are holding onto a broker for too long, i.e. performing
lots of operations with a single broker, sometimes this makes sense,
but if you have distinct code units then sometimes it is better to
treat a broker as a hot-potato and just grab it when you absolutely
have to, and release it as soon as possible. The above pattern works
well in this scenario, and if you do want to switch user, the pattern
could be extended to accomodate that with a
executeWithBrokerAsUser(BrokerOperation, User user) function.

> Now that I think about it, this may not be an issue in the trunk (I think I
> recall seeing some broker.setUser(user) commented out in finally
> statements). Maybe this is an easy port back to 1.4.1? I'd be happy to do it
> if so...

Could you be more exact?

>
> Cheers,
>
> Patrick
>
> On Tue, May 31, 2011 at 2:43 PM, Patrick Bosek <patrick.bosek <at> jorsek.com>
> wrote:
>>
>> Follow up question... Do I need to  release or destroy the contexts I
>> create somehow?
>>
>> On Tue, May 31, 2011 at 2:34 PM, Patrick Bosek <patrick.bosek <at> jorsek.com>
>> wrote:
>>>
>>> Hi everyone,
>>>
>>> I'm trying to figure out when I should and should not release a broker,
>>> as it seems to be critical to eXist running well.
>>>
>>> So, right now I'm getting these is my exist.log:
>>> 2011-05-31 13:21:32,314 [P1-8] ERROR (BrokerPool.java [release]:1312) -
>>> release() has been called from the wrong thread for broker null
>>>
>>> Which I assume is because I'm releasing a broker when I shouldn't. These
>>> are appearing most notably after creating a new broker and a new context,
>>> then releasing the broker. Like this:
>>>
>>> class myUtil {
>>>
>>>     public static XQueryContext getNewContextAsUser(String username){
>>>
>>>         NativeBroker broker = getBrokerAsUser(username);
>>>
>>>         XQueryContext newContext = new XQueryContext(broker,
>>> AccessContext.XMLRPC);
>>>
>>>         return newContext;
>>>     }
>>>
>>> public static NativeBroker getBrokerAsUser(String username)
>>>     {
>>>         BrokerPool pool;
>>>         NativeBroker broker = null;
>>>         try {
>>>             pool = BrokerPool.getInstance();
>>>
>>>             org.exist.security.SecurityManager security =
>>> pool.getSecurityManager();
>>>             User asUser = security.getUser(username);
>>>
>>>             broker = new NativeBroker(pool, pool.getConfiguration());
>>>
>>>             broker.setUser(asUser);
>>>
>>>         } catch (EXistException e) {
>>>             System.out.println("Exception in getBrokerAsUser: " + e);
>>>             e.printStackTrace();
>>>
>>>             return null;
>>>         }
>>>
>>>         return broker;
>>>     }
>>>
>>>     public static void brokerReleaseWrapper(DBBroker broker)
>>>     {
>>>         StackTraceElement[] trace = new Throwable().getStackTrace();
>>>
>>>         existLOG.info("brokerReleaseWrapper being called");
>>>         for(int i = 0; i < 4; i++)
>>>             existLOG.info(trace[i]);
>>>
>>>         broker.getBrokerPool().release(broker);
>>>     }
>>>
>>> }
>>>
>>> // Code that's causing the error
>>>
>>>         XQueryContext context =
>>> myUtil.getNewContextAsUser(org.exist.security.SecurityManager.DBA_USER);
>>>
>>>         NativeBroker broker = (NativeBroker) context.getBroker();
>>>
>>>         try{
>>>
>>>            *** DO STUFF ****
>>>         } finally{
>>>             broker.setUser(currentUser);
>>>             myUtil.brokerReleaseWrapper(broker);
>>>         }
>>>
>>>
>>>
>>> In general, when do I need to be sure I release a broker?
>>>
>>>
>>> Does the XQueryContext release it's broker automatically somehow?
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>> --
>>> Patrick Bosek
>>> Jorsek Software
>>> Cell (585) 820 9634
>>> Office (585) 239 6060
>>> Jorsek.com
>>>
>>
>>
>>
>> --
>> Patrick Bosek
>> Jorsek Software
>> Cell (585) 820 9634
>> Office (585) 239 6060
>> Jorsek.com
>>
>
>
>
> --
> Patrick Bosek
> Jorsek Software
> Cell (585) 820 9634
> Office (585) 239 6060
> Jorsek.com
>
>
> ------------------------------------------------------------------------------
> Simplify data backup and recovery for your virtual environment with vRanger.
> Installation's a snap, and flexible recovery options mean your data is safe,
> secure and there when you need it. Data protection magic?
> Nope - It's vRanger. Get your free trial download today.
> http://p.sf.net/sfu/quest-sfdev2dev
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>



--
Adam Retter

eXist Developer
{ United Kingdom }
adam <at> exist-db.org
irc://irc.freenode.net/existdb



--
Patrick Bosek
Jorsek Software
Cell (585) 820 9634
Office (585) 239 6060
Jorsek.com

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Permalink | Reply | 5 comments |
headers
Adam Retter | 1 Jun 2011 02:56

Re: GET of query files

> What's the trick ?  What can it do that I can't using REST/GET ? and how ?
>
>
>
> I understand the desire for security but if I'm a user that has the
> permissions to write (PUT) a file, surely I should be able to GET it back ?

Indeed! This is a legacy hangover from eXist-db's permissions system.
At present you cannot get back the query from the REST API, this was
done for as you guessed security. The problem is that eXist-db's
permission system did not support an execute bit, so you had to grant
read access to ALL users for all users to be able to execute your
query. Typically you dont just want anyone also grabbing the source of
your application.

The permissions system in eXist-db has been undergoing some intensive
modifications, whilst the flags are still rwu, it should be fairly
trivial in the near future to change this to rwx and then modify the
xquery engine to support the execute bit check.

Cheers Adam.

--

-- 
Adam Retter

eXist Developer
{ United Kingdom }
adam <at> exist-db.org
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Permalink | Reply | 1 comment |
headers
Dmitriy Shabanov | 1 Jun 2011 08:30
Picon
Gravatar

Re: Stop XQuery caching

This should be fixed for trunk + XQuery servlet configuration have an option to disable caching.

On Wed, Jun 1, 2011 at 5:52 AM, Adam Retter <adam <at> exist-db.org> wrote:
>Some sort of configuration was what I
> was looking for. The only way I could find to ensure that all the included
> queries were compiled was to change the parent query.

The 'query-pool' settings in conf.xml should allow you to disable
caching in your dev environment should you wish...


--
Dmitriy Shabanov
------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Permalink | Reply | 0 comments |
headers
Marc Monroy | 1 Jun 2011 09:14
Picon

Re: fn:distinct-values() for nodes?

Markus Kaindl <markus.kaindl <at> bsb-muenchen.de> writes:

> 
> Hi,
> 
> I need to return a list of nodes where all those nodes which have the
> same text value were removed.
> I tried fn:distinct-values($arg) but that would only return the text
> values, not the unique node list.
> Unfortunately, the nodes arent identical (subelements / attributes
> differ). Any quick solutions to this?
> 
> Cheers,
> Markus
> 
> -----------------------------------------------------------------------------
-
> Simplify data backup and recovery for your virtual environment with vRanger. 
> Installation's a snap, and flexible recovery options mean your data is safe,
> secure and there when you need it. Data protection magic?
> Nope - It's vRanger. Get your free trial download today. 
> http://p.sf.net/sfu/quest-sfdev2dev
> 

Hi Markus,

Here are functions i use to select distinct nodes, i think this is what you 
need:

declare function local:distinct-deep($nodes as node()* ) as node()* 
{
	for $seq in (1 to count($nodes))
	return $nodes[$seq][not(local:is-node-in-sequence-deep-equal(.,$nodes
[position() < $seq]))]
};

declare function local:is-node-in-sequence-deep-equal($node as node()?, $seq 
as node()*) as xs:boolean
{
	some $nodeInSeq in $seq satisfies deep-equal($nodeInSeq,$node)
};

Cheers,
Marc

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 0 comments |
headers
Johan Mörén | 1 Jun 2011 10:07
Picon

Re: xmdb:store ignores serialization options for x-includes = off

I changed the serialization option in conf.xml  to
--> enable-xinclude="no"

and my use-case now works.

But I guess I'm out of luck if I deploy another application that want
to expand x-includes during xmldb:store()?

Is it possible to overide the global configuration regarding expansion
of x-includes in some way when storing documents to the database?

Regards,
Johan

On Tue, May 31, 2011 at 4:51 PM, Johan Mörén <johan.moren <at> gmail.com> wrote:
> Hi!
>
> I'm having trouble with storing documents with xmldb:store. It ignores
> my serialization options and expands xincludes in my document.
>
> declare option exist:serialize "expand-xincludes=no";
>
> If i comment out the store and just return the result-document the
> x:includes are not expanded.
>
> I'm using 1.4.1dev-rev14490-20110522 and inoke a stored query via the
> REST-API using method POST.
>
> From a search on MarkMail this looks like a problem that has appeared
> before and was then related to an indexing-bug.
>
> /Johan
>

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
Exist-open mailing list
Exist-open <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/exist-open
Permalink | Reply | 0 comments |
headers
Ralf Jung | 1 Jun 2011 11:38
Picon
Picon

Re: Starting exist as service (/etc/init.d/exist) vs bin/startup.sh

Hi,

actually, only the logging configuration file changes, not the "main" eXist one. 
I solved this by editing tools/wrapper/conf/wrapper.conf and changing the line 
starting with
wrapper.java.additional.5
to
wrapper.java.additional.5=-Dlog4j.configuration=file:../../../log4j.xml

Kind regards,
Ralf

Am Mittwoch 01 Juni 2011, um 00:48:21 schrieb Alister Pillow:
> Hi David,
> When started using the wrapper, the configuration files are in
> /{$exist-home}/tools/wrapper/conf/
> and on the Admin page, it shows my eXist Home as
> /exist141/tools/wrapper/bin/../../..
> 
> The log files are in
> /{$exist-home}/tools/wrapper/logs
> 
> On 01/06/2011, at 7:46 AM, David Lee wrote:
> > When I start exist as a service (on linux) using /etc/init.d/exist script
> > as documented, it creates different logging files then when I start
> > directly via bin/startup.sh
> > 
> > I dug deep enough to find the service uses "wrapper" which is a native
> > exe program.
> > 
> > What is the difference between these startup methods ? Is this documented
> > anywhere ? I find I cannot get the log in exist.log output if I start as
> > a service but can if started via startup.sh
> > 
> > Thanks for any pointers.
> > 
> > 
> > ----------------------------------------
> > David A. Lee
> > dlee <at> calldei.com
> > http://www.xmlsh.org
> > 
> > -------------------------------------------------------------------------
> > ----- Simplify data backup and recovery for your virtual environment with
> > vRanger. Installation's a snap, and flexible recovery options mean your
> > data is safe, secure and there when you need it. Data protection magic?
> > Nope - It's vRanger. Get your free trial download today.
> > http://p.sf.net/sfu/quest-sfdev2dev______________________________________
> > _________ Exist-open mailing list
> > Exist-open <at> lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> ---------------------------------------------------------------------------
> --- Simplify data backup and recovery for your virtual environment with
> vRanger. Installation's a snap, and flexible recovery options mean your
> data is safe, secure and there when you need it. Data protection magic?
> Nope - It's vRanger. Get your free trial download today.
> http://p.sf.net/sfu/quest-sfdev2dev
> _______________________________________________
> Exist-open mailing list
> Exist-open <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 0 comments |
headers
Stéphane S. | 1 Jun 2011 11:48
Picon
Picon
Favicon

Access to $exist:path (i.e. URLRewriting variables) from imported XQuery modules in controller.xql

Hi,

In a controller.xql file I am importing an XQuery module with an instruction such as:

import module namespace gen = "http://blahblah.com/generator" at "myapp/mylib/generator.xqm";

The imported module makes references to URLRewriting exist set variables such as $exist:path as
explained in [1] 

The problem is that I get the following error at runtime:

An error occurred while executing the urlrewrite query: XPDY0002 : variable '$exist:path' is not set. [at
line 177, column 29]

I then tried to declare the variable as external as suggested in [1]:

declare variable $exist:path as external;

Which gives:

An error occurred while executing the urlrewrite query: error found while loading module gen: Error while
loading module myapp/mylib/generator.xqm: error found while loading module from
myapp/mylib/generator.xqm: org.exist.xquery.XPathException: err:XPST0003 in line 15, column 41:
unexpected token: ;

I can get rid of that error removing the "as" which suggest the explanation in [1] has a syntactic error
(extra "as" which does not appear in Priscilla Walmsley XQuery book p168)

declare variable $exist:path external;

But once again after declaring all the $exist:path, $exist:root, $exist:prefix, $exist:resource as
external, I still get the same error:

An error occurred while executing the urlrewrite query: XPDY0002 : variable '$exist:path' is not set. [at
line 182, column 29] (*)

I am using eXist 1.4.0-rev10440-20110131

Does that mean that it is not possible to use these variables in an imported module from controller.xql ?

Then how can I use them in my library module ? Should I pass them explicitely as parameters to any library
function called from controller.xql ?

Thanks in advance,

Stéphane S.
---

[1] http://www.exist-db.org/urlrewrite.html#N10181

(*) Note that curiously if I only declare $exist:path, I get the error message for the $exist:root, which is
the second one I use, then declaring all of them I return back to the error about the first one which is used,
maybe this gives a clue
------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 1 comment |
headers
Wolfgang Meier | 1 Jun 2011 11:57
Gravatar

Re: Access to $exist:path (i.e. URLRewriting variables) from imported XQuery modules in controller.xql

> Does that mean that it is not possible to use these variables in an imported module from controller.xql ?

The variables are indeed only visible from within controller.xql.
Imported modules do not inherit the context from the importing module.

> Then how can I use them in my library module ? Should I pass them explicitely as parameters to any library
function called from controller.xql ?

You could use parameters, yes. If I need the exist:* variables in
other queries within a pipeline, I also often set them as request
attributes, thus making them available within the entire request
processing pipeline.

Wolfgang

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger. 
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today. 
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 0 comments |

Gmane