headers
Torben Nehmer | 4 Oct 22:57
Gravatar

MidCOM performance plan of action

Hi community,

as especially Henri noticed, the current MidCOM development strain
suffers from major performance problems. This mail intents to a)
describe the sources of these problems and proposes b) both a short- and
long-term roadmap how to solve these problems.

A. Sources of MidCOM performance issues, in their probable order of
"magnitude":

1. MidCOM Metadata framework (2.4 and previous implementation)

This is the part currently slowing down MidCOM most. Simple benches done
by Henri indicate that 2/3s of the runtime is lost here. The reason for
this is quite simple actually: MidCOM Metadata is held in simple
parameters, nothing that could be queried on the SQL level easily
(especially as all of these parameters are optional).

This means, that in every place where metadata is queried, we have at
least three parameter selects (hide, schedule_start and schedule_end) in
the default configuration which filters "hidden" objects. If you enable
approval, another select is added to that list.

Metadata is queried a) by NAP when building the NAP entry lists, b) by
the MidCOM QueryBuilder wrapper on-site, which automatically includes
this "filtering function" without you having to take care about it and
c) in several pre 2.5 tech components, which add metadata checks
manually in their view / NAP code.

2. MidCOM ACL system
(Continue reading)

Permalink | Reply | 10 comments |
headers
Henri Bergius | 5 Oct 06:51
Favicon
Gravatar

Re: MidCOM performance plan of action

On Oct 4, 2005, at 23:57, Torben Nehmer wrote:
> as especially Henri noticed, the current MidCOM development strain
> suffers from major performance problems.

Well, depends what you make of these runtimes of requesting a fairly  
typical
site front page on a biggish website, without caching:

MidCOM 2.2: ~2s (estimated, about half of 2.4 runtime)
MidCOM 2.4: 4s
MidCOM CVS HEAD: 1m 35s
MidCOM CVS HEAD (hiding disabled): 36s
MidCOM CVS HEAD (hiding disabled, mmcache): 25s

> 1. Disable the Metadata system by default
>
> This should relieve a good deal of load from most systems, as  
> scheduling
> and hiding is not in much use anyway. The show_hidden_objects
> configuration parameter, set to false by default, will be set to true
> from now on, unconditionally showing all objects.

+1, but with a condition: if metadata is disabled on a site, then the  
metadata
toolbar shouldn't be shown in AIS either. Otherwise it will just  
unnecessarily
confuse users.

> All components, especially the legacy ones need to be rewritten to  
> work
(Continue reading)

Permalink | Reply | 0 comments |
headers
Piotras | 6 Oct 10:36

1.7.2 release


	Hi all!

1.7.2 "Plugged" should be released on monday or tuesday ( October 10th or 11th).
Do not ask when exactly and why not yet because I spent my free private time to make
releases :)

I would like to ask everyone to test latest branch-1-7. 
Two important memory fixes:

	* mgd_include_snippet
	* style and page elements engine ( with cache )  

Probably the second one was responsible for memleaks in 1.6.x releases.
There are no new features in latest branch, and we should finally make sure 
if midgard triggers memory leaks.

Piotras
Permalink | Reply | 0 comments |
headers
Piotras | 6 Oct 10:58

Re: MidCOM performance plan of action

Torben Nehmer <torben@...> wrote:

> 1. MidCOM Metadata framework (2.4 and previous implementation)
> 
> This is the part currently slowing down MidCOM most. Simple benches done
> by Henri indicate that 2/3s of the runtime is lost here. The reason for
> this is quite simple actually: MidCOM Metadata is held in simple
> parameters, nothing that could be queried on the SQL level easily
> (especially as all of these parameters are optional).

* Use $object->domain_name instead $object->parameter(domain,name);

I think midcom parameters usage is abused. At least midgard log file produces 
something like :

$object = mgd_get_xxx()
$object->parameter(domain, name);

* Can we use QB for parameters?
Temporary solution with Qb should be much faster then old API one.

Piotras
Permalink | Reply | 8 comments |
headers
Eero af Heurlin | 6 Oct 11:03
Favicon
Gravatar

Re: MidCOM performance plan of action


Piotras wrote:
> 
> * Use $object->domain_name instead $object->parameter(domain,name);
> 

Isn't "autoloading" of parameters deprecated in MgdSchema ? I think we
agreed in spring that it should be.

/Rambo
Permalink | Reply | 4 comments |
headers
Piotras | 6 Oct 11:08

Re: MidCOM performance plan of action

Eero af Heurlin <eero.afheurlin@...> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
> 
> Piotras wrote:
> > 
> > * Use $object->domain_name instead $object->parameter(domain,name);
> > 
> 
> Isn't "autoloading" of parameters deprecated in MgdSchema ? I think we
> agreed in spring that it should be.

Yes, it is , but I wrote about :

$object  = mgd_get_xxx();
$object->parameter();

MgdSchema objects retrieved according to parameters is trivial.
A bit of trick but it is easy and doable. 

Of course we still think about Midgard2 while using Midgard 1.7 , right?

Piotras
Permalink | Reply | 3 comments |
headers
Eero af Heurlin | 6 Oct 11:19
Favicon
Gravatar

Re: MidCOM performance plan of action


Piotras wrote:
>
> Yes, it is , but I wrote about :
>
> $object  = mgd_get_xxx();
> $object->parameter();
>

True, but those will change (probably sooner than later) to use MidCOM
DBA, which uses MgdSchema. Thus removing changing the parameter call to
point to "autoloaded" parameter is not the solution.

>
> Of course we still think about Midgard2 while using Midgard 1.7 , right?
>

Sure we do, unless someone tries to use those features for real they'll
never stabilize.

/Rambo
Permalink | Reply | 0 comments |
headers
Torben Nehmer | 6 Oct 12:07
Gravatar

Re: MidCOM performance plan of action


Hi,

--Piotras wrote on 2005-10-06 10:58:
>> 1. MidCOM Metadata framework (2.4 and previous implementation)
>> 
>> This is the part currently slowing down MidCOM most. Simple benches done
>> by Henri indicate that 2/3s of the runtime is lost here. The reason for
>> this is quite simple actually: MidCOM Metadata is held in simple
>> parameters, nothing that could be queried on the SQL level easily
>> (especially as all of these parameters are optional).
> 
> * Use $object->domain_name instead $object->parameter(domain,name);
> 
> I think midcom parameters usage is abused. At least midgard log file produces 
> something like :
> 
> $object = mgd_get_xxx()
> $object->parameter(domain, name);

As rambo already pointed out, this is not an option. If we touch these code
places, then only to rewire them to DBA classes instead of Legacy Midgard ones.

> * Can we use QB for parameters?
> Temporary solution with Qb should be much faster then old API one.

To the contrary. DBA uses QB currently to query parameters, but it is actually
slower then the original ->parameter() calls. Reason we use DBA: Ability for
access control, extended syntax with cleaner error reporting (compared to legacy
Midgard). This solution showed to be quite slower then the classic pre-MgdSchema
(Continue reading)

Permalink | Reply | 2 comments |
headers
Piotras | 6 Oct 12:51

Re: MidCOM performance plan of action

Torben Nehmer <torben@...> wrote:

> > * Can we use QB for parameters?
> > Temporary solution with Qb should be much faster then old API one.
> 
> To the contrary. DBA uses QB currently to query parameters, but it is actually
> slower then the original ->parameter() calls.

/* pseudo code */

$object = new someclass();

$qb = new midgardquerybuilder("midgard_paramater");
$qb->add_constraint("table", "=", $object->__table__);
$qb->add_constraint("domain", "=", $value);

etc etc 

$array  = $qb->execute();

$object->get_by_id($array[$i]->oid);

That one is slower?

Piotras
Permalink | Reply | 1 comment |
headers
Torben Nehmer | 6 Oct 13:00
Gravatar

Re: MidCOM performance plan of action


Hi,

--Piotras wrote on 2005-10-06 12:51:
>> > * Can we use QB for parameters?
>> > Temporary solution with Qb should be much faster then old API one.
>> 
>> To the contrary. DBA uses QB currently to query parameters, but it is actually
>> slower then the original ->parameter() calls.
> 
> /* pseudo code */
> 
> $object = new someclass();
> 
> $qb = new midgardquerybuilder("midgard_paramater");
> $qb->add_constraint("table", "=", $object->__table__);
> $qb->add_constraint("domain", "=", $value);
> 
> etc etc 
> 
> $array  = $qb->execute();
> 
> $object->get_by_id($array[$i]->oid);
> 
> That one is slower?

There are two scenarios here, the one you have just outlined is very special and
only for cases where you need to filter objects by parameters.

In many cases parameters are options, not filters, the example you had earlier,
(Continue reading)

Permalink | Reply | 0 comments |

Gmane