headers
Brian | 1 Jul 01:01
Picon
Favicon

Re: On templates and programming languages

On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde <rarohde <at> gmail.com> wrote:
>
> However,
> given the nastiness of template syntax, I would expect no end of wiki
> authors willing to help convert the commonly used stuff.
>
> -Robert Rohde

I was curious just how terrible of a task conversion can be expected
to be. This is just a heuristic I came up with..

# Simple English parser functions
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
22,211

# Simple English templates
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
416,126 - 22,211 = 393,915

# English parser functions
$  bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
430,980

# English templates
$  bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
44,928,358 - 430,980 = 44,497,378
Permalink | Reply | 25 comments |
headers
Andrew Garrett | 1 Jul 01:16
Picon

Re: On templates and programming languages


On 30/06/2009, at 9:42 PM, Aryeh Gregor wrote:

> On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibber<brion <at> wikimedia.org>  
> wrote:
>> IMO by the time you've implemented your whitelisting parser you  
>> might as
>> well just interpret it rather than eval()ing.
>
> I don't think so.  You'd only have to do the whitelisting once, on
> page save.  After that you could just execute with no extra overhead.

That's just scary. We'd definitely want to do the validation as close  
as possible to the actual eval()ing, to minimise backdoors like  
Special:Import et al.

--
Andrew Garrett
Contract Developer, Wikimedia Foundation
agarrett <at> wikimedia.org
http://werdn.us
Permalink | Reply | 1 comment |
headers
The Cunctator | 1 Jul 01:17
Picon

Wikipedia downtime?

Was that planned downtime?
Permalink | Reply | 2 comments |
headers
jidanni | 1 Jul 01:23
Favicon
Gravatar

zh.wikipedia.org says "you are now logged in as $1"

I logged in to zh.wikipedia.org and it says I am $1. en.wikipedia.org
didn't have that problem.

您現在以「$1」的身分登入維基百科。

如果您對維基百科有任何問題的話，請參看幫助頁。

您已登入其他的維基媒體計劃：

_______________________________________________
Wikitech-l mailing list
Wikitech-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Permalink | Reply | 0 comments |
headers
Hay (Husky | 1 Jul 01:24
Picon
Gravatar

Re: On templates and programming languages

I would opt for Javascript.

PHP and Python are intended for large and complex applications and
come with a huge standard library people probably expect to be
available. Security concerns are a problem too, so a subset would
probably be necessary So, in essence you get a crippled-down language
that isn't really useful for templates.

Making our own language, either by 'fixing' the template language or
by inventing something new would only mean we introduce a new language
that'll be specific to our own platform and nobody knows outside of
Mediawiki developers.

XSLT is not meant to be written or read by humans. It's a
Turing-complete language stuffed into horrendous XML statements. Let's
not go down that road.

That leaves us to Lua and Javascript, which are both small and
efficient languages meant to solve tasks like this. Remember, i'm
talking about 'core' Javascript here, not with all DOM methods and
stuff. If you strip that all out (take a look at the 1.5. core
reference at Mozilla.com:
https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you
get a pretty nice and simple language that isn't very large. Both
would require a new parser and/or installed compilers on the
server-side. Compared to the disadvantages of other options, that
seems like a pretty small loss for a great win.

Javascript is a widely understood and implemented language, with lots
of efforts to get it even faster in modern browsers. Every Wikipedia
(Continue reading)

Permalink | Reply | 1 comment |
headers
Dan Collins | 1 Jul 01:36
Picon

Re: Wikipedia downtime?

On Tue, Jun 30, 2009 at 7:17 PM, The Cunctator<cunctator <at> gmail.com> wrote:
> Was that planned downtime?
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l <at> lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

No.

Apparently, for whatever reason, the master database server for enwiki
got overloaded. This was following a few updates, which may have (I
don't think they're sure yet) caused the problem. What actually
happened was the database server had a large number of queries stuck
in the 'statistics' status, leading to overload, leading to wiki down.
Enwiki was set to read only, and the Almighty Tim, Patron Saint of
Master Databases, arrived on the scene to heroically run the
master-database switch script. The S1 (enwiki) master database was
changed from db14 to db16, and db14 was removed from the slave
rotation. From what I understand, db14 will need a swift kick to the
power button to make it all jolly and happy again.

For the latest information when the wiki is down, you can always visit
#wikipedia on irc.freenode.net.

--

-- 
DCollins/ST47
Administrator, en.wikipedia.org
Permalink | Reply | 1 comment |
headers
Trevor Parscal | 1 Jul 01:44
Picon

Re: On templates and programming languages

I personally agree entirely. Now we just need to revive J4P5 
(http://j4p5.sourceforge.net) :)

- Trevor

On 6/30/09 4:24 PM, Hay (Husky) wrote:
> I would opt for Javascript.
>
> PHP and Python are intended for large and complex applications and
> come with a huge standard library people probably expect to be
> available. Security concerns are a problem too, so a subset would
> probably be necessary So, in essence you get a crippled-down language
> that isn't really useful for templates.
>
> Making our own language, either by 'fixing' the template language or
> by inventing something new would only mean we introduce a new language
> that'll be specific to our own platform and nobody knows outside of
> Mediawiki developers.
>
> XSLT is not meant to be written or read by humans. It's a
> Turing-complete language stuffed into horrendous XML statements. Let's
> not go down that road.
>
> That leaves us to Lua and Javascript, which are both small and
> efficient languages meant to solve tasks like this. Remember, i'm
> talking about 'core' Javascript here, not with all DOM methods and
> stuff. If you strip that all out (take a look at the 1.5. core
> reference at Mozilla.com:
> https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you
> get a pretty nice and simple language that isn't very large. Both
(Continue reading)

Permalink | Reply | 0 comments |
headers
Brion Vibber | 1 Jul 01:56
Picon
Gravatar

Re: Wikipedia downtime?

Dan Collins wrote:
> Apparently, for whatever reason, the master database server for enwiki
> got overloaded. This was following a few updates, which may have (I
> don't think they're sure yet) caused the problem. What actually
> happened was the database server had a large number of queries stuck
> in the 'statistics' status, leading to overload, leading to wiki down.
> Enwiki was set to read only, and the Almighty Tim, Patron Saint of
> Master Databases, arrived on the scene to heroically run the
> master-database switch script. The S1 (enwiki) master database was
> changed from db14 to db16, and db14 was removed from the slave
> rotation. From what I understand, db14 will need a swift kick to the
> power button to make it all jolly and happy again.

Ah excellent, you did my summary post for me. :)

Lots of threads being in the "statistics" state seems to be MySQL's way 
of saying "I've fallen and I can't get up". It's unclear exactly what 
set it off, but basically nothing works well until you restart it.

At 52 minutes from start of event, this took us a bit longer than I'd 
like to resolve -- we had to percolate through a couple levels of alert 
calls. (Sorry to wake you up early Tim!)

A similar event in future should be fixable within a few minutes, thanks 
to Tim's work on making the master-switch system more foolproof. We're 
fixing up our internal documentation so all our site ops will now know 
how to run the database master switch script next time!

Only en.wikipedia.org was affected, other than a couple of minutes where 
we threw the whole site to read-only while figuring out what was going on.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Robert Rohde | 1 Jul 02:09
Picon

Re: On templates and programming languages

On Tue, Jun 30, 2009 at 4:01 PM, Brian<Brian.Mingus <at> colorado.edu> wrote:
> On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde <rarohde <at> gmail.com> wrote:
>>
>> However,
>> given the nastiness of template syntax, I would expect no end of wiki
>> authors willing to help convert the commonly used stuff.
>>
>> -Robert Rohde
>
> I was curious just how terrible of a task conversion can be expected
> to be. This is just a heuristic I came up with..
>
> # Simple English parser functions
> $ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
> 22,211
>
> # Simple English templates
> $ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
> 416,126 - 22,211 = 393,915
>
> # English parser functions
> $  bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
> 430,980
>
> # English templates
> $  bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
> 44,928,358 - 430,980 = 44,497,378

I assume we are primarily talking about replacing template code and
not template calls, per se.
(Continue reading)

Permalink | Reply | 24 comments |
headers
Brian | 1 Jul 02:28
Picon
Favicon

Re: On templates and programming languages

On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohde<rarohde <at> gmail.com> wrote:
> In other words, I assume things like "{{fact}}" and "{{msg | foo is
> bar }}" will be be basically unchanged on the article side but
> rewritten on the implementation side in Template: space.  If that is
> correct, it would be more useful to simply ask how large Template:
> space is rather than counting all the template calls.
>
> -Robert Rohde

Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.
Permalink | Reply | 23 comments |

Gmane