headers
Manuel Schneider | 30 Apr 2009 06:25
Picon

[openZIM dev-l] WG: [LT2009] Project "openZIM" accepted

Dear all,

we managed to get a booth at LinuxTag in Berlin this year (June 24th - 27th).

This is good news! But it also means work - now we have to get ready for the Wikipedia DVD, the ISO must be ready
by May 31st.

Please keep me up to date with the current status.

Tommi, did you hear about your talk submission?

Cheers,

Manuel

-- Urspr. Mitt. --
Betreff: [LT2009] Project "openZIM" accepted
Von: projects@...
Datum: 29.04.2009 21:03

Dear Manuel Schneider,

we are excited to inform you, that your application for the project 
  openZIM 
qualified for a sponsored booth at this year's LinuxTag. Please note
this is just the confirmation of our general acceptance of your
submission. The size and location of the booth will be decided later
mainly depending on the overall amount of accepted project submissions
and the quality of the exhibition concept you provide us using the
virtual conference center and in dialogue with the project committee.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Tommi Mäkitalo | 1 May 2009 21:32

Re: [openZIM dev-l] WG: [LT2009] Project "openZIM" accepted

On Donnerstag 30 April 2009 06:25:32 Manuel Schneider wrote:
> Dear all,
>
> we managed to get a booth at LinuxTag in Berlin this year (June 24th -
> 27th).
>
> This is good news! But it also means work - now we have to get ready for
> the Wikipedia DVD, the ISO must be ready by May 31st.
>
> Please keep me up to date with the current status.
>
> Tommi, did you hear about your talk submission?
>
> Cheers,
>
> Manuel
>
Hi,

unfortunately I have not yet received any information other than that my own 
project tntnet  was accepted also. I have to make sure, that we really get a 
shared booth.

I was busy the last weeks, but I just continued to work on the full text 
indexing. It looks so far very promising. I may finish that in the next days.

Tommi
Permalink | Reply | 0 comments |
headers
Tommi Mäkitalo | 4 Apr 2009 16:10

[openZIM dev-l] new zimlib code checked in

Hi,

I would like to inform you, that I have reached a major milestone with the new 
zim format: I created successfully a zim file and read it with zimDump.

The changes are:
	*	rewritten large parts
	*	updated the zim file format
	*	redesigned zimwriter

Let me say some words about these changes and why I did this.

* Rewritten large parts:

Rewrite helped me to improve code quality. With my knowledge of today and my 
experience with the zeno file format, it was possible to clean up the library 
code.

* Updated the zim file format:

Since we decided to leave the compatibility I rethought some parts of the zeno 
file format. The zeno file format did not support clustering of articles to get 
better compression. I did a minor change and added a offset and size to the 
directory entry of the article. The offset to the data blob was left in the 
article. But now multiple articles pointed to the same blob. In the new format 
I added another datastructure: the chunk, which is a collection of blobs. We 
have a pointerlist similar to the directory pointer list, which points to the 
chunks. The article addresses his blob by chunk number and blob number. Also 
redirect entries do not need these pointers at all. I just skipped them. This 
saves some bytes for each redirect.
(Continue reading)

Permalink | Reply | 5 comments |
headers
Manuel Schneider | 4 Apr 2009 18:22
Picon

Re: [openZIM dev-l] new zimlib code checked in

Hi Tommi and the rest of the team,

thanks for committing your changes and of course thank you very much for your 
work on the project.
I am currently sitting at the Wikimedia Board Panel at the Wikimedia 
Conference (chapters meeting) in Berlin.

Yesterday I had an interestng discussion with Asaf Bartov. He is volunteer at 
Wikimedia Israel and as a professional he is developper of transactional 
systems für banks.
Wikimedia Israel has plans to get the hebrew Wikipedia onto the OLPC and is 
looking for technical solutions. Asaf already worked on Bzreader which is a 
Reader of the bzip2ed XML dumps the WMF provides. Bzreader runs on 
ReactOS-compatibles and has to parse the MW syntax by itself and has a 
built-in browser. Asaf already fixed some bugs, but when he heard me in the 
introductionery session mentioning openZIM he started thinking about better 
investing his effort into something which already works.
We looked into openZIM yesterday, checked out the stuff from Subversion, 
compiled it and played with it. I am pretty sure that Asaf will be quite 
happy with the update now and I will try to get him on board of the 
development team.

This would then also require more frequently commits, so Asaf is able to work 
on current code and submit patches.

He also had Kiwix on his hard drive and we updated from Subversion, so we 
realised that also Emmanuel did some work on his code, zim replaced zeno now 
etc.

There are two other Wikipedia DVD projects on this conference:
(Continue reading)

Permalink | Reply | 3 comments |
headers
Tommi Mäkitalo | 7 Apr 2009 22:54

[openZIM dev-l] openzim file

Hi,

I created a openzim file from the german wikipedia dump I became from Josch 
last year. The file contains all articles from the german wikipedia without 
images and its size is 1,3G (or more precisely 1302052315 bytes). Generation 
took only about 1:10 on our server with the new zimwriter. It could be even 
improved by parallizing the compression phase, since this is CPU bound and 
takes the most time, but I feel, that it is not necessary. There are more 
important task to do.

You can download the file from http://www.openzim.org/download/dewiki.zim.

The zimreader (the tntnet based webapplication) is almost working with that 
file. There are some bugs to fix, but this will be done soon.

Emmanuel: the file is updated. I fixed some bugs. The zimDump crashed when 
reading redirects and the writer failed to generate redirects correctly.

Josch: do you have an updated dump?

Tommi
Permalink | Reply | 4 comments |
headers
Manuel Schneider | 8 Apr 2009 12:05
Picon

Re: [openZIM dev-l] openzim file

These are great news.

Am Dienstag, 7. April 2009 22:54:41 schrieb Tommi Mäkitalo:

> I created a openzim file from the german wikipedia dump I became from Josch
> last year. The file contains all articles from the german wikipedia without
> images and its size is 1,3G (or more precisely 1302052315 bytes).
> Generation took only about 1:10 on our server with the new zimwriter. It
> could be even improved by parallizing the compression phase, since this is
> CPU bound and takes the most time, but I feel, that it is not necessary.
> There are more important task to do.

How much is 1:10 now in absolute numbers?
Can you please have a look into 
http://www.openzim.org/Status_and_next_steps and strike the things which have 
been done or add other things, which aren't yet on the list?

> You can download the file from http://www.openzim.org/download/dewiki.zim.

Please allow me to rename that file soon to wikipedia-de.zim. I try to 
establish a scheme for filenames which take into account that we will later 
host all kind of different ZIM files.

> The zimreader (the tntnet based webapplication) is almost working with that
> file. There are some bugs to fix, but this will be done soon.

Is it just the zimreader or in libzim?
What about the status of Kiwix?

> Josch: do you have an updated dump?
(Continue reading)

Permalink | Reply | 3 comments |
headers
Tommi Mäkitalo | 8 Apr 2009 14:26

Re: [openZIM dev-l] openzim file

Hi,

On Mittwoch 08 April 2009 12:05:14 Manuel Schneider wrote:
> These are great news.
>
> Am Dienstag, 7. April 2009 22:54:41 schrieb Tommi Mäkitalo:
> > I created a openzim file from the german wikipedia dump I became from
> > Josch last year. The file contains all articles from the german wikipedia
> > without images and its size is 1,3G (or more precisely 1302052315 bytes).
> > Generation took only about 1:10 on our server with the new zimwriter. It
> > could be even improved by parallizing the compression phase, since this
> > is CPU bound and takes the most time, but I feel, that it is not
> > necessary. There are more important task to do.
>
> How much is 1:10 now in absolute numbers?

It is one hour and 10 minutes. I feel, that this is pretty fast.

> Can you please have a look into
> http://www.openzim.org/Status_and_next_steps and strike the things which
> have been done or add other things, which aren't yet on the list?
>

I will update the status page. I have to reorganize some of the tasks.

> > You can download the file from
> > http://www.openzim.org/download/dewiki.zim.
>
> Please allow me to rename that file soon to wikipedia-de.zim. I try to
> establish a scheme for filenames which take into account that we will later
(Continue reading)

Permalink | Reply | 1 comment |
headers
Manuel Schneider | 8 Apr 2009 15:05
Picon

Re: [openZIM dev-l] openzim file

Am Mittwoch, 8. April 2009 14:26:25 schrieb Tommi Mäkitalo:
> > How much is 1:10 now in absolute numbers?
>
> It is one hour and 10 minutes. I feel, that this is pretty fast.

ah, I see. I interpreted 1:10 as 1 10th of the time before.
As far as I know it was around 25 hours before, right?

> > Can you please have a look into
> > http://www.openzim.org/Status_and_next_steps and strike the things which
> > have been done or add other things, which aren't yet on the list?
>
> I will update the status page. I have to reorganize some of the tasks.

Thx.

> > Please allow me to rename that file soon to wikipedia-de.zim. I try to
> > establish a scheme for filenames which take into account that we will
> > later host all kind of different ZIM files.
>
> You can do that. Can you also give me/us some advise, how to put zim files
> for download. I just created a download-directory in mediawiki. Is there a
> better way?

I think www.openzim.org/download/ is a good URL, isn't it?

I thought about categorysing the ZIM files, 
like "knowledge", "literature", "news" etc. We will definitely do that in the 
Wiki, but maybe we should have the same directory structure as well?
(Continue reading)

Permalink | Reply | 0 comments |
headers
Tommi Mäkitalo | 11 Apr 2009 12:30

[openZIM dev-l] status - ZimReader now working

Hi,

the ZimReader is fixed and I can browse through the zim file I created from 
Josch's dewiki-Dump. There were some smaller bugs in the reader as well as in 
the library, but the main bug was in cxxtools. So if you want to test the 
reader please update cxxtools.

There are still some tasks to do in the reader. There is many hardcoded stuff 
from the old wikipedida DVD, like the text "DVD-ROM-Ausgabe 2007" in the title 
area and a reference to Directmedia.

I also updated the status and next steps page in our wiki.

Tommi
Permalink | Reply | 0 comments |
headers
Manuel Schneider | 11 Apr 2009 12:37
Picon

Re: [openZIM dev-l] status - ZimReader now working

Great!

I am also curious hearing from Kiwix. Will it be ready so we can put both 
readers onto the DVD?

Greets,

Manuel

Am Samstag, 11. April 2009 schrieb Tommi Mäkitalo:
> Hi,
>
> the ZimReader is fixed and I can browse through the zim file I created from
> Josch's dewiki-Dump. There were some smaller bugs in the reader as well as
> in the library, but the main bug was in cxxtools. So if you want to test
> the reader please update cxxtools.
>
> There are still some tasks to do in the reader. There is many hardcoded
> stuff from the old wikipedida DVD, like the text "DVD-ROM-Ausgabe 2007" in
> the title area and a reference to Directmedia.
>
> I also updated the status and next steps page in our wiki.
>
> Tommi
> _______________________________________________
> dev-l mailing list
> dev-l@...
> https://intern.openzim.org/mailman/listinfo/dev-l

--

--
(Continue reading)

Permalink | Reply | 501 comments |

Gmane