Wikimedia XML data dumps

headers Christine Bush | 21 May 2013 17:44 · /dumps/

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Dear Ariel,

I submitted the patches upstream.

0) Review

The changes may be found at <https://gerrit.wikimedia.org/r/64343/>.

1) Git

In the hope that others may find it useful, I am posting the sequence
of Git commands used, organized as a Makefile.

#-----------------------------------------------------------------------------+
# Makefile for submitting patches to the Wikimedia Foundation                 |
# Copyright (C) 2013 Dr. Kent L. Miller.  All rights reserved.                |
#                                                                             |
# This program is free software: you can redistribute it and/or modify        |
# it under the terms of the GNU General Public License as published by        |
# the Free Software Foundation, either version 3 of the License, or (at       |
# your option) any later version.                                             |
#                                                                             |
# This program is distributed in the hope that it will be useful, but         |
# WITHOUT ANY WARRANTY; without even the implied warranty of                  |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU           |
# General Public License for more details.                                    |
#                                                                             |
# You should have received a copy of the GNU General Public License           |
# along with this program.  If not, see <http://www.gnu.org/licenses/>")      |
#-----------------------------------------------------------------------------+
GERRIT   = wpmirrordev <at> gerrit.wikimedia.org
PORT     = 29418
WMFLABS  = wpmirrordev <at> bastion.wmflabs.org

all: clone hooks pull checkout branch edit diff add commit rebase review

clone:
        # create directory `dumps' and initialize a repository in it
        # copy all commit objects and head references (from remote to local)
        # add `remote repository reference' named `origin' (saves typing)
        # add `remote heads' named `origin/[head-name]'
        # add `HEAD' to track `origin/master'
        git clone ssh://$(GERRIT):$(PORT)/operations/dumps

hooks:
        # get `pre-commit-hook' to add `change id' to commit summary
        scp -p -P $(PORT) $(GERRIT):hooks/commit-msg ~/dumps/.git/hooks/.
        cd dumps; git review -s

pull:
        # list `remote heads'
        cd dumps; git branch -r
        # setup tracking branch `ariel'
        cd dumps; git branch --track ariel origin/ariel
        # add new commit objects (if any)
        # update `remote heads'
        cd dumps; git fetch origin
        # update `local heads' (`master' and `ariel') to `remote-heads'
        # merge `origin/HEAD' into `HEAD'
        cd dumps; git pull origin

checkout:
        # point `HEAD' to `ariel's commit object
        cd dumps; git checkout ariel
        cd dumps; git status

branch:
        # create head `wpmirrordev'
        # point `wpmirrordev' to `ariel's commit object
        cd dumps; git branch wpmirrordev ariel
        # point `HEAD' to `wpmirrordev's commit object
        cd dumps; git checkout wpmirrordev
        cd dumps; git status

edit:
        # apply patched files
        cp temp/* dumps/xmlfileutils/.

diff:
        # diff files (but not added files) against `HEAD'
        cd dumps; git diff
        # list changed files against `HEAD'
        cd dumps; git status

add:
        # stage the files to be committed
        cd dumps/xmlfileutils; git add mwxml2sql.c sql2txt.c sqlfilter.c
        cd dumps/xmlfileutils; git add Makefile
        # diff added files against `HEAD'
        cd dumps; git diff --cached
        # list changed files against `HEAD'
        cd dumps; git status

commit:
        # create `commit object'
        # point `HEAD' to the new `commit object'
        cd dumps; git commit -m "Fix for compatibility with help2man
and Debian Policy"
        # list all commits from `HEAD' back to initial commit
        cd dumps; git log

rebase:
        # add new commit objects (if any)
        # update `remote head' `origin/ariel'
        # merge `origin/ariel' into `ariel'
        # point `ariel' to `origin/ariel's commit object
        cd dumps; git pull origin ariel
        # rebase `wpmirrordev' branch on updated `ariel' head
        cd dumps; git rebase ariel

review:
        # push changes to Gerrit
        cd dumps; git review -R ariel

#-----------------------------------------------------------------------------+

shell:
        ssh -A $(WMFLABS)

purge:
        rm -r dumps

clean:
        rm -f *~

Sincerely Yours,
Kent

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Dear Ariel,

I am having some trouble submitting the patches.  On-line examples
that I have seen do not cover the case of submitting patches to a
branch.  Here is what I have tried so far:

0) clone, hooks, and review setup

(shell) git clone ssh://wpmirrordev <at> gerrit.wikimedia.org:29418/operations/dumps
(shell) scp -p -P 29418
wpmirrordev <at> gerrit.wikimedia.org:hooks/commit-msg ~/dumps/.git/hooks/.
(shell) cd dumps
(shell) git review -s
(shell) git status

1) pull the ariel branch

(shell) git pull origin master
(shell) git branch ariel
(shell) git pull origin ariel  # throws errors
(shell) git status # emits a long list of revisions

2) commit

(shell) git commit -a # needed to quell errors from the `pull'
(shell) git checkout ariel
(shell) git status

3) create branch for the patches

(shell) git branch wpmirrordev master
(shell) git checkout wpmirrordev
(shell) git status

4) apply patches

(shell) cp ../patched-files/* xmlfileutils/.
(shell) git diff
(shell) git status

5) commit the patches

(shell) cd xmlfileutils
(shell) git add mwxml2sql.c sql2txt.c sqlfilter.c Makefile
(shell) git diff --cached
(shell) git commit
(shell) git status

6) rebase

(shell) git pull origin master
(shell) git rebase master
(shell) cat .gitreview
gerrit]
host=gerrit.wikimedia.org
port=29418
project=operations/dumps.git
<<<<<<< HEAD
defaultbranch=master
=======
defaultbranch=ariel
>>>>>>> 3b82bbea24f999f1a5af721d37ec0684615bc3ae
(shell) git checkout ariel  # need to fix error in .gitreview

7) submit for review

(shell) review -R ariel
Enter passphrase for key '/home/wikimedia/.ssh/id_rsa':
Creating a git remote called "gerrit" that maps to:
        ssh://wpmirrordev <at> gerrit.wikimedia.org:29418/operations/dumps.git
Enter passphrase for key '/home/wikimedia/.ssh/id_rsa':
You have more than one commit that you are about to submit.
The outstanding commits are:

bf268a2 (HEAD, origin/master, origin/HEAD, gerrit/master, ariel) pep8
whitespaces fixing
6dff615 pep8: E302 expected 2 blank lines, found 1
671beaa .pep8 configuration file
1235eaa Merge "Add .gitreview file"
9a814c2 README is obsolete :(
67a61fa Add .gitreview file
47b6db7 add CC-BY_SA license for text, plus pointer to terms of use
689fa7c dump iwlinks table
e4bc572 Kill .cvsignore, svn ignore is doing the same
25f4a46 svn:eol-style native

Is this really what you meant to do?
Type 'yes' to confirm: yes
Enter passphrase for key '/home/wikimedia/.ssh/id_rsa':
X11 forwarding request failed on channel 0
remote: Processing changes: refs: 1, done
To ssh://wpmirrordev <at> gerrit.wikimedia.org:29418/operations/dumps.git
 ! [remote rejected] HEAD -> refs/publish/ariel/ariel (no new changes)
error: failed to push some refs to
'ssh://wpmirrordev <at> gerrit.wikimedia.org:29418/operations/dumps.git'
make: *** [review] Error 1
wikimedia <at> darkstar-7:~$ QDBusConnection: session D-Bus connection
created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before
QCoreApplication. Application may misbehave.
QSystemTrayIcon::setVisible: No Icon set
Connecting to deprecated signal
QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString)

Any help is welcome.

Sincerely Yours,
Kent

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

I'm seeing some issues with the history phase of the wikidata dumps
taking a huge amount of memory and causing the server they are on to
swap. I've shot the jobs and left less worker running on the one host
for now; I'll investigate in depth tomorrow.

Ariel

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Dear Ariel,

0) INTRO

I am close to releasing WP-MIRROR 0.6.  It will exhibit reliability
and performance improvements in all areas of operation.

As a part of the development process, I have been testing `mwxml2sql'
with a view towards using it to replace `importDump.php' in WP-MIRROR
0.6.  These tests have worked out well.

There are however some issues that I should discuss with you.

1) Packaging

I distribute WP-MIRROR as a DEB package.  In order to use `mwxml2sql',
I would have to package your tools as a separate DEB package.  This I
have done.  However, in the process, I had to apply some patches; and
the question now arises as to how to submit them upstream.

2) Makefile

I have patched the `Makefile' that you distribute with `mwxml2sql'
because:  a) the `install' target must use `install' rather than `mv';
and b) it lacked a `deinstall' target.  Both changes are required by
Debian policy.

3) Man pages

Man pages are also required by Debian policy.  To that end, I have
written man pages for `mwxml2sql', `sql2txt', and `sqlfilter'.
However, the better approach would be to patch those tools so that man
pages could automatically be generated using `help2man'.  The later
approach has the benefit of eliminating duplication, and hence, helps
keep code and documentation in sync.

4) Upstream

I would like to know:

a) if patches are welcome upstream; and, if so,
b) what procedures you prefer for submitting, reviewing, and applying
patches; and
c) whether you would prefer that I submit the man pages I wrote, or
submit patches to your utilities to make them compatible with
`help2man'.

Sincerely Yours,
Kent

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

Hi,

I noticed that some pagecounts data files are missing, namely the files in the 
interval (20090921160000 - 20091001000000) (ends excluded).

See http://dumps.wikimedia.org/other/pagecounts-raw/2009/2009-09/

Does anybody know the reason why these data are missing?

Best,

--

-- 
Giovanni Luca Ciampaglia

Postdoctoral fellow
Center for Complex Networks and Systems Research
Indiana University

✎ 910 E 10th St ∙ Bloomington ∙ IN 47408
☞ http://cnets.indiana.edu/
✉ gciampag <at> indiana.edu

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

32	May 2013
23	April 2013
36	March 2013
22	February 2013
37	January 2013
9	December 2012
17	November 2012
18	October 2012
16	September 2012
3	August 2012
18	July 2012
48	June 2012
21	May 2012
40	April 2012
18	March 2012
13	February 2012
34	January 2012
8	December 2011
11	October 2011
19	September 2011
14	August 2011
5	July 2011
25	June 2011
6	May 2011
16	April 2011
32	March 2011
12	February 2011
8	January 2011
16	December 2010
20	November 2010
10	October 2010
5	September 2010
12	August 2010
21	July 2010
14	June 2010
3	May 2010
19	April 2010
31	March 2010
1	October 2009
5	September 2009
2	August 2009
5	July 2009
10	June 2009
5	May 2009
1	February 2007
1	March 2006
1	January 2002
1	June 2001
2	May 2001
1	February 2001

Inquiry re: XML data

mwxml2sql

mwxml2sql

wikidata dumps woes

please about image

mwxml2sql

(no subject)

Pagecounts data missing (2009/09/21 - 2009/10/01)

Incremental XML dumps GSoC proposal

Encodage with french dump

GSoC 2013 - Incremental XML Dumps

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31