Wikimedia Toolserver users' discussion ()

headers Marcin Cieslak | 1 Jan 2009 18:28

Hi,

Big thank you for providing IPv6 for the stable toolserver.

I think I was using it unknowingly since around Dec 14th, excellent idea 
- so thanks again.

--Marcin

yarrow (s1/s3) is now up.  as it wasn't replicating during the downtime, the
replication lag is fairly high for the moment.

	- river.

DaB. wrote:

I am following up on a discussion here in October:

>> I run a script that downloads 1200 MB every night
> 
> if you do this, please save the data at 
> 
> /mnt/user-store/
> 
> (create a directoy there). So every usercan use the data and they have to 
> downloaded one 1 time.

Since I am becoming involved with statistics too, I have setup such a 
scheme in /mnt/user-store/stats.  Data files starting from 1 October 
2008 are currently available (emijrp asked if I could get older files 
too, which should be doable but I haven't looked into it yet). I still 
have to fine tune the update process, but basically a cron task will 
take care of this at least every day (probably more often, but I have to 
see when the original files are actually updated)

Let me know if anyone else is interested in using this data.

> Perhaps there is a better way (rsync or something) to get the data from the 
> source.

I use wget; it will not download files twice unless they have been 
modified (which should not happen). Also, files are already gz'ipped, so 
compression would not be of much use here. Even though rsync is a better 
solution on paper, all in all, I don't think it would improve the 

hi,

as Wikimedia deleted the MySQL binlogs required for replicating s3 on the
Toolserver, replication is now halted.  a full dump/import is required to fix
replication.  because disk space on yarrow is limited at the moment, and we are
about to receive new servers, i might wait until this before reimporting.
(however, if that's likely to take a long time, i might do it sooner.)

	- river.

<div>
<p>Hello, River,</p>
<div><br></div>
<div>could you please to clarify "about to receive" and "long time" let say selecting a proper time measure: days, weeks, months. Our services depend on replicated data, so the delay might influence us and our users somehow.</div>
<div><br></div>
<div>Mashiah<br><br><div class="gmail_quote">2009/1/5 River Tarnell <span dir="ltr">&lt;<a href="mailto:river <at> loreley.flyingparchment.org.uk">river <at> loreley.flyingparchment.org.uk</a>&gt;</span><br><blockquote class="gmail_quote">
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br><br>
hi,<br><br>
as Wikimedia deleted the MySQL binlogs required for replicating s3 on the<br>
Toolserver, replication is now halted. &nbsp;a full dump/import is required to fix<br>
replication. &nbsp;because disk space on yarrow is limited at the moment, and we are<br>
about to receive new servers, i might wait until this before reimporting.<br>
(however, if that's likely to take a long time, i might do it sooner.)<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;- river.<br>
-----BEGIN PGP SIGNATURE-----<br><br>
iD8DBQFJYWeSIXd7fCuc5vIRAo03AJ9ouEHS98oTwKPugzcUsc8Lp+LmSgCffFmd<br>
gK3xul+k9UinKPPHH5Q3PWg=<br>
=pnlB<br>
-----END PGP SIGNATURE-----<br><br>
_______________________________________________<br>
Toolserver-l mailing list<br><a href="mailto:Toolserver-l <at> lists.wikimedia.org">Toolserver-l <at> lists.wikimedia.org</a><br><a href="https://lists.wikimedia.org/mailman/listinfo/toolserver-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/toolserver-l</a><br>
</blockquote>
</div>
<br>
</div>
</div>

Mashiah Davidson schrieb:
> Hello, River,
> 
> could you please to clarify "about to receive" and "long time" let say
> selecting a proper time measure: days, weeks, months. 

It will be about a month until the new boxes are online.

-- daniel

Il giorno 05/gen/09, alle ore 12:23, Daniel Kinzler ha scritto:

> Mashiah Davidson schrieb:
>> Hello, River,
>>
>> could you please to clarify "about to receive" and "long time" let  
>> say
>> selecting a proper time measure: days, weeks, months.
>
> It will be about a month until the new boxes are online.
>
> -- daniel

Hmm... that's a long time. Can you restart the replication before  
please?

Pietrodn
powerpdn <at> gmail.com

Pietrodn schrieb:
> Il giorno 05/gen/09, alle ore 12:23, Daniel Kinzler ha scritto:
> 
>> Mashiah Davidson schrieb:
>>> Hello, River,
>>>
>>> could you please to clarify "about to receive" and "long time" let  
>>> say
>>> selecting a proper time measure: days, weeks, months.
>> It will be about a month until the new boxes are online.
>>
>> -- daniel
> 
> Hmm... that's a long time. Can you restart the replication before  
> please?

It's not a matter ofrestarting. The process is: take a slave server out of
rotation and make a dump (I don't have access on the main cluster, so I can't do
that), then copy the dump over, import it, and *then* start replication. This
takes quite some time (a week, i'd guess), and is a lot of hassle. Also, during
import, s3 is going to be completly unavailable for several days, instead of
just having old data.

Doing this twice is no fun. So it's not an easy choice. I leave it to river to
make it :)

-- daniel

Il giorno 05/gen/09, alle ore 12:34, Daniel Kinzler ha scritto:

> Pietrodn schrieb:
>> Il giorno 05/gen/09, alle ore 12:23, Daniel Kinzler ha scritto:
>>
>>> Mashiah Davidson schrieb:
>>>> Hello, River,
>>>>
>>>> could you please to clarify "about to receive" and "long time" let
>>>> say
>>>> selecting a proper time measure: days, weeks, months.
>>> It will be about a month until the new boxes are online.
>>>
>>> -- daniel
>>
>> Hmm... that's a long time. Can you restart the replication before
>> please?
>
> It's not a matter ofrestarting. The process is: take a slave server  
> out of
> rotation and make a dump (I don't have access on the main cluster,  
> so I can't do
> that), then copy the dump over, import it, and *then* start  
> replication. This
> takes quite some time (a week, i'd guess), and is a lot of hassle.  
> Also, during
> import, s3 is going to be completly unavailable for several days,  
> instead of
> just having old data.
>
> Doing this twice is no fun. So it's not an easy choice. I leave it  
> to river to
> make it :)
>
> -- daniel

Oh, I understand. I didn't think it was such a long and difficult  
process.

Pietrodn
powerpdn <at> gmail.com

How long are the binlogs kept for on Wikimedia servers?

Surely it would be possible to take a dump now, import it to s3, start
replication, then import the same dump onto the new server, and let it
catch up from a month of replag?

Of course, this wouldn't be possible if the binlogs are not kept for that long.

2009/1/5 Pietrodn <powerpdn <at> gmail.com>:
> Il giorno 05/gen/09, alle ore 12:34, Daniel Kinzler ha scritto:
>
>> Pietrodn schrieb:
>>> Il giorno 05/gen/09, alle ore 12:23, Daniel Kinzler ha scritto:
>>>
>>>> Mashiah Davidson schrieb:
>>>>> Hello, River,
>>>>>
>>>>> could you please to clarify "about to receive" and "long time" let
>>>>> say
>>>>> selecting a proper time measure: days, weeks, months.
>>>> It will be about a month until the new boxes are online.
>>>>
>>>> -- daniel
>>>
>>> Hmm... that's a long time. Can you restart the replication before
>>> please?
>>
>> It's not a matter ofrestarting. The process is: take a slave server
>> out of
>> rotation and make a dump (I don't have access on the main cluster,
>> so I can't do
>> that), then copy the dump over, import it, and *then* start
>> replication. This
>> takes quite some time (a week, i'd guess), and is a lot of hassle.
>> Also, during
>> import, s3 is going to be completly unavailable for several days,
>> instead of
>> just having old data.
>>
>> Doing this twice is no fun. So it's not an easy choice. I leave it
>> to river to
>> make it :)
>>
>> -- daniel
>
> Oh, I understand. I didn't think it was such a long and difficult
> process.
>
> Pietrodn
> powerpdn <at> gmail.com
>
>
> _______________________________________________
> Toolserver-l mailing list
> Toolserver-l <at> lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>

--

-- 
Regards,

Simon Walker
User:Stwalkerster on all public Wikimedia Foundation wikis
Administrator on the English Wikipedia
Developer of Helpmebot and the ACC tool

Your donations keep Wikipedia running! Support the Wikimedia
Foundation today: http://www.wikimediafoundation.org/wiki/Donate

Mon	Tue	Wed	Thu	Fri	Sat	Sun

			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

74	May 2013
125	April 2013
69	March 2013
96	February 2013
39	January 2013
53	December 2012
78	November 2012
56	October 2012
179	September 2012
50	August 2012
48	July 2012
90	June 2012
50	May 2012
56	April 2012
115	March 2012
46	February 2012
47	January 2012
50	December 2011
83	November 2011
46	October 2011
95	September 2011
85	August 2011
36	July 2011
80	June 2011
30	May 2011
103	April 2011
54	March 2011
112	February 2011
74	January 2011
86	December 2010
197	November 2010
5	October 2010
97	September 2010
47	August 2010
32	July 2010
74	June 2010
94	May 2010
83	April 2010
106	March 2010
132	February 2010
60	January 2010
76	December 2009
49	November 2009
37	October 2009
113	September 2009
173	August 2009
94	July 2009
88	June 2009
42	May 2009
70	April 2009
53	March 2009
73	February 2009
106	January 2009
71	December 2008
33	November 2008
40	October 2008
63	September 2008
36	August 2008
56	July 2008
86	June 2008
43	May 2008
6	April 2008
9	March 2008
77	February 2008
23	January 2008
36	December 2007
95	November 2007
144	October 2007
32	September 2007
45	August 2007
48	July 2007
37	June 2007
33	May 2007
25	April 2007
22	March 2007
8	February 2007
22	January 2007
37	December 2006
20	November 2006
27	October 2006
79	September 2006
48	August 2006
68	July 2006
51	June 2006
42	May 2006
90	April 2006
62	March 2006
9	February 2006
37	January 2006
41	December 2005
23	November 2005
33	October 2005

IPv6 on stable - thank you

Re: [Toolserver-announce] maintenance 28th to 31st Dec: all servers

Statistics files on the toolserver (was: Re: 1200 MB every night)

s3 replication is halted

Re: s3 replication is halted

Re: s3 replication is halted

Re: s3 replication is halted

Re: s3 replication is halted

Re: s3 replication is halted

Re: s3 replication is halted