Gergo Tisza | 28 Jan 01:37 2015
Picon

Google Code-in accomplishments

Hi,

we had a great Google Code-in contest this year. Of the 28 multimedia-related tasks we had, 23 were closed, thanks to three very talented students, Danny (unicodesnowman), Geoffrey (Sn1per) and Mateusz (m4tx). Thanks all for participating and improving MediaWiki, it was a pleasure to work with you! I was continuously impressed by the high quality of the patches.

The full list of improvements/fixes:
  • Media Viewer now handles custom attributions (T67445) and the {{multiple image}} template (T85354), displays copy-pastable file names (T76680), uses the alt text if available (T66519, T75923), and some small UX/DX/metrics issues have been fixed (T71233T77272T68329T76923T75962). Progress was made towards warnings for trademarks (T77717) and a guided tour (T77726), and a prototype was made to test alternative history navigation strategies (T64266).
  • CommonsMetadata returns most dates in a sane format (T66014T63701) and is more flexible about styling in descriptions (T69887).
  • JSHint is now voting for VipsScaler (T63643).
  • Images are now served with a Timing-Allow-Origin header to expose more timing information (T76020).
  • Several bugs were fixed (T74084T86389T86032).

_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Gilles Dubuc | 15 Jan 15:30 2015
Picon

Swift thumbnail storage stopgap measure idea

Erik came up with the following suggestion on the whole thumbnails-stored-in-swift issue: would it be possible to store thumbnails in a separate Swift cluster than originals, and have the replica count setting for the thumbnail swift cluster set to 1?

The rationale being that this would solve the main issue of storage space waste without having to learn/test/deploy a new tool.
_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Gabriel Wicke | 13 Jan 21:44 2015
Picon

Wikia's thumbnailer service

Hi,

Wikia have developed a thumbnailer service called vignette, and are currently using to to serve about 50% of their thumbnail traffic:

https://github.com/Wikia/vignette

It's based on Clojure and imagemagick. License is eclipse, so pretty liberal.

Gabriel
_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Alexandros Kosiaris | 13 Jan 17:10 2015
Picon

Re: [Ops] Image serving performance discoveries

> cp1067 miss (0), cp1054 frontend miss (0)
> and not
> cp1053 miss (0), amssq43 miss (0), amssq48 frontend miss (0)
>
> (This might be a good time to ask, what do three varnish entries even mean?
> AFAIK we only have to tiers, front and back.)

Two tiers per DC (caching or otherwise). However it is a design
decision that the varnishes in the caching DCs (like esams where
amssq43,48 live) instead of passing directly to the mediawiki
appservers in the main DCs, go through the varnishes in the main DCs
(eqiad right now)

--

-- 
Alexandros Kosiaris <akosiaris <at> wikimedia.org>

_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Brian Wolff | 13 Jan 02:36 2015
Picon

Re: [Ops] Image serving performance discoveries


> which is an error in our logging (or log aggregation) system - some requests (apparently most of the slow ones) only pass through two tiers of varnish cache and XCache will look like 
> cp1067 miss (0), cp1054 frontend miss (0)
> and not
> cp1053 miss (0), amssq43 miss (0), amssq48 frontend miss (0)
>
> (This might be a good time to ask, what do three varnish entries even mean? AFAIK we only have to tiers, front and back.)

I believe its whether you hit eqiad or not. I believe non eqiad varnish hits eventually look things up in an eqiad varnish, so in essence there are three layers for some people.

--bawolff

_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Gilles Dubuc | 9 Jan 10:46 2015
Picon

Image serving performance discoveries

Hi everyone,

I recently looked very closely at client-gathered statistics about image serving performance from within Media Viewer. Looking more specifically at the effect of thumbnail pre-rendering at upload time (which has been live for a few months) and thumbnail chaining (which was live for a few weeks and has now been turned off). The main question I was looking to answer is whether either of those techniques improved performance as experienced by users.

Chaining, when combined with pre-rendering, had no noticeable effect on performance experienced by viewers. This is logical because pre-rendering means that the thumbnail generating gains only happen at upload time, therefore clients requesting the image later won't be affected. As for the effect on image scalers load, it was so insignificant that it couldn't be measured. Chaining is probably still useful for people requesting non-standard thumbnail sizes, which I'm not measuring since I've only been looking at Media Viewer, but the priority of addressing the community concerns over JPG sharpening in order to redeploy chaining seems much lower to me now if that's the only use case chaining will be useful for.

The big discovery in my research is that we set out to do pre-rendering based on a wrong assumption. When looking at performance statistics earlier last year, we clearly saw that Varnish misses performed a lot worse than Varnish hits (well, duh) and so we set out to deploy pre-rendering the thumbnail sizes Media Viewer needs in order to get drasticaly reduce the amount of the Varnish misses. The reduction didn't happen.

The wrong assumption was that each varnish miss is a case where the thumbnail requested has to be generated on the fly by the backend. The data I've just discovered shows that this is very rare for the thumbnail sizes Media Viewer currently uses. The vast majority of Varnish misses merely pulls from Swift a thumbnail that has already been rendered at some earlier point in time and just happens to not have been requested for a while. And that Swift pull + Varnish re-add is what's making the majority of Varnish misses perform worse than hits, not the need to generate the thumbnail with ImageMagick. The bottom line is that the thumbnail prerendering provided insignificant performance gains for this set of sizes. Infrequently requested thumbnails is the main problem, not the fact that they are rendered on the fly the first time they are requested.

It seems like the only way to increase image serving performance in our current setup is to increase the expiry value in Varnish and/or increase Varnish capacity. Right now 17% of image requests in Media Viewer are Varnish misses, and 99.5% of those are pulling an existing thumbnail from Swift. Varnish misses are twice as slow as hits on average.

I plan to disable pre-rendering next week in order to confirm these findings and determine for certain what percentage of image requests pre-rendering is useful for on the set of sizes Media Viewer currently uses.

If you want to dig into the data, the relevant tables on the analytics DB are MultimediaViewerNetworkPerformance* and more specifically the event_varnish*, event_timestamp and event_lastModified columns.

_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Fabrice Florin | 7 Jan 19:26 2015
Picon

Joining the Communications team

Hi folks,

I am happy to announce that I will be joining the Wikimedia Foundation's Communications team as Movement
Communications Manager for a period of six months. 

I have really enjoyed my work as product manager at WMF in the past 3 years, leading the development of new
tools like Notifications, Thanks, Beta Features, Media Viewer and other multimedia products [1]. 

One of the lessons I learned during that time is that 'better communications' are really important to make
our movement more effective. I also think that growing a 'culture of kindness' is key if we want to engage a
broader community [2]. I hope to support both of these goals in my new role. 

Starting this week, I will be working with the communications team on the Wikimedia blog and general
WMF/movement communications. My focus for the blog will include improving contributor workflow for
community members and WMF staff, and providing editorial guidance for contributors. I will also act as
the main contact point for authors submitting new blog posts, and shepherd the publication process.
Within movement communications, I will work to improve the availability, distribution, and timeliness
of communications from the WMF to the broader community. I will also manage a range of movement
communications tasks previously handled by Tilman Bayer, who is transitioning to the WMF Product team as
Senior Analyst. 

Going forward, senior engineer Gilles Dubuc will lead the multimedia team and take on some of my
responsibilities as product owner, until a new product manager is assigned to that team. Please contact
him directly with any questions related to our multimedia work.

I’d like to thank all my colleagues on the multimedia and product teams -- as well as our many community
champions -- for being such wonderful collaborators over the past few years. I am proud of what we
accomplished together, and I hope that the features we created can help many more people share knowledge
productively in years to come. Special thanks to Erik Moeller for his thoughtful guidance throughout the
past three years -- and for making this transition possible.

I'm delighted to take on this new assignment, and look forward to more collaborations with you all in the
coming year.

Onward!

Fabrice Florin
Movement Communications Manager
Wikimedia Foundation

[1] Fabrice's user page:
https://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF)

[2] 'A Culture of Kindness' blog post:
https://blog.wikimedia.org/2014/12/23/a-culture-of-kindness/

_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Andrew Lih | 19 Dec 20:06 2014

[Wikivideo-l] Opinions: Category for video?

What are your opinions of having a category for "articles with video" in WP? The one was created for en.wp is being considered for deletion. 

When the numbers were small, it made sense to try to track these, but perhaps Phoebe's suggestion that it stay a parent category and we label sub-categories may work? What have others in other WP communities been doing?


-Andrew

_______________________________________________
Wikivideo-l mailing list
Wikivideo-l <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
Derk-Jan Hartman | 11 Dec 16:55 2014
Picon

What to do with TimedMediaHandler

So for a while now, I have been toying a bit with TimedMediaHandler/MwEmbed/TimedText, with the long term goal of wanting it to be compatible with VE, live preview, flow etc.

There is a significant challenge here, that we are sort of conveniently ignoring because stuff 'mostly works' currently and the MM team having their plate full with plenty of other stuff:

1: There are many patches in our modules that have not been merged upstream
2: There are many patches upstream that were not merged in our tree
3: Upstream re-uses RL and much infrastructure of MW, but is also significantly behind. They still use php i18n, and their RL classes themselves are also out of date (1.19 style ?). This makes it difficult to get 'our' changes merged upstream, because we need to bring any RL changes etc with it as well.
4: No linting and code style checks are in place, making it difficult to assess and maintain quality.
5: Old jQuery version used upstream
6: Lot's of what we consider deprecated methodologies are still used upstream.
7: Upstream has a new skin ??
8: It uses loader scripts on every page, which really aren't necessary anymore now that we can add modules to ParserOutput, but since I don't fully understand upstream, i'm not sure what is needed to not break upstream in this regard.
9: The JS modules arbitrarily add stuff to the mw. variables, no namespacing there.
10: The RL modules are badly defined, overlap each other and some script files contain what should be in separate modules
11: We have 5 'mwembed' modules, but upstream has about 20, so they have quite a bit more code to maintain and migrate.
12: Brion is working on his ogvjs player which at some point needs to integrate with this as well (Brion already has some patches for this [1]).
13: Kaltura itself seems very busy and doesn't seem to have too much time to help us out. Since some of the code is however highly specific to their use cases, it becomes difficult to validate changes.

Oh and the file trees are disjunct between us and upstream, making git merging a lot more troublesome than it should be (anyone got tips ?).

This is maintenance hell, we need to come up with a plan here or we are going the way of getting so far out of sync that the cheapest solution will be to start from scratch...

So my questions:
1: Is there anything in upstream that we actually want ? I've been hearing about the 'update' that was still coming from there for over a year now, but due to how far both trees are now out of sync, i'm not really holding my breath for that anymore. The last 'proper sync' seems to have been 'Kaltura 1.7' in july 2012.[2] They are now at v2.21.9 
2: Who can think of a strategy to fix this ?
3: Or should we just split off our modules and let upstream sort this out ?
4: Should we consider starting something from scratch ?

DJ

I have done a bit of cleanup [3] with jshint and jscs on the modules that we use. There are some remaining problems [4], some of which are true bugs in the code. I don't intend to propose these changes for merging any time soon, since it will probably make consolidation of the two variants even more complicated, but I'll try to keep it up to date and maybe try to fix some of these bugs upstream or in MW.


_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Mark Holmquist | 9 Dec 00:34 2014
Picon

CORS on betalabs

Hi all!

I have a curious mystery on my hands in the mediawiki-config repo.

In the prod settings[0], CORS is turned on for public wikis. In the labs
settings[1], CORS is enabled for all subdomains of beta.wmflabs.org. I
was under the impression that unset variables in the labs settings were
set to whatever the prod settings had them at, but maybe I'm wrong...
or maybe I'm missing something that has turned CORS off.

If anyone has any thoughts, I'd appreciate them - I'm toying with a small
gadget to test cross-domain uploading.

[0] http://git.wikimedia.org/blob/operations%2Fmediawiki-config/master/wmf-config%2FInitialiseSettings.php#L13652
[1] http://git.wikimedia.org/blob/operations%2Fmediawiki-config/master/wmf-config%2FCommonSettings-labs.php#L179

--

-- 
Mark Holmquist
Software Engineer, Multimedia
Wikimedia Foundation
mtraceur@...
https://wikimediafoundation.org/wiki/User:MHolmquist
_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia
Dan Garry | 8 Dec 20:29 2014
Picon

CommonsMetadata API returning HTML?

Greetings, Multimedia Team!

Background: The Mobile Apps Team is working on a restyling of the way content the first fold of content is presented in the Wikipedia app. You can see this image to see what this looks like. Having a high-resolution image so prominently at the top of the page will likely drive a lot of clicks, so we're working on a lightweight image viewer to deal with file pages, which are poorly styled monstrosities on the mobile app. We're going to use the CommonsMetadata API to help us out. :-)

Problem: The CommonsMetadata API can sometimes return HTML [1]. Having HTML in the API response is a bit problematic for us. Native apps make next to no use of HTML when creating links or layouts, so we have to strip the HTML from every API response, lest it be displayed as plaintext to the user. In the short term this is fine, we can strip it and throw the information away. But in the long run it'd be better if the API didn't return HTML.

Our ask: Can the CommonsMetadata API please not return HTML in its responses? :-)

Thanks,
Dan

[1]: Run this query, and look at "artist" key. The API response has an HTML link in it.

--
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
_______________________________________________
Multimedia mailing list
Multimedia <at> lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/multimedia

Gmane