Brion Vibber | 27 Jun 20:36 2016

ogv.js update & various TMH updates coming

I'm landing a big update to the ogv.js Ogg player shim for Safari/IE/Edge; ogv.js 1.1.3 fixes some seek bugs with the future video.js frontend code, and has a bunch of performance & correctness updates.

On slower machines (like older iPhones, or old Windows machines running IE) ogv.js now makes better use of dual-core CPUs to overlap decoding one frame with drawing another, etc. It's also much more aggressive about keeping audio and video synchronized, making short delays instead of letting it get wildly out of sync followed by a weird catchup period.

With the seek & sync bugs mostly cleaned up, I'm also turning some of my A/V attention to iOS, making updates to my OGVKit Objective-C library which uses native builds of the Ogg and VPX libraries, able to handle higher resolutions on slower phones than ogv.js can. I'll track down the current iOS mobile app folks and see what the best way to work with them on this is. :)

Back to TimedMediaHandler -- Derk-Jan has been doing some more updates on the Video.JS-based new frontend code, which I'm reviewing through this week and hope to have merged and cleaned up for further testing. In addition to looking more modern and being more actively maintained, video.js is more mobile-friendly -- deploying the new frontend when it's ready will get us more consistent media playback on Android, and ogv.js-based playback on iPhone and iPads in the mobile view.

I've also got a number of back-end fixes to finish up such as a new maintenance script for re-running video transcodes. It currently needs just one more option, which is a 'throttle' so we can background-run it unattended without worrying it'll fill up the job queues.

In the closer future, 720p and 1080p Ogg transcodes should start rolling out in the next few weeks for manual switching, and we'll see how it all goes. :)

Farther out in the future -- I did testing last month on transcoding, and some preliminary planning for how to do adaptive streaming (seamless resolution switching as necessary). I've got a pretty good handle on how it will work for native WebM playback, and a good theoretical model for how it should be possible with ogv.js and Ogg playback by creating an analogue of the native Media Source Extensions APIs, and encoding / dividing up the packetized files _just so_.

Note that creating adaptive-switchable Ogg transcodes will require customizing ffmpeg2theora and/or creating a separate similar tool specific to our needs. Shouldn't require any new daemons/services though.

-- brion
Multimedia mailing list
Multimedia <at>
Brion Vibber | 21 Jun 18:22 2016

TMH & ogg tweaks coming up

I've gone ahead and self-merged a couple small tweaks to TimedMediaHandler's .ogv handling, which can go out with the next release if there's no rush to deploy them earlier:

Enabled 720p and 1080p .ogv output, using existing variable bitrate configuration:

This will enable new generation of high-definition .ogv transcodes for files whose sources are high-definition. The files will be smaller than they would be at constant bitrate, running somewhere from 'about the size of existing webm' to 'up to 40% larger than webm' depending on which configuration the webms were generated against. :)

Disabled forcing of audio output to max 44.1 kHz for .ogv output due to bugs in the resampling leading to a/v sync mismatch on long videos:

Once existing .ogv transcodes are redone, this will fix a/v sync drift on the .ogv versions of longer videos with 48kHz audio, such as

In the next week I'm going to try to finish up a couple more things:

Update ogv.js to 1.1.2 release (just released alpha.7, probalby final):

I've made major improvements to ogv.js on slower machines, with better audio/video sync maintenance and more effective use of multiple CPU cores. This should improve performance on old 32-bit iOS devices as well as older Windows machines running IE.

Maintenance script to re-run video transcodes in bulk:

I'm replacing several half-working maintenance scripts in TimedMediaHandler with a new one that actually works. Once I've added a 'throttle' mode I'll commit it, and we can bulk re-run transcodes in production to update things to current configurations.

Re-tuning .webm and low-res .ogv transcodes for VBR:

Transcode settings other than the HD .ogvs are currently specifying constant bitrates, which are higher than they need to be for a lot of material in order to provide enough headroom for high-complexity scenes. A cleaner variable bitrate configuration will save space on most material, while using more bits only when needed. (That will free up more space for the HD .ogv transcodes.)

...And then try to finish up more of the video.js front-end with TheDJ and anybody else who's interested. We're so close! And that'll get us video on mobile as well... :D

-- brion

Multimedia mailing list
Multimedia <at>
Brion Vibber | 15 Jun 04:19 2016

Exploring VP9 as a progressive still image codec

I spent some time between projects today exploring the idea of progressive image decoding using the VP9 video codec -- sort of a mashup of progressive JPEG and WebP.

Like a progressive JPEG, each resolution step (a separate frame of the "video") encodes only the differences from the previous resolution step. Like WebP, it's more space-efficient than the ancient JPEG codec.

This sort of technique might be useful for lazy-loading images in our modern internet, where screen densities keep going up and network speeds can vary by a factor of thousands. On a slow network the user sees immediate feedback during load, and on a fast network they can reach full resolution quickly, still in less bandwidth than a JPEG. And since JS would have control of loading, we can halt cleanly if the image scrolls offscreen, or pick a maximum resolution based on actual network speed.

Detail notes on my blog:

Sample page if you just want to look at some decoded images at various resolutions (loading not optimized for slow networks yet!):

It looks plausible, and should be able to use native VP9 decoding in Firefox, Chrome, and eventually MS Edge in some configurations with a JavaScript fallback for Safari/etc. Currently my demo just plops all the frames into a single .webm, but to avoid loading unneeded high-resolution frames they should eventually be in separate files.

-- brion
Multimedia mailing list
Multimedia <at>
Brion Vibber | 24 May 18:44 2016

Variable bit rate encoding for video transcodes?

In the past, we've had a mixture of fixed bitrates and quality-based settings for producing video transcodes.

Each has its advantages: fixed bitrates are more predictable for watching while streaming, while fixed quality settings allow for reducing the bitrate on low-complexity scenes to save bandwidth (and increasing it on high-complexity scenes to keep quality up!)

Since "download and watch it later" is less of a thing on today's internet than "stream it right now!", I'd been leaning for a while towards moving more things to fixed bitrates. However, I'm starting to come down on the side of a fixed quality setting with a variable bitrate...

Overall variable rate encoding should lead to lower bandwidth usage for most parts of most files, while still maintaining high quality on scenes that need it.

The downside is that a high-complexity scene encoded at a higher bitrate might cause buffering to run out during playback that had been working ok on earlier scenes at a lower bitrate.

Once we support adaptive streaming (using MPEG-DASH, or something like it) the system should be able to provide a detailed enough manifest[1] to show which segments of the file are low-bandwidth and which are high-bandwidth, so if there's a bandwidth limitation that stops us from viewing one particular segment at the current resolution, we can bump down and then bump resolution back up again when the bandwidth usage goes down.

If there's no strong objection, I'm going to tinker with the quality settings for WebM and Ogg Theora video transcodes to try to find quality settings I'm happy with that result in reasonable bandwidth averages.

[1] An MPEG-DASH manifest (.mpd) specifies a target bitrate on each resolution representation, but the actual segments can be different sizes. When they're specified as byte ranges of a source file, the exact segment size is conveniently available!

-- brion

Multimedia mailing list
Multimedia <at>
Brion Vibber | 5 May 15:49 2016

Reviving SVG client-side rendering task

For the last decade we've supported uploading SVG vector images to MediaWiki, but we serve them as rasterized PNGs to browsers. Recently, display resolutions are going up and up, but so is concern about low-bandwidth mobile users.

This means we'd like sharper icons and diagrams on high-density phone displays, but are leery of adding extra srcset entries with 3x or 4x size PNGs which could become very large. (In fact currently MobileFrontend strips even the 1.5x and 2x renderings we have now, making diagrams very blurry on many mobile devices. See - fix in works.)

Here's the base bug for SVG client side rendering:
I've turned it into an "epic" story tracking task and hung some blocking tasks off it; see those for more details.

TL;DR stop reading here. ;)

One of the basic problems in the past was reliably showing them natively in an <img>, with the same behavior as before, without using JavaScript hacks or breaking the hamlet caching layer. This is neatly resolved for current browsers by using the "srcset" attribute -- the same one we use to specify higher-resolution rasterizations. If instead of PNGs at 1.5x and 2x density, we specify an SVG at 1x, the SVG will be loaded instead of the default PNG.

Since all srcset-supporting browsers allow SVG in <img> this should "just work", and will be more compatible than using the experimental <picture> element or the classic <object> which deals with events differently. Older browsers will still see the PNG, and we can tweak the jquery.hidpi srcset polyfill to test for SVG support to avoid breaking on some older browsers.

This should let us start testing client-side SVG via a beta feature (with parser cache split on the user pref) at which point we can gather more real-world feedback on performance and compatibility issues.

Rendering consistency across browser engines is a concern. Supposedly modern browsers are more consistent than librsvg but we haven't done a compatibility survey to confirm this or identify problematic constructs. This is probably worth doing.

Performance is a big question. While clean simple SVGs are often nice and small and efficient, it's also easy to make a HUGEly detailed SVG that is much larger than the rasterized PNGs. Or a fairly simple small file may still render slowly due to use of filters.

So we probably want to provide good tools for our editors and image authors to help optimize their files. Show the renderings and the bandwidth balance versus rasterization; maybe provide in-wiki implementation of svgo or other lossy optimizer tools. Warn about things that are large or render slowly. Maybe provide a switch to run particular files through rasterization always.

And we'll almost certainly want to strip comments and white space to save bandwidth on page views, while retaining them all in the source file for download and reediting.

Feature parity also needs more work. Localized text in SVGs is supported with our server side rendering but this won't be reliable in the client; which means we'll want to perform a server side transformation that creates per-language "thumbnail" SVGs. Fonts for internationalized text are a big deal, and may require similar transformations if we want to serve them... Which may mean additional complications and bandwidth usage.

And then there are long term goals of taking more advantage of SVGs dynamic nature -- making things animated or interactive. That's a much bigger question and has implementation and security issues!

-- brion
Multimedia mailing list
Multimedia <at>
Brion Vibber | 25 Apr 13:41 2016

Hypermedia linking in new media types ("Myst for Wikipedia")

At Wikimedia Conference in Berlin I met with Felix from Wikimedia Ghana, who is super interested in getting more immersive media available such as 360-degree panoramic photos ("photo spheres"); I showed him the tool labs widget using panellum to do WebGL spherical photo viewing -- see -- and he was very excited to see that it's something we could probably work out how to integrate in the nearish term.

That got me thinking more generally about new media types (video, panos, stereoscopic photos/videos/panos, 3D models, interactive diagrams, etc) and how we can extend them to support annotations and linking in a way that could create immersive visual experiences with the same kind of rich information and interlinking that Wikipedia is famous for in the world of text articles.

Ladies and gentlemen, I give you: "Epic saga: immersive hypermedia (Myst for Wikipedia)"

I would be real interested to hear y'all's ideas on medium to long term feasibility and desirability of this sort of system, and what we can pull more directly into the short term.

For instance I would love to get the panoramic / spherical viewers integrated in MMV, which is much easier than figuring out how to do clickable annotations in 3d environment. ;)

Medium term, I would also love to see us look at the annotation system that's on Commons done in site JS, and see if we can build a future-extensible system that's more integrated into the wiki and can be used in MMV.

Longer term, I think it'll just be nice to have these kinds of long-term goals to work towards.

Thoughts? Ideas? Am I crazy, or just crazy enough? ;)

-- brion
Multimedia mailing list
Multimedia <at>
Brion Vibber | 13 Apr 21:41 2016

Preparing for VP9/Opus video/audio transcodes

In addition to the smaller bandwidth requirements for VP9 video encoding versus Theora or VP8, Microsoft is adding support for VP9 video and Opus audio in WebM to Windows 10 in the summer 2016 update.

Currently in Win10 preview builds this only works in Edge when using Media Source Extensions, and VP9 is disabled by default if not hardware-accelerated, but it's coming along. :)

If the final version lands with suitable config, users of Edge version 15 and later shouldn't need the ogv.js JavaScript decoding shim to get media playback on Wikipedia. Neat!

Things still to do in TimedMediaHandler:
* add transcode output for audio-only files as Opus in WebM container (Brion)
* keep working on the Kaltura->VideoJS front-end switch to make our lives easier fixing the UI (Derk-Jan & Brion) and to prep for...
* eventually we'll want to use MPEG-DASH manifests and Media Source Extensions to implement playback that's responsive to network and CPU speed and can switch resolutions seamlessly. This may or may not be a prerequisite for Win10 Edge playback if MS sticks with the MSE requirement.
* consider improving the transcode status overview at Special:TimedMediaHandler; it reports errors in a way that doesn't scale well.

Things still to do in Wikimedia site config:
* add VP9/Opus transcodes to our config (audio, 240p, 360p, 480p, 720p, 1080p definitely; consider 1440p and 2160p for Ultra-HD videos)
* consider dropping some VP8 sizes (desktop browsers that support VP8 should all support VP9 now; old Android versions that don't grok VP9 might be main target remaining for VP8)

Things to consider:
* VP9 is slower to encode than VP8, and a transition will require a lot of back-running of existing files. We *will* need to assign more video scalers, at least temporarily.
* I started writing a client-side bot to trigger new transcodes on old files. Prefer I should finish that, or prep a server side script that someone in ops will have to babysit?
* in future, the ogv.js JS decoder shim will still be used for Safari and IE 11, but I may be able to shift it from Theora to VP9 after making more fixes to the WebM demuxer. Decoding is slower per pixel but at a lower resolution you often get higher quality because of better compression and handling of motion -- and bandwidth usage is much better, which should make it a win on iPhones. This means eventually we may be able to reduce or drop the Ogg output. This will also tie in with MPEG-DASH adaptive streaming, so should be able to pick best size for CPU speed more reliably than current heuristic.
* longer term, AOMedia codec will arrive (initial code drop came out recently, based on VP10) with definite support from Google, Mozilla, and Microsoft. This should end up supplementing VP9 in a couple years, and should be even more bandwidth-efficient for high resolutions.

-- brion

Multimedia mailing list
Multimedia <at>
Gilles Dubuc | 13 Apr 15:11 2016

Image scaling on the fly (closed source, AWS only)

Claims resizing JPEGs "any size" in 25ms or less on m3.medium AWS instances.

While this is closed source, and is part of a worrying trend of closed SaaS frameworks one has to pay by the hour, the author reveals enough technical details to figure out how it works. Namely:

- the inspiration for the code is an unnamed Japanese paper which describes how to process the Y, U and V components of a JPEG in parallel
- it uses a similar technique as the jpeg:size option of ImageMagick, whereby only parts of the JPEG are read, instead of every pixel, according to the needed target thumbnail size
- it leverages "vector math" in the processor, which I assume means AVX instructions and registries

Essentially, it's parallelized decoding and resizing of JPEGs, using hardware-specific instructions for optimization.

Of course writing something similar would be a large undertaking. Let's hope that the folks who work on ImageMagick/GraphicsMagick take note and try to do just that :)

I have confirmation that it's extremely fast from pals at deviantArt (whose infrastructure is on AWS) who tried it out. To the point that they're likely getting rid of their storage of intermediary resized images.

I have a feeling that we'll be seeing more of this sort of hardware-optimized JPEG decoding/transcoding once Intel releases their first CPUs with integrated FPGAs, which is supposed to happen soon-ish. Unfortunately these Xeon CPUs will be released "in limited quantities, to cloud providers first". Here's that annoying trend again...
Multimedia mailing list
Multimedia <at>
Gilles Dubuc | 13 Apr 14:54 2016

Facebook surround 360

Facebok just announced this 360 video camera, whose hardware and stitiching software will be open source and released this summer.
Multimedia mailing list
Multimedia <at>
Daniel Mietchen | 18 Mar 12:54 2016

Candidature for the European Commission Open Science Policy Platform

Dear all,

the European Commission is setting up an advisory group for its Open
Science Agenda:

This group - called 'Open Science Policy Platform' (OSPP) - has an
open call for candidates, and I am amongst those who have publicly
declared their interest in serving on this group.

Since many of my Open Science activities have a Wikimedia component, I
posted my candidacy at
and would welcome endorsements from individuals and groups, especially
if you are active at the interface of Open Science with Wikimedia or
other open movements.

I also set up
in case others here would like to serve as candidates as well, which I
would welcome.

Thanks and cheers,


Multimedia mailing list
Multimedia <at>
Claudia Garád | 10 Mar 13:23 2016

[Wikivideo-l] Invitation: Explainer video workshop during Wikimania Pre-Conference

Hi everyone,

are you coming to Esino Lario in June and looking for some interesting things to do during Wikimania Pre-Conference? Well, perhaps this is for you: like last year, we plan to host a workshop for creating animated explainer videos for Wikimedia projects:

Videos have become a popular format for the dissemination of information - especially short animated explainer videos which provide a quick overview over complex topics. The same principle applies to Wikipedia: videos can enrich exisiting content and provide an entertaining and easily comprehensible access to free knowledge. Together with experts from the simpleshow foundation Wikimedia Österreich offers workshops for Wikimedians which are designed to learn the tricks of the trade of screenwriting for explainer videos.
The workshop might also be interested for people who attended last year, as we will also introduce a new freeware, which enables you to produce the videos yourself!

Like the other submissions we need at least 3-5 interested Wikimedians to sign up for the workshop. So please do so at the link above if you are interested :-)

Content of the workshop

   * Interested participants will learn the basic skills for creating plots of explainer videos: how to explain complex topics in a short and comprehensible way, how to work with cut-out animation, storyboard conception and visualization.
   *  The participants can create and refine their scripts and storyboards together with the simpleshow foundation after the workshop.
   *  Depending on the topic & previous research participants can turn their scipts directly into an ilustrated video using “mysimpleshow”-freeware.
    * The realization and production of the finished storyboard could be done either using the “mysimpleshow”-freeware or by the simpleshow foundation. The result will be published under the CC-BY-SA license on Wikimedia Commons.
   *  The screenwriters can use the videos for Wikipedia articles and other purposes.

What do you need?

   * Interest in storytelling
   * A topic for an explainer video
   * First research on the topic (Wikipedia article niveau)
   * No previous knowledge in screenwriting or video production necessary

Please sign up here:

Claudia Garád
Executive Director

Wikimedia Österreich - Verein zur Förderung Freien Wissens
Siebensterngasse 25/15
1070 Wien
0699 141 28615
Wikivideo-l mailing list
Wikivideo-l <at>