I don’t think I’ve boasted about developing Ultrawidify much on this blog. Maybe I should have, but then again: the audience of this blog is a) people who know me and b) people who use Ultrawidify and were bored enough to click that link in update notes. The point is — you’re all familiar with this extension.
A while back, I’ve noticed an issue on Twitch. It turned out that the video was a bit … well, twitchy. However, the issue seemed to be fairly uncommon, so I made a note in my test videos file and decided to kick the can a little farther down the road. “No big deal, it surely can wait.”
Well, turns out that the issue was a bigger deal than I thought. I’ve recently accidentally visited my facebook feed, and the twitching issue appeared — except worse. I tend to avoid twatter as well, but long story short: I stumbled on a tweet with embedded video.
On the plus side, the issue happens very often (more often than auto-detection interval), and it happens equally often even when the video is paused. Here we get the first (and perhaps the only) bit of good news for the day: auto-detection isn’t to blame — and since there’s exactly one other thing that could cause this behaviour, this means I already know where the problem is.
In Search of the Problem
In order to understand whats and whys of the problem, we have to take a quick look at how Ultrawidify crops the video. It’s very simple: it finds the video element and basically tells the browser: “Make sure the video is this wide and this tall and then enlarge it by this much,” where “this much” is whatever number auto-detection script (or user intervention) spat out. In programmer jargon, that’s called setting style string.
For technical reasons,1Blame the video alignment feature for that! Ultrawidify’s auto-detection will also “correct” the aspect ratio when there’s no need — in cases like this, the video would be enlarged by a factor of 1: same size as before.
This should, in theory, do the job just fine. In practice though, Ultrawidify isn’t the only thing doing that. Some sites will also tell the browser to make the video element that big because something on the page changed (example: switching between normal and theater mode on youtube). This effectively undoes any changes Ultrawidify has made to the page. And we really don’t want that, since that has the potential to uncrop the unnecessary letterbox.
Solution to this problem is easy enough at the first glance: we’re just gonna tell Ultrawidify to watch for sites trying to meddle with the video size. If the website tries to change anything, Ultrawidify will undo that change immediately. I think you can see where this is going.
Yes. Twitter is also watching for anything that would meddle with video sizes. If it detects that something changed how big the video element is supposed to be, it will undo that change.
Developer tools seem to agree with this assessment. In inspector view, video element is blinking like there’s no tomorrow while in the console, Ultrawidify is seen setting the same style string over and over again, and the zoom factor is always one. At this point you may wonder why the twitchy video if ultrawidify sets the zoom factor to one, and the answer is simple: twitter doesn’t.
With Twitter insisting that the video should be zoomed by a factor of 1.005, Ultrawidify wanting a zoom factor of 1, and neither being very keen on letting go. And this spells trouble for us.
In Search of Solution
If the site will just undo our changes, what can we do? Well, it turns out that there’s a way. As it turns out, there’s actually two sorts of CSS styles: author styles — which is CSS defined by the website you’re visiting; and user styles.
Through the magic of user styles, you — the user — have the final say over how the browser will display the site. If Facebook says the background of the page needs to be white, and you have user style that says the page background should be whatever meme is popular this week … well, tough luck Facebook. Nothing can override user styles, which makes them the perfect “fuck you, you’ll do what I tell you” card. We’ll take it.
There’s another piece of good news: you don’t have to define the styles in advance — you can make them up on the spot and tell the browser to use that. WebExtension API allows us to do that. There is a few caveats, though. Besides making up the style, you also have to make up a class name, attach it to the element and hope that the site won’t remove it. If you want to edit your style, you have to throw the old style at the browser and tell it that you want it removed. You also have to create a brand new style, tell the browser to use that.
Fortunately for us, it currently seems that the sites aren’t as thorough with removing “unauthorized” CSS classes from elements as they are with “unauthorized” styles, but that’s all it is for now: a rule of thumb that everyone seems to follow, until someone doesn’t.
Well, that was a fairly easy fix, wasn’t it? After all, it required very little work from us (the user-style injecting is already implemented — as a part of dealing with vimeo and its special snowflakey bullshit). Performance seems comparable to what it was before (albeit a tiny bit slower to react to changes) and all is well.
Further testing reveals that tie twitching image issue was fixed on Twitter, probably on Facebook as well. At least as far as Firefox is concerned.
Now it’s time to test in Chrome.
Hotel California
So I whip up my “variable aspect ratio” video example on youtube and start sweating profusely. Will the extension work?
Because there is one problem with my solution: Chrome doesn’t support tabs.removeCss(), which turns programmatically generated styles into a bunch of unsuspecting fellows checking into a hotel California. Sure, you can insert the style at any time you want, but it can never leave.
Not being able to yank the previous style when adding a new one is problematic the same reason you telling your kids to wear a white t-shirt where your partner told them to wear a red one a minute ago is problematic. It’s also problematic the same reason putting a new highway tolling sticker on your windshield without removing the old one.
Fortunately for us, Chrome takes the “common sense” (and standard-compliant) approach to handling the first problem: in case you have multiple conflicting definitions, it’s gonna respect the last one. This makes it a bit easier to ignore the second problem, though having tens or in the very worst case (frequent aspect ratio changes, frequent resizings of the player and browser window and watching a video in the same time for long periods of time) even hundreds of conflicting styles shouldn’t be too much of a problem.
Nonetheless, we’ll have to settle for this very terrible practice when in Google Chrome. I’ve ran out of fucks to give and the new ones are expected to arrive only in about a month.
Now, I could probably spend another day trying to deal with Chrome shittiness and try to invent new workarounds, it’s easier to just sit back, REEEEE at Chrome, use my userbase as a glorified guinea pigs and hope that people who watch youtube without closing/refreshing/opening video in a different tab won’t have their performance degraded.
And that’s a very dangerous game. I should know: the first update where autodetection was introduced had major performance issues if you watched videos in a tab continuously for long amount of time (Chrome was a complete lagfest in ~5 minutes of watching video, while performance in Firefox was somewhat better (and very dependent on your hardware) — just good enough that it escaped my testing. It’s been two years now and my ratings on Chrome Web Store still haven’t recovered.
I’ll still take the gamble though, against my better judgement.
At this point, I suppose it’s time for a PSA:
Public service announcement: Google Chrome is garbage (and so are all other Chromium-based browsers). If you aren’t already, you should really use Firefox instead.
Thanks for coming to my TEDx talk.
The Case of Twitchy Twitch
So the extension stopped doing twitch things on twitter and the new system for resizing video works. But after a quick visit to the out of season April fools joke (a.k.a. the immortal Blizzcon stream from hell) it turns out that the video still appears … twitchy, which means that twitching on twitch was a result of a different problem.
The ball is now back in the court of automatic aspect ratio detection. To be fair, that twitch issue is a problem with autodetection was known for a wihle. After all, most streams don’t have this problem, and the fact that this doesn’t happen when autodetection detects proper letterbox (or that aspect ratio correction works at all — on Twitter, it wouldn’t!) should be plenty of evidence to support that:
This issue smells like a video with a very tiny, nigh unnoticeable black border. Not wide enough for you to notice, but just enough to trigger autodetection. 30 seconds of (local only) defacing, it turns out that our intuition was correct:
Turns out that this is a very interesting edge case, but to understand how things went wrong we’re gonna need a quick crash-course in how auto-detection works. The key steps are:
- Start counting rows that contain nothing else than black pixels, from top and bottom to the middle. Each row has a number: top row is row 0, last row has a number that’s one less than the number of rows. Due to performance reasons, we only pick a few columns of each row that we actually check for the presence of a (non-)black pixel.
- Remember the first row that contains black pixel (on both ends)
- Calculate the height of the black bars
- Check whether top and bottom black bar are roughly of equal thickness.
We don’t require black bars to match exactly because:- in theory, the “not black bars” portion of the video is not guaranteed to be of even height. In cases like this, the top and bottom black bar could differ by 1px
- in practice, not all pre-letterboxed videos are exactly centered. Case in point: first few seconds of this video.
- Calculate and apply aspect ratio. If top and bottom thickness are different, you’ve got two options. Either you over-cut or you end up with a thin black border on either top to bottom. I’m not exactly 100% sure which strategy I’m using: twitch issue suggests I’m using strategy A (overcut), but the previous video suggests black edges are preferable.
That’s the basic idea behind aspect ratio detection, but reality is far more complicated in order to crack down on false positives. This means that we want to avoid unnecessary checks if possible.
Once we determine the correct aspect ratio, we can take a shortcut. Since we know where black bars end and the image begins, we can just check these four rows:
The reason why we aren’t checking the rows exactly on either side of the edge is to dodge compression artifacts and blurred edges.
Naming and shaming: Disney (Star Wars: Solo trailer).
So fine, we save numbers of these four rows. Top outer row is the last row in which we failed to detect a non-black pixel minus a pixel or two for safety, top inner row is the first row in which a non-black pixel was detected plus a pixel or two of safety margin.
There is, of course, a problem. If the top black border is one pixel thick and bottom black border doesn’t exist, the extension would have us check rows that don’t actually exist. Since the rows we need to check don’t exist, we can assume that black bars either don’t exist or are too thin to actually annoy anyone. When that problem happens, the extension resets the aspect ratio back to what it was originally.
Here’s a fun fact tho: this step — saving numbers of the four rows — happened only after extension already corrected aspect ratio. Normally that’s not a problem, because videos don’t have one pixel thick black bars on the top and no black bar at the bottom. They would either have substantial letterbox, or none at all.
In this particular case, though, things were a bit different. Ultrawidify would detect the one pixel letterbox and correct it. Then it would check where the black bar edges are and apply the safety margins. Safety margins would be out of bounds.
Would say Ultrawidify and undo the correction right away, and this cycle would repeat forever (or at least as long as one-pixel and zero-pixel letterbox was present).
Fortunately enough for us, the fix for that is easy enough: we try to save the edge rows, and only issue aspect ratio correction if the result isn’t bogus.
And that’s about it for today, really.