Ultrawidify and Chrome (2021 edition, episode 2)

Back in January, Chromium browsers had a bit of a fucky-wucky on Windows — though when I say “had,” I actually mean “this issue still happens.” The fucky-wucky in question causes the video to incorrectly stretch instead of cropping away the black bars. Here’s a pic for quick illustration.

The issue first started to appearing mid-January and it started to appear across all extensions that do the “crop away the black bar on ultrawide videos pls” thing. By mid February, most extension had this issue fixed to varying degrees.

The workarounds have not fixed the issue for everyone, though (I have been getting complaints that had this issue smell on them rather recently). Since Chromium devs don’t seem to be in any sort of hurry to actually fix their bugs (and since I need to pad out my blog), I guess that some sort of semi-official statement needs to get posted somewhere. Yes, even though it’s been about 8 months by now.

What Went Wrong

Google fucked up their DirectX11 and DirectX12 hardware acceleration.

Since this explanation may not be very accessible to folks who aren’t technically minded, time for an ELI5-level crash course on computer graphics.

Displaying a video — especially when it’s also high resolution — requires a lot of calculations. However, as it turns out, that’s too much for the processors, which can generally do one about calculation at the time1asterisks apply — with advent of multi-threading, CPUs can do more things at once, but CPUs with thread counts in double digits at a consumer-affordable price are a fairly recent thing. So at some point in time — probably about a decade or so ago, but hard to verify with 5 second google search — someone had a bright idea. “What if we don’t do video calculations on the CPU? What if we instead do them on the graphics card?” GPUs are, after all, a simpler, dumbed down version of a CPU. But they have a lot of said dumbed down cores, which makes them much faster at solving certain problems. Decoding video is one of those problems, so off it goes.

But to do things with the graphics card, your program needs to tell the graphics card what needs to do. It can do that in a few different ways, using a few different so-called APIs. On the “I don’t like Microsoft” side of things you have OpenGL (older) and Vulkan (new), on the “Microsoft <3” side of thing there’s DirectX 9, at this point ancient enough that it can run on everything that doesn’t belong to a garbage bin (and even on some things that do), and then there’s brand new DirectX 112Not actually new — it’s well over a decade old at this point and DirectX 12.

Since each API does things differently and offers different goodies, hardware acceleration using DirectX 9 will also do things different than the hardware acceleration approach that uses DirectX 11. This means that you can have bugs that only happen with DirectX 11 but not with Directx9 (or vice versa).

The neat thing about this is since almost all browsers now copy Google’s homework, you are guaranteed to run into this problem no matter what browser you use. Chrome and Chromium both have this issue. So do Edge and Opera. The only browser with an userbase worth considering that doesn’t have this issue is Firefox. Firefox (and a few other browsers that are too irrelevant to warrant a mention) don’t copy Google’s homework and are therefore not affected.

And How Do I Fix That?

There’s two ways.

I am not Afraid of chrome://flags Edition

  • Open chrome://flags/ (or edge://flags/ if using Edge)
  • Search for Choose ANGLE graphics backend
  • Pick D3D9 or OpenGL.
  • Restart
  • Optionally do the exact opposite of the next section
  • Enjoy

I don’t Want to Mess With My Settings

  • Open extension settings from the popup
  • Scroll down to ‘Browser quirk mitigations‘ section
  • Ensure ‘Limit Zoom’ option is checked (if you followed the previous section, you can untick that option and stop reading)
  • If the option is ticked, try decreasing the Limit zoom to % of width option to 0.99 or 0.98 or even lower. Once you lower the number enough, the cropping should stop stretching the video at the expense of some black on all four sides.
  • If you need to decrease the ‘Limit zoom‘ box below 0.90, try unchecking ‘limit zoom only while in fullscreen’ box instead.

That’s gonna be all.

Related reading

Both links appeared earlier in this post.

Ultrawidify and Chrome (2021 edition)

Note (2021-08-19): Chrome has annoyed me into making a follow-up article, which contains a slightly more up-to-date information.

As if Microsoft Edge screwing me over in 2020 wasn’t enough, as soon as Microsoft fixed their bugs, Google decided to introduce their own. In second-ish half of Januray, Chrome and Edge both received updates that broke pretty much every single extension for correcting video aspect ratios on ultrawide (and super-ultrawide) monitors.

Instead of the black bars getting cropped away, the video would get squished and stretched. Less than optimal.

This was accompanied with an uptick of one-star reviews on Chrome Web Store, which happened on most extensions like Ultrawidify — a sign that the problem is not being caused by extensions, but by Chrome.

Side note: while I am singling out Chrome in this post, this issue is present on ALL Chromium-based browsers, such as Edge or Opera.

Firefox is unaffected by this problem.

What’s going on

The bug only appears under certain conditions:

  1. You’re using Windows
  2. You’re using a nVidia GPU (may be specific to RTX 20xx and 30xx series)
  3. You’re viewing a video in fullscreen mode
  4. <video> element is both wider-ish and taller-ish than the monitor.

Item #4 gives us a possible workaround (other than “just use Firefox”) for the problem. Since the issue only appears if we zoom in too much, we can tell the extension to avoid zooming in past a certain point. And that’s exactly what I did.

The fix

If you’re using Chrome, the extension will always leave a thin (3-10px) wide black border at the left and right side of the monitor while in full screen. The fix for this problem came out around January 20th. It was designed to only activate in cases where the bug may appear.

Update (2021-04-04): as of Ultrawidify version 5.0.1, it is possible to manually enable or disable this workaround.

I am not affected by this issue. Can I disable this workaround?

Open the extension popup, select “Advanced settings” in the menu at the left side of the popup. At the bottom of advanced settings, there will be the Browser quirk mitigations section:

To disable the workaround, uncheck the limit zoom checkbox.

The video is still being incorrectly stretched for me

This is improbable: it appears that the issue is fixed for both Chrome and Edge. Ensure your browser, GPU drivers, and Windows are up to date. If you’re having this issue, chances are your system is out of date in some way. However, if you’re sure everything is up to date, there are further things that you can do.

First of all, check that ‘limit zoom’ option is checked (in: Extension popup > advanced settings > browser quirk mitigations; see the screenshot above). There’s a slight possibility that I decided to disable the workaround by default after publishing this post but before uploading new version of the addon to the Chrome Web Store.

If that didn’t work — it is possible that Chrome is unable to detect whether you’re using full screen mode or not. Try unchecking the ‘limit zoom only while in fullscreen’ checkbox.

Finally, if the issue still persists, you can also try decreasing the ‘Limit zoom to % of width’ number by a few percent.

If using Edge or Opera, please consider reporting the bug to browser developers

You can do so by pressing alt + shift + i. (Or look for ‘Report an issue’ option in browser’s menu. The option may be nested under ‘help’ option). This should open up a dialog like this:

In the ‘describe what’s happening’ box, enter something along the lines of the following:

If — while in fullscreen mode — CSS property transform: scale(x) causes video to be both taller and wider than the monitor, the video will be stretched to exactly fill the monitor. This is incorrect behaviour.

* If video and monitor are of different aspect ratios, aspect ratio of the video will not be preserved. (Expectation: transform: scale(x) should preserve aspect ratio if only one factor is provided)
* transform: scale(x) will not scale video to a size that’s strictly bigger than the monitor. (Expectation: it should do so if x is big enough)

This issue only happens when the following are met:
* Windows
* Full screen and watching a video
* Hardware acceleration: enabled
* nVidia GPU. This issue appears primarily on RTX cards (20xx and 30xx). It likely does not happen on non-RTX cards (this issue does not seem to be present on a GTX 965m)
* Console: closed or in detached window. If console is opened and docked to the side, the problem does not appear.
* transform: scale(x) or transform: scale(x,y) is used to scale the video to a size bigger than the monitor

Ultrawidify and Edge (2020 edition)

It’s been a nice autumn evening when my phone buzzes. One new email, says the notification. It’s github. Ultrawidify doesn’t work on Edge at all. Not on youtube, not on Netflix, nowhere. Curious, I think to myself and wow to not take the Valve strategy this time around. Which, of course, happens anyway because it turns out that my weekends are considerably less free than I thought they would be (Not to mention that my brain sometimes works the way Beinoff, Weiss and DeBlois say it should — something something kinda forgot). Recent reviews on Chrome Web Store seem to echo that issue (I’m at least 99.7% sure at least one, but likely more, of recent one-star reviews came from an Edge user, who has absolutely no business leaving reviews on CWS), though the majority is about issues that can be reproduced in Chrome.

So let’s see why Ultrawidify is broken on Edge, and why I pulled it from the Edge’s extension store, and why Edge users are getting this big ass red popup when they try to watch their Netflix.

Oh yea, spoiler alert: Ultrawidify is not broken on Edge. Edge is broken on Edge.

ELI3: Webdev 101

Long story extremely short: websites are just a bunch of rectangles. Sometimes said rectangles are inside each other, sometimes they’re next to each other, and it’s developer’s job to tell them where they should be and how big should they be.

Webdev: you don’t think it be like this, but (sometimes) it do

And this is what Ultrawidify does, ultimately: it tells one of the boxes to get bigger.

The Player and the Video

Ultrawidify only cares about two of these rectangles: the player and the video. The video is a special kind of rectangle: you can control its size, but it’s kind of a black box. You can’t control anything inside it, and it’s the browser’s job to handle displaying things inside the box.

In normal, functional browsers, things look like this:

Blue rectangle is the player, red rectangle is the video.

And when we encounter a video with black bars, things look like this:

This time, the video element also gets a blue tint.

As you can see, two things happen:

  • Red square gets taller than the blue square
  • Browser makes video fit the red square

But if you open youtube in Edge, you’ll find that youtube behaves in exactly the same way as it does in Firefox and Chrome. How come Netflix and the likes don’t?

Turns out: same reason why autodetection doesn’t work on Netflix (and Hulu and Disney and everything else).

Secret Ingredient: DRM

There’s lots of money spent on making movies. So much, in fact, that most movies never make a profit to the point Disney cannot even afford to pay writers their dues.

Obviously, the studios try to make as much of their money back as possible. And that brings us to a problem. Once the movie is on the internet for free, it gets a bit harder to make money off it. After all, why would you pay for Netflix or Hulu or Disney+ when you can just get a browser extension that records (or even downloads!) the movie or show you want to watch, subscribe to the Netflix’ one month free trial, download everything you want and then cancel before you start getting billed.

This is why streaming sites attempt to ensure that recording and downloading the shows and movies they host is as difficult as possible. And while such DRM measures are not very effective — as new movie and show releases very quickly find their way to thepiratebay, YIFY, whatever’s the tracker of the hour right now — the DRM measures still raise the tech barrier significantly enough to be worth it.

DRM magic happens inside the <video> tag (or the red square), and it’s up to browser to handle it. Different browsers handle DRM differently, which is why Edge gets 4K netflix while Firefox and Chrome don’t.

Which Brings us to Microsoft Edge

Yeah, Edge’s DRM implementation is completely broken.

Okay, it’s not completely broken. If the video element (red square) fits completely within the player (blue square), things will display correctly:

BTW, remember the thing about “DRM measures aren’t very effective?” Yeah, through the magic of RDP from Linux machine you can screenshot Netflix videos as much as you want.

However, the moment the video (red rectangle) becomes taller or wider than the inside of the browser window (in this case, also dashed blue square) … Well.

Height of the video is set to 132%, and the video element itself appears to respect that. Video stream, on the other hand, is a different story.

Remember: red square (top and bottom are outside the window boundaries in this example) is the video tag. We cannot control how things inside the red square appear, at all.

It appears that internally, Edge thinks that video (the red square) cannot be bigger than what’s displayed on the screen. This becomes super-evident when we push the video 25% off-screen:

Notice how the black bar on the top is of the same thickness as the black bar on the bottom.

Both the player and the video are pushed 25% towards the bottom. This means that the bottom 25% of the video should be cropped off — however, as you can see by the size of the top and letterbox, Edge simply resized the video so no cropping occurs.

The same thing happens if you push the video 25% over the top edge: edge will resize the video to 75% of the original size so no cropping occurs … except it’ll also align the stream with the top of the video tag / red box, meaning you’re now getting twice the cropping.

All in all, I see Microsoft’s acquisition of Bethesda is already paying dividends.

Dealing with keyboards is mini-hell

It’s been a while since I’ve written about programming, but lately I’ve gotten back to working on Ultrawidify. With no major bugs or problems that require immediate fix, I can finally get to work on bugs and features that I’ve been kicking down the line for a while.

Problem of the week are, of course, keyboards (or keyboard layouts).

In javascript, you have roughly two ways of identifying the key the user has pressed. There’s event.keyCode, which gives you some number that’s associated with the key. Pro of using keyCode is that it’s going to be the same for a given key, regardless of keyboard layout. Con of this method is that keyCode is going to be the same for a given key, regardless of keyboard layout (+ a few other downsides).

To elaborate a bit further on the problem: a significant chunk of Europe uses QWERTZ layout. QWERTZ is much like QWERTY layout Americans (and the rest of Europe except France) are using, except ‘Z’ and ‘Y’ trade places. French weren’t content with swapping only two letters and came up with with AZERTY layout instead. Then you also have things like Dvorak and Colemak layouts, because some people insist it makes their typing much faster. As letters trade places around the keyboard with each different layout, the keycodes don’t.

As a result, you can never be sure what letter the end user get from pressing what key. On some layouts, keycode 90 will give you Y, on others Z. This means that if you base your keyboard shortcuts on a QWERTZ layout and have ‘Z’ as a shortcut for anything, French and QWERTY users will wonder why they have to press the Y key, and Dvorak users will just shout at you that keyboard shortcuts don’t work.

If your goal is to have keyboard shortcuts that won’t be flat out wrong for people using a different keyboard layout than your own, then keyCode is not the way. Handling different keyboard layouts is an easy road to code spaghettification. Most importantly, it’s a major pain in the ass. You’ll spend a lot of time on it, but with very little gain.

Fortunately for us, modern javascript provides a solution to that. Events for key handling come with a property event.key, which gives us the character we pressed.

Neat. Now I can just use this property for all my keyboard shortcuts. Since event.key gives us a specific character, I no longer have to pay attention. I can just say “press Z to zoom.” After all, the ‘z’ that event.key gives me is exactly the same, regardless of whether the user uses QWERTZ, QWERTY or something more exotic. Foolproof, isn’t it?

Cyka blyat1and yes, I know ‘cyka blyat’ is just a DotA2/CSGO meme and doesn’t actually make sense in russian.

Wrong, sir, wrong.

True, event.key will return the same letter regardless of what keyboard layout you pressed said letter on. However, some letters are unpressable on certain keyboard layouts. If you’re using cyrillic (or anything non-latin), you’ll quickly find that keyboard shortcuts using even the standard ASCII letters no longer work.

Certainly a mild oversight on my part, but in my defence: the only reason I’ve started developing this extension is because at the time, there was no extension for fixing aspect ratio available for Firefox (Chrome did have a fair share of aspect ratio fixers such as Ultrawide Video, but those hadn’t been ported to Firefox until way later) and I really wanted the functionality. And when all you want is a swingset, why build a rollercoaster? The good old days.

But let’s get back to the topic at hand. If we want to fix the Russian problem, we’re in a bit of a tough spot. event.keyCode is starting to look better and better by the minute … except it doesn’t, really.

What can be done?

The options roughly boil down to the following:

1. Use event.keyCode to determine keys.

This option brings a lot of problems. Not only will there be issues with people using non-QWERTZ layouts (unless I spend unreasonable amount of time working on getting around that), using event.keycode would mean I have to rewrite lots of the existing code. More importantly — since all keys have been fully rebindable for a while in extension settings, I would have to decide between writing something that will correctly preserve keyboard shortcuts for existing users (annoying, quick StackOverflow recon didn’t give encouraging results), or reset keyboard shortcuts to default for everyone (easy but rather unacceptable. I don’t want another Nosedive).

2. Use event.keyCode to determine keys for new users, event.key for existing ones

This one offers some benefits over purely keycode solution — I don’t have to write code to port keyboard shortcuts to the new system, I don’t have to wipe settings of existing users. Still has some drawbacks that I don’t like, though — namely, the fact that I’ll have to deal with displayed keyboard shortcuts being wrong for non-QWERTZ keyboard layouts.

3. Keep using event.key and fall back to event.keycode if event.key doesn’t contain an ASCII character

Hey look, this is the quick and lazy solution we’ve been looking for. It’s also dirty, but it’s going to work. Maybe not on custom shortcuts, but we’ll see.

And all will be fine.

Ultrawidify and the Improper Cropping

Yet another day, yet another post about stuff going wrong. This time, I’ve got a bug report that “videos are jumping around” on Facebook and some other pages. I tried to verify the problem … and everything worked fine for me. Then I decided to boot up Windows and there it was — the problem as described. So nice — we have a problem that happens on some operating systems and doesn’t on others, even though that shouldn’t be the case in theory.

But eventually, the issue was reproduced and that’s all that matters. The issue appears very familiar — it has been observed on reddit before.

A video and a player

In a very ELI5 way, every webpage is made out of a bunch of rectangles (layers, elements), one within another. In order to properly crop a video, we must know which of these elements is actually the player (‘player’ element is to our video what picture frame is to a picture), and we need to know which element is the player element. Picking the wrong element can result in extension cropping to little, too much, or moving the video out of the picture altogether.

We can’t just assume that the first element above the video is a player, either: sometimes sites put addiitonal elements between the two. This is why we need ‘guess’ the player element by looking at the size.

Side note: not all extensions use that approach. Some seem to just assume you use a 21:9 monitor and slap a ‘enlarge this element by 1.3’ on the video element. Great and foolproof strategy for fullscreen. Less great for youtube’s theater mode, twitch with chat opened at the side, or non-fullscreen Netflix.

Legacy and technical debt

The code for determining which element is the player element has some weird quirks thanks to the history of the addon. Most notably, the extension used to work by determining how tall and how wide the video should be back in the day when it was only focused on Youtube and Netflix. This method has a few drawbacks, with most notable ones being:

  1. If you ask browser to tell you the size of the video, it’ll tell you the dimensions you specified
  2. It worked for youtube and netflix, but not for everything else

In general, we can assume that initial size of the player will be exactly as wide as the video or exactly as tall as the video. However, since we actually changed the size of the video (as opposed to telling browser to just enlarge the video by some factor), we couldn’t check for that as if the video was cropped, browser would tell us the post-crop size (and post-crop size is useless for that purpose). Some wonky code was written to deal with this issue and it worked well enough for Youtube and Netflix and sometimes even other sites. However, said code is — in retrospect — pretty bad. Looking at it invokes a few questions that every programmer sometimes asks themselves: “the hell was I trying to do with this shit” and “how the fuck did this even work at all?”

You could farm karma at /r/badcode with this.

Due to problems with #2, a better solution to resizing the video needed to be implemented, and eventually it was in the form of transform: scale(x,y). Using this to crop video (as opposed to modifying width and height attributes of the video) has some nifty advantages: it’s possible to get the size of the video without taking transform into account. This allows us to rewrite the player detect loop in a way that will correctly detect the player element.

Dealing with duplicates

Another thing worth addressing is “duplicates” — that is, what happens when more than one element on our way from video element to the root of the page has the same size. I haven’t figured out what to do in this case, since the correctness of picking innermost over outermost element for player may differ from site to site. In absence of better options, I decided to score every element that could be our player. Rules of the game:

  • Every element that matches our criteria gets 100 base points
  • Elements with 'player' in their ID get 75 bonus points
  • Elements with 'player' in their classlist get 50 bonus points
  • The farther the element is from our video, the more penalty points it gets. First match gets 0 penalty points, second gets one, third gets two and so on.

I haven’t had the chance to test this thoroughly, so results may vary.

That’s it for the day.

Ultrawidify: The Twitchy Twitch Problem

Ultrawidify has been seeing some issues with constant aspect ratio readjustments. This post examines and explains why and how these issues happened.

I don’t think I’ve boasted about developing Ultrawidify much on this blog. Maybe I should have, but then again: the audience of this blog is a) people who know me and b) people who use Ultrawidify and were bored enough to click that link in update notes. The point is — you’re all familiar with this extension.

A while back, I’ve noticed an issue on Twitch. It turned out that the video was a bit … well, twitchy. However, the issue seemed to be fairly uncommon, so I made a note in my test videos file and decided to kick the can a little farther down the road. “No big deal, it surely can wait.”

Well, turns out that the issue was a bigger deal than I thought. I’ve recently accidentally visited my facebook feed, and the twitching issue appeared — except worse. I tend to avoid twatter as well, but long story short: I stumbled on a tweet with embedded video.

What a good time for a #literallyshaking joke.

On the plus side, the issue happens very often (more often than auto-detection interval), and it happens equally often even when the video is paused. Here we get the first (and perhaps the only) bit of good news for the day: auto-detection isn’t to blame — and since there’s exactly one other thing that could cause this behaviour, this means I already know where the problem is.

In Search of the Problem

In order to understand whats and whys of the problem, we have to take a quick look at how Ultrawidify crops the video. It’s very simple: it finds the video element and basically tells the browser: “Make sure the video is this wide and this tall and then enlarge it by this much,” where “this much” is whatever number auto-detection script (or user intervention) spat out. In programmer jargon, that’s called setting style string.

For technical reasons,1Blame the video alignment feature for that! Ultrawidify’s auto-detection will also “correct” the aspect ratio when there’s no need — in cases like this, the video would be enlarged by a factor of 1: same size as before.

This should, in theory, do the job just fine. In practice though, Ultrawidify isn’t the only thing doing that. Some sites will also tell the browser to make the video element that big because something on the page changed (example: switching between normal and theater mode on youtube). This effectively undoes any changes Ultrawidify has made to the page. And we really don’t want that, since that has the potential to uncrop the unnecessary letterbox.

Solution to this problem is easy enough at the first glance: we’re just gonna tell Ultrawidify to watch for sites trying to meddle with the video size. If the website tries to change anything, Ultrawidify will undo that change immediately. I think you can see where this is going.

Yes. Twitter is also watching for anything that would meddle with video sizes. If it detects that something changed how big the video element is supposed to be, it will undo that change.

Twitter’s behaviour would also break crop and stretch functionalities, though I don’t think improperly cropped videos are common on the platform. I have noticed improperly cropped videos on facebook, though.

Developer tools seem to agree with this assessment. In inspector view, video element is blinking like there’s no tomorrow while in the console, Ultrawidify is seen setting the same style string over and over again, and the zoom factor is always one. At this point you may wonder why the twitchy video if ultrawidify sets the zoom factor to one, and the answer is simple: twitter doesn’t.

Through the magic of “inspect element” we can see that Twitter is zooming the video ever so slightly (by a factor of 1.005).

With Twitter insisting that the video should be zoomed by a factor of 1.005, Ultrawidify wanting a zoom factor of 1, and neither being very keen on letting go. And this spells trouble for us.

Ultrawidify vs. twitter, 2019 (colorized)

In Search of Solution

If the site will just undo our changes, what can we do? Well, it turns out that there’s a way. As it turns out, there’s actually two sorts of CSS styles: author styles — which is CSS defined by the website you’re visiting; and user styles.

Through the magic of user styles, you — the user — have the final say over how the browser will display the site. If Facebook says the background of the page needs to be white, and you have user style that says the page background should be whatever meme is popular this week … well, tough luck Facebook. Nothing can override user styles, which makes them the perfect “fuck you, you’ll do what I tell you” card. We’ll take it.

There’s another piece of good news: you don’t have to define the styles in advance — you can make them up on the spot and tell the browser to use that. WebExtension API allows us to do that. There is a few caveats, though. Besides making up the style, you also have to make up a class name, attach it to the element and hope that the site won’t remove it. If you want to edit your style, you have to throw the old style at the browser and tell it that you want it removed. You also have to create a brand new style, tell the browser to use that.

Fortunately for us, it currently seems that the sites aren’t as thorough with removing “unauthorized” CSS classes from elements as they are with “unauthorized” styles, but that’s all it is for now: a rule of thumb that everyone seems to follow, until someone doesn’t.

Well, that was a fairly easy fix, wasn’t it? After all, it required very little work from us (the user-style injecting is already implemented — as a part of dealing with vimeo and its special snowflakey bullshit). Performance seems comparable to what it was before (albeit a tiny bit slower to react to changes) and all is well.

Further testing reveals that tie twitching image issue was fixed on Twitter, probably on Facebook as well. At least as far as Firefox is concerned.

Now it’s time to test in Chrome.

Hotel California

So I whip up my “variable aspect ratio” video example on youtube and start sweating profusely. Will the extension work?

Because there is one problem with my solution: Chrome doesn’t support tabs.removeCss(), which turns programmatically generated styles into a bunch of unsuspecting fellows checking into a hotel California. Sure, you can insert the style at any time you want, but it can never leave.

Chrome and chromium pls. The worst thing is that the “feature request” for this was opened in 2016, and the patch has existed for over a year and a half at this point (since February 2018).

Not being able to yank the previous style when adding a new one is problematic the same reason you telling your kids to wear a white t-shirt where your partner told them to wear a red one a minute ago is problematic. It’s also problematic the same reason putting a new highway tolling sticker on your windshield without removing the old one.

It can get messy real fast.

Fortunately for us, Chrome takes the “common sense” (and standard-compliant) approach to handling the first problem: in case you have multiple conflicting definitions, it’s gonna respect the last one. This makes it a bit easier to ignore the second problem, though having tens or in the very worst case (frequent aspect ratio changes, frequent resizings of the player and browser window and watching a video in the same time for long periods of time) even hundreds of conflicting styles shouldn’t be too much of a problem.

This is a bigger mess than How To Train Your Dragon: The Hidden World, even though that theoretically shouldn’t be possible.

Nonetheless, we’ll have to settle for this very terrible practice when in Google Chrome. I’ve ran out of fucks to give and the new ones are expected to arrive only in about a month.

Now, I could probably spend another day trying to deal with Chrome shittiness and try to invent new workarounds, it’s easier to just sit back, REEEEE at Chrome, use my userbase as a glorified guinea pigs and hope that people who watch youtube without closing/refreshing/opening video in a different tab won’t have their performance degraded.

And that’s a very dangerous game. I should know: the first update where autodetection was introduced had major performance issues if you watched videos in a tab continuously for long amount of time (Chrome was a complete lagfest in ~5 minutes of watching video, while performance in Firefox was somewhat better (and very dependent on your hardware) — just good enough that it escaped my testing. It’s been two years now and my ratings on Chrome Web Store still haven’t recovered.

I’ll still take the gamble though, against my better judgement.

At this point, I suppose it’s time for a PSA:

Public service announcement: Google Chrome is garbage (and so are all other Chromium-based browsers). If you aren’t already, you should really use Firefox instead.

Thanks for coming to my TEDx talk.

“You’re just like that coffee machine, y’know … from bean to cup, you fuck up.”

The Case of Twitchy Twitch

So the extension stopped doing twitch things on twitter and the new system for resizing video works. But after a quick visit to the out of season April fools joke (a.k.a. the immortal Blizzcon stream from hell) it turns out that the video still appears … twitchy, which means that twitching on twitch was a result of a different problem.

The ball is now back in the court of automatic aspect ratio detection. To be fair, that twitch issue is a problem with autodetection was known for a wihle. After all, most streams don’t have this problem, and the fact that this doesn’t happen when autodetection detects proper letterbox (or that aspect ratio correction works at all — on Twitter, it wouldn’t!) should be plenty of evidence to support that:

Look at how twitching stops as soon as WoW cinematic begins about 12 seconds into this loop.

This issue smells like a video with a very tiny, nigh unnoticeable black border. Not wide enough for you to notice, but just enough to trigger autodetection. 30 seconds of (local only) defacing, it turns out that our intuition was correct:

Notice this thin black line at the top of the video? It’s only about a pixel or two wide on the source stream, but with video and page background set to white and gray it quickly becomes apparent. Video element is outlined with red.

Turns out that this is a very interesting edge case, but to understand how things went wrong we’re gonna need a quick crash-course in how auto-detection works. The key steps are:

  1. Start counting rows that contain nothing else than black pixels, from top and bottom to the middle. Each row has a number: top row is row 0, last row has a number that’s one less than the number of rows. Due to performance reasons, we only pick a few columns of each row that we actually check for the presence of a (non-)black pixel.
  2. Remember the first row that contains black pixel (on both ends)
  3. Calculate the height of the black bars
  4. Check whether top and bottom black bar are roughly of equal thickness.
    We don’t require black bars to match exactly because:
    1. in theory, the “not black bars” portion of the video is not guaranteed to be of even height. In cases like this, the top and bottom black bar could differ by 1px
    2. in practice, not all pre-letterboxed videos are exactly centered. Case in point: first few seconds of this video.
  5. Calculate and apply aspect ratio. If top and bottom thickness are different, you’ve got two options. Either you over-cut or you end up with a thin black border on either top to bottom. I’m not exactly 100% sure which strategy I’m using: twitch issue suggests I’m using strategy A (overcut), but the previous video suggests black edges are preferable.

That’s the basic idea behind aspect ratio detection, but reality is far more complicated in order to crack down on false positives. This means that we want to avoid unnecessary checks if possible.

Once we determine the correct aspect ratio, we can take a shortcut. Since we know where black bars end and the image begins, we can just check these four rows:

Where ‘n px’ and ‘X%’ are some adjustable thresholds.

The reason why we aren’t checking the rows exactly on either side of the edge is to dodge compression artifacts and blurred edges.

Naming and shaming: Disney (Star Wars: Solo trailer).

So fine, we save numbers of these four rows. Top outer row is the last row in which we failed to detect a non-black pixel minus a pixel or two for safety, top inner row is the first row in which a non-black pixel was detected plus a pixel or two of safety margin.

There is, of course, a problem. If the top black border is one pixel thick and bottom black border doesn’t exist, the extension would have us check rows that don’t actually exist. Since the rows we need to check don’t exist, we can assume that black bars either don’t exist or are too thin to actually annoy anyone. When that problem happens, the extension resets the aspect ratio back to what it was originally.

Here’s a fun fact tho: this step — saving numbers of the four rows — happened only after extension already corrected aspect ratio. Normally that’s not a problem, because videos don’t have one pixel thick black bars on the top and no black bar at the bottom. They would either have substantial letterbox, or none at all.

In this particular case, though, things were a bit different. Ultrawidify would detect the one pixel letterbox and correct it. Then it would check where the black bar edges are and apply the safety margins. Safety margins would be out of bounds.

Would say Ultrawidify and undo the correction right away, and this cycle would repeat forever (or at least as long as one-pixel and zero-pixel letterbox was present).

Fortunately enough for us, the fix for that is easy enough: we try to save the edge rows, and only issue aspect ratio correction if the result isn’t bogus.

And that’s about it for today, really.