Since the introduction of automatic aspect ratio detection in Ultrawidify, a performance issue would float by me. “Hey, when I watch a high resolution video on youtube, I get frame drops.” These issues tended to be rather common in the early days of autodetection.
At some point, I remember thinking: “hey, webgl has been a thing for a while now. Instead of processing video data on the CPU, why don’t I write a shader that would do everything on the GPU?”
So I tried, but there was a problem. Never have I done that, not even during the computer graphics class at the university.01Long story short: through the magic of course work consisting of two assignments that boiled down to “write a game in js/webgl” and “now, write that same game in a real game engine” respectively, I managed to completely dodge any actual graphics programming. I worked mainly on procedural generation, UI, and quests.. Naturally, when I tried to get my shaders to work, nothing ever worked at all. So I left that idea “for later,” did few optimizations on the current code, and called it a day. During that time, browsers got some neat performance gains as well. As result, autodetection worked well enough for me, and I stopped pursuing the webgl side of things. Instead, I decided to prioritize UI — specifically, tools that would make easier to get figure out how to make Ultrawidify to work with other websites.
Except that post-corona, I got a bunch of new hobbies and exciting side projects, so the UI wasn’t really that much of a priority.
While doing the UI, I decided to figure out whether sporadic performance issue reports are caused by the browser, or by processing data in javascript. I put a timer around ctx.drawImage(video)
& ctx.getImageData()
— which is “browser fault”, and another two timers around two tests used to process the data browser gives me, and determine the aspect ratio.

With the ‘draw’ phase being the biggest contributor to processing time, any lagginess is clearly browser fault, right?
Well, lately that question started to irk me again, so I decided to give WebGL another try. Excecpt this time, the plan wasn’t to do aspect ratio processing in the shader. The experiment was limited solely to using WebGL to render high-resolution (3440×1440) video frames to <canvas>
element (640×480; same as Ultrawidify’s internal canvas).

… well hot dang.
Now, that’s just the _draw_ phase, without corresponding getImageData()
. Let’s try to get data from canvas into javascript so we can process it.

Well that is a pretty big performance improvement. From 80-90 ms to 4-6 ms to get image data.
Maybe I shouldn’t have given up on WebGL before I even really started.