which is often (but not always) followed to a link that makes me silently judge people’s music tastes. Back in the day, such requests were ignored “because performance.” However, now that autodetection uses webgl, performance issues have been resolved to a large extent, and with autodetection taking only ~1-3 ms per run … that leaves us quite some space in our 16 ms budget. So let’s do subtitle autodetection.<\/p>\n\n\n\n
But how do we do that?<\/p>\n\n\n\n
Now, I could probably train a computer vision model and take great care to never ever call it AI in public in front of normies. However, CV probably isn’t free to run, and I’d have a lot of work cut out for me if I went this route. So let’s see if we can bolt something on top of our existing algorithm.<\/p>\n\n\n\n
Going this route means we have to actually define how our algorithm that can only check one (sub)pixel at a time should differentiate between image and subtitles. So let’s go and find a sample video with baked subtitles. <\/p>\n\n\n\n
This took a bit (especially since modern music lyrics videos just put the subtitles in the subtitle track instead of baking them into the video file), but eventually I manage to find an example. <\/p>\n\n\n\n So let’s see what we have. First of all, we have some letters in the upper right corner. That looks a lot like channel “logo”, which we don’t care about and should ideally be cropped. The one thing that makes it very noticeably different from subtitles is that the logo is in the corner, not in the center. <\/p>\n\n\n\n Subtitles, in general, tend to be centered. They also tend to be very bright, at least almost full brightness in at least one of the color channels. There also tend to be some other neat properties:<\/p>\n\n\n\n If all these assumptions were correct, things would be super convenient for us. However, the universal laws of the universe state that things that seem convenient often aren’t particularly true. We know that arabic and hebrew are written from right-to-left. Fortunately for us, that doesn’t really change much in terms of item #2. However: if there’s cultures who didn’t get the memo and opted for writing in the opposite direction … could there also be cultures that decided that perpendi\u2014<\/p>\n\n\n\n Okay I’ll spare you the theatrics. If you’re from any country with access to the not-so-modern-anymore technology like movies or TV or especially the internet, then you have probably learned through osmosis that at least traditionally, China and Japan have done their writing vertically by the time you were 15. <\/p>\n\n\n\n In practice, ICNU tends to list vertical writing systems as critically endangered, sometimes even borderline extinct in the wild. In modern contexts \u2014 and especially when used as subtitles \u2014 even vertical languages tend to write things out in horizontal direction (but exceptions exists \u2014 apparently, japan will do vertical subtitles for certain movies and\/or artistic productions). More importantly, if a video has vertical subtitles … they aren’t gonna be in the bottom or top letterbox. Which means they’re in no danger of actually being cropped by Ultrawidify. (That is, unless I decide to implement autodetection for pillarbox as well, but pillarbox autodetection is so niche you’ll have to manually enable it). <\/p>\n\n\n\n So turns out we can assume #2 without any risk. <\/p>\n\n\n\n What does all of this mean for our subtitle detection algorithm? <\/p>\n\n\n\n When we capture a frame of video, we could start scanning rows, in the following manner:<\/p>\n\n\n\n The neat part of this algorithm is that it’s very compatible with how the image is stored in computer memory. In computer memory, image is stored as one row of pixels:<\/p>\n\n\n\n This makes it very easy to check the value of one pixel to the left or one pixel to the right, while checking pixels above or below requires slightly bit more work.<\/p>\n\n\n\n This approach generally works well on “ideal subtitles” like on the video somebody linked me. Subtitles seem to get detected and everything seems to work. If I bring up my debugging tools, the “subs confirmed” seems to stay for most of the time. <\/p>\n\n\n\n Now, sometimes subs won’t get explicitly detected, with extension instead giving “can’t determine aspect ratio, will keep whatever I have” status, but this doesn’t seem to cause too many issues right now. <\/p>\n\n\n\n I wonder what happens if I put on a fan-subtitled video of a song by a lesser band.<\/p>\n\n\n\n Those subtitles look a lot <\/em>like they should be getting detected. However, the cursor graph (where the trail is being drawn) is almost all the way on the right, but there’s no red near the cursor. <\/p>\n\n\n\n On the other hand, I also had the opposite problem: subtitle detection very sporadically got triggered in random places that didn’t contain any subtitles (though I don’t have a screenshot for this \u2014 I resolved some of those issues before <\/em>I even got the idea of writing this blog). So how do we throw out spurious subtitle detections?<\/p>\n\n\n\n We’ll gonna be leveraging the “subtitles dont move” but from earlier. This is gonna be suboptimal for people who build their subtitles word-by-word instead of flashing the entire line, but there’s no one-size-fits-all solution (short of training a computer vision model, and even then). There’s only balancing acts, tradeoffs and compromises. If we track detections in the last n <\/em>lines that we check across the last m <\/em>frames that subtitle detection ran on. For every line, we track up to 16 letter detections (specifically their starting position and their sizes). If those detections match between all last m <\/em>runs on all n <\/em>lines (and if it obeys all the other subtitle rules), then this is surely a subtitle. <\/p>\n\n\n\n This fixes the issue of spurious detections, but we still got dem type II errors. So what gives?<\/p>\n\n\n\n Because autodetection doesn’t run on full frame (it first scales down the video to 360p), the edge of the subtitles might become blurred a bit. This can cause the following situation. First, subtitle detection hits the lower edge of the subtitle. Because subtitles got blurred, none of the pixels registered bright enough to count as a subtitle. However, they were more than bright enough to register as image, so extension was like “yes, we found the edge of the video frame.” <\/p>\n\n\n\n The solution to this problem has two components. First one is to define three thresholds:<\/p>\n\n\n\n then, you count how many times a row switches between these three states. If the state changes often, then you’re very likely looking at a subtitle. <\/p>\n\n\n\n The other part of the solution is to ignore any short changes that exceed thresholds by minimal amounts, as those could be compression artifacts.<\/p>\n\n\n\n I go an implement those two things, and suddenly both aspect ratio detection and subtitle detection seem to work remarkably well (on this one video that I hope I never ever have to see again). Aspect ratio is stable and bang on, detected right through these subs, despite the fact that the video is over-compressed garbage with compression artifacts that can be seen from mars.<\/p>\n\n\n\n So, now that we got things working, what’s the cost? Because algorithm is no good if it causes everything to lag.<\/p>\n\n\n\n![\"subtitle1\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/subtitle1.avif\")
\n
\n
\n
![\"image-en\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/image-en.avif\")
![\"aard-results\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/aard-results.avif\")
![\"aard2\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/aard2.avif\")
\n
![\"aard-final\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/aard-final.avif\")
![\"4k\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/4k.avif\")
![\"yikes\"](\"https:\/\/stuff.tamius.net\/sacred-texts\/wp-content\/uploads\/2026\/10\/yikes-1024x383.avif\")