OpenCanvas

The official music video of any song, playing full-screen behind your player and locked to the song. It jumps when the listener seeks, starts with the next track and never loops. It runs inside your app on Android and desktop JVM, with no server, no helper process and no API key.

What it does

A streaming app knows the song it is playing. OpenCanvas finds that song's official music video, checks that it really is the same recording, works out how the song's timeline maps onto the video's, and then keeps the picture on the right second for as long as the song plays.

• It starts at once. The video's stream is known about half a second after the song starts, and the map is ready after about a second. A sharper stream replaces the first one when it arrives.
• It stays locked, and it does not loop. A music video usually has an intro and often edits. OpenCanvas measures one offset per stretch of the song, so the picture stays right through the edits and is hidden where the video has no matching scene.
• Seeks and skips just work. Seek in the song and the video jumps to the matching second. Start the next track and its video begins immediately.
• It is light. Everything is learned from the headers of two small MP4 files, about 0.3 MB each, and a few milliseconds of arithmetic. Nothing is decoded.
• It is self-contained. One library, no daemon, no server of ours, no account. It talks to YouTube's own servers the way YouTube's apps do, and nothing else.

How it works

Each step below feeds the next. The first picture can start before the map is finished, which is why a song never waits for the measurement.

Why frame sizes?

An AAC audio track is a sequence of frames, and the size of each frame follows the loudness and busyness of the music. The table of frame sizes sits in the file's header, so reading about 300 KB of each file gives a loudness-like curve for the song and for the video, without downloading or decoding any audio.

The two curves are aligned with FFT cross-correlation in 10-second windows, and a Viterbi pass picks one offset per window, charging for every change of offset. The candidate that lines up best wins, which is also what separates the official video from a lyric video, a live take or a re-upload.

Why two tiers?

YouTube serves a 360p stream almost instantly and a taller 720p or 1080p stream through a second client a moment later. Both are requested at once. The canvas starts on the first and is re-published with the sharper one.

Why ranged reads?

YouTube throttles an open-ended download of its adaptive streams to about 140 KiB/s, but serves bounded ranges at line rate. OpenCanvas starts with a small first range that grows, and answers a seek with a small range at the new position first, so playback never waits for a whole file.

Measured, not claimed

Everything here was measured on one Windows 11 machine on a home connection, with the final code, from an empty cache. The raw reports come from the manual tests in the repository.

Use it in your app

Add the dependency, configure once at app start, then hand OpenCanvas the song that is playing.

repositories {
    mavenCentral()
    maven { url = uri("https://jitpack.io") }
}
dependencies {
    implementation("com.github.Vijay-Janarthanan:OpenCanvas:1.0.0")
    // optional Compose player
    implementation("com.github.Vijay-Janarthanan.OpenCanvas:opencanvas-compose:1.0.0")
}

OpenCanvas.configure(
    OpenCanvasConfig(
        cacheDirectory = File(context.filesDir, "opencanvas"), // measured maps live here
    )
)

The full guide, with every option and the Compose player, is in the repository README.