Building Setto 1: Reinventing Music Discovery with an iOS App

It's a lot of math and statistics

Apr 09, 2026

Howdy y’all! In this new series, Building Setto, I’ll be sharing my experience building an AI DJ that uses slot machine math to find your next favorite songs, and it runs entirely on your iPhone.

Let’s get into it.

Motivations

Ever since Apple released the Fast VLM Demo, I’ve always thought that since we now have the ability to run Video-Language Models with super fast inference on-device, we should have more responsive software that can react in real time.

In particular, I wanted to make an app that takes into account all the signals from my iPhone and Mac to create a real time soundtrack as I live my life.

Similar to being a main character in a movie.

Apple’s Fast VLM Demo Gif from ml-fastvlm GitHub repo

Setto V0.1: The Foundation

My initial approach involved polling the user’s camera, Health Data and Location to generate a snapshot of their current situation. However, getting the VLM to give me good data and deciding which frames to use was a rather difficult challenge with diminishing returns.

Thus, I stripped down to the concept and Setto v0.1 used Apple’s Foundation Language Models and a simple weighted algorithm for recommending music, plus health data to figure out the user’s heart rate.

I was inspired by the Tinder swipe mechanism as I thought it was a great way to communicate likes or dislikes for a vibe.

Moreover, the swipe interaction gives the algorithm explicit signals. Most streaming platforms have to infer your preferences from implicit behavior. Did you skip? How long did you listen? Did you add it to a playlist? With Setto, users literally swipe left or right. That’s as clear as feedback gets.

Pandora has thumbs up and thumbs down, and Spotify Stations (the app, not the feature) explored something similar, but neither combined it with the real time, continuous DJ experience I was going for.

The app also took into account time of day and moods to try and predict your vibe. Unfortunately, as the nocturnal person I am, the app always, and I mean always, chose Lo-Fi Hip Hop at 11 PM or assumed I was in the mood for heavy rock because my heart rate was slightly elevated in the morning. The HealthKit integration was more noise than signal.

Setto V0.2: The Pivot

Hence, to try and ground the algorithm, I added a chat box to allow users to guide the Foundation Model directly. However, I hit a couple of problems.

First, music is really hard to describe. Try telling someone what makes a song feel like a late night drive versus a Sunday morning. You know it when you hear it but words don’t quite capture it.
Second, music nuance is really hard for an LLM to deduce. You can tell it you want “chill jazz” but the difference between Nujabes (debatably a Nu-Jazz artist) and Kamasi Washington is enormous, and both are arguably “chill jazz” depending on your frame of reference.
Third, even if it understands the concepts, how does the model actually pick and choose music? You need a catalog to search through, and natural language to catalog mapping is a hard problem.

To address that third issue, I gave the language model access to Apple Music’s search, which allows natural language queries. But that didn’t work out well due to problems one and two.

What makes this fascinating is that Apple Music released a similar feature in beta a few months later in March 2026 with Playlist Playground, and I can sense the same issues. Curious to see how the team approaches it.

Setto V1.0: What If We Just Use ML

At some point, I accepted that the AI DJ would work better with a traditional Machine Learning approach rather than Generative AI. Another benefit of this approach was that I could support iOS 17+ rather than only the newest iPhones, which meant way more people could actually use the app.

My initial approach was to use a Bayesian system. In simple terms, it works like this: if I know you like Pop, then I suspect you want more Pop. If you don’t like Pop, I’ll weight it less. On top of that, I hardcoded a lot of semantic rules into the app. Artists and related music were found via Apple MusicKit’s Related Artists API. This is essentially the version 1.0 that I shared on Reddit.

It kind of worked but had a lot of limitations. Since music discovery was powered entirely through MusicKit’s Related Artists, if all your seed artists led to the same cluster of results, the app would literally have no more leads. It would just keep playing variations of the same few artists. The DJ was stuck in a loop and would give up.

The Reddit Launch

I posted Setto to r/AppleMusic and it got 22,000 views in about 9 hours, 49 upvotes and 27 comments. The UX feedback was overwhelmingly positive:

“Very smooth and the UX is awesome” — LostInAwe55
“Really nicely designed! Promising” — Lambor14
“Love this app! Recommendations are great” — Inner-Gold-5196

The SwiftUI and interaction design were validated. But the number one complaint, reported independently by multiple different users, was that the algorithm latches onto one artist and won’t branch out. One user said they picked an artist, hit “Surprise Me” as a discovery tactic, and the DJ started playing an alias of the same artist.

The app treated them as completely different artists when they’re essentially the same creative persona. The UI, UX & Product was solid, but the DJ algorithm needed a complete rebuild.

The 48 Hour Hotfix (V1.0.1)

I shipped v1.0.1 within 24 hours, covering the easy wins from the Reddit thread: playlist saving, a scrubber, playback controls, Keep Playing mode, a New Songs Only filter, and a listening history list.

Setto V1.1: A Lot of Research Papers

The main complaint that wasn’t an easy fix was the algorithm itself. People loved the idea and UX but the recommendations weren’t good enough.

Fortunately, I also work as a research assistant and enjoy reading research papers. I discovered the ACM RecSys conference and wow, thank you so much to Spotify, Stanford, Deezer and all the people in academia who publish papers.

The Multi-Armed Bandit Problem

In the simplest way possible, the Setto algorithm (the good one) is a modified implementation of the Multi-Armed Bandit problem and Thompson Sampling, optimized for music search using my experience as a DJ.

The core idea behind this problem is that you have a slot machine (an armed bandit) with multiple levers (arms) and a certain combination of them maximizes your return. But you want to balance how far you explore, so as to minimize losses (bad experiences) while also increasing your returns (good experiences). This stats question maps to a lot of the recommendation algorithms that power the modern internet, YouTube videos, brain rot, ads, and of course music discovery.

Image Sources: https://hackintoshrao.com/2017/12/12/the-exploration-exploitation-dilemma-in-multiarm-bandit-problem/; https://i.stack.imgur.com/04Ggq.jpg

In Setto’s case, each music genre is an “arm” of the bandit. When you swipe right on a jazz track, the jazz arm gets rewarded. When you skip a pop track, the pop arm takes a hit. The algorithm uses Thompson Sampling to decide which genre or artist to explore next. Instead of just picking the genre with the best track record, it samples from a probability distribution for each genre and picks the highest sample. This means genres you haven’t tried yet have high uncertainty, so they occasionally get sampled high and the DJ plays something unexpected. Genres you’ve consistently liked get played reliably. And genres you’ve consistently skipped get naturally suppressed without being permanently blocked.

If you want to understand Thompson Sampling in depth, I highly recommend this Stanford tutorial by Russo et al.which covers the theory neatly. There’s also a great YouTube video that gives a more audio visual explanation if papers aren’t your thing.

The best thing about Setto is that since users explicitly swipe left and right, they’re giving the algorithm explicit signals rather than the implicit approach that other streaming platforms have to use. The Tinder cofounders actually explored a similar idea with an app called Next in the 2010s but didn’t figure out the algorithm.

Hence, if I had to distill what makes Setto special, it’s the algorithm, and the music discovery is a feature.

The Cold Start

One thing I’m particularly proud of is how the algorithm handles the very first batch of songs, when it knows almost nothing about you.

Rather than just playing random stuff and hoping for the best, the app looks at the overlap between your seed artists. It finds “bridge” artists (people who appear in the similar artists of multiple seeds) and uses them as a starting point. This feels like the DJ heard your picks, understood the thread connecting them, and extended it outward. From there, it mixes in some discovery picks and the occasional wildcard from an unexplored genre. The result is a first experience that feels intentional rather than random.

This was inspired by how Gnoosic handles artist recommendation. I love the idea behind that website but I’m not a huge fan of the UX.

After the cold start, the algorithm falls into a rhythm of familiar, related, and then unexpected. Users tend to experience this as the DJ “getting them,” which is exactly the feeling I was going for.

Beyond Bandits: One More Thing

The Multi-Armed Bandit is the foundation, but the v1.1 algorithm is really a collection of systems working together. I want to give a general overview here and might go deeper in future issues of Building Setto, if there’s interest.

Grouping Side Projects: I needed a way to tell the algorithm that side projects, aliases and related acts are basically the same creative person. At session start, the app clusters these together so that if you skip one, the others go on cooldown too.

Preventing Genre Flooding: If you like one reggaeton track out of five total, a naive algorithm might flood you with 80% reggaeton. I found a really elegant solution for this in the RecSys literature. The idea is to ensure the genre distribution of your recommendations roughly matches the actual distribution of your preferences. So one like out of five stays proportional, not dominant. As you give the algorithm more data, it trusts your preferences more and relies less on keeping things uniform.

Smarter Tag Matching: Generic tags like “rock” and “alternative” appear on thousands of artists and carry almost no signal. Specific tags like “psychedelic funk” or “Midwest emo” are much more discriminating. I borrowed a classic technique from information retrieval (TF-IDF) that weights rare tags louder and generic tags quieter. So if you like artists tagged “math rock” and “Midwest emo”, the algorithm picks up on that specific taste, not just “rock.”

Skip Timing Matters: Not all skips are equal. A 2 second instant skip says “absolutely not.” A 45 second skip means the user sampled enough and moved on, that’s barely negative feedback at all. The algorithm treats skip timing as a spectrum of intent rather than a binary yes/no. I found some really interesting research from Spotify on skip prediction that informed the approach.

Together, these systems replaced the simple Bayesian score from v1.0 with a proper recommendation pipeline. Generate candidates, score them across multiple signals, then rerank to ensure diversity. All of it runs on device, no cloud, no accounts, no data leaving your phone.

What’s Next

In the next issue, I’ll talk about how Sets work and why building them was way harder than I expected. People wanted a way to have multiple algorithms that don’t contaminate each other (your indie folk mood shouldn’t bleed into your techno workout set), and the system design that went into making that happen is a whole article on its own.

If you have questions about Setto, music discovery, recommendation algorithms, or anything in between, I’d love to hear from you. Subscribe so you don’t miss it.

Camy's Broadcast

Discussion about this post

Ready for more?