How Searching Photos by Description Actually Works — A Plain-English Guide

The first time you search a photo library by describing a picture, it feels slightly impossible. Nobody tagged anything. The file is called U2EgGd.webp. And yet typing three words finds it.

Understanding roughly how this works is genuinely useful — not out of curiosity, but because it tells you exactly how to phrase searches, why some fail, and why the match percentages look alarmingly low when the results are in fact correct.

The old way: matching letters

Traditional search compares text to text. You type "cat", the computer looks for the letters c-a-t in filenames and tags, and returns what contains them. It has no idea what a cat is. It is comparing strings of characters.

This is why it fails on photos. There is no text to compare against — just pixels and a meaningless filename. The search isn't broken; it has nothing to work with.

The new way: turning both into positions on a map

Here is the central idea, and it is genuinely the whole trick.

Imagine an enormous map where every possible meaning has a location. Not a map of places — a map of concepts. Things that mean similar things sit close together. Photos of kittens cluster in one region. Photos of beaches cluster somewhere else entirely, with sunsets nearby, because they often appear together.

Now the important part: the words go on the same map. The phrase "a white cat sleeping" gets a location too — and it lands right in the middle of the region where sleeping-cat pictures live.

That is why it feels like understanding. The software isn't reading your mind — it is measuring distance. Your description and the right photo end up in the same neighbourhood because they mean the same thing.

Where the map comes from

The map was learned, not designed. Models like this are trained on enormous collections of images paired with the text that accompanied them online — captions, alt text, descriptions. Shown hundreds of millions of examples, the model gradually arranges things so that each image sits near the words that tend to describe it.

Nobody wrote a rule saying "cats have whiskers". The arrangement emerged from the pattern of which words accompanied which pictures, at a scale no human could review.

Two consequences follow directly from this, and they explain most of the behaviour you will encounter:

• It knows what was common on the internet. Everyday objects, scenes, animals, activities, and clothing are represented very well. Your specific colleague, your particular breed of houseplant, or a niche piece of industrial equipment much less so.
• It learned mostly from English captions. Which is why English queries land more accurately on the map than other languages — the English words have far more precise positions.

What this means for videos

A video is treated as what it physically is: a very long sequence of still pictures. To make one searchable, the software takes frames from it at intervals — say, one per second — and places each on the map individually.

So an hour of video becomes a few thousand positions. When you search, you are not searching "the video", you are searching every sampled moment inside it. That is why results come back with timestamps rather than just a filename.

Why near-identical frames get filtered out

Video is enormously repetitive. In a static shot, a thousand consecutive frames may be visually identical. Indexing every one wastes time and floods your results with a thousand copies of the same moment.

So frames that look nearly the same as the one before are skipped, keeping the ones where the picture meaningfully changed. This is why your results show distinct moments rather than the same scene twenty times in a row — a filter, not a limitation.

Why the match percentages look so low

This confuses almost everyone. You search for something, you get exactly the right photo, and it says 26%. It looks like a failure. It isn't.

The score is a measure of how close your phrase landed to that picture on the map. But a five-word phrase can never fully describe a photograph containing hundreds of details — lighting, background, composition, everything you didn't mention. Perfect overlap is impossible in principle, so scores cluster in a modest range even for excellent matches.

What is worth noticing is the shape of the scores. If the top result is well clear of the rest, the search found something distinctive. If everything sits within a point or two of each other, the model is saying "nothing here particularly matches" — that's your cue to rephrase.

Four rules for writing searches that work

Everything above leads to practical advice. Since your words become a position on a map of appearances, write words that describe appearances.

1. Describe what a stranger would see, not what you know. The model sees one frame with no context. a man in a blue shirt at a laptop works. Dave's first day cannot, because Dave and first-days have no visual appearance.
2. Add detail to narrow, remove detail to widen. Each extra concrete detail pulls the position toward a more specific neighbourhood. Too many results for beach? Try people with umbrellas on a crowded sandy beach. Nothing at all? Strip back to the single most distinctive element.
3. Use plain, common words. The map was built from ordinary internet captions. dog is better located than canine; bag beats satchel. Technical or unusual vocabulary lands in vaguer territory.
4. Describe the scene, not the emotion. happy is nearly useless — it has no consistent appearance. people laughing around a dinner table describes the same thing in a way the model can locate.

Searching with a picture instead of words

Because pictures and words share one map, you can skip the words entirely. Hand the system an example image, and it computes that image's position and returns whatever sits nearest.

This is the better tool whenever the thing you want is hard to put into words — a particular pattern, an unusual object, a specific visual style, a shade of colour. "Find more that look like this" sidesteps the whole problem of describing it.

What it genuinely cannot do

Being clear about the limits saves a lot of frustration:

• Reading small text. It recognises that something looks like a document or a receipt, but does not read the words on it. That is OCR, a different technology.
• Counting reliably. "Three people" and "four people" sit very close together on the map. Expect approximate results.
• Identifying specific individuals. It finds "a woman in a red coat", not a named person. It has no concept of who anyone is.
• Understanding sequence. Each frame is judged alone, so "the moment after she stood up" isn't expressible. Describe the state you want to find instead.
• Precise spatial relationships. "The cup to the left of the laptop" is weakly represented. Both objects will be matched; their arrangement often won't be.

Why this can run on your own computer

A reasonable assumption is that anything described as AI must run in a data centre. For this particular job, it doesn't have to — and the reason is worth knowing.

The heavy work — building the map — was done once, in advance, by whoever trained the model. What happens on your machine is much cheaper: converting a picture or a phrase into its position, then measuring distances. That is small enough for an ordinary laptop.

The practical upshot is that your photos and videos never need to leave your computer. The same searches, the same results, with no upload, no account, and no third party holding your family video or your security footage. For most people that is not a technical preference — it is the difference between using this on their real library and not using it at all.

Frequently asked questions