On-device OCR on a Mac, with no server in the loop

For most of the last decade, "add OCR to this" meant one of two things: a Python stack with Tesseract and a lot of preprocessing, or an API key and a per-thousand-pages bill. On macOS, neither is necessary any more. The Vision framework has shipped with the OS since Mojave, does neural text recognition entirely on-device, and costs nothing.

What Vision actually gives you

The whole API surface for text recognition is one request type, VNRecognizeTextRequest, and a handful of knobs that matter:

• recognitionLevel — .accurate runs a neural recogniser and is what you want for anything you'll paste somewhere. .fast exists for live video frames.
• usesLanguageCorrection — a language model that fixes plausible misreads in prose. It genuinely helps on sentences and actively hurts on serial numbers, licence keys and hashes, where it will happily "correct" a valid string into a more word-like invalid one. Worth exposing as a toggle rather than hardcoding.
• recognitionLanguages and automaticallyDetectsLanguage — ordering matters; the first language is preferred on ambiguous glyphs.
• Results come back as VNRecognizedTextObservations with bounding boxes and ranked candidates, so you get position data for free if you want to preserve layout.

No entitlement, no account, no network. It runs on Apple Silicon and Intel alike.

Why on-device matters more here than almost anywhere else

The privacy argument for local processing is usually abstract. For OCR it isn't, because of why people reach for OCR in the first place: you run it on things that aren't selectable text. In practice that means screenshots, and screenshots of the highest-sensitivity material you handle.

• An error dialog with an internal hostname and a stack trace.
• An invoice, a payslip, a bank statement someone sent as an image.
• A one-time code, because the sender screenshotted their authenticator.
• An identity document attached to a support ticket.

A cloud OCR service processes all of that on someone else's hardware under a retention policy you can't verify. For a tool that lives on a global hotkey and gets used dozens of times a day, that's an unreasonable amount of trust to ask for a capability the machine already has built in.

And unlike a policy, the local claim is checkable. otool -L on the binary will show you whether it links a networking framework at all. If there's no CFNetwork and no URLSession symbols, it cannot upload your screenshot, regardless of what the marketing page says.

The recognition is the easy part

The genuinely fiddly engineering in a screen-OCR tool is everything around the recogniser.

Capture at native pixels. A region-selection overlay gives you coordinates in points, not pixels. On a Retina display, capturing at point dimensions hands the recogniser a 2x downsampled, softened image and accuracy falls off a cliff, especially at small font sizes. You have to multiply by the target screen's backingScaleFactor, and on a multi-display setup the target screen may not be the main one.

Screen Recording permission. Any screen capture requires the TCC grant, and macOS binds that grant to the code signature. Re-sign the binary with a different identity — which happens quietly if you switch between ad-hoc and Developer ID signing during development — and the grant stops applying without a useful error. The app just captures black.

Getting the result somewhere. The clipboard is the right default roughly always, because the next action is a paste. Anything that shows the text in a window and makes you select and copy it has added a step to a flow whose whole value proposition is removing steps.

Where it's honestly still worse than the cloud

Vision is very good at printed text on screens, which is the actual use case. It is less good at:

• Handwriting — supported, but cursive remains rough.
• Dense tables — you get the cell contents and bounding boxes, but no layout model, so reconstructing a table is on you.
• Low-resolution or heavily compressed source images — a JPEG screenshot that's been through a messaging app twice will beat any recogniser.
• Less common scripts — language coverage is good and growing, but not universal.

For pulling text off your own screen, though, on-device isn't a compromise you accept for privacy. It's just the correct implementation, and the privacy is a side effect of not needing a server.