Summarization, proofreading, classification and short rewrites no longer need a server round-trip: Chrome ships built-in AI APIs, Android exposes Gemini Nano through ML Kit GenAI, and iOS 26 opens the Foundation Models framework. The architecture that ships is hybrid — detect capability, run on-device when you can, degrade gracefully to a cloud path when you cannot.
Tier: Advanced · Workflow area: AI features in the client
This playbook builds the capability-gated hybrid: private, zero-marginal-cost, offline-capable inference for supported devices, with one abstraction so product code never knows which path answered.
Skip it for open-ended chat quality, long-context reasoning, or anything needing tools and retrieval — small on-device models are task engines, not frontier assistants.
LanguageModelSession, guided generation via @Generable, on Apple Intelligence-enabled devices.Write the capability gate first — availability is a spectrum, not a boolean. Every platform reports more than yes/no; your gate must handle unavailable, downloadable (model not yet fetched), downloading, and ready. Web:
const availability = await Summarizer.availability();
// "unavailable" | "downloadable" | "downloading" | "available"
if (availability === "downloadable") {
summarizer = await Summarizer.create({
monitor(m) { m.addEventListener("downloadprogress", e =>
updateUI(e.loaded)); },
});
}
On Android, the GenAI API's status check plays the same role (with AICore managing the Nano download); on iOS, check SystemLanguageModel.default.availability, which reports reasons like Apple Intelligence disabled or model not ready. The download states matter for UX: trigger downloads at an opportune moment (onboarding, Wi-Fi), never block a user action on a surprise multi-hundred-MB fetch.
Abstract the task, not the model. Product code calls summarize(text, opts) — a small module resolves the best available engine per call: on-device if ready, cloud if allowed, "unavailable" UI otherwise. Keep the interface task-shaped (summarize/rewrite/classify) rather than chat-shaped; task APIs across all three platforms take options (length, tone, format), and a task interface lets each backend use its native strengths. Record which engine served each request in your analytics (engine name, not content) — you will need the split for quality and cost analysis.
Engineer for small-model reality. On-device models are 3-4B-class: keep prompts short and directive; constrain outputs structurally — Foundation Models' @Generable guided generation gives you typed Swift structs instead of parsed prose, and the Prompt API supports JSON schema constraints; validate outputs with the same rigor as cloud calls (Zod or codable types) because small models drift more, not less; and set expectations in UX copy for generative surfaces ("quick summary" framing, with an explicit "try the fuller cloud version" affordance where policy allows).
Define the fallback policy as product policy, not just error handling. Decide and document, per feature: privacy-first (on-device or nothing — the honest choice if you market "never leaves your device"; the fallback is a graceful unavailable state), quality-first (cloud when reachable, on-device offline), or cost-first (on-device whenever capable, cloud for overflow cases like inputs exceeding the on-device context window). Then implement fallback triggers uniformly: capability-unavailable, inference error, timeout, output-validation failure, and input-too-large all route to the same policy switch. If privacy claims are made, the cloud path must require explicit user consent per surface — silently "upgrading" a private feature to cloud is a trust breach, not a fallback.
Test on the real capability matrix. Emulators and dev machines lie about availability. Your matrix: a supported flagship (full on-device path, including first-run download UX), an unsupported/older device (fallback or unavailable UI), airplane mode on both (offline on-device still works; cloud path degrades honestly), and mid-download states. Add a CI-able contract test for the abstraction: mock each engine state and assert the policy routes correctly — the matrix run is manual per release, the routing logic is tested forever.
The unit of design is the task plus a capability gate plus a written fallback policy. On-device is a privacy and cost feature only if the fallback respects the promise — detect honestly, download politely, degrade explicitly.
LanguageModelSession with guided generation and streaming; sessions are stateful — create per task, watch context limits, and handle the guardrail-violation error as a first-class refusal state (pb-51's rubric applies on-device too).On-device inference is the restricted environment's best friend: content never leaves the device, satisfying most data-residency and confidentiality constraints by construction. Configure the abstraction's policy to privacy-first (no cloud path) for restricted deployments — the same codebase serves both postures via configuration. Verify with egress monitoring rather than documentation, and note that OS-level features (Apple Intelligence's own cloud escalation, browser sync) are outside your app's control — scope your claims to what your code does.
"On-device" claims in marketing and consent screens are compliance statements. Have the actual data flow — including fallback triggers and any OS-mediated processing — reviewed against the claim before launch, not after a user reads the network log.