Foundation Models guardrails and untrusted input boundaries
Parent: Apple Foundation Models framework · Published reference · snapshot 2026-10-23 · viability 4.45 / 5 (research)
↓ Facts as markdownall context files
Source-anchored research for Foundation Models guardrails and untrusted input boundaries. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Guided generation constrains output structure but does not establish semantic correctness or authorization. Apple describes generating typed structures, while FoundationModelsBench separates decodable structure from grading the values inside the response; validate values and caller permissions before downstream actions. (medium confidence) [source]
- Apple's Foundation Models safety guidance says on-device SystemLanguageModel guardrails check the input prompt and generated output; a violation at either point raises LanguageModelError.guardrailViolation. Apple also warns that context-dependent harms may pass both built-in safety layers, so developers should add protections tailored to their app's use case. (high confidence) (verified as of 2026-08-06) [source]
Problems, failure modes and limitations
- Apple says the on-device SystemLanguageModel guardrails check both the prompt sent to the model and the generated output. Apple’s safety talk counts developer instructions, user prompts, and tool calls among model inputs, and the framework raises a guardrail-violation error if an input or output fails its check. (high confidence) (verified as of 2026-10-23) [source]
- Apple says Instructions take priority over prompts and warns that putting user or other unverified content in Instructions creates prompt-injection risk. Keep Instructions app-authored and treat user or external text as data; OWASP warns that prompt labels alone do not enforce a trust boundary. (high confidence) [source]
- Treat retrieved pages and tool results as untrusted context. Apple’s agentic security guidance identifies tool results as injection sources and demonstrates spotlighting them in historyTransform; OWASP says labeling alone does not enforce trust and recommends validating tool arguments, enforcing caller permissions outside the model, and requiring approval for high-risk actions. (high confidence) [source]
- Apple documents LanguageModelError.refusal as the model refusing to answer and LanguageModelError.guardrailViolation as a safety check triggered by prompt or response content. A plain string response can itself be a refusal without a reliable programmatic marker; guided generation reports refusal with a refusal error instead. (high confidence) (verified as of 2026-10-22) [source]
- Apple recommends a user-facing recovery path for guardrail errors: rephrase a built-in prompt that triggers one, and explain the limit and offer a different prompt for user-created input. A proactive feature can suppress an interruption; for a user-initiated feature, show feedback and alternatives. For a guided-generation refusal, show the refusal explanation and let the person retry. (high confidence) [source]
- In an April 9, 2026 report, RSAC researchers said a Neural Exec payload combined with Unicode right-to-left override bypassed then-current Apple Intelligence on-device input/output filters and internal guardrails in their tests. They reported 76% success across 100 random prompts, said Apple shipped hardening in iOS 26.4 and macOS 26.4, and reported no evidence of in-the-wild exploitation. This is one lab's reported pre-fix sample, not a current bypass rate or a framework-wide failure rate. (low confidence) (verified as of 2026-04-09) [source]
- Apple warns developers not to put user or unverified external content in Instructions because session instructions take precedence over later prompts and untrusted text there creates prompt-injection risk. Keep instructions developer-controlled; put external content in task data, and use fixed choices or guided generation when the app can safely narrow the accepted input or output. (high confidence) (verified as of 2026-08-06) [source]
- For a user-initiated request that triggers a guardrail error, Apple recommends explaining that the feature cannot process the input and offering a different prompt or another action. For guided-generation refusal, the app can request an explanation asynchronously and fall back to refusal.debugDescription if that request fails; for string responses, design the UI to handle a refusal message that may not be distinguishable from ordinary text. (high confidence) (verified as of 2026-08-06) [source]
Comparisons and alternatives
- Apple documents permissiveContentTransformations as a mode for processing potentially unsafe source text: String generation does not throw LanguageModelError.guardrailViolation, though the model may still return a refusal string. Non-String guided generation follows default guardrail behavior and can throw guardrailViolation or refusal errors, so the permissive mode changes the string-generation checks rather than disabling every safety behavior. (high confidence) (verified as of 2026-08-06) [source]
Facts and statements
- Apple says input and output safety guardrails cover only supported languages and locales. If language detection misses a short unsupported phrase mixed into a supported-language prompt, the guardrails may miss it too; check supportsLocale and disable the affected feature or provide an alternative experience for unsupported locales. (high confidence) [source]
- Apple's safety material identifies prompts, instructions, tool calls, and unverified external sources as safety inputs, but it does not establish a separate trust-isolation guarantee for tool-returned text. For that boundary, OWASP's MCP guidance says to treat every tool response as untrusted and validate outputs and subsequent authorization in trusted application code. (medium confidence) (verified as of 2026-10-21) [source]
- OWASP advises treating model output and retrieved or tool-provided content as untrusted, enforcing permissions and parameter validation outside the model, applying least privilege, and requiring human approval for sensitive actions. Prompt labels, deny lists, or pattern filters can help screen content but do not prove that remaining text is safe or authorize a tool action. (high confidence) (verified as of 2026-10-22) [source]
Children
- No children recorded.