# Execution-guided text-to-MQL refinement > Source-anchored research for Execution-guided text-to-MQL refinement. Run status: COMPLETED. Parent: [Natural language to MongoDB query generation](https://llms-explorer.com/tree/natural-language-to-mongodb-query-generation/) · 4 facets · 17 facts · page: https://llms-explorer.com/tree/execution-guided-text-to-mql-refinement/ ## How it works - Execution-based optimization must define result matching carefully. EvoMQL weakens rewards for excessive returned fields because fuzzy matching can be exploited; SAG uses bounded column tolerance and strict row/value comparison to constrain similar evaluation shortcuts. (medium confidence) — [source](https://arxiv.org/html/2604.13045v1) - SAG uses agreement between independently generated candidates in result space because differently written pipelines can return equivalent results. Such consistency is a selection heuristic; its live execution feedback does not expose the gold answer. (low confidence) — [source](https://arxiv.org/html/2502.11201v3) ## Problems, failure modes and limitations - A practical repair loop needs an explicit stopping policy. SAG bounds repair rounds and returns a diagnostic when attempts fail; Cisco’s agent README routes recoverable execution errors to a refiner and fatal errors to the user. (medium confidence) — [source](https://arxiv.org/html/2502.11201v3) - Successful execution does not establish semantic correctness. SAG reports executable pipelines with wrong rows or values, and the Oracle Forge engineering account identifies structurally valid but semantically wrong results as an unresolved failure mode. (medium confidence) — [source](https://arxiv.org/html/2502.11201v3) - MongoDB’s tutorial places mongodb_query_checker before mongodb_query execution and describes syntax and field-reference checks. That pre-execution check is distinct from Cisco’s documented repair after an execution error; the tutorial alone does not establish execution-guided self-correction. (high confidence) — [source](https://www.mongodb.com/company/blog/technical/natural-language-agents-mongodb-text-mql-langchain?tck=natural_language_blog) - Cisco’s reference architecture routes recoverable MongoDB execution errors to an LLM query refiner and then re-executes the revised query. It routes fatal errors to the user and documents a maximum retry limit of one. This is README-level architecture evidence, not a measured repair success rate. (low confidence) (verified as of 2026-10-30) — [source](https://github.com/cisco-open/mongodb-nl-query-ai-agent) ## Comparisons and alternatives - Execution-guided refinement can implement a draft-execute-revise loop: database failures become feedback for another generated MongoDB pipeline, rather than ending the request after the first failed query. (medium confidence) — [source](https://arxiv.org/html/2502.11201v3) - Oracle Forge reports a concrete empty-result cause: MongoDB Yelp business IDs used businessid_ while DuckDB review references used businessref_. Its recovery required documented prefix normalization, illustrating why a retry may need new domain evidence rather than a syntax rewrite. (low confidence) — [source](https://app.readytensor.ai/publications/building-oracle-forge-what-it-actually-takes-to-make-an-ai-agent-work-on-real-enterprise-data-team-GctvZPl2Os2N) - EvoMQL v1 reports correct-output-fuzzy scores of 0.766 on EAI and 0.831 on its TEND MQL subset. These are paper-specific results under fuzzy output matching, rather than a universal production accuracy estimate. (low confidence) (verified as of 2026-03-11) — [source](https://arxiv.org/html/2604.13045v1) ## Facts and statements - SAG localizes empty outputs by probing pipeline prefixes with $count to find the stage that first empties the stream or errors; it can add actual distinct filter values to the repair feedback. (low confidence) — [source](https://arxiv.org/html/2502.11201v3) - EvoMQL couples draft-conditioned schema/value retrieval with execution-based online policy optimization across Draft-Refine-Optimize cycles. This updates model parameters across training rounds, whereas SAG’s bounded repair revises candidates during inference. (medium confidence) — [source](https://arxiv.org/html/2604.13045v1) - Refinement should operate within database resource and access controls. SAG uses read-only probes and per-query timeouts; Vision Nexera describes typed validation, plan-based cost rejection, hard execution timeouts, and rejection reasons returned to the model. (medium confidence) — [source](https://arxiv.org/html/2502.11201v3) - Vision Nexera claims that rejection-with-reason improved correction during development, but explicitly withholds benchmark figures pending a public methodology. Its case study supports an architecture pattern, not a quantified accuracy gain. (low confidence) — [source](https://www.visionnexera.com/work/mongodb-mcp-server) - Execution feedback can guide text-to-MQL improvement at different stages: Cisco documents runtime repair of failed queries, while EvoMQL uses database probes during refinement and execution-result rewards during training. (high confidence) — [source](https://github.com/cisco-open/mongodb-nl-query-ai-agent) - EvoMQL extracts field–value predicates from a draft, expands candidate fields semantically, and runs lightweight MQL probes to find matching values. It supplies successful probe evidence to the model to refine field–value alignment. (low confidence) — [source](https://arxiv.org/html/2604.13045v1) - EvoMQL separates execution success from result correctness. Its fuzzy output metric tolerates extra fields and somewhat relaxed numeric constraints; the authors identify excessive field inclusion as reward hacking and reduce the reward for it. Thus executability and benchmark success require separate interpretation. (low confidence) — [source](https://arxiv.org/html/2604.13045v1) - A bounded refinement loop should distinguish recoverable query errors from terminal failures and constrain resource use. Cisco documents separate recoverable/fatal routes and a retry cap; MongoDB recommends timeouts for both model calls and database queries plus fallback handling. (medium confidence) — [source](https://github.com/cisco-open/mongodb-nl-query-ai-agent)