AWS says fine-tuning Qwen3.6-27B with SageMaker AI multi-turn reinforcement learning improved retrieval on three of four tests and sharply reduced some failures.
AWS published results on October 2, 2026, from fine-tuning a search agent with multi-turn reinforcement learning (MTRL) in Amazon SageMaker AI. The experiment used Qwen3.6-27B in an enterprise-search setup, where the agent could choose between keyword-based BM25 search and vector search, then refine its queries over multiple turns.
The training approach scores the outcome of a complete search sequence rather than judging each response in isolation. AWS used normalized discounted cumulative gain at rank 10 (nDCG@10) as its retrieval reward and penalized runs that exceeded turn or token limits. Training data came from six datasets, including FRAMES, BRIGHT, and an enterprise retrieval-augmented generation collection; four separate datasets were held out for testing.
In AWS’s evaluation, the fine-tuned model improved nDCG@10 on WixQA, Wands, and BrowseComp-Plus, while its score dipped slightly on FreshStack. The largest reported relative gains were 23.7% on BrowseComp-Plus and 18.4% on WixQA. On BrowseComp-Plus, the failure rate fell from 22.89% for the base model to 0.68% after fine-tuning. The WixQA failure rate also declined, from 0.67% to 0.17%; Wands recorded no failures for either version.
AWS says the training job changed three main settings from their defaults: one training epoch, a global batch size of 128, and rollout concurrency of 32. The post describes SageMaker’s MTRL service as serverless and priced per token, with support for inspecting training trajectories and evaluating models before deployment to SageMaker endpoints or Amazon Bedrock.
The results are AWS’s own experiment, not an independent comparison, and the FreshStack regression shows the gains were not uniform across tasks. The example nevertheless illustrates how teams might train a smaller, specialized agent against their own retrieval tools and task-specific outcome measures.