Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses

对话式大语言模型代理的网络搜索特征研究：从搜索决策与策略到结果与响应

Abstract: Conversational LLM agents increasingly rely on Web search, yet the end-to-end lifecycle of agentic search remains poorly understood. We present the first study of Web search across four major conversational platforms (ChatGPT, Claude, Grok, and DeepSeek), combining real-world user interactions (invivo) with controlled experiments using the same platform’s models by their APIs (invitro).

摘要： 对话式大语言模型（LLM）代理正日益依赖网络搜索，然而人们对代理搜索的端到端生命周期仍缺乏深入了解。我们针对四大主流对话平台（ChatGPT、Claude、Grok 和 DeepSeek）的网络搜索进行了首次研究，结合了真实用户交互（体内实验，invivo）与使用相同平台模型 API 进行的受控实验（体外实验，invitro）。

We investigate the quality of agentic decisions to invoke Web search, their strategies to formulate queries, the potential domain preferences in the search results they receive, and the choices they make when transforming search results into grounded responses.

我们调查了代理调用网络搜索的决策质量、其构建查询的策略、搜索结果中潜在的领域偏好，以及它们在将搜索结果转化为基于事实的响应时所做出的选择。

We find that Web-search decisions vary substantially across platforms and models, while more frequent Web-search invocation does not necessarily yield better response quality. We further show that conversational agents employ different complex querying strategies and that platform specific search engines return search results from their preferred domains.

研究发现，不同平台和模型之间的网络搜索决策存在显著差异，且更频繁地调用网络搜索并不一定能带来更好的响应质量。我们进一步表明，对话式代理采用了不同的复杂查询策略，且特定平台的搜索引擎会返回其偏好领域内的搜索结果。

Finally, although responses are largely grounded in search results, some claims rely on uncited search results, raising concerns about attribution and reliability. Our findings have important implications for the design of future AI agents and Web search tools optimized for conversational retrieval.

最后，尽管响应在很大程度上基于搜索结果，但部分论断依赖于未引用的搜索结果，这引发了对归因和可靠性的担忧。我们的研究结果对未来 AI 代理的设计以及针对对话式检索优化的网络搜索工具具有重要意义。