Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
基于超网络的大语言模型知识注入缩放定律
Abstract: Injecting factual knowledge into large language models (LLMs) reliably and at scale remains an open challenge. Hypernetworks provide a promising solution to large-scale knowledge injection. Although hypernetworks are typically applied for test-time adaptation, we explore their use in train-time knowledge injection, where, given a large corpus of facts, we train a hypernetwork to generate a fixed LoRA adapter that, when inserted into the target model, enable the model to answer questions about those facts.
摘要： 如何可靠且大规模地向大语言模型（LLM）注入事实性知识仍然是一个未解决的挑战。超网络（Hypernetworks）为大规模知识注入提供了一种有前景的解决方案。尽管超网络通常用于测试时适应（test-time adaptation），但我们探索了其在训练时知识注入中的应用：给定一个庞大的事实语料库，我们训练一个超网络来生成一个固定的 LoRA 适配器；当该适配器插入目标模型时，能够使模型回答关于这些事实的问题。
In this work, we investigate whether hypernetworks can be used to perform train-time knowledge injection and how this ability varies with scale. The scaling behavior of hypernetworks remains largely unstudied. Our design decouples the hypernetwork’s injection capacity from the target model’s general capability, enabling, for the first time, a rigorous study of scaling laws for hypernetwork architectures. We characterize how loss, reasoning accuracy, and out-of-distribution (OOD) generalization vary with hypernetwork depth, width, and target network size.
在这项工作中，我们研究了超网络是否可用于执行训练时知识注入，以及这种能力如何随规模变化。超网络的缩放行为在很大程度上尚未得到研究。我们的设计将超网络的注入能力与目标模型的通用能力解耦，从而首次实现了对超网络架构缩放定律的严谨研究。我们刻画了损失、推理准确率以及分布外（OOD）泛化能力如何随超网络的深度、宽度和目标网络的大小而变化。
We construct a large-scale dataset, called MegaWikiQA, containing tens of millions of multi-hop question-answer examples across 39 domains constructed from examples in Wikidata5M. Our results reveal: (i) hypernetwork-based injection exhibits broadly predictive power law scaling along all architecture axes; and (ii) hypernetworks are capable of reliable OOD generalization at increasing scales, suggesting that hypernetwork provides a promising alternative to other train-time adaptation methods such as LoRA finetuning and full fine-tuning, exhibiting steeper scaling exponents in all OOD evaluations.
我们构建了一个名为 MegaWikiQA 的大规模数据集，其中包含从 Wikidata5M 示例中构建的、跨越 39 个领域的数千万个多跳问答示例。我们的结果表明：（i）基于超网络的注入在所有架构维度上都表现出广泛的可预测幂律缩放；（ii）超网络在规模扩大时具备可靠的 OOD 泛化能力，这表明超网络是 LoRA 微调和全参数微调等其他训练时适应方法的有力替代方案，并在所有 OOD 评估中表现出更陡峭的缩放指数。
Together, these results establish hypernetworks as a principled and scalable substrate for train-time adaptation, and provide the first empirically grounded scaling laws to guide hypernetworks for factual reasoning in large language models.
总之，这些结果确立了超网络作为训练时适应的一种原则性且可扩展的基底，并提供了首个基于实证的缩放定律，以指导大语言模型中的事实推理超网络设计。