RAG Powered Apps with Amazon Bedrock, Part 2: Automating the RAG Pipeline with Terraform

查看原文 / View Original 编辑 / Edit

RAG Powered Apps with Amazon Bedrock, Part 2: Automating the RAG Pipeline with Terraform

基于 Amazon Bedrock 的 RAG 应用，第 2 部分：使用 Terraform 自动化 RAG 流水线

Before you start: This picks up where Part 1 left off. From part 1, you would’ve learned how to setup a Bedrock Knowledge Base in the console. In addition to that, you should have a general understanding of how the ingestion and query pipeline works. 在开始之前：本文承接第 1 部分的内容。在第 1 部分中，你已经学习了如何在控制台中设置 Bedrock 知识库。此外，你应该对摄取（Ingestion）和查询（Query）流水线的工作原理有一个大致的了解。

Introduction & Motivation

介绍与动机

I started this project with a singular goal: to build a comprehensive Terraform module that allows developers to deploy the entire infrastructure for a “Chat with PDF” application faster. When Amazon Bedrock was first unveiled in April 2023, I jumped in immediately. Like many of you, I built several proof-of-concepts (PoCs) through the AWS Console. The UI is amazing for building quick pocs, but once I moved into experimentation, I realized it would be best to quickly setup and tear down the infra. An example use case was testing if there were any cost savings in using S3 Vectors vs OpenSearch and how much cost savings exactly. None of the Terraform modules I found on GitHub (at the time) seemed to cover the end-to-end pipeline I was looking for, so I decided to build mine. I’m also big on learning so why not. 我启动这个项目的目标非常明确：构建一个全面的 Terraform 模块，让开发人员能够更快地部署“与 PDF 聊天”应用程序的完整基础设施。当 Amazon Bedrock 在 2023 年 4 月首次发布时，我立即投入了研究。和你们许多人一样，我通过 AWS 控制台构建了几个概念验证（PoC）。UI 对于快速构建 PoC 非常棒，但当我进入实验阶段时，我意识到快速搭建和销毁基础设施才是最佳实践。一个典型的用例是测试使用 S3 Vectors 与 OpenSearch 相比是否有成本优势，以及具体能节省多少成本。当时我在 GitHub 上找到的 Terraform 模块似乎都没有涵盖我所需要的端到端流水线，所以我决定自己动手构建。我也非常热衷于学习，所以何乐而不为呢？

What Are We Building?

我们要构建什么？

A couple of terraform modules to automate everything we clicked through manually in Part 1. One terraform apply brings up the full stack: 我们将构建几个 Terraform 模块，以自动化我们在第 1 部分中手动点击完成的所有操作。执行一次 terraform apply 即可部署整个堆栈：

  • S3 Bucket: your document store. Encrypted at rest, versioning on, zero public access. S3 存储桶：你的文档存储库。开启静态加密、版本控制，且禁止任何公共访问。
  • OpenSearch Serverless: the vector database. Stores the embeddings Bedrock generates during ingestion. OpenSearch Serverless：向量数据库。用于存储 Bedrock 在摄取过程中生成的嵌入（Embeddings）。
  • Bedrock Knowledge Base: orchestrates the chunking, embedding, and storage of documents, and retrieval at query time. Bedrock 知识库：编排文档的分块、嵌入、存储以及查询时的检索工作。
  • Ingestion Lambda: triggered automatically when you upload a file to S3. Starts a Bedrock ingestion job so documents are chunked, embedded, and indexed without ClickOps. 摄取 Lambda：当你上传文件到 S3 时自动触发。启动 Bedrock 摄取作业，从而无需手动点击即可完成文档的分块、嵌入和索引。
  • Query Lambda: accepts a natural language question, calls RetrieveAndGenerate, and returns an answer with source citations. 查询 Lambda：接收自然语言问题，调用 RetrieveAndGenerate，并返回带有来源引用的答案。

Full source code + ReadMe: Bedrock Project. If you run into issues or want to extend the module, feel free to open an issue. 完整源代码 + ReadMe：Bedrock Project。如果你遇到问题或想要扩展该模块，欢迎提交 Issue。

Architecture

架构

(Mermaid Diagram omitted for brevity, representing the flow from S3 upload to Ingestion Lambda, Bedrock, OpenSearch, and finally the Query Lambda response.) (此处省略 Mermaid 图表，展示了从 S3 上传到摄取 Lambda、Bedrock、OpenSearch，最后到查询 Lambda 响应的流程。)

In Part 3 we’ll put an API Gateway in front of the query Lambda. For now we’re invoking it directly from the CLI. 在第 3 部分中，我们将在查询 Lambda 前面放置一个 API Gateway。目前，我们直接通过 CLI 调用它。

Project Structure

项目结构

rag-bedrock-project/
├── main.tf
├── variables.tf
├── outputs.tf
├── backend.tf
├── terraform.tfvars.example
├── bootstrap/
└── modules/
    ├── storage/
    ├── opensearch/
    ├── bedrock/
    └── lambda/

Each module owns one piece of the infrastructure and exposes what other modules need through outputs. The root main.tf wires them together by passing outputs from one module as inputs to another. To be honest, I went back and forth on this architecture, and granted having multiple modules might be overkill. But designing this took me back to my Node.js applications days where I would put everything in a single server.js which made it difficult to debug errors. I learned about MVC which changed the way I build software. Terraform modules clicked the same way for me. One module per function. The Lambda module does not need how OpenSearch is set up. It just gets the IDs it needs through variables. As the architect, you know how each module communicates with the others. 每个模块负责基础设施的一部分，并通过输出（outputs）暴露其他模块所需的信息。根目录下的 main.tf 通过将一个模块的输出作为输入传递给另一个模块，将它们连接在一起。老实说，我曾反复考虑过这种架构，诚然，拥有多个模块可能有些过度设计。但这种设计让我想起了我写 Node.js 应用程序的日子，那时我会把所有东西都塞进一个 server.js 文件中，这使得调试错误变得非常困难。后来我学习了 MVC 模式，这改变了我构建软件的方式。Terraform 模块对我来说也是同样的道理：每个功能对应一个模块。Lambda 模块不需要知道 OpenSearch 是如何设置的，它只需要通过变量获取所需的 ID。作为架构师，你清楚每个模块是如何与其他模块通信的。

Implementation

实现

Step 1: Bootstrap Remote State First

第 1 步：首先引导远程状态（Remote State）

Before running terraform apply on anything, you need somewhere to store your Terraform state. Hold up? State what? Terraform state essentially tells Terraform what infrastructure already exists. Every resource it creates gets recorded in a terraform.tfstate file. Without it, Terraform can’t tell what’s already deployed. If your local state file is ever lost or corrupted, Terraform loses track of everything it deployed. Storing it in S3 keeps it versioned and safe. Terraform 1.10 introduced native S3 state locking so you don’t need a seperate DynamoDB table. 在运行任何 terraform apply 之前，你需要一个地方来存储 Terraform 状态。等等？什么状态？Terraform 状态本质上是告诉 Terraform 哪些基础设施已经存在。它创建的每个资源都会被记录在 terraform.tfstate 文件中。没有它，Terraform 就无法判断哪些资源已经部署。如果你的本地状态文件丢失或损坏，Terraform 就会失去对所有已部署资源的追踪。将其存储在 S3 中可以保持版本控制并确保安全。Terraform 1.10 引入了原生的 S3 状态锁定功能，因此你不再需要单独的 DynamoDB 表。

The bootstrap/ directory sets this up. Run it once before anything else. bootstrap/ 目录用于设置此项。在进行任何其他操作之前，请先运行它。

(Commands and configuration details follow…) (后续为命令和配置详情…)

← 返回新闻列表 / Back to news list