I Added a Tag in AWS. Terraform Removed It Automatically.

查看原文 / View Original 编辑 / Edit

I Added a Tag in AWS. Terraform Removed It Automatically.

我在 AWS 中添加了一个标签，Terraform 自动将其删除了。

I opened the AWS console, selected an EC2 instance, added a tag, and saved it. The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another. That was the test. 我打开 AWS 控制台，选择了一个 EC2 实例，添加了一个标签并保存。Terraform 中并不存在这个标签。此时，已部署的基础设施与代码描述出现了不一致。这就是我的测试。

My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually. So I added a separate remediation pipeline. 我的漂移检测器已经能够发现这种差异，并将此次 EC2 更新归类为“低风险”（LOW）。缺失的一环是闭环处理：一旦系统识别出该变更属于低风险，它就应该在无需我手动运行 Terraform 的情况下恢复基础设施。因此，我增加了一个独立的修复流水线。

The Stack 技术栈 Manual EC2 tag added in AWS ↓ Drift CodeBuild runs terraform plan ↓ SNS publishes the structured change ↓ Lambda classifies the update as LOW ↓ Remediation CodeBuild runs terraform apply ↓ Terraform removes the unmanaged tag 在 AWS 中手动添加 EC2 标签 ↓ 漂移 CodeBuild 运行 terraform plan ↓ SNS 发布结构化变更 ↓ Lambda 将更新归类为“低风险” ↓ 修复 CodeBuild 运行 terraform apply ↓ Terraform 移除未受管理的标签

Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources. 检测和修复使用不同的 CodeBuild 项目。检测器负责读取基础设施并报告差异。修复项目拥有独立的 IAM 角色，因为它具备修改 AWS 资源的权限。

Step 1: Make EC2 Updates Eligible for Remediation

第一步：使 EC2 更新符合修复条件

The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories. An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation. The resource type alone was not enough. The action mattered too. 分类器已经将资源类型分为高、中、低三个类别。EC2 实例属于 MEDIUM_RISK_TYPES（中风险类型）。起初，这意味着即使是标签更新也会被归类为“中风险”，从而在修复前被拦截。仅凭资源类型是不够的，操作类型同样重要。

I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH: 我更新了分类器，使得对“中风险”资源的就地更新（in-place update）被归类为“低风险”，而创建操作保持为“中风险”，删除或替换操作则变为“高风险”：

def classify_change(resource_type, actions):
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        elif "update" in actions:
            return "LOW"
        return "MEDIUM"
    else:
        return "LOW"

For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path. 在本次测试中，Terraform 将手动添加的 EC2 标签报告为一次更新。这使其进入了“低风险”处理路径。

Step 2: Keep Deletions Out of the Automatic Path

第二步：将删除操作排除在自动路径之外

LOW did not automatically mean “start Terraform.” I also filtered out deletions: “低风险”并不自动意味着“启动 Terraform”。我还过滤掉了删除操作：

low_changes_to_remediate = [
    change for change in classified["LOW"]
    if "delete" not in change.get("actions", [])
]

This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path. If an eligible LOW change remains, Lambda starts the remediation CodeBuild project: 这种区分很重要。更新现有资源以匹配代码，与重新创建某人故意删除的内容是两码事。删除操作仍保留在人工审核路径中。如果存在符合条件的“低风险”变更，Lambda 将启动修复 CodeBuild 项目：

if low_changes_to_remediate:
    codebuild.start_build(
        projectName=os.environ.get(
            "REMEDIATION_PROJECT_NAME", "terraform-drift-remediation",
        )
    )

The Lambda role only needs permission to start that specific project: Lambda 角色仅需拥有启动该特定项目的权限：

{
  Effect = "Allow"
  Action = ["codebuild:StartBuild"]
  Resource = aws_codebuild_project.remediation.arn
}

Lambda makes the routing decision. CodeBuild performs the repair. Lambda 负责路由决策，CodeBuild 负责执行修复。

Step 3: Run Remediation Separately

第三步：独立运行修复

I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled. The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration: 我没有将 terraform apply 添加到现有的漂移检测器中。即使修复失败或被禁用，检测也应保持只读状态。第二个构建任务在一个干净的环境中启动，安装 Terraform，克隆 Three-Tier 仓库，并应用声明的配置：

version: 0.2
phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin
  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init
  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - terraform apply -auto-approve -lock=false

The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project. That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply. 修复项目使用其自己的 CodeBuild 角色，该角色对 Three-Tier 项目管理的 AWS 服务拥有写权限。在实施过程中，Terraform 在评估和更改资源时暴露出一些缺失的权限，因此我扩展了该实验性角色以覆盖整个项目。该角色特意与检测器分离开来。仅运行 terraform plan 的作业不应继承 terraform apply 所需的权限。

The Repository Was Part of the Control Loop

仓库是控制循环的一部分

One remediation run tried to include unrelated resource actions because CodeBuild was cloning a version of the Three-Tier repository that did not match the configuration I intended to operate. The problem was not the classifier. Lambda correctly identified the triggering drift. The remediation build was applying the complete repository it cloned from GitHub. I pushed the current Three-Tier configuration before running the test again. 有一次修复运行尝试包含不相关的资源操作，因为 CodeBuild 克隆的 Three-Tier 仓库版本与我打算操作的配置不匹配。问题不在于分类器，Lambda 正确识别了触发漂移的变更。修复构建任务应用的是它从 GitHub 克隆的完整仓库。我在再次运行测试之前，推送了当前的 Three-Tier 配置。

This is easy to miss in CI. Terraform does not know what exists only on a developer’s laptop. If CodeBuild clones GitHub, the committed branch is the configuration Terraform will enforce. 在 CI 中这很容易被忽略。Terraform 不知道仅存在于开发人员笔记本电脑上的内容。如果 CodeBuild 从 GitHub 克隆，那么提交的分支就是 Terraform 将要强制执行的配置。

The Tag Disappeared

标签消失了

With the current repository in GitHub, I repeated the test: 在 GitHub 仓库更新后，我重复了测试：

  1. Opened the EC2 instance in the AWS console.

  2. Added a tag manually and saved it.

  3. Started the Terraform drift detector.

  4. The plan reported an update to the EC2 instance.

  5. Lambda classified the update as LOW.

  6. Lambda started the remediation CodeBuild project.

  7. CodeBuild ran Terraform apply.

  8. 在 AWS 控制台中打开 EC2 实例。

  9. 手动添加标签并保存。

  10. 启动 Terraform 漂移检测器。

  11. 计划报告了 EC2 实例的更新。

  12. Lambda 将更新归类为“低风险”。

  13. Lambda 启动了修复 CodeBuild 项目。

  14. CodeBuild 运行了 Terraform apply。

I returned to the EC2 Tags view and confirmed that the manually added tag was gone. Terraform restored the infrastructure to what was declared in Git. 我回到 EC2 标签视图，确认手动添加的标签已经消失。Terraform 将基础设施恢复到了 Git 中声明的状态。

Git configuration: tag absent AWS resource: tag added manually Terraform result: tag removed Git 配置：无标签 AWS 资源：手动添加标签 Terraform 结果：标签被移除

The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it. MEDIUM, HIGH, and deletion paths still stop for human review. That boundary is intentional. Automatic remediation is useful only when the system can distinguish a small correction from a change that deserves a person. 该系统不再仅仅是向我报告漂移。对于我测试的“低风险”更新，它检测到了变更、进行了分类并修复了它。中风险、高风险和删除路径仍然会停止并等待人工审核。这种界限是有意为之的。只有当系统能够区分“小修正”和“需要人工介入的变更”时，自动修复才是有意义的。

What Comes Next

下一步计划

The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs. The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS logs. 基础设施已恢复到预期状态，但证据分散在 CodeBuild 和 CloudWatch 日志中。系统的下一部分将在数据库中保留修复历史记录，通过 API 公开，并在 Grafana 中进行可视化。我希望能够直接看到发生了什么变更、它是如何被分类的、修复是否运行以及是否成功，而无需从分散的 AWS 日志中重新拼凑整个过程。

← 返回新闻列表 / Back to news list