本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Self-Hosted Databricks Alternative: Homelab Data Lakehouse

Self-Hosted Databricks Alternative: Homelab Data Lakehouse Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow and OpenMetadata on RKE2, with highly available state on Ceph and CloudNativePG, secrets from Vault and sign-in through Keycloak. Raw data comes in at one door, gets built and inspected on the line, and leaves as tables and models I can trust. Nobody sends an invoice.

自托管 Databricks 替代方案：家庭实验室数据湖仓。这是家庭实验室构建系列的一部分。在 RKE2 上运行 Trino、Apache Iceberg、Project Nessie、dbt、Spark、MLflow 和 OpenMetadata，利用 Ceph 和 CloudNativePG 实现高可用状态，通过 Vault 管理密钥，并使用 Keycloak 进行登录。原始数据从一端进入，在流水线上进行构建和检查，最终转化为我所信任的表和模型。没有人会给我寄账单。

Introduction A factory you rent by the hour, or one you own Raw data is worth about as much as raw steel. Nobody wants it until someone has cut it, joined it, checked it and stamped it. Snowflake, Databricks and BigQuery will rent you an excellent factory for that, and the meter runs for as long as the lights are on. I wanted the factory without the meter, so I built one on three mini-PCs in my homelab, on the same RKE2 cluster as everything else in this series.

引言：你是租用按小时计费的工厂，还是拥有自己的工厂？原始数据的价值大约等同于原材料钢材。在有人对其进行切割、连接、检查和盖章之前，没人想要它。Snowflake、Databricks 和 BigQuery 可以为你租用一个优秀的工厂，只要灯亮着，计费器就会一直运行。我想要一个没有计费器的工厂，所以我用家庭实验室里的三台迷你电脑构建了一个，它运行在与本系列其他项目相同的 RKE2 集群上。

Every station is a real open-source tool doing a real job. It’s built to make one thing: machine-learning models trained on data I can name down to the exact version. One is already serving, a football predictor that a scheduled pipeline refits and re-tests every day. Here is the floor, in the order the material moves: The loading dock: Ceph object storage. Everything that arrives lands here. The stock ledger: Project Nessie. Every table is an Apache Iceberg table, and Nessie records every version it has ever had. It’s Git for data.

每个工作站都是一个执行实际任务的真实开源工具。它的构建目标只有一个：训练出基于我能精确到具体版本的数据所构建的机器学习模型。目前已经有一个模型在运行，这是一个足球预测器，通过定时流水线每天进行重新拟合和测试。以下是按物料移动顺序排列的生产车间：装卸码头：Ceph 对象存储。所有到达的数据都存放在这里。库存账本：Project Nessie。每一张表都是 Apache Iceberg 表，Nessie 记录了它曾经拥有的每一个版本。它就是数据的 Git。

The assembly line and quality control: dbt. Tables are built and tested on a branch, so a batch that fails inspection never reaches the shelf. The front counter: Trino. Anyone with access asks for what they need in plain SQL. The showroom: Superset. Charts and dashboards over the finished tables, for people who’d rather not write SQL. It asks for data in their name, never its own. The records office: OpenMetadata. It traces any table back to its raw material.

装配线与质量控制：dbt。表在分支上构建和测试，因此未通过检查的批次永远不会上架。前台：Trino。任何有权限的人都可以用简单的 SQL 查询他们所需的内容。展示厅：Superset。基于成品表的图表和仪表板，适合那些不想写 SQL 的人。它以用户的名义请求数据，从不以自身名义。记录办公室：OpenMetadata。它能将任何表追溯到其原始物料。

The machine shop: Kubeflow. Notebooks, pipelines and hyperparameter searches, on GPUs when a job needs them. Finished goods: MLflow and KServe. Every run is tracked, and the model that passes its checks is served over HTTP. The utilities were already in the building: Ceph, CloudNativePG, Vault, Keycloak, Istio and ArgoCD. It’s highly available where it holds state. Postgres runs three instances, one in each datacenter, the ledger runs two replicas and search runs three nodes.

机械车间：Kubeflow。笔记本、流水线和超参数搜索，在任务需要时使用 GPU。成品：MLflow 和 KServe。每一次运行都被跟踪，通过检查的模型会通过 HTTP 提供服务。基础设施已经在楼里了：Ceph、CloudNativePG、Vault、Keycloak、Istio 和 ArgoCD。在存储状态的地方，它实现了高可用。Postgres 运行三个实例，每个数据中心一个；账本运行两个副本，搜索运行三个节点。

Trino, Superset, OpenMetadata and MLflow run a single replica each, and I come back to that below. One bay is still empty: a Hermes AI copilot that will read the lineage over OpenMetadata’s MCP interface, waiting on a DGX Spark that hasn’t arrived. If you know Databricks, you already know the layout:

Trino、Superset、OpenMetadata 和 MLflow 各运行一个副本，我将在下文讨论这一点。有一个工位仍然空着：一个 Hermes AI 副驾驶，它将通过 OpenMetadata 的 MCP 接口读取血缘关系，目前正在等待尚未到货的 DGX Spark。如果你了解 Databricks，你已经知道这个布局了：

It’s an open-source equivalent, not Databricks, which is a product and a trademark, and there’s no affiliation. This is the second design: the first used Hive Metastore and Airflow, and I say below why I replaced both. The brief The brief I set myself: an MLOps and big-data platform on the existing homelab RKE2 cluster that a real data team could plausibly use. The homelab’s existing backends (Rook-Ceph, CloudNativePG, Vault, Keycloak) had to thread through every component, and each piece had to land in GitOps and come back after a node reboot with nobody touching it.

这是一个开源等价物，而不是 Databricks（那是一个产品和商标），两者没有任何关联。这是第二个设计版本：第一个版本使用了 Hive Metastore 和 Airflow，我将在下文说明为什么要替换掉它们。任务简报：我为自己设定的任务是：在现有的家庭实验室 RKE2 集群上构建一个真实的团队可以使用的 MLOps 和大数据平台。家庭实验室现有的后端（Rook-Ceph、CloudNativePG、Vault、Keycloak）必须贯穿每个组件，每个部分都必须通过 GitOps 部署，并且在节点重启后无需人工干预即可自动恢复。

What landed is two tiers. The lakehouse: Ceph RGW object storage, Apache Iceberg tables, a Project Nessie catalog (state in CNPG nessie_db), Trino from Stackable’s coordinated 25.3.0 release, dbt-trino for tested marts, and OpenMetadata for governance, with the Kubeflow spark-operator installed for the day the data outgrows SQL. MLOps on top: MLflow Tracking and the Model Registry, backed by CNPG mlflow_backend_db and the mlflow-artifacts bucket, behind oauth2-proxy for the browser and reachable directly over in-cluster DNS from Kubeflow notebooks.

最终落地的是两个层级。湖仓层：Ceph RGW 对象存储、Apache Iceberg 表、Project Nessie 目录（状态存储在 CNPG nessie_db 中）、来自 Stackable 协调版本 25.3.0 的 Trino、用于测试数据集市的 dbt-trino，以及用于治理的 OpenMetadata，并安装了 Kubeflow spark-operator 以备数据量超出 SQL 处理能力时使用。上层的 MLOps：MLflow 跟踪和模型注册中心，由 CNPG mlflow_backend_db 和 mlflow-artifacts 存储桶支持，浏览器端由 oauth2-proxy 保护，并可从 Kubeflow 笔记本通过集群内 DNS 直接访问。

The constraints that shaped every decision: No HDFS, no ZooKeeper. Ceph RGW is my S3, and nothing in the lakehouse needs ZK. The RAM saving alone justified leaving both out; the complexity avoided mattered more. No bundled databases. Every chart wants to ship its own MySQL or Postgres or Mongo. I ripped each one out of the lakehouse and MLflow charts and pointed them at CNPG. There is one Postgres cluster for this stack (postgres-mlops-cnpg-cluster, three instances across three DCs, each on its own zone-local volume) and it hosts the lakehouse and MLflow databases.

塑造每一个决策的约束条件：没有 HDFS，没有 ZooKeeper。Ceph RGW 是我的 S3，湖仓中没有任何组件需要 ZK。仅节省的内存就足以证明放弃这两者的合理性；而避免的复杂性更为重要。没有捆绑数据库。每个图表都想自带 MySQL、Postgres 或 Mongo。我将它们从湖仓和 MLflow 图表中全部移除，并指向 CNPG。该堆栈只有一个 Postgres 集群（postgres-mlops-cnpg-cluster，跨三个数据中心运行三个实例，每个实例位于各自的区域本地卷上），它托管了湖仓和 MLflow 数据库。

(Kubeflow is the exception; it keeps its bundled MySQL, katib-mysql and Model Registry’s Postgres for now.) No v1/Secret literals in Git. Every credential (application user passwords, OAuth2 client secrets, OIDC cookie keys) flows from Terraform into a Vault path, then into a VaultStaticSecret CR, then into the destination namespace. Kustomize may declare a placeholder Secret to keep references valid, but VSO always wins on the data values.

（Kubeflow 是个例外；它目前保留了其捆绑的 MySQL、katib-mysql 和模型注册中心的 Postgres。）Git 中没有 v1/Secret 字面量。每个凭据（应用程序用户密码、OAuth2 客户端密钥、OIDC cookie 密钥）都从 Terraform 流入 Vault 路径，然后进入 VaultStaticSecret CR，最后进入目标命名空间。Kustomize 可以声明一个占位符 Secret 以保持引用有效，但在数据值方面，VSO 始终具有最高优先级。

Mirror production patterns, even when it’s harder. Real OIDC, real mTLS, real RBAC. No “developer convenience” backdoors. The rest of this article covers how each piece lands, what I cut, and which traps cost the most time. The architecture If the word is new, the lakehouse is the third step in a short history. A warehouse gives fast SQL over the tables you planned for, in a format you rent. A lake keeps any file cheaply but has no table rules, so teams copy the data into a warehouse anyway and end up with two versions of the truth. A lakehouse puts the table rules (Iceberg) and the versions (Nessie) on the open files.

即使更困难，也要模仿生产环境的模式。真正的 OIDC、真正的 mTLS、真正的 RBAC。没有“开发者便利”的后门。本文的其余部分将介绍每个组件如何落地、我删减了什么，以及哪些陷阱耗费了最多的时间。架构：如果这个词对你来说很陌生，那么湖仓是简短历史中的第三步。数据仓库以你租用的格式，为你预先规划的表提供快速 SQL 查询。数据湖可以廉价地保存任何文件，但没有表规则，因此团队最终还是会将数据复制到数据仓库中，导致出现两个版本的真相。湖仓将表规则（Iceberg）和版本控制（Nessie）应用于开放文件。