what if my git host were a static site generator?
What if my git host were a static site generator?
如果我的 Git 托管平台是一个静态网站生成器会怎样?
i have been running several personal git forges for, at this point, almost half my life :o i like running my own dev infrastructure, not only because i’m almost always 120ms+ away from us-east-1, but because sysadmin is just plain fun :3 in 2015 i had a Gogs instance which became a Gitea instance which became a Forgejo instance, and i’ve also deployed GitLab/Forgejo several other times for various groups i’ve been a member of. 到目前为止,我已经运行了几个个人 Git 托管平台,几乎占据了我人生的一半 :o 我喜欢运行自己的开发基础设施,不仅因为我距离 us-east-1(AWS 美国东部区域)总是超过 120ms,还因为系统管理本身就很有趣 :3 2015 年我有一个 Gogs 实例,后来变成了 Gitea,再后来变成了 Forgejo,我还为我加入的各种组织部署过多次 GitLab/Forgejo。
i like the communal collaborative git forge, and Forgejo is great at this! but my Forgejo server keeps running out of disk space (from crashing while repacking git repos that haven’t updated) and falling over / OOMing under ambient scraper load. for my needs it’s clear that this is just the wrong size of thing: on the tiny machines i use for personal infrastructure, the software can’t stand up to the internet’s cosmic microwave background radiation. 我喜欢这种社区协作式的 Git 托管平台,Forgejo 在这方面做得很好!但我的 Forgejo 服务器总是磁盘空间不足(因为在重新打包未更新的 Git 仓库时崩溃),并且在环境爬虫的负载下频繁宕机或内存溢出(OOM)。对于我的需求来说,这显然是“大材小用”了:在我用于个人基础设施的小型机器上,这种软件根本无法抵御互联网的“宇宙微波背景辐射”(指无处不在的爬虫流量)。
i also kinda wanna simplify my experience by only exposing features i’ll actually use: Forgejo and its ilk do way more than i need them to: issues, PRs, releases, wikis - a bunch of GitHub feature-compatibility that i don’t care about, and pay some sort of cost for anyway :( 我还想简化我的使用体验,只展示我真正会用到的功能:Forgejo 之类的软件做了太多我不需要的功能:Issue、PR、Release、Wiki——这些我并不关心的 GitHub 兼容性功能,却让我为此付出了代价 :(
publishing to the open web the usual antidote prescribed for Forgejo resource exhaustion is to block scrapers via a web application firewall like Anubis, which aims to gate access to the webapp behind a JavaScript proof of work challenge. but this is counter to, like, the philosophy of the open web, right? the browser, ostensibly the “user agent”, is coerced into user-unfriendly behavior, executing near-useless code that taxes the user’s device (the point of the challenge is to spin!) - were it to refuse, no user-relevant information could be displayed at all. 在开放网络上发布内容时,针对 Forgejo 资源耗尽的常用解药是通过像 Anubis 这样的 Web 应用防火墙来拦截爬虫,它旨在通过 JavaScript 工作量证明(Proof of Work)挑战来限制对 Web 应用的访问。但这违背了开放网络的理念,对吧?浏览器,作为所谓的“用户代理”,被迫执行对用户不友好的行为,运行几乎无用的代码来消耗用户设备的资源(挑战的目的就是让 CPU 旋转!)。如果拒绝执行,用户就无法看到任何相关信息。
alternative browsers that don’t support JavaScript (or just don’t support JITted JavaScript) are either completely blocked off or locked behind a truly intrusive wait time. this deepens the oligoculture of the modern web, which i think is a bad thing. additionally, deployment of such a thing is an admission of defeat - we surrender to the assumption that the fronted application does not work correctly when met with real-world internet traffic: isn’t this kind of ridiculous when we have a workload where reads so heavily outnumber writes? 不支持 JavaScript(或不支持 JIT JavaScript)的替代浏览器要么被完全屏蔽,要么被困在极其恼人的等待时间里。这加剧了现代网络的寡头文化,我认为这是一件坏事。此外,部署这种东西是一种失败的承认——我们屈服于一种假设,即前端应用在面对真实互联网流量时无法正常工作:考虑到我们的工作负载中读取量远超写入量,这难道不是很荒谬吗?
serving write-sparse data ought to be super cheap in practice: all of github pages ran on one machine for years!! why not have a git host where everything is static files? git repo views with minimal server compute at its core, sorcery is shaped like a static site generator: when it receives an update to a git repo, it will rebuild a bunch of on-disk HTML for that repo - an overview page, the directory tree the tip commit of each branch, and syntax-highlighted source code renderings for each file in the tips. 实际上,提供这种写入稀疏的数据应该非常廉价:GitHub Pages 曾多年运行在一台机器上!!为什么不做一个所有内容都是静态文件的 Git 托管平台呢?Sorcery 的核心理念是极简的服务器计算,它就像一个静态网站生成器:当它收到 Git 仓库的更新时,会为该仓库重建一堆磁盘上的 HTML 文件——包括概览页面、目录树、每个分支的最新提交,以及每个文件最新版本的语法高亮渲染。
this allows us to pay a fixed upfront cost for serving many future requests, which means we are resilient against scraper load (because a sendfile-and-forget has basically negligible cost). however, since it would be expensive to render out HTML ahead of time for every revision of every file, we choose not to serve static historical views of the repo.
这使我们能够通过一次性的固定成本来服务未来的大量请求,这意味着我们能够抵御爬虫负载(因为 sendfile 操作的成本几乎可以忽略不计)。然而,由于为每个文件的每个版本预先渲染 HTML 代价太高,我们选择不提供仓库历史记录的静态视图。
repo history viewing is an integral feature of a git web interface, though, so we serve the .git directory directly, implement a basic read-only git client in JavaScript, and then client-side render all the “rich views” of the repository - the repo site generator does also need to emit some supplementary JSON data to aid the git client, since we can’t reliably list directories in the git repo, but that’s still static!
不过,查看仓库历史是 Git Web 界面不可或缺的功能,因此我们直接提供 .git 目录,用 JavaScript 实现一个基础的只读 Git 客户端,然后在客户端渲染仓库的所有“富视图”。仓库站点生成器确实需要输出一些辅助的 JSON 数据来帮助 Git 客户端(因为我们无法可靠地列出 Git 仓库中的目录),但这仍然是静态的!
since browsing around history can mean many fetches to different objects (commits, trees [i.e. repo directory listings], blobs [i.e. file contents]), a high-latency connection can cause direct object fetching and traversal to feel really slow. even moreso when git stores these objects compressed in delta-encoded packfiles: a naïve fetch of a packfile index in linux.git to view the diff of one commit would use over 400MiB of bandwidth!! 由于浏览历史意味着要多次获取不同的对象(提交、树 [即仓库目录列表]、Blob [即文件内容]),高延迟连接会导致直接获取和遍历对象时感觉非常缓慢。当 Git 将这些对象压缩在增量编码的包文件(packfiles)中时,情况更糟:为了查看一个提交的差异,简单地获取 linux.git 的包文件索引就会消耗超过 400MiB 的带宽!!
and smartly scanning ranges would kill cache hit rates and also waterfall out to a bunch of requests that depend on the data in prior requests. so, as a non-static optimization, we also provide a serverside route to fetch specific git objects by a list of object IDs (QUERY /
additionally, even for loose object repos, we can still optimize roundtrips by providing “smart fetch” modes which traverse for referenced oids for a given access pattern (e.g. traversing commit history via commit.parent->parent->parent->…, or accessing all blobs in the trees of a pair of commits in order to diff them). round-trip reduction with the help of the server! 此外,即使对于松散对象仓库,我们仍然可以通过提供“智能获取”模式来优化往返次数,该模式针对特定的访问模式遍历引用的 OID(例如通过 commit.parent->parent->parent->… 遍历提交历史,或访问一对提交树中的所有 Blob 以进行差异对比)。在服务器的帮助下减少了往返次数!
non-features so sorcery is a “git repo viewer” and not a “git forge” because it omits user accounts, ssh/gpg key management, and issues+patches entirely. in fact, sorcery proper is entirely read-only! repos are only ever written via git over ssh, which is separate to sorcery. you can run sorcery-ssh as your git user’s ssh ForceCommand, and it will provide “autocreate repo on first push” + the ability to edit repo descriptions.
非功能特性:Sorcery 是一个“Git 仓库查看器”而非“Git 托管平台”,因为它完全省略了用户账户、SSH/GPG 密钥管理以及 Issue 和补丁功能。事实上,Sorcery 本身是完全只读的!仓库只能通过 SSH 协议进行 Git 写入,这与 Sorcery 是分离的。你可以将 sorcery-ssh 作为 Git 用户的 SSH ForceCommand 运行,它将提供“首次推送时自动创建仓库”以及编辑仓库描述的功能。
the benefit of this setup is that your public deployment is as secure as its sshd, which is reassuring in big 26 :) historical views do require JavaScript, but i’m not super into blanket js allergy on the web (because unless you live in Ashburn, running UI code on-device is basically always better!) - sorcery uses my own frontend microframework + a bunch of built-in web platform affordances + intentional codesplitting to create a rich clientside experience in a super lightweight manner! 这种设置的好处是,你的公共部署的安全性等同于其 SSHD 的安全性,这在 2026 年让人感到安心 :) 历史视图确实需要 JavaScript,但我并不太赞同网络上那种对 JS 的全面抵制(因为除非你住在 Ashburn,否则在设备上运行 UI 代码通常总是更好的选择!)——Sorcery 使用了我自己的前端微框架 + 一堆内置的 Web 平台功能 + 有意的代码分割,以一种超轻量的方式创造了丰富的客户端体验!
e.g. loading a project overview page transfers about 9kb of gzipped JS to support recent commit pagination / language filtering / links through to commit diffs - the largest part of the site ends up being the syntax highlighting grammars; i thin 例如,加载一个项目概览页面只需传输约 9kb 的 Gzipped JS,即可支持最近提交的分页、语言过滤以及跳转到提交差异的链接——网站最大的部分反而是语法高亮规则;我想……