How I Built a Multi-Agent System for Interrupted Time Series Analysis (ITSA)

如何构建用于中断时间序列分析 (ITSA) 的多智能体系统

One month ago your company decided to launch a brand new checkout process. The alternative hypothesis was very simple: a three-step checkout would drive more purchases and revenue than the previous one-step checkout, based on industry benchmarks and good UX/UI practices. Just to make sure this decision really moves the needle (and with the aim of quantifying it), you suggested that an A/B test would be the best solution to measure the effect and reject the null hypothesis. The result: you couldn’t convince the stakeholders to run a properly designed A/B test and the decision of launching this new checkout was made anyway.

一个月前，你们公司决定推出一套全新的结账流程。备择假设非常简单：基于行业基准和良好的 UX/UI 实践，三步结账流程将比之前的一步结账流程带来更多的购买量和收入。为了确保这一决策确实能产生实质性影响（并对其进行量化），你建议通过 A/B 测试来衡量效果并拒绝零假设。结果是：你没能说服利益相关者进行设计合理的 A/B 测试，而推出新结账流程的决定还是被执行了。

However, one month later, right now, those stakeholders ask questions such as: “what’s the effect of the new checkout?” “Did it work?” or “Did we lose money with it?”… I am very sure that most of you have been in this situation and might know some quasi-experimental techniques such as Difference-in-differences or Synthetic Control Methods but… what can you do when there are no other markets to compare against, and the company isn’t big enough to build a credible synthetic control group?

然而，一个月后的今天，这些利益相关者开始问：“新结账流程的效果如何？”“它奏效了吗？”或者“我们因此亏钱了吗？”……我敢肯定，你们大多数人都遇到过这种情况，可能也了解一些准实验技术，如双重差分法 (Difference-in-differences) 或合成控制法 (Synthetic Control Methods)，但是……当没有其他市场可供对比，且公司规模不足以建立可信的合成对照组时，你该怎么办？

I want to introduce ITSA (Interrupted Time Series Analysis), a method I’ve been using during the last four years in situations where you can’t run a randomized control trial and you have few options between all of the vast amount of techniques related to observational studies. Also, I will introduce how we created a multi-agent system to run ITSA, the architectural, statistical and software decisions (and trade-offs) we needed to make to develop this data product and the benefits this solution offers.

我想介绍一下 ITSA（中断时间序列分析），这是我在过去四年中一直使用的一种方法，适用于无法进行随机对照试验且在海量观测研究技术中选择有限的情况。此外，我还将介绍我们如何创建一个多智能体系统来运行 ITSA，以及我们在开发该数据产品时所做的架构、统计和软件决策（及权衡），并探讨该方案所带来的益处。

ITSA and why it matters: A brief introduction

ITSA 及其重要性：简要介绍

Measurement in digital business has long trusted the pre- versus post-intervention approach for exactly these situations. I’m sure you’ve heard something like: “just compare what happened after the change with the period before.” Maybe you’ve even believed it’s a nice approach. Respectfully, let me show you why it falls short.

在数字业务中，针对这些情况，人们长期以来一直依赖“干预前与干预后”的对比方法。我相信你一定听过类似的话：“只需比较变化发生后和发生前的情况即可。”也许你甚至认为这是一种不错的方法。恕我直言，让我来告诉你为什么这种方法存在缺陷。

(Figure 1: Line chart of daily orders rising steadily; a naive pre/post comparison around a new checkout launch shows a +10% lift that is just the trend. Image by author.) (图 1：每日订单稳步上升的折线图；围绕新结账流程发布进行的简单前后对比显示出 10% 的增长，但这仅仅是趋势使然。图片由作者提供。)

Simulated daily orders climb steadily from about 100 to 140 over 130 days. A new checkout ships on day 80, but the series shows no break: the upward trend simply continues. Comparing the pre-period average (days 40–80) with the post-period average (days 80–120) suggests a +10% lift that the checkout didn’t cause; the pre-existing trend did. This is the naive pre/post trap that Interrupted Time Series Analysis (ITSA) is designed to avoid.

模拟的每日订单在 130 天内从约 100 单稳步攀升至 140 单。新结账流程在第 80 天上线，但序列并未出现断点：上升趋势仍在持续。比较干预前（第 40-80 天）和干预后（第 80-120 天）的平均值，会得出 10% 的增长，但这并非由结账流程引起，而是由既有趋势导致的。这就是中断时间序列分析 (ITSA) 旨在避免的“简单前后对比陷阱”。

From a statistical point of view, comparing the mean before and after the intervention is a naive approach: it ignores pre-existing trends and seasonality, does not account for autocorrelation when quantifying uncertainty, and cannot distinguish the intervention from concurrent shocks. In the Figure 1 scenario, this comparison would lead you to conclude that the intervention increased daily orders by 10%. And you’d be wrong.

从统计学角度来看，比较干预前后的均值是一种简单粗暴的方法：它忽略了既有趋势和季节性，在量化不确定性时未考虑自相关性，也无法将干预效果与同期发生的其他冲击区分开来。在图 1 的场景中，这种比较会让你得出“干预使每日订单增加了 10%”的结论。而你是错的。

(Figure 2: Same daily orders series with the pre-launch trend projected forward as a counterfactual; observed orders stay on that projection, so the effect is zero. Image by author.) (图 2：同一每日订单序列，将发布前的趋势作为反事实向前预测；观察到的订单量保持在该预测线上，因此效果为零。图片由作者提供。)

The same simulated daily orders series, now analysed with a counterfactual. After the new checkout ships on day 80, the pre-intervention trend is projected forward as a dashed line with an uncertainty band: what would have happened without the change. Observed orders stay within that band, so the estimated effect is essentially zero. In Interrupted Time Series Analysis, the effect is the gap between the observed series and the counterfactual, not between the pre and post averages.

同样的模拟每日订单序列，现在通过反事实进行分析。在新结账流程于第 80 天上线后，干预前的趋势被作为虚线向前预测，并带有不确定性区间：即如果没有发生变化，情况会如何。观察到的订单量保持在该区间内，因此估计的效果基本为零。在中断时间序列分析中，效果是指观察序列与反事实之间的差距，而不是干预前后的平均值之差。

In Figure 2 you can see a different approach: instead of comparing the average value from pre and post-intervention, you can compare what happened post-intervention (Observed) with what would have happened if we hadn’t launched the three-step checkout (Counterfactual). In this case there is no positive or negative effect attributed to our intervention. Why? because the effect is measured by subtracting the counterfactual from the observed value.

在图 2 中，你可以看到一种不同的方法：与其比较干预前后的平均值，不如将干预后发生的情况（观察值）与如果我们没有推出三步结账流程可能发生的情况（反事实）进行比较。在这种情况下，我们的干预没有产生正面或负面的影响。为什么？因为效果是通过从观察值中减去反事实值来衡量的。

(Figure 3: Equation: the effect at time t equals the observed outcome Y_t(1) minus the estimated counterfactual Y_t(0). Image by author.) (图 3：公式：t 时刻的效果等于观察到的结果 Y_t(1) 减去估计的反事实 Y_t(0)。图片由作者提供。)

The potential-outcomes definition of the effect: τ_t = Y_t(1) − Ŷ_t(0). Y_t(1) is what actually happened after the intervention; Ŷ_t(0) is the estimated counterfactual, what would have happened without it. Every causal inference method, including Interrupted Time Series Analysis, differs mainly in how it estimates that counterfactual.

效果的潜在结果定义为：τ_t = Y_t(1) − Ŷ_t(0)。Y_t(1) 是干预后实际发生的情况；Ŷ_t(0) 是估计的反事实，即如果没有干预会发生的情况。每种因果推断方法（包括中断时间序列分析）的主要区别在于它们如何估计该反事实。

How can you estimate the counterfactual? Well, that’s the whole point: it depends on the identification and estimation technique you use. You can use Difference-in-difference if you can assume parallel trends, you can use Synthetic Control Methods if you can build a properly weighted synthetic control group, and so on.

你该如何估计反事实？嗯，这正是关键所在：它取决于你使用的识别和估计技术。如果你能假设平行趋势，可以使用双重差分法；如果你能建立一个加权合理的合成对照组，可以使用合成控制法，等等。

ITSA uses a different approach: it uses the outcome’s own pre-intervention trajectory to estimate what would’ve happened after the intervention in its absence. In its simplest form, a segmented regression estimates the baseline level and the pre-existing trend, then projects that trajectory into the post-intervention period. The intervention effect is not the difference between the post- and pre-intervention periods, but the difference between the observed post-intervention trajectory and this projected counterfactual.

ITSA 使用了不同的方法：它利用结果自身的干预前轨迹来估计在没有干预的情况下，干预后会发生什么。最简单的形式是，分段回归估计基准水平和既有趋势，然后将该轨迹投影到干预后的时期。干预效果不是干预后时期与干预前时期的差值，而是观察到的干预后轨迹与该投影反事实之间的差值。

ITSA assumptions and when to use it (and when you shouldn’t)

ITSA 的假设及其适用场景（以及不适用场景）

This method has its own assumptions, and you need to know them before using it to make the best decision possible. In a nutshell, here are the ten most important assumptions when using ITSA:

该方法有其自身的假设，在决定使用它之前，你需要了解这些假设以做出最佳决策。简而言之，使用 ITSA 时最重要的十个假设如下：

Counterfactual continuity: Absent the intervention, the pre-intervention trend would have continued unchanged. Extrapolating that trend is the counterfactual in a single-group design. This is the assumption that carries all the weight and it is fundamentally untestable, since you never observe the counterfactual.

反事实连续性：如果没有干预，干预前的趋势将保持不变。在单组设计中，外推该趋势即为反事实。这是最具分量的假设，且从根本上是无法验证的，因为你永远无法观察到反事实。