数据分析公司 PostHog 宣布将其数据仓库架构从基于 DuckDB 的方案迁移至基于 Trino 的新架构,并推出了自研的开放表格式 HogLake 1。此次重构历时三个月,旨在解决原有系统在云环境下处理 PB 级数据时面临的性能瓶颈、扩展性限制及资源预测难题 1。尽管单节点工具如 DuckDB 在特定场景下表现优异,但 PostHog 指出其在大规模分布式场景中存在着显著短板 1。
PostHog 早在五个月前曾尝试使用 DuckDB 重建仓库,但目前决定再次进行重构并彻底移除 DuckDB 1。旧架构下的数据规模已达到约 2 PB,包含大约 1 万亿行数据和 1530 万个快照 1。在此规模下,DuckDB 的扩展方案 DuckLake 出现了严重问题,单次写操作耗时超过 3 分钟,且即使清理统计信息后行数仍会重新增长 1。此外,连接元数据库在共享 Postgres 实例上开销过高导致写入速度变慢,而直接提供 Postgres 端口 5432 连接不仅难以扩展,还面临负载均衡工具匮乏及客户端兼容性维护成本高昂的问题 1。新方案将重点优化用户体验和规模化能力,团队强调不应过度炒作底层技术,而应关注最终用户获得的体验 1
Analytics company PostHog has announced a major architectural shift in its data warehouse, moving away from its previous DuckDB-based solution to a new system built on Trino and a custom open table format called HogLake 1. This decision comes just five months after the team had originally rebuilt their warehouse using DuckDB 1. The rapid reversal was driven by scalability challenges encountered when processing approximately 2 petabytes of data, which includes around 1 trillion rows and 15.3 million snapshots 1. The transition aims to address specific performance bottlenecks that emerged as the company’s data volume grew beyond the effective limits of single-node tools in a distributed cloud environment 1.
The primary catalyst for this migration was the operational failure of DuckLake, an extension associated with DuckDB, at PostHog’s scale 1. Single write operations frequently exceeded three minutes in duration, and statistics cleanup processes resulted in row counts that continued to grow rather than stabilize 1. Furthermore, the overhead of connecting to the metadata database became prohibitive when running on shared Postgres instances, significantly slowing down write speeds 1. The engineering team also highlighted difficulties in scaling direct connections to Postgres port 5432, citing a lack of suitable load-balancing tools and high maintenance costs for ensuring client compatibility 1. By adopting Trino and developing HogLake, PostHog intends to optimize user experience and improve scalability, emphasizing that end-user outcomes should take precedence over hype surrounding underlying technologies 1.
评论
还没有评论,欢迎留下第一条。