Spotify开发了Random Access Parquet（RAP）技术，旨在为数据湖中的点查询提供低延迟的访问方式[1]。该方案通过外部索引将键直接映射到文件位置和精确字节范围，有效规避了传统分布式SQL引擎的查询规划开销[1]。这一技术使得AI代理和在线服务能够从大规模数据中快速检索信息[1]。
Spotify的数据基础设施规模庞大，其中Petabytes级别的数据存储在Bigtable中用于在线业务场景，而Exabytes级别的数据存储在谷歌云存储（GCS）的数据湖中[1]。从GCS直接查询数据会产生30至100毫秒的延迟[1]。RAP方案在现有Parquet文件基础上运行，无需数据复制或ETL转换[1]。该技术采用"单键一页"的优化策略，通过减少读取操作和解压开销来进一步降低延迟[1]。
索引规模随数据量级递增，Terabytes数据量产生Gigabytes级别的索引，而Petabytes数据量则产生Terabytes级别的索引[1]。S3 Express One Zone和GCS Rapid Storage等存储服务可提供单位数毫秒的延迟[1]。
Spotify has developed Random Access Parquet (RAP), a technique designed to enable low-latency point queries directly within data lakes, addressing performance constraints faced by AI agents and online services accessing large-scale datasets [1]. The technology uses external indexing to map keys directly to file locations and precise byte ranges, eliminating the query planning overhead inherent in traditional distributed SQL engines and avoiding the need for chain-reading operations [1].
The company operates at massive scale, storing petabytes of data in Bigtable for online scenarios and exabytes in Google Cloud Storage (GCS) data lakes [1]. Standard GCS access introduces latency of 30–100 milliseconds per request, though newer storage options such as S3 Express One Zone and GCS Rapid Storage have reduced this to single-digit milliseconds [1]. RAP operates on existing Parquet files without requiring data duplication or ETL transformations, making it practical for immediate deployment [1].
The indexing approach follows consistent scaling patterns: terabytes of data generate gigabytes of indexes, while petabytes of data produce terabytes of indexes [1]. A key optimization involves maintaining one page per key, which reduces both the number of read operations and decompression overhead [1]. This design allows AI agents and online services to rapidly retrieve data from massive datasets while maintaining efficiency at scale.