据一名前Jane Street系统团队员工透露,该公司位于纽约1 New York Plaza的唯一数据中心曾因一次误报烟雾警报险些遭遇水灾 1。事件起因是隔壁房间施工产生的灰尘触发了烟雾报警器,进而联动关闭了计算机房空调(CRAC),导致室内温度迅速升高 1。虽然喷淋头并未立即喷水,但管道已充压,一旦室温超过150°F便会自动喷洒 1。
面对这一紧急情况,该员工在室温升至143.6°F并停止攀升之前,手动拔掉了机柜中所有设备的电源线,从而避免了灾难性损失 1。由于事发时正值美股休市,停机时间极短,服务器于次日上午恢复运行 1。为防止类似事故再次发生,该员工随后使用OCaml编写了一个监控脚本,每分钟轮询Linux服务器内部传感器温度并计算平均值以进行监控 1。
A former employee of the Jane Street systems team recounted a critical incident where a false smoke alarm nearly resulted in the flooding of the firm’s sole New York City data center, located at 1 New York Plaza 1. The cascade began when dust from construction work in an adjacent room triggered the facility's smoke detectors 1. This activation automatically shut down the Computer Room Air Conditioning (CRAC) units, causing internal temperatures to rise rapidly while simultaneously pressurizing the fire suppression sprinkler pipes 1.
The situation reached a precarious threshold as the ambient temperature climbed toward 150°F, the point at which the sprinklers were designed to discharge water automatically 1. In response, the author manually disconnected power cables from all equipment within the racks just before the temperature peaked at 143.6°F and stopped rising 1. Because the incident occurred during a U.S. stock market holiday, the resulting downtime was minimal, with operations restored by the following morning 1. To prevent future occurrences, the team subsequently developed a monitoring solution using OCaml that polls Linux server sensors every minute and calculates average temperatures for real-time alerts 1.
评论
还没有评论,欢迎留下第一条。