1Password旗下Off-by-1 Labs发布的论文《FLAWED》遭到安全研究人员的严厉批评，指控其存在多项学术不规范问题。1批评者指出，该论文仅引用19条源，其中多为企业博客和非学术资源，而同期的PatchBench论文则包含73条引用且主要来自学术文献。1论文还遗漏了内容高度相关的NDSS会议论文《Chasing Shadows: Pitfalls in LLM Security Research》。1
在具体错误方面，论文存在多处不准确之处。1关于Patch the Planet项目的描述不完整，仅指向Trail of Bits的工程师，忽视了Calif和HackerOne的参与。1此外，论文还包含图表标记错误、算术错误和内部文本不一致等问题。1论文声称该领域"几乎没有先前工作"的表述与现有文献不符。1
批评者认为该论文采用了严谨研究的形式和表述方式，却缺乏必要的学术严谨性，已对研究议程和学术注意力造成负面影响。1他们主张FLAWED应发布勘误或撤回论文，并指出目前的更新未能解决根本性批评。1批评者呼吁将强化研究规范和学术完整性作为评估行业研究的重要标准。1
A security researcher has raised substantial concerns about the academic rigor of "FLAWED," a paper published by 1Password's Off-by-1 Labs, arguing that it employs the trappings of rigorous research while falling short on fundamental scholarly standards.1 The researcher identified multiple critical issues, including inadequate citations, omitted relevant literature, factual errors, and internal inconsistencies that have undermined the credibility of the work's findings.
The paper's citation practices proved notably sparse compared to contemporary research in the field.1 FLAWED references only 19 sources, predominantly consisting of corporate blogs and informal content like XKCD, whereas a contemporaneous paper titled PatchBench cites 73 sources with a focus on peer-reviewed academic literature.1 Most significantly, the researcher found that FLAWED failed to cite or engage with an NDSS conference paper titled "Chasing Shadows: Pitfalls in LLM Security Research," despite its direct relevance to the paper's subject matter.1 Additionally, regarding the Patch the Planet initiative, the paper inaccurately attributed the project solely to Trail of Bits engineers while neglecting contributions from Calif and HackerOne.1
The analysis further documented factual problems within the paper itself, including mislabeled charts, arithmetic errors such as claiming 2.8 equals 4, and contradictory statements in its body text.1 The researcher noted that the paper's assertion that the field contains "almost no prior work" contradicts the existing body of literature.1 In response to these deficiencies, the critic has called for FLAWED to either issue a comprehensive erratum or retract the publication, while emphasizing that subsequent updates have not addressed the underlying substantive criticisms.1 The researcher further advocates for strengthened standards prioritizing research rigor and academic integrity when evaluating industry-produced research.1
评论
还没有评论，欢迎留下第一条。