一位开发者基于三个月的实际使用经验，对Claude等大语言模型在编程中的应用进行了系统评估[1]。评测显示，这些工具在某些场景下确实有用，但整体表现存在显著局限。
LLM在代码搜索、内部知识库检索和代码理解方面相对有帮助[1]。然而，在实际代码生成任务上表现平庸，这些模型倾向于过度设计，频繁应用不必要的面向对象编程模式[1]。更为关键的是，LLM存在严重的幻觉问题，对专业技术细节的回答常常涉及实际不存在的API或功能[1]。
训练数据的不足也限制了LLM的适用范围。游戏开发和原生编程领域的训练数据质量较差，最新的开源AAA游戏参考数据已停留在2004年的《毁灭战士3》[1]。从经济角度看，输出token的成本是输入的五至十倍，这使得代码生成任务成本高昂[1]。评测者指出，当前AI公司亏损运营，token价格难以持续，未来价格预计将上升而非下降[1]。对此，该评测者引用了一句评论："有趣的是，AI在我专业领域内不断提供虚假信息和错误说法，但对我一无所知的话题却总是准确无误"[1]。
A comprehensive assessment of large language models in software development reveals a mixed picture of capabilities and limitations. Based on three months of hands-on experience with tools like Claude, the review finds that LLMs prove moderately useful for natural language search, internal knowledge base retrieval, and code comprehension tasks [1]. However, their performance in actual code generation falls short of expectations, as these models tend toward overengineering and the unnecessary application of object-oriented programming patterns [1]. The systems also suffer from a well-documented hallucination problem, frequently generating references to non-existent APIs and features when answering questions about specific technical details [1].
The cost-benefit analysis presents additional concerns for developers considering AI-assisted programming at scale. Output token expenses run five to ten times higher than input costs, making code generation tasks economically burdensome [1]. Moreover, the underlying business models appear unsustainable; AI companies currently operate at a loss, suggesting that token pricing will likely rise rather than decline in the future [1]. The training data limitations further constrain utility in specialized domains—game development and native programming suffer from sparse historical data, with the most recent open-source AAA game in the training corpus dating to 2004's Doom 3 [1].
The paradox of AI reliability was captured in an observation: "It's interesting how AI is constantly providing false information and incorrect statements about my area of expertise. Fortunately, it's very useful and always right about topics I know very little about" [1]. This encapsulates a fundamental challenge: LLMs perform better on subjects where users cannot easily verify accuracy, while confidently generating errors in specialized fields where expert judgment is available.