一名开发者利用周末时间和仅约10美元的成本，成功构建了一个包含56万个域名的个人搜索引擎[1]。该项目采用自主爬虫、语言模型自动总结和队列加权等技术手段，有效过滤了企业文档和SEO垃圾内容，建立了针对个人创意作品、艺术项目和小型独立网站的搜索索引[1]。
项目的技术指标令人瞩目[1]。最终编目了560,183个主页，索引大小控制在1GB以下[1]。GPU租赁成本约为每小时0.35美元，平均每10万个网站约1美元[1]。该引擎包含671个不同类别和121,000个标签，其中非营利组织类别单独收录了近27,000个真实组织[1]。
在爬虫运行过程中，约15%的内容被标记为"空"，原因是这些页面采用了JavaScript渲染或检测墙等反爬技术[1]。自动阻止检测器在周末运行中独立阻止了177个问题域名[1]。
A developer has created a personal search engine indexing over 560,000 domains using minimal resources and cost.[1] The project was completed in a weekend with total expenditure of approximately $10 and GPU rental fees.[1] The final index encompasses 560,183 home pages across 671 different categories and 121,000 tags, with the entire indexed content occupying less than 1 gigabyte of storage.[1]
The search engine specifically targets makers, artists, and independent websites while filtering out corporate documents and SEO spam content.[1] GPU rental costs averaged approximately $0.35 per hour, with processing roughly one million websites per dollar of rental expense.[1] The index includes nearly 27,000 legitimate nonprofit organizations in its dedicated category.[1] During the weekend crawl operation, approximately 15% of encountered content was flagged as "empty," representing JavaScript-rendered pages or sites with access detection mechanisms.[1] An automated blocking detection system independently identified and excluded 177 problematic domains without manual intervention.[1]