Transforming AI queries with precise search infrastructure
Exa is revolutionizing AI search infrastructure with its embeddings-based search engine, designed to enhance the accuracy of AI model responses by filtering the internet for precise knowledge. Headquartered in Lower Haight, San Francisco, Exa has raised $112.2 million in funding through several succ...
Exa offers competitive salaries, equity options, and a flexible remote work policy. Employees enjoy generous PTO, parental leave, and a budget for pro...
Exa fosters a culture focused on innovation and precision in AI search technology. The team values collaboration and is committed to building a unique...
Exa • Singapore
Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5M H200 GPU cluster that regularly lights up tens of thousands of machines.
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!
You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
You have experience with some high performance language (C++, Rust, etc.)
You’re comfortable optimizing a system to an exceptional degree
You care about the problem of finding high quality knowledge and recognize how important this is for the world
Build a distributed crawler that can handle 100M+ pages per day
Optimize crawl politeness and rate limiting across thousands of domains
Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
This is an in-person opportunity in Singapore.
Apply now or save it for later. Get alerts for similar jobs at Exa.
