Transforming AI queries with precise search infrastructure
Exa is revolutionizing AI search infrastructure with its embeddings-based search engine, designed to enhance the accuracy of AI model responses by filtering the internet for precise knowledge. Headquartered in Lower Haight, San Francisco, Exa has raised $112.2 million in funding through several succ...
Exa offers competitive salaries, equity options, and a flexible remote work policy. Employees enjoy generous PTO, parental leave, and a budget for pro...
Exa fosters a culture focused on innovation and precision in AI search technology. The team values collaboration and is committed to building a unique...
Exa • Remote
Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5M H200 GPU cluster that regularly lights up tens of thousands of machines.
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!
You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
You have experience with some high performance language (C++, Rust, etc.)
You’re comfortable optimizing a system to an exceptional degree
You care about the problem of finding high quality knowledge and recognize how important this is for the world
Build a distributed crawler that can handle 100M+ pages per day
Optimize crawl politeness and rate limiting across thousands of domains
Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
This is an in-person opportunity in San Francisco. We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3).
Apply now or save it for later. Get alerts for similar jobs at Exa.
