Applied Scientist, AGI Info, AGI - World Information Service (WIS), AGI - World Information Service (WIS)

Applied Scientist

Amazon is seeking an exceptional Applied Scientist to join AGI Info Content team. In this role, you will be at the forefront of developing and enhancing the intelligence of AmazonBot crawler and content processing. The team is a key enabler of Amazon's AGI initiatives such as data pipelines for Nova model training and collecting data for AGI Info grounding services. Our systems operate on web scale. This requires great combination of innovation to utilize all SOTA ML techniques in combination with model optimization to operate on 100k+ requests/decision per second. Your work will directly impact the quality and efficiency of our data acquisition efforts, ultimately benefiting millions of customers worldwide.

Key job responsibilities:

• Design, develop, and implement advanced algorithms and machine learning models to improve the intelligence and effectiveness of our web crawler and content processing pipelines.
• Collaborate with cross-functional teams to identify and prioritize crawling targets, ensuring alignment with business objectives.
• Analyze and optimize crawling strategies to maximize coverage, freshness, and quality of acquired data while minimizing operational costs as well as dive deep into data to select the highest quality data for LLM model training and grounding.
• Conduct in-depth research to stay at the forefront of web acquisition and processing.
• Develop and maintain scalable, fault-tolerant systems to handle the vast scale of Amazon's web crawling operations.
• Monitor and analyze performance metrics, identifying opportunities for improvement and implementing data-driven optimizations.
• Mentor and guide junior team members, fostering a culture of innovation and continuous learning.