Testing & Assurance Engineer

Testing & Assurance Engineer

The Testing & Assurance Engineer will design and execute comprehensive verification, validation, and assurance workflows to ensure agentic AI systems are reliable, compliant, and ready for secure deployment in NGA CORE and other National Geospatial-Intelligence Agency (NGA) environments. This role focuses on validating system performance and compliance through benchmark, regression testing, component verification, and similar. Specific responsibilities include:

• Design and maintain benchmark datasets and test scenarios to validate agentic AI system behavior under real-world conditions.
• Track performance metrics over time to detect accuracy drops, bias emergence, or unintended behavior.
• Develop and execute detailed component-level test plans for individual models, agents, data pipelines, and microservices.
• Verify compliance with interface contracts (APIs, schemas, protocols) and system requirements prior to release.
• Integrate automated tests into CI/CD pipelines (GitLab CI/GitHub Actions/Jenkins) to enable early defect detection.
• Produce validation artifacts, test results, and documentation required for Authority to Operate (ATO) packages and re-accreditation reviews.
• Work closely with AI/ML engineers, agentic systems engineers, and DevSecOps teams to incorporate testability into system design.
• Provide feedback and recommendations to improve reliability, safety, and mission-alignment of the agentic AI platform.

Required qualifications:

• Bachelor's degree in Computer Science, Software Engineering, or a related field; additional years of experience may be substituted.
• 3+ years of hands-on experience in software/system testing, validation, and assurance of AI/ML systems.
• Strong proficiency in Python for building automated test suites, analysis tools, and reporting frameworks.
• Experience designing regression and performance tests for distributed microservice systems.
• Familiarity with AI/ML model evaluation, scoring metrics, and explainability methods.
• Knowledge of software assurance, IV&V, and compliance processes in secure environments.

Desired qualifications:

• Master's degree in Software Engineering, Testing & QA, or a related field.
• Experience with: model and agentic AI assurance frameworks, benchmark design, and test harness development.
• Operating in NGA CORE or other NGA/Intelligence Community (IC) secure data environments.
• Contributing to or leveraging open-source testing frameworks or model governance tools.
• Familiarity with ATO/re-accreditation pipelines, automated compliance validation, and RMF/NIST 800-53 security controls.
• Active TS/SCI clearance (or ability to obtain one).