AI Automation Engineer
Vienna, VA
Experience: 0
Category: Information Technology
Contractor Work Model: Hybrid
Brand: System One
Job ID: 381540
Date Posted: 10/06/2026
Shortcut: http://jobs.systemone.com/VuwXXZ
Job Title: AI Test Engineer
Location: Vienna, VA
Type: Contract
Work Model: Hybrid – onsite and remote
Hours: 40.0
Responsibilities
• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (DevEx) solutions.
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.
• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services.
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns.
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.
• Support root-cause analysis, troubleshooting, and validation activities for AI-related defects and incidents.
• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non-technical stakeholders.
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.
• Support delivery of strategic DevEx initiatives and AI-enabled platform capabilities across the software development lifecycle.
Requirements
• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Experience designing and executing automated test strategies for complex software applications.
• Strong programming and scripting skills in Python.
• Experience building automated test frameworks and test pipelines.
• Experience working with APIs, automated integration testing, and CI/CD environments.
• Understanding of machine learning, generative AI, large language models (LLMs), or AI-powered applications.
• Experience performing data validation, test result analysis, and defect investigation.
• Familiarity with modern testing methodologies, quality engineering practices, and software delivery lifecycles.
• Knowledge of statistical analysis and evaluation techniques used to validate system performance.
• Experience using source control systems and collaborative development practices.
• Strong analytical, problem-solving, and troubleshooting skills.
• Excellent written and verbal communication skills.
• Ability to work independently in a fast-paced, agile environment while collaborating effectively across teams.
Preferred Qualifications:
• Experience testing generative AI applications, LLM-based solutions, or Retrieval-Augmented Generation (RAG) systems.
• Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or similar tools.
• Knowledge of LLM observability, monitoring, and traceability practices.
• Experience working with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms.
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.
• Understanding of Responsible AI principles and AI governance frameworks.
• Experience with performance testing, scalability testing, and reliability engineering.
• Experience supporting developer platforms, DevEx initiatives, or software engineering productivity tools.
• Experience working within enterprise or regulated technology environments.
Ideal Candidate Profile:
The ideal candidate is a hands-on quality engineer who enjoys building testing capabilities rather than simply executing predefined test cases. They are curious about how AI systems behave, comfortable analyzing large volumes of AI-generated outputs, and passionate about improving reliability through automation. They bring a strong software testing foundation, practical automation experience, and a desire to help create scalable AI quality processes from the ground up. They thrive in collaborative environments, communicate findings clearly, and can translate evaluation results into actionable improvements for engineering teams.
System One, and its subsidiaries including Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.
System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.
#LI-EL1
#M2
Ref: #851-Rockville-S1
-
Jr. Software Test Engineer
Huntsville, AL
Job Title: Jr. Software Test Engineer (Contract-to-Hire) Location: Huntsville, AL (On-site) Security Clearance: Active Secret preferred OR ability to obtain Pay Rate: $62-68/hr (W2) About the role Join a team supporting defense/tactical systems where y...
Date Posted: 09/29/2026 Recommended
-
Mid Embedded Software Engineer (C/C++)
Huntsville, AL
Location: Huntsville, AL (Onsite) Type: Contract-to-hire Pay Rate: $97-102/hr. (W2) Clearance: Active Secret clearance (or ability to obtain) Overview We’re looking for a mid-level Embedded Software Engineer to help build and integrate mission-critical...
Date Posted: 09/29/2026 Recommended
-
Embedded Software Engineer C/C++
Huntsville, AL
Job Title: Embedded Software Engineer (C/C++) Location: Huntsville, AL (Onsite) Type: Contract-to-hire Compensation: $77-80/HR Clearance: Active Secret clearance or ability to obtain Experience: 4+ years (or 2+ with a Master’s) About the role We’re hir...
Date Posted: 09/29/2026 Recommended
-
Sr. Embedded Software Engineer C/C++
Huntsville, AL
Location: Huntsville, AL (On-site) Type: Contract-to-hire Clearance: Active Secret clearance (or ability to obtain) Pay: $116–$126/hr (W2) About the role We’re hiring a Senior Embedded Software Engineer to build and support mission-critical embedded so...
Date Posted: 09/29/2026 Recommended
-
Senior Software Engineer
Huntsville, AL
Senior Software Engineer (Onsite) — Huntsville, AL Type: Contract-to-hire Work schedule: 40 hours/week Security clearance: Active Secret clearance or ability to obtain Pay rate: $117-125/HR Overview We’re hiring a Senior Software Engineer to lead and b...
Date Posted: 09/29/2026 Recommended
-
DevSecOps Engineer
McLean, VA
DevSecOps Engineer US Citizenship and the ability to obtain Public Trust is required 100% remote W2 only, no C2C Responsibilities/Requirements: • Implement and maintain modern DevSecOps practices, including secure CI/CD pipelines, automated build and d...
Date Posted: 08/25/2026 Recommended
-
Senior Software Test Engineer
Huntsville, AL
Type: Contract-to-hire Pay: $117–$125/hr (W2) Location: Huntsville, AL Clearance: Active Secret preferred, or ability to obtain Summary We’re hiring a Senior Software Test Engineer to help build and run automated test solutions for mission-critical sof...
Date Posted: 09/29/2026 Recommended
-
Test Engineer
Lebanon, Ohio
Job Title: Test Engineer Location: Lebanon, Ohio Type: Direct Hire Compensation: $80,000.00 - $90,000.00 Work Model: Onsite Must already live within a commutable distance; no sponsorship or C2C Responsibilities Collaborates with Engineering, Product Ma...
Date Posted: 09/03/2026 Recommended
-
Mid Software Test Engineer
Huntsville, AL
Mid Software Test Engineer (Automated Testing) Location: Huntsville, AL (On-site) Type: Contract (Contract-to-hire) Pay: $77–$85/hr (W2) Clearance: Active Secret preferred; must be able to obtain/maintain Must have: Candidate must possess at least a ba...
Date Posted: 09/29/2026 Recommended
-
Mechanical Engineer
Germantown, Maryland
Job Title: Mechanical Engineer Location: Germantown, Maryland Onsite Responsibilities Lead end-to-end hardware development for AERO mobile solution, from conceptual design through production and in-service operation Interact with various stakeholders t...
Date Posted: 07/14/2026 Recommended
-
Mechanical Test Engineer
Tulsa, OK
Job Title: Mechanical Test Engineer Location: Tulsa, OK Type: Direct Hire Compensation: $72,000.00 - $100,000.00 Work Model: Onsite Hours: 40.0 We are seeking a Mechanical Test Engineer to develop and execute test plans for complex machinery and equipm...
Date Posted: 09/30/2026 Recommended
-
Manufacturing Automation/ Commissioning Engineer
York, PA
Manufacturing Automation / Commissioning Engineer (Robotics) Location: York, PA (Onsite) Type: Direct Hire | Schedule: Full-time (40 hrs/week) Travel: Up to 25% Join a team building and launching robotic automation systems—and be the person who brings ...
Date Posted: 09/06/2026 Recommended
-
Senior Test Automation Engineer – Payments
Salt Lake City, Utah
Job Title: Senior Test Automation Engineer – Payments Location: Salt Lake City, UT Type: Contract to hire Responsibilities • Design, develop, enhance, and maintain scalable and reusable test automation frameworks for complex enterprise applications. • ...
Date Posted: 10/05/2026 Recommended
-
Senior OpenShift/Kubernetes GitOps and CI/CD Engineer
Washington, DC
Job Title: Senior OpenShift/Kubernetes GitOps and CI/CD Engineer Location: Washington, DC Type: Contract Compensation: $75-80/hr W-2 Work Model: Remote – offsite Hours: 40.0 Security Clearance: Must be able to obtain a Public Trust clearance Responsibi...
Date Posted: 09/30/2026 Recommended
-
AI Engineer
Atlanta, Georgia
Job Title: AI Engineer Location: Atlanta, GA or the Washington, DC / Maryland / Virginia area Type: Direct Hire Compensation: $155,000.00 Work Model: Hybrid – onsite and remote Responsibilities Build and enhance production generative AI applications an...
Date Posted: 09/28/2026 Recommended
-
Electrical Engineer
Pittsburgh, PA
Job Title: Electrical Engineer Location: Pittsburgh, PA Overview We are seeking a talented and hands-on Electrical Engineer to contribute to the design, development, fabrication, assembly, and testing of advanced electrical systems and technology-drive...
Date Posted: 09/11/2026 Recommended
-
Software Developer
McLean, VA
Software Developer Practice: Software Development & Support / Digital Company: Guidehouse Location: Remote Security and Work Authorization Requirements • Must be able to obtain and maintain a Federal or Department of Defense Public Trust. • Candidates ...
Date Posted: 08/25/2026 Recommended
-
Startup Manager
Houston, TX
Job Title: Deputy Commissioning Manager CQV Location: Houston, TX Type: Contract Compensation: DOE Work Model: Onsite – onsite Hours: 40.0 Security Clearance: Not specified System One is seeking a high-caliber Deputy Commissioning Manager to ensure sta...
Date Posted: 09/24/2026 Recommended
-
Senior Safety Verification and Validation Engineer
Bellevue, Washington
Job Title: Senior Safety Verification and Validation Engineer Type: Contract (12-months) Contractor Work Model: Remote About the Role: As a Senior Safety V&V Engineer, you will be instrumental in verifying and validating safety assessment models and so...
Date Posted: 09/28/2026 Recommended
-
Manufacturing Engineer III
Huntsville, AL
Job Title: Manufacturing Engineer III Location: Huntsville, AL Compensation: $72 per hour Type: Contract (6-month assignment) As a Manufacturing Engineer, you will translate engineering intent, drawings, and configurations into operational steps and pr...
Date Posted: 09/01/2026 Recommended
-
Platform Engineer
McLean, VA
Platform Engineer US Citizenship and the ability to obtain Public Trust is required 100% remote W2 only, no C2C Responsibilities/Requirements: • Implement and maintain modern DevSecOps practices, including secure CI/CD pipelines, automated build and de...
Date Posted: 08/25/2026 Recommended
-
Distribution Systems Test Engineer
Lubbock, TX
Distribution Systems Test Engineer Location: Lubbock, TX (Hybrid Schedule) Schedule: Full-Time, 40 Hours per Week Duration: Contract with Potential for Long-Term Opportunity Hourly Pay Rate: $48.00 - $50.00 System One is seeking a Distribution Systems ...
Date Posted: 09/22/2026 Recommended
-
Fuel Handling Mechanical Design Engineer
Bellevue, WA
Job Title: Fuel Handling Mechanical Design Engineer Location: Bellevue, WA Type: Contract Work Model: Remote Responsibilities Design of nuclear fuel handling equipment, and fuel handling components from requirements development through final design, ma...
Date Posted: 09/24/2026 Recommended
-
Mechanical Designer
Houston , Texas
Mechanical Designer Location: North Houston, TX (77032) Position Type: Contract-to-Hire Compensation: $75,000 / year Position Overview We are seeking a Mechanical Designer to join our client's engineering team in North Houston. This role focuses on des...
Date Posted: 08/07/2026 Recommended
-
Maintenance Mechanic
Eatontown, NJ
Job Title: Maintenance Mechanic / Technician Location: Eatontown, NJ Type: Full-time, direct hire Compensation: $65,000.00 to $68,000.00 Work Model: Onsite. Some travel outside NJ once or twice per year for training (company paid). All tools supplied. ...
Date Posted: 09/28/2026 Recommended
-
Lead AI Agent Engineer
Birmingham, Alabama
Job Title: Lead AI Agent Engineer Location: Birmingham, AL, Knoxville, TN or Lafayette, LA Type: Permanent Full Time Responsibilities Implement, administer, secure, and support enterprise AI coding agent and developer productivity platforms within a la...
Date Posted: 10/02/2026 Recommended
-
Engineer - Manufacturing III
Huntsville, AL
Manufacturing Engineer III Location: Huntsville, Alabama A-Shift (5:00AM to 3:30PM) Monday to Friday Compensation: starting at $72 per hour Contract / On-site Position Overview As a Manufacturing Engineer, you will translate engineering intent, drawing...
Date Posted: 07/28/2026 Recommended
-
Intermediate Capital Cost Estimator (Electrical & Instrumentation)
Greentree, PA
Intermediate Capital Cost Estimator (Electrical & Instrumentation) Greentree, PA, United States Responsibilities Perform detailed material take-offs from single-line diagrams, instrument indexes, cable schedules, and loop diagrams for power, control, a...
Date Posted: 09/21/2026 Recommended
-
Electrical Designer
Cranberry Township, Pennsylvania
Job Title: Electrical Designer Location: Cranberry Township, PA Type: Direct Hire Compensation: $70,000.00 - $85,000.00 Work Model: Onsite – onsite Responsibilities Design electrical and control systems for automated manufacturing equipment Work with p...
Date Posted: 09/09/2026 Recommended
-
Manufacturing Engineer
Tulsa, OK
Job Title: Manufacturing Engineer Location: Tulsa, OK Type: Direct Hire Compensation: $55,000.00 - $75,000.00 Work Model: Onsite – onsite Hours: 40.0 Overview We are seeking a Manufacturing Engineer for a leading manufacturer in the Tulsa, OK area. Res...
Date Posted: 09/30/2026 Recommended
⚠️ Protecting Job Seekers from Identity Scams
At System One, your privacy and security are important to us. As job-related scams continue to target job seekers, we want to help you recognize and avoid common recruitment fraud. Identity theft scams often involve individuals, also known as "scammers," posing as recruiters or employers to obtain sensitive personal information, such as Social Security numbers, bank account details, or copies of driver’s licenses, with the goal of stealing identities, accessing financial accounts, or creating fraudulent accounts in a victim’s name.