AI Evaluation Specialist
Atlanta, Georgia
Experience: 5
Category: Information Technology
Contractor Work Model: Hybrid
Brand: System One
Compensation Range: 155000.00
Job ID: 382967
Date Posted: 11/03/2026
Shortcut: http://jobs.systemone.com/HscRgy
Job Title: AI Evaluation Specialist
US Citizenship or Green Card required per government contract
Clearance: Must be able to obtain Public Trust
Salary: $155K
Job Description:
In this position, you’ll evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. You’ll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.
Key Job Responsibilities
- Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas.
- Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
- Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.
- Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders.
- Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost.
- Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable.
- Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program.
- Engage in knowledge sharing and contribute to building a strong, collaborative team environment.
- 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services
- Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
- Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
- Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
- Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
- Bachelor's degree
- Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder)
- Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions
- Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing
- Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows
- Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results
- Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
- Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
- Master’s degree preferred
#M1
#LI-VH1
Ref: #851-Rockville-S1
-
Biology/Public Health Surveillance AI Specialist
Atlanta, Georgia
Biology/Public Health Surveillance AI Specialist US Citizenship or Green Card required per government contract Clearance: Must be able to obtain Public Trust Salary: $155K Job Description: In this position, you’ll support the next generation of public ...
Date Posted: 11/03/2026 Recommended
-
AI Engineer
Atlanta, Georgia
Job Title: AI Engineer Location: Atlanta, GA or the Washington, DC / Maryland / Virginia area Type: Direct Hire Compensation: $155,000.00 Work Model: Hybrid – onsite and remote Responsibilities Build and enhance production generative AI applications an...
Date Posted: 10/28/2026 Recommended
-
Enterprise AI Architect
knoxville, Tennessee
Job Title: Enterprise AI Architect Job Category: Architecture Job Type: Permanent Full Time Location: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL. Description: Enterprise AI Architect * Position Description Seeking an experienced Enterpr...
Date Posted: 09/10/2026 Recommended
-
AI Automation Engineer
Vienna, VA
Job Title: AI Test Engineer Location: Vienna, VA Type: Contract Work Model: Hybrid – onsite and remote Hours: 40.0 Responsibilities • Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-power...
Date Posted: 10/06/2026 Recommended
-
AI Scrum Master with Python - 100% remote
REMOTE, Virginia
Job Title: AI Scrum Master with Python skills MUST HAVE PYTHON PROGRAMMING SKILLS Location: 100 % remote Type: Contract To Hire Compensation: W-2 Negotiable Work Model: Remote Hours: 40 hours a week Responsibilities Serve as SAFe Agile Scrum Master for...
Date Posted: 10/07/2026 Recommended
-
AI/ML Data Scientist
McLean, VA
Job Title: AI/ML Data Scientist Location: 100% REMOTE Clearance: Ability to obtain and maintain a Public Trust clearance. Type: Contract-to-hire Contact: Crystal.dinnocenti@systemone.com WHAT YOU WILL DO: Partner with stakeholders to define and deliver...
Date Posted: 09/28/2026 Recommended
-
Generative AI Developer
Knoxville, Tennessee
Job Title: Generative AI Developer Location: Lafayette, LA, Knoxville, TN, Columbia, SC or Birmingham, AL Type: Direct Hire Compensation: $108,600.00 Work Model: Onsite – onsite Hours: 40.0 Security Clearance: (not specified) Overview Leave placeholder...
Date Posted: 09/10/2026 Recommended
-
Senior GenAI & Data Engineer
Reston, Virginia
Job Title: Senior GenAI & Data Engineer Location: Reston, Virginia Type: Direct Hire Work Model: Hybrid – onsite and remote Overview System One is seeking a Senior GenAI & Data Engineer to develop and integrate AI-enabled solutions within a large-scale...
Date Posted: 09/25/2026 Recommended
-
Senior GenAI & Data Engineer
Reston, Virginia
Job Title: Senior GenAI & Data Engineer Location: Reston, Virginia Type: Direct Hire Work Model: Hybrid – onsite and remote Overview System One is seeking a Senior GenAI & Data Engineer to develop and integrate AI-enabled solutions within a large-scale...
Date Posted: 09/19/2026 Recommended
-
Lead AI Agent Engineer
Birmingham, Alabama
Job Title: Lead AI Agent Engineer Location: Birmingham, AL, Knoxville, TN or Lafayette, LA Type: Permanent Full Time Responsibilities Implement, administer, secure, and support enterprise AI coding agent and developer productivity platforms within a la...
Date Posted: 10/02/2026 Recommended
-
Senior Atlassian Tools Administrator
Columbia, South Carolina
Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham, AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator to support and enhance a large scale Atlassian Cloud env...
Date Posted: 08/28/2026 Recommended
-
Senior Atlassian Tools Administrator
Knoxville, Tennessee
Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham, AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator to support and enhance a large scale Atlassian Cloud env...
Date Posted: 08/28/2026 Recommended
-
Senior Atlassian Tools Administrator
Birmingham, Alabama
Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham, AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator to support and enhance a large scale Atlassian Cloud env...
Date Posted: 08/28/2026 Recommended
-
Senior Atlassian Tools Administrator
Lafayette, Louisiana
Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham, AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator to support and enhance a large scale Atlassian Cloud env...
Date Posted: 08/28/2026 Recommended
-
SDET – AI Test Automation (AWS, Playwright, CloudBees)
Columbia, MD
Job Title: SDET – AI Test Automation (AWS, Playwright, CloudBees) Location: Columbia, MD Type: Contract Compensation: $62.40 Work Model: Hybrid – onsite and remote Hours: 40.0 Responsibilities • Design, develop, and maintain automated test scripts and ...
Date Posted: 10/15/2026 Recommended
-
Senior AWS Full Stack Software Engineer
Knoxville , Tennessee
Senior AWS Full Stack Engineer Locations: Knoxville, TN | Columbia, SC | Lafayette, LA | Birmingham, AL Work Arrangement: Hybrid — candidates must be able to work from one of the listed locations Industry: Financial Services Experience: 5+ years We are...
Date Posted: 09/04/2026 Recommended
-
Senior AWS Full Stack Software Engineer
Knoxville , Tennessee
Senior AWS Full Stack Engineer Locations: Knoxville, TN | Columbia, SC | Lafayette, LA | Birmingham, AL Work Arrangement: Hybrid — candidates must be able to work from one of the listed locations Industry: Financial Services Experience: 5+ years We are...
Date Posted: 08/31/2026 Recommended
-
Senior Cyber AI Operations Engineer (Onsite)
Washington, DC
Cyber Artificial Intelligence Operations Engineer 100% ONSITE in Washington DC Per Federal contract U.S. Citizenship is required Must be able to pass enhanced background screen (criminal, financial, drug) for Public Trust clearance Long Term Multi-Year...
Date Posted: 08/28/2026 Recommended
-
AI Engineer Agentic Systems - must be local in Pittsburgh, PA
Pittsburgh, Pennsylvania
*** This position is open only to local candidates in Pittsburgh, PA. The role is fully on-site with no remote work. We are unable to offer relocation help. We are also unable to offer any visa sponsorship or green card sponsorship for this role *** Fo...
Date Posted: 10/26/2026 Recommended
-
AWS Cloud Security Architect
Birmingham, Alabama
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
AWS Cloud Security Architect
Knoxville , Tennessee
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
AWS Cloud Security Architect
Plano, Texas
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
AWS Cloud Security Architect
Knoxville, Tennessee
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 09/12/2026 Recommended
-
AWS Cloud Security Architect
Birmingham, Alabama
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 09/12/2026 Recommended
-
AWS Cloud Security Architect
Lafayette, Louisiana
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
AWS Cloud Security Architect
Reston, Virginia
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
AWS Cloud Security Architect
Columbia, South Carolina
Job Title: AWS Cloud Security Architect Location: Plano, TX/Reston, VA/Columbia, SC/Knoxville, TN/Lafayette, LA/Birmingham, AL Work Model: Onsite – onsite Responsibilities Provide security architecture guidance for enterprise applications, AWS cloud en...
Date Posted: 08/31/2026 Recommended
-
Senior AWS Java/Python Developer with Microservices & API experience
Knoxville, Tennessee
Job Title: Senior AWS Java/Python Developer with Microservices & API experience Location: Knoxville, TN Type: Direct Hire Compensation: $98,000.00 Work Model: Onsite – onsite Hours: 40.0 Security Clearance: (not specified) Overview Leave placeholder te...
Date Posted: 09/09/2026 Recommended
-
AWS AI/Data Engineer
Knoxville, Tennessee
Job Title: AWS AI/Data Engineer Location: Knoxville, Tennessee Type: Direct Hire Work Model: Hybrid – onsite and remote Overview System One is seeking an AWS AI/Data Engineer to support the development of AI-driven and analytical capabilities for enter...
Date Posted: 09/21/2026 Recommended
-
Senior AWS Full Stack Engineer
Pittsburgh , Pennsylvania
Senior AWS Full Stack Engineer Full-time Hybrid – Knoxville, TN | Columbia, SC | Lafayette, LA | Birmingham, AL Position Overview Seeking a Senior AWS Full Stack Engineer to develop cloud-based enterprise applications across frontend, backend, database...
Date Posted: 08/28/2026 Recommended
⚠️ Protecting Job Seekers from Identity Scams
At System One, your privacy and security are important to us. As job-related scams continue to target job seekers, we want to help you recognize and avoid common recruitment fraud. Identity theft scams often involve individuals, also known as "scammers," posing as recruiters or employers to obtain sensitive personal information, such as Social Security numbers, bank account details, or copies of driver’s licenses, with the goal of stealing identities, accessing financial accounts, or creating fraudulent accounts in a victim’s name.