Senior Splunk & Observability Engineer
Birmingham, Alabama
Experience: 0
Category: Information Technology
Contractor Work Model: Onsite
Brand: System One
Compensation Range: 90000.00 - 160000.00
Job ID: 380104
Date Posted: 09/11/2026
Shortcut: http://jobs.systemone.com/ThgoQh
*** Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer )*****
Job Title: Senior Splunk & Observability Engineer
Duration: Full Time / Permanent Position
Location: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL
Work Mode: 5 Days Onsite
Position Description
Systemone is seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation.
The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions.
The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation.
Your future duties and responsibilities
- Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
- Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2.
- Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery.
- Develop and optimize complex SPL queries and troubleshoot query performance issues.
- Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
- Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues.
- Restore data flow and replay data following ingestion or platform interruptions.
- Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs.
- Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts.
- Develop Python based automation for observability and platform operations.
- Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms.
- Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation.
- Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments.
- Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
- Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes.
Required qualifications to be successful in this role
- 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments.
- Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery.
- Advanced SPL query development and performance optimization skills.
- Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
- Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.
- Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases.
- Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration.
- Experience building or maintaining OpenTelemetry agent and collector packages.
- Working knowledge of Python for automation and operational tooling.
- Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.
- Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery.
- Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes.
- Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams.
Educational Requirements:
Bachelor's degree in Computer Science, Information Systems, or a related field.
Ref: #404-IT Pittsburgh
-
Senior Splunk & Observability Engineer
Columbia , South Carolina
Job Title: Senior Splunk & Observability Engineer Job Type: Permanent Full Time Position Description Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based productio...
Date Posted: 09/11/2026 Recommended
-
Splunk Observability Engineer
Knoxville, Tennessee
For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to denis.potapenk...
Date Posted: 09/11/2026 Recommended
-
Splunk Observability Engineer
Columbia, South Carolina
For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to denis.potapenk...
Date Posted: 09/11/2026 Recommended
-
Splunk Observability Engineer
Lafayette, Louisiana
For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to denis.potapenk...
Date Posted: 09/11/2026 Recommended
-
Splunk Observability Engineer
Birmingham, Alabama
For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to denis.potapenk...
Date Posted: 09/11/2026 Recommended
-
AWS Production Support Engineer
Lafayette, Louisiana
Job Title: AWS Production Support Engineer Location: Knoxville, TN Type: Contract To Hire Compensation: $75,000.00 Work Model: Onsite – onsite Hours: 40.0 Security Clearance: Not specified Overview Leave placeholder text here for recruiter to input Res...
Date Posted: 09/10/2026 Recommended
-
AWS Production Support Engineer
Birmingham, Alabama
Job Title: AWS Production Support Engineer Location: Knoxville, TN Type: Contract To Hire Compensation: $75,000.00 Work Model: Onsite – onsite Hours: 40.0 Security Clearance: Not specified Overview Leave placeholder text here for recruiter to input Res...
Date Posted: 09/10/2026 Recommended
-
DevSecOps Engineer
McLean, VA
DevSecOps Engineer US Citizenship and the ability to obtain Public Trust is required 100% remote W2 only, no C2C Responsibilities/Requirements: • Implement and maintain modern DevSecOps practices, including secure CI/CD pipelines, automated build and d...
Date Posted: 08/25/2026 Recommended
-
Senior API Platform Software Engineer
Columbia, South Carolina
Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior API P...
Date Posted: 08/28/2026 Recommended
-
Senior API Platform Software Engineer
Lafayette, Louisiana
Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior API P...
Date Posted: 08/28/2026 Recommended
-
Senior API Platform Software Engineer
Birmingham, Alabama
Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior API P...
Date Posted: 08/28/2026 Recommended
-
Senior API Platform Software Engineer
Knoxville, Tennessee
Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior API P...
Date Posted: 08/28/2026 Recommended
-
Platform Engineer
McLean, VA
Platform Engineer US Citizenship and the ability to obtain Public Trust is required 100% remote W2 only, no C2C Responsibilities/Requirements: • Implement and maintain modern DevSecOps practices, including secure CI/CD pipelines, automated build and de...
Date Posted: 08/25/2026 Recommended
-
Senior API Platform Software Engineer
Pittsburgh , Pennsylvania
Senior API Platform Software Engineer Fulltime Hybrid – Knoxville, TN | Columbia, SC | Lafayette, LA | Birmingham, AL Position Overview Seeking a Senior API Platform Software Engineer to build and enhance backend services, APIs, and API platform capabi...
Date Posted: 08/28/2026 Recommended
-
Cloud Engineer (onsite)
Washington, DC
Cloud Engineer Washington, DC – 100% Onsite Per Federal contract U.S. Citizenship or Green Card is Required Must be able to pass enhanced background screen (criminal, financial, drug) for Public Trust clearance W-2 or C2C Application Deadline: 8/28/202...
Date Posted: 08/11/2026 Recommended
-
Senior AWS Data Engineer
Pittsburgh , Pennsylvania
Senior AWS Data Engineer Full-time Hybrid – Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Position Overview Seeking a Senior AWS Data Engineer to develop, enhance, and maintain enterprise risk analytics applications in a financial servi...
Date Posted: 08/28/2026 Recommended
-
Senior AWS Data Engineer
Pittsburgh , Pennsylvania
Job Title: Senior AWS Data Engineer Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Description: Senior AWS Data Engineer to support the development, enhancement, and maintenance of enterprise risk analytics applications within ...
Date Posted: 08/28/2026 Recommended
-
Network Engineer
Salt Lake City, Utah
Job Title: Network Engineer Location: Salt Lake City UT Type: Contract to Hire Work Model: Onsite – onsite Hours: 40.0 Security Clearance: Not specified Responsibilities • Design, implement, and maintain scalable, secure, resilient, and highly availabl...
Date Posted: 08/26/2026 Recommended
-
AWS Data Engineer
McLean, VA
Job Title: AWS Data Engineer Location: McLean, VA Type: Contract-to-Hire Compensation: $150,000.00 - $170,000.00 Work Model: Remote – offsite Security Clearance: US Citizen with the ability to obtain a public trust System One is looking for a strong AW...
Date Posted: 09/01/2026 Recommended
-
Lead Devops Engineer
Cincinnati, OH
Job Title: Lead DevOps Engineer Location: Cincinnati, OH (Hybrid – 3 days onsite/week) Type: Direct Hire Compensation: $160,000 + Range (based on experience) We’re seeking an experienced Lead DevOps Engineer to join a small, high-impact infrastructure ...
Date Posted: 09/11/2026 Recommended
-
Senior Infrastructure Engineer
Pittsburgh , Pennsylvania
Senior Infrastructure Engineer Job Type: Permanent Full Time Cleveland, OH, Pittsburgh, PA, or Dallas, TX and is 5 days onsite. Description: Senior Infrastructure Engineer Position Description We are seeking a highly motivated Sr. Infrastructure Engine...
Date Posted: 06/29/2026 Recommended
-
Systems Engineer
Tulsa , Oklahoma
Successful candidates must be able to provide proof of ability to work in the U.S. without sponsorship. This position is not open to corp-to-corp, subcontractor or independent consulting arrangements. About the Role We are seeking a talented Systems En...
Date Posted: 09/10/2026 Recommended
-
Senior F5 Network Engineer
Birmingham, Alabama
Job Title - Senior F5 Network Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior F5 Network Engin...
Date Posted: 08/28/2026 Recommended
-
Senior F5 Network Engineer
Knoxville, Tennessee
Job Title - Senior F5 Network Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior F5 Network Engin...
Date Posted: 08/28/2026 Recommended
-
Senior F5 Network Engineer
Columbia, South Carolina
Job Title - Senior F5 Network Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment Type: Full time Experience Required: 5+ years Position Overview We are seeking a Senior F5 Network Engin...
Date Posted: 08/28/2026 Recommended
-
Platform Engineer (DevOps)
Salt Lake City, Utah
Job Title: Platform Engineer (DevOps) Location: Salt Lake City, UT Responsibilities • Design, build, and optimize enterprise CI/CD pipelines using GitHub Actions and other modern DevOps platforms • Administer enterprise Source Code Management (SCM) rep...
Date Posted: 08/06/2026 Recommended
-
Platform Engineer (DevOps)
Salt Lake City, Utah
Job Title: Platform Engineer (DevOps) Location: Salt Lake City, UT Work Mode : 5 Days Onsite Duration : Contract to Hire Responsibilities • Design, build, and optimize enterprise CI/CD pipelines using GitHub Actions and other modern DevOps platforms • ...
Date Posted: 08/06/2026 Recommended
-
Senior AWS Full Stack Software Engineer
Pittsburgh , Pennsylvania
Job Title: Senior AWS Full Stack Software Engineer Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Description: Senior AWS Full Stack Software Engineer to develop, modernize, and support enterprise applications within a large sc...
Date Posted: 08/28/2026 Recommended
-
Senior Azure Cloud & GitHub Platform Engineer
Merrifield, VA
Title: Senior Azure Cloud & GitHub Platform Engineer Location: Vienna, VA Work Schedule: hybrid, onsite 3 days a week Pay Rate: Open to both W2 and established 1099's Position Type: Multiyear Contract This group is looking for a Senior Azure Cloud & Gi...
Date Posted: 08/17/2026 Recommended
-
AWS Application Production Support Engineer
Birmingham, , Alabama
Job Title: AWS Application Production Support Engineer Location: Lafayette, Louisiana, United States Type: Direct Hire Responsibilities Monitor AWS hosted applications and respond to production alerts and incidents. Perform incident triage using AWS co...
Date Posted: 09/11/2026 Recommended