Location: Hybrid 3 days onsite per week Sheffield, Birmingham, or London (UK)

Contract Duration:8 months

Day Rate: £450 £500 per day (Inside IR35)

Role Overview

This is a specialist role focused on designing, deploying, and optimising Splunk Enterprise and Splunk IT Service Intelligence (ITSI) in complex hybrid Kubernetes/OpenShift environments. You will handle large-scale data onboarding, build advanced ITSI service models and monitoring views, tune platform performance, implement secure governance, and integrate with modern observability pipelines. The position supports critical observability, reliability, and cost management for containerised workloads in a high-stakes enterprise setting.

Key Responsibilities

• Design, deploy, and operate Splunk Enterprise and ITSI in hybrid Kubernetes/OpenShift environments.
• Onboard data at scale using HEC, Universal Forwarders/Deployment Server; align to Common Information Model (CIM); enforce RBAC, retention policies, and cost guardrails.
• Build ITSI service decompositions, KPIs (including multi-KPI), adaptive/time-based thresholds, NEAP policies, glass tables, deep dives, and service health scoring.
• Create OpenShift-specific executive and operations views: cluster health (API/etcd), node readiness/pressure, pod restart hotspots, network/storage errors, capacity, quotas, and bursting visibility.
• Tune search/platform performance: workload rules, concurrency limits, Data Model Acceleration (DMA), summary indexing, and scheduling optimisation.
• Implement alerting, event enrichment, routing to ITSM/ChatOps, suppression windows, maintenance schedules, and runbook automation.
• Govern data ingest and security: allow/deny lists, PII handling, TLS/mTLS, token/cert governance, index/role mapping, and data quality SLAs.
• Integrate upstream sources/pipelines: OpenTelemetry (OTLP), Prometheus exporters, Fluentd/Fluent Bit/Vector, Kafka (with TLS), CMDB/ITSM enrichments, and AIOps/ML anomaly detection.

Essential Skills & Experience

• Deep Splunk Enterprise expertise: SPL mastery, CIM alignment, KV stores/lookups/macros, saved searches, index/retention/RBAC design, search performance tuning.
• Advanced Splunk ITSI knowledge: Service trees/decompositions, KPIs/thresholds (adaptive/time-based), NEAP tuning, glass tables, deep dives, Service Analyzer configuration.
• Strong OpenShift/Kubernetes observability: Cluster/control-plane metrics, kube events/logs, workload/node/network/storage correlations, capacity/noisy-neighbor detection.
• Experience with data pipelines/collectors: OpenTelemetry, Prometheus scraping, Fluentd/Fluent Bit/Vector, Kafka (TLS-secured), HEC/UF/DS onboarding.
• Reliability & SLOs: Golden signals, rollout/rollback health checks, SLO/KPI mapping to namespaces/apps, executive/ops dashboards.
• Performance & cost optimisation: Workload rules, DMA, summary indexing, schedule hygiene, license/cost guardrails.
• Security & compliance: TLS/mTLS, token/cert management, PII controls, auditability, role/index mappings.
• Automation & integrations: ITSM/ChatOps routing, runbooks, CMDB enrichment, webhook/AIOps integrations.

Preferred / Desirable

• Hands-on experience in regulated/financial services environments.
• Certifications: Splunk Enterprise Certified Architect, Splunk ITSI Certified Admin, or equivalent.
• Familiarity with AIOps/ML features in Splunk for anomaly detection.
• Previous work with container platforms (Kubernetes/OpenShift) for observability at scale.

Success Measures

• High-quality, scalable Splunk/ITSI deployments with optimised performance and cost controls.
• Effective service health monitoring via ITSI (accurate KPIs, glass tables, deep dives).
• Reduced alerting noise, improved incident response through enriched routing and automation.
• Strong governance, security compliance, and traceability in data ingest/observability pipelines.

This role is ideal for a Splunk specialist with proven expertise in ITSI and container observability, who can deliver robust, production-grade monitoring solutions in dynamic hybrid environments. Applications must be PAYE via Umbrella.