Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared
Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required.
The Role:
We are looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data‑driven applications and pipelines in a collaborative Dev and DevOps environment.
The role focuses on writing and improving application code while working closely with DevOps engineers to support automated deployments, infrastructure management, system monitoring, reliability, and scaling. The post holder collaborates with Product Owners, Business Analysts, and users to translate business needs into robust technical solutions, operates and improves production services, analyses data and system issues to identify root causes, and champions engineering best practices. They also provide technical leadership through coaching and mentoring junior colleagues and contributing to continuous improvement across development, delivery, and operational processes.
Responsibilities:
Engineers will contribute to research, development and delivery across:
æ Design, build, and operation of data ingest and publishing pipelines
æ Workflow orchestration and task scheduling using managed services
æ Collaboration with Product Owners, Business Analysts, and users to shape technical solutions
æ Production support, monitoring, and continuous improvement of system resilience, stability, and performance
æ Data analysis and investigation to identify root causes of defects and operational issues
æ DevOps collaboration, including supporting automated deployments, infrastructure management, monitoring, and scaling
æ Coaching and mentoring junior engineers and promoting engineering best practices
Skills & Experience:
æ Understanding of data processing using Apache Spark
æ Use of Python, SQL, and familiarity with PySpark
æ Experience using Apache Airflow for task orchestration
æ Understanding of EMR and reviewing output logs
æ Use of Jupyter notebooks and/or Amazon Athena to query and validate data
æ Data analysis to identify root cause of issues
æ Understanding of dimensional data models and slowly changing dimensions/historic data capture
æ Use of AWS console and services such as, but not limited to; CloudWatch, IAM, S3, Glue, ECR, EC2, EMR, Dynamo DB, LakeFormation
æ Familiarity with Amazon Textract and Comprehend
æ Understanding of both server-side and client-side encryption
æ Use of GitLab for source code management pipelines for CI/CD
æ Use of GitLab Tags for component versioning in shared repositories
æ Understanding of Docker and containerization of solutions
æ IaC using Terraform
æ Experience of understanding how customer expectations transition to applied functionality
æ Familiarity with, and implementation of Engineering best practices
æ Use of gitlab for release tagging and deployments
æ Familiarity with basic data structures for constructing a solution
æ Active BPSS, SC clearance or eligible for clearance
Desirable skills:
æ Experience supporting AI or data driven platforms
æ Knowledge of cyber security or fraud prevention domains
æ Experience working within government or critical national infrastructure environments
Find out more: or check out our LinkedIn page
Salary description
£0.00 - £59000.00 per year