Business-Critical Rail Data Pipelines

Goal

Modernise business-critical rail data processing by replacing legacy desktop and manual workflows with reliable, scalable and governed AWS data pipelines.

The programme needed to move important operational datasets — including ticket sales, ticket scans, reservation availability, customer-experience feeds and delay-repay data — onto a centrally managed cloud platform without disrupting existing reporting or downstream consumers.

The objective was not simply to “move data to AWS”. It was to create production-ready data services that could be trusted, monitored, maintained and extended by the wider engineering team.

Role

As a Contract AWS Data Engineer, I designed, built, tested and operationalised production data pipelines across LNER’s AWS data platform.

My work covered the full delivery lifecycle: understanding existing business processes, analysing legacy outputs, designing cloud-native replacements, building ingestion and transformation logic, implementing data-quality controls, supporting user acceptance testing, deploying infrastructure and resolving production issues.

Legacy-to-cloud modernisation. Replaced Alteryx and manually operated data processes with automated AWS pipelines designed for central ownership, scalability and long-term maintainability.
Production data engineering. Built ingestion and transformation workflows across APIs, files and databases using Python, SQL, PySpark, AWS Glue, Lambda, Step Functions, S3, Athena and related AWS services.
Incremental and historical processing. Designed pipelines capable of processing new data incrementally while also handling large historical backfills and safely bringing legacy history into the same curated datasets.
Data quality and reconciliation. Implemented completeness, uniqueness, schema and business-rule validation so data problems were detected before reaching downstream users. Outputs were reconciled against existing processes during UAT to prove that the new platform could safely replace them.
Production reliability. Designed restartable processing, pagination and continuation behaviour, deterministic record keys, retries and operational controls so pipelines could recover safely from failures rather than requiring manual rebuilding.
Secure external data delivery. Built controlled outbound feeds to external partners using encryption, managed credentials and restricted cloud access rather than exposing internal data stores directly.
Infrastructure and deployment. Delivered cloud configuration and environments using Terraform/Terragrunt and CI/CD, supporting repeatable deployment across development, staging and production.
Stakeholder collaboration. Worked directly with engineers, analysts, data providers and business users to understand requirements, resolve discrepancies and turn existing operational processes into maintainable technical solutions.
Technical ownership and documentation. Produced detailed pipeline and operational documentation so systems could be understood, supported and extended after delivery.

The work ranged from straightforward scheduled extracts to complex multi-stage pipelines combining incremental APIs, historical data, transformation, quality controls and curated AWS data stores.

Capability

The resulting capability moved business-critical processing away from individually operated desktop workflows and into centrally managed cloud services.

Business Impact

Ticket scan data. Replaced a legacy Alteryx process with a governed AWS pipeline, giving teams a more reliable source for ticket-scan analysis and reporting. Historical migration covered 100+ million scan records.
Ticket sales data. Brought historical and ongoing ticket-sales data into one automated cloud pipeline, improving consistency for reporting and reconciliation while reducing reliance on manual processing.
Reservation and ticket availability. Replaced legacy reservation workflows with a cloud-based data service, giving the business more reliable visibility of seat and ticket availability across services, journeys and booking horizons.
Customer-experience punctuality feed. Automated the daily delivery of disrupted-train data into the customer-experience platform, supporting timely analysis of how disruption affects passenger experience and removing dependence on the legacy feed.
Delay Repay data delivery. Automated the daily flow of rail transaction data into Delay Repay processing, replacing repeated manual extraction with a faster, more reliable and controlled data feed.
Safer migration. Reconciled new cloud outputs against existing processes before cutover, reducing the risk of disrupting reporting or downstream users.
Better resilience. Built pipelines to recover safely from failures, rerun where needed and reduce manual intervention.
Stronger data quality. Embedded validation so incomplete, duplicate or invalid data could be caught before reaching analysts, reports or external consumers.
Easier future change. Replaced individually operated desktop workflows with centrally managed, documented cloud services that were easier to support and extend.

How

Scalable cloud processing. AWS-native services supported growing data volumes, historical backfills and recurring production workloads.
Built-in data quality. Completeness, uniqueness, schema and business-rule checks were embedded into processing.
Recoverable processing. Incremental state, deterministic keys and restartable workflows supported safe continuation after interrupted runs.
Secure data delivery. Encryption, AWS IAM, Secrets Manager and governed external delivery patterns protected business data.
Operational transparency. Logging, orchestration and documented run behaviour made failures easier to diagnose and recover.
Maintainable engineering. Infrastructure as Code, CI/CD and reusable platform patterns supported repeatable deployment across environments.
Controlled migration. Legacy outputs were reconciled against new AWS pipelines before production adoption.

Tech Stack

Programming & Data Engineering: Python, SQL, PySpark
AWS Data Processing: AWS Glue, Lambda, Step Functions
AWS Storage & Analytics: S3, Athena, Redshift
Data Lake Technologies: Apache Hudi, Parquet
Orchestration & Scheduling: AWS Step Functions, EventBridge
Security & Access: AWS IAM, Secrets Manager, Lake Formation
Infrastructure as Code: Terraform, Terragrunt
DevOps: Git, CI/CD, GitHub Actions
Data Integration: REST APIs, JDBC, CSV, JSON, Parquet, SFTP
Data Quality: Schema validation, completeness checks, uniqueness controls, reconciliation, deterministic record keys
Secure Delivery: PGP encryption, SSH/SFTP, cross-account AWS delivery
Collaboration & Delivery: Jira, Confluence, technical documentation, UAT and stakeholder reconciliation

Year

2026