Goal

Future-proof public sector client’s data management processes (collection, sharing, use, storage and publication). Consolidate data assets from multiple source systems for analysts to gain better insights and enrich their analyses.

Role

• Developed and optimized AWS-based ETL processes: Designed and implemented 5 AWS Lambda functions and multiple AWS Glue ETL jobs, enhancing data ingestion and processing efficiency. This resulted in a 30% reduction in data pipeline execution time by automating data mapping, data asset initialization, and data preparation workflows.
• Implemented robust data sanitization and compliance measures: Enhanced data quality and regulatory compliance by debugging and extending AWS Glue ETL jobs to perform comprehensive data sanitization, personally identifiable information (PII) identification and redaction. This ensured all data handling adhered to GDPR standards, securing sensitive information across all datasets.
• Streamlined data migration and documentation: Authored high-level and low-level design documents and testing plans to ensure clear, maintainable code and reliable data migration processes. Developed 25 custom scripts for efficient data migration into AWS PostgreSQL, facilitating the seamless transition of critical data assets with minimal downtime.

Capability

Data-driven decision-making and reporting enabled via custom built:

1. Data Integration: Combined data from various sources into a unified data warehouse.
2. Data Transformation: Standardised and sanitised data for consistency and accuracy.
3. Data Accessibility: Provided a user-friendly interface for accessing and querying data.
4. Reporting and Analytics: Enabled comprehensive reporting and analytics capabilities.

Tech Stack

• Python – coding language for data pipelines
• SQL – DQL, DML, DDL
• AWS Cloud Services
  • Aurora RDS – To host a data warehouse
  • Glue – for data pipelines
  • Lambda – serverless compute platform
  • Step Function – workflow orchestrator
  • S3 – cloud storage
  • Comprehend – Machine Learning powered, continuoulsy trained Natural Language Processing (NLP) service. Used to identify and redact the personally identifiable information (PII).
  • CodeCommit – for repository management
  • CloudShell – browser-based CLI environment
• Apache Airflow – for orchestration of data pipelines using DAGs
• Docker Engine / Desktop – to create containers for airflow environment and postgresql instance
• DevOps – GitHub Actions, Terraform
• Git – for repository management
• Pycharm – IDE
• Atlassian Toolset – Jira / Confluence
• Sharepoint – content collaboration and workspace
• Mural – content collaboration / design
• Microsoft Visio – solution design

Year

2024

Goal

Client wanted to continuously quantify how the biggest and 200+ most polluting companies compare to each other (on the 0-100 scale) when it comes to their reputation + performance in causing less pollution + pro-activity in the space of general sustainability.

Role

• Leveraged python OOP code across various tasks, reducing time-to-delivery by 60%. The code executes competitive benchmarking and sentiment analysis, thus improving the client’s campaign accuracy.
• Co-developed python data pipelines for web crawling, ETL, data validation, and testing. To our client this serves as an extensive and efficient social listening system. The system scans corporate social media, annual reports for the 200+ companies, processes the data and feeds into the charts and KPIs at the front-end.
• Quantified the reputation, sustainability and pollution reduction efforts across 200 target companies. Outcome was key in the acquisition of 2 new high-profile clients;

Capability

Multi-source and multi-keyword data pipelines driving sentiment analysis, competitive benchmarking and effective resource allocation in multinational advertising and public relations company.

Tech Stack

Python (OOP) ; PyTest ; data pipelines building (Python + Luigi) ; Regular Expressions (Regex) ; Git ; Data validation (Great Expectations); Data ETL ; Web crawling (pages and apis) ; AWS S3

Year

2022

Goal

To future-proof and centralise their data assets, client embarked on a cloud-based data migration and transformation journey.

Role

• Supported the client’s cloud-based data migration and centralization initiative, resulting in a 25% reduction in data access time and saving 000s of pounds in annual maintenance costs by:
  • Collaborating  on PySpark and SQL-based Azure Synapse notebooks;
  • Scripting CI/CD integration for data loading, transformation, validation, and writing into the ‘Strategic Data Platform’ SDP) for HM Courts and Tribunals Service, UK Ministry of Justice.

Tech Stack

SQL ; PySpark ; Azure Data Factory (ADF) ; Synapse Analytics; CI/CD ; Regular Expressions (Regex) ; Git ; Microsoft Azure Cloud

Year

2023

Goal

Derive meaningful insights and knowledge from large volumes of text data.

Role

I developed a python and spark based tools to recognise, process and summarise untructured text data.

Capability

Highly customisable toolkit for rapid and high quality exploration of text patterns:

• Profile unstructured text data by recognizing and quantifying either all text strings or user-defined ones.
• Capture context variations around text patterns of interest
• Conduct semantic grouping at scale to enhance clarity regarding patterns in text and improve the accuracy of conclusions.
• Produce analytics-ready summarized data.

Tech Stack

Regex, Spark, Python, Big Data

FROM:

Polluted + unnecessarily dilluted view of text patterns

TO:

Contextual focus + meaningful relative scale

Year

2022

Data collection tool for producing accurate forecasts of the total supply in the UK’s natural gas market.

Goal

Client asked me to fix 3 resource drains:

1. Time drain. Time consuming data collection.
2. Money drain. Manually collected data was used in the natural gas demand forecasting process. Poor data quality and insufficient data collection frequency resulted in suboptimal forecasting results and significant revenue loss.
3. Stress. Unlock new time for more rewarding work and reduce human stress.

Role

I automated tedious data processes with python (retrieval, harmonization, reformatting, sharing), saving 2 FTE hours per day, improving forecasting accuracy, maximizing company revenue, and creating more time for rewarding work.

Capability

I delivered 3 benefits for the client:

1. Time gain. 2 hours (1 FTE equivalent) of time savings per day;
2. Money gain. Better data quality, higher frequency and faster collection process significantly improved demand forecasting deliverables and protected key revenue stream.
3. Reduced human stress. New time for more rewarding work unlocked / stress reduced.

Preview

Tech Stack

Python, API development, Database design, Data pipelines, ETL, Web scraping, Data quality automation.

Year

2019

Goal

Power reporting process to create a lot of visibility and impact + deliver time and money savings on production.

Role

I developed a python tool to automate tedious data process with python (retrieval + sharing) for the FTSE 100 company. This has boosted their reporting efficiency + unlocked new time for a more rewarding work.

Capability

Produces a user-friendly and comprehensive presentation to the top management. Python tool actions below for 100+ reports:

• authenticates access to an in-house web-based system
• opens necessary pages
• downloads page contents as offline HTML files
• uploads HTML files onto SharePoint site

Tech Stack

Python, Microsoft SharePoint.

Year

2021

Goal

Automatically recognise, process and evaluate structured text data. Tool builds up an integrity score (0-100 score) for a professional advisor based on the attributes of the reviews they receive.

Role

I developed a python tool to recognise, process and evaluate structured text data.

Capability

Python tool reads input text data line by line and calculates an integrity score based on a number of factors:

• Lots to say: Genuine reviewers tend to say less – knock 0.5% points off for each review that contains more than 100 words.
• Burst: If a number of reviews come in within the same time frame – knock 40% points off if 2 or more come through in the same minute, 20% points if they come through in the same hour.
• Same Device: We have a system that forms a readable tag (e.g. LB4-6WR) based on the browser/device/location. If we are seeing multiple reviews coming from the same device knock 30% points off each time.
• All-Star: Non-genuine reviews are likely to have a five-star rating take 2% points off the integrity score for each review that has 5 stars; quadruple the penalty if the average is under 3.5 stars.
• Solicited: If the review was left by someone who was invited by the professional then add 3% points to the integrity score.

Tech Stack

Python, API development

Year

2020