Lakehouse

FMCG Data Engineering & Analytics Pipeline

Scalable sales, product, and customer analytics platform for FMCG data

FMCG data pipeline showing sales, product and customer data processed through Bronze, Silver and Gold layers

Project Overview

Designed and deployed a comprehensive FMCG data pipeline utilizing Databricks and PySpark to process fast-moving consumer goods transactions. The pipeline handles data profiling, cleansing, deduplication, and business-rule processing across Bronze, Silver, and Gold layers to produce metrics across sales performance, product turn rates, and customer purchasing patterns.

The Problem

FMCG business teams were unable to track sales KPIs or perform customer segmentation due to fragmented transaction logs and unvalidated product catalog entries.

The Solution & Architecture

Implemented a unified Delta Lake pipeline using Spark SQL and PySpark that ingests raw FMCG data, cleanses bad records, executes Delta MERGE incremental updates, and presents normalized sales and customer analytics in Databricks dashboards.

System Data Flow:

Raw FMCG Data→Bronze Layer→Silver Layer→Gold Layer→Sales & Business Analytics

Key Engineering Highlights

End-to-end FMCG data pipeline architecture

Cleansing and schema validation routines

Data transformation and duplicate removal logic

Business-rule processing tailored for FMCG metrics

Gold sales datasets for reporting

Product analytics and customer purchasing insights

Delta MERGE operations for incremental upserts

Spark SQL optimizations and Databricks KPI dashboards

Technologies Used

DatabricksPySparkSpark SQLDelta LakeMedallion Architecture

Interested in this Pipeline?

I can build similar production-grade Lakehouse or API data ingestion systems for your organization.

Contact Owais Ahmad