blob: 9c7baee5ea1e022451a1ff92224674f1e5a95b77 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# ToteSys - Data Engineering Project
# Summary
The project aims to implement a data platform that can extract data from an
operational database, archive it in a data lake, and make it easily accessible
within a remodelled OLAP data warehouse.
The solution showcases our skills in:
- Python
- PostgreSQL
- Database modelling
- Amazon Web Services (AWS)
- Agile methodologies
# Main Objective
Our goal is to create a reliable ETL (Extract, Transform, Load) pipeline that
can:
1. Extract the data from the `totesys` operational database
2. Store the data in AWS S3 buckets, that will form our data lake
3. Transform the data into a suitable schema for the data warehouse
4. Load the transformed data into the data warehouse hosted on AWS
|
