| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-27 | 2 | -41/+39 | |
| | | | | | | | | | | | | | This commit fixes the style issues introduced in c68f63f according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/97 | |||||
| * | | wip: refactored fact payment function | lian-manonog | 2024-08-27 | 2 | -8/+9 | |
| | | | ||||||
| * | | wip: added a bracket in dataframes | lian-manonog | 2024-08-27 | 2 | -3/+4 | |
| | | | ||||||
| * | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-23 | 1 | -35/+41 | |
| | | | | | | | | | | | | | This commit fixes the style issues introduced in 8f75a47 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/96 | |||||
| * | | Merge branch 'test/test_transform_lambda' into test/dataframes | bulve-ad | 2024-08-23 | 6 | -248/+194 | |
| |\ \ | ||||||
| | * | | wip: pushing again | lian-manonog | 2024-08-23 | 2 | -6/+7 | |
| | | | | ||||||
| | * | | Merge pull request #94 from ajschofield/feature/transform-fact-sales-order | lian-manonog | 2024-08-23 | 2 | -5/+484 | |
| | |\ \ | | |/ | |/| | PR: rebased the fact-sales branch with the datafrrames branch - now merging into development | |||||
| | | * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-23 | 1 | -3/+6 | |
| | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 3ff2182 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/93 | |||||
| | | * | trying to resolce git index issue conflicts - commiting was the only solution | lian-manonog | 2024-08-23 | 1 | -9/+4 | |
| | | | | ||||||
| | | * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-23 | 2 | -157/+236 | |
| | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 8e20c5c according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/93 | |||||
| | | * | Deleted the fact_table schema py files | lian-manonog | 2024-08-23 | 4 | -233/+157 | |
| | | | | | | | | | | | | | | | | | | | Completed Lambda_handler for transform_lambda - and other helper functions. Testing is still to be done. Need to implement lambda layer to share helper functions across all lambdas | |||||
| | * | | fix: improve error handling for list_existing_s3_files and tests | Alex Schofield | 2024-08-22 | 1 | -6/+10 | |
| | | | | ||||||
| | * | | fix: handle no buckets properly | Alex Schofield | 2024-08-22 | 1 | -0/+3 | |
| | | | | ||||||
| | * | | fix: change default parameters for bucket_name and client | Alex Schofield | 2024-08-22 | 1 | -2/+9 | |
| | | | | ||||||
| * | | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-23 | 1 | -96/+154 | |
| | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 821e241 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/96 | |||||
| * | | | tests: additional tests written (pass) for dim tables transformation. Fact ↵ | Ang Bel | 2024-08-23 | 1 | -14/+16 | |
| | | | | | | | | | | | | | transformation functions not yet tested | |||||
| * | | | fix: refactoring for create_dim_date to include all date columns from all ↵ | Ang Bel | 2024-08-23 | 1 | -20/+13 | |
| | | | | | | | | | | | | | fact dfs, tested on dummy data. Tests are not written | |||||
| * | | | removed duplicate functions | T-Aji | 2024-08-23 | 1 | -89/+28 | |
| | |/ |/| | ||||||
| * | | dataframes combined into one file | T-Aji | 2024-08-23 | 1 | -0/+238 | |
| | | | ||||||
| * | | Merge branch 'feature/transform-fact-sales-order' of ↵ | T-Aji | 2024-08-23 | 2 | -5/+53 | |
| |\ \ | | | | | | | | | | https://github.com/ajschofield/de-project-bentley into feature/transform-fact-sales-order | |||||
| | * | | wip: wrote pseudocode for lambda handler in writing df to parquet file ↵ | lian-manonog | 2024-08-22 | 2 | -5/+53 | |
| | | | | | | | | | | | | | format and uploading the parquet files | |||||
| * | | | payment schema added | T-Aji | 2024-08-23 | 2 | -2/+46 | |
| |/ / | ||||||
| * | | complete code for tables for purchase schema including a scrape for ↵ | Ang Bel | 2024-08-22 | 1 | -18/+48 | |
| | | | | | | | | | currency table. Test to be done | |||||
| * | | add fact table | T-Aji | 2024-08-22 | 1 | -50/+0 | |
| | | | ||||||
| * | | Merge branch 'feature/transform-fact-sales-order' of ↵ | T-Aji | 2024-08-22 | 1 | -23/+30 | |
| |\ \ | | | | | | | | | | https://github.com/ajschofield/de-project-bentley into feature/transform-fact-sales-order | |||||
| | * | | refactored approach to writing transformation as functions per df. WIP | Ang Bel | 2024-08-22 | 1 | -23/+30 | |
| | | | | ||||||
| * | | | add fact table | T-Aji | 2024-08-22 | 1 | -4/+31 | |
| |/ / | ||||||
| * | | passing tests create_dim_design and create_dim_staff | T-Aji | 2024-08-22 | 1 | -0/+50 | |
| | | | ||||||
| * | | refactor df creation into func | T-Aji | 2024-08-22 | 1 | -70/+34 | |
| | | | ||||||
| * | | wip: creating dim dataframes | T-Aji | 2024-08-22 | 3 | -9/+58 | |
| |\ \ | ||||||
| | * | | wip: fact_purchase_order schema | lian-manonog | 2024-08-21 | 3 | -3/+37 | |
| | | | | ||||||
| | * | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-21 | 1 | -18/+68 | |
| | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 20a3bd8 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/85 | |||||
| * | | | began dim_date df | T-Aji | 2024-08-21 | 1 | -18/+17 | |
| |/ / | ||||||
| * | | 5 dim tables created | T-Aji | 2024-08-21 | 1 | -15/+33 | |
| | | | ||||||
| * | | add dim tables: design, staff, currency, location (wip) | Ellie | 2024-08-21 | 1 | -0/+54 | |
| |/ | ||||||
| * | Merge branch 'development' into feature/transform_lambda | bulve-ad | 2024-08-21 | 4 | -96/+86 | |
| |\ | ||||||
| | * | chore: tidy-up repository & remove unused files | Alex Schofield | 2024-08-20 | 1 | -49/+0 | |
| | | | ||||||
| | * | feat: add more logging for debugging | Alex Schofield | 2024-08-20 | 1 | -2/+5 | |
| | | | ||||||
| | * | refactor: change rows output to debug logger output | Alex Schofield | 2024-08-20 | 1 | -2/+1 | |
| | | | ||||||
| | * | feat: add additional logging and exclude unnecessary table | Alex Schofield | 2024-08-20 | 1 | -5/+7 | |
| | | | ||||||
| | * | feat: add advanced logging | Alex Schofield | 2024-08-20 | 1 | -5/+11 | |
| | | | ||||||
| | * | feat: conditional logic for if bucket is empty | Alex Schofield | 2024-08-20 | 1 | -9/+23 | |
| | | | ||||||
| | * | feat: revert s3 streaming to previous implementation for uploading | Alex Schofield | 2024-08-20 | 1 | -31/+25 | |
| | | | ||||||
| | * | fix(extract_lambda): fix UnboundLocalError when db is called before it is ↵ | Alex Schofield | 2024-08-20 | 1 | -0/+1 | |
| | | | | | | | | | assigned a value | |||||
| | * | refactor: optimise s3 streaming & file naming | Alex Schofield | 2024-08-19 | 1 | -11/+6 | |
| | | | ||||||
| | * | feat: use buffers for s3 upload instead of csv files | Alex Schofield | 2024-08-19 | 1 | -17/+33 | |
| | | | ||||||
| | * | refactor: remove print statements in process_and_upload_tables | Alex Schofield | 2024-08-19 | 1 | -2/+0 | |
| | | | ||||||
| | * | refactor: add timestamp function in extract_lambda.py | Alex Schofield | 2024-08-19 | 1 | -12/+17 | |
| | | | ||||||
| | * | chore: reorganise imports in extract_lambda | Alex Schofield | 2024-08-19 | 1 | -5/+5 | |
| | | | ||||||
| | * | refactor(retrieve_secrets): add logging for ClientError | Alex Schofield | 2024-08-19 | 1 | -0/+1 | |
| | | | ||||||
 |
index : de-project-bentley | |
| Northcoders project - ETL pipeline with AWS | Alex Schofield |
| aboutsummaryrefslogtreecommitdiffstats |