| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | wip: fact_purchase_order schema | lian-manonog | 2024-08-21 | 3 | -3/+37 |
| | | |||||
| * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-21 | 1 | -18/+68 |
| | | | | | | | This commit fixes the style issues introduced in 20a3bd8 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/85 | ||||
| * | 5 dim tables created | T-Aji | 2024-08-21 | 1 | -15/+33 |
| | | |||||
| * | add dim tables: design, staff, currency, location (wip) | Ellie | 2024-08-21 | 1 | -0/+54 |
| | | |||||
| * | Merge branch 'development' into feature/transform_lambda | bulve-ad | 2024-08-21 | 4 | -96/+86 |
| |\ | |||||
| | * | chore: tidy-up repository & remove unused files | Alex Schofield | 2024-08-20 | 1 | -49/+0 |
| | | | |||||
| | * | feat: add more logging for debugging | Alex Schofield | 2024-08-20 | 1 | -2/+5 |
| | | | |||||
| | * | refactor: change rows output to debug logger output | Alex Schofield | 2024-08-20 | 1 | -2/+1 |
| | | | |||||
| | * | feat: add additional logging and exclude unnecessary table | Alex Schofield | 2024-08-20 | 1 | -5/+7 |
| | | | |||||
| | * | feat: add advanced logging | Alex Schofield | 2024-08-20 | 1 | -5/+11 |
| | | | |||||
| | * | feat: conditional logic for if bucket is empty | Alex Schofield | 2024-08-20 | 1 | -9/+23 |
| | | | |||||
| | * | feat: revert s3 streaming to previous implementation for uploading | Alex Schofield | 2024-08-20 | 1 | -31/+25 |
| | | | |||||
| | * | fix(extract_lambda): fix UnboundLocalError when db is called before it is ↵ | Alex Schofield | 2024-08-20 | 1 | -0/+1 |
| | | | | | | | | | assigned a value | ||||
| | * | refactor: optimise s3 streaming & file naming | Alex Schofield | 2024-08-19 | 1 | -11/+6 |
| | | | |||||
| | * | feat: use buffers for s3 upload instead of csv files | Alex Schofield | 2024-08-19 | 1 | -17/+33 |
| | | | |||||
| | * | refactor: remove print statements in process_and_upload_tables | Alex Schofield | 2024-08-19 | 1 | -2/+0 |
| | | | |||||
| | * | refactor: add timestamp function in extract_lambda.py | Alex Schofield | 2024-08-19 | 1 | -12/+17 |
| | | | |||||
| | * | chore: reorganise imports in extract_lambda | Alex Schofield | 2024-08-19 | 1 | -5/+5 |
| | | | |||||
| | * | refactor(retrieve_secrets): add logging for ClientError | Alex Schofield | 2024-08-19 | 1 | -0/+1 |
| | | | |||||
| | * | refactor(retrieve_secrets): improve error handling when retrieving secrets | Alex Schofield | 2024-08-19 | 1 | -0/+3 |
| | | | |||||
| | * | refactor(lambda_handler): remove unnecessary else statement | Alex Schofield | 2024-08-19 | 1 | -8/+7 |
| | | | |||||
| | * | refactor(retrieve_secrets): use aws recommended method for retrieving secrets | Alex Schofield | 2024-08-19 | 1 | -8/+11 |
| | | | |||||
| | * | refactor(log): implement logging ancestry - avoid using root logger | Alex Schofield | 2024-08-19 | 1 | -1/+1 |
| | | | |||||
| | * | Merge branch 'development' into feature/test-process-upload-tables-v2 | Alex | 2024-08-19 | 1 | -0/+1 |
| | |\ | |||||
| | | * | Merge branch 'development' into extract-lambda-code-update | Alex | 2024-08-19 | 3 | -17/+18 |
| | | |\ | |||||
| | | * | | wip: fixed broken tests; hashed out test_error_retrieving_object | lian-manonog | 2024-08-19 | 1 | -0/+1 |
| | | | | | |||||
| | * | | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-19 | 1 | -5/+3 |
| | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 4f629e5 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/65 | ||||
| | * | | | add working process and upload tables test | Ellie | 2024-08-19 | 1 | -2/+2 |
| | | | | | |||||
| | * | | | wip: fixing last test | Ellie | 2024-08-19 | 1 | -2/+3 |
| | | |/ | |/| | |||||
| | * | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-19 | 3 | -17/+18 |
| | |/ | | | | | | | | | | | This commit fixes the style issues introduced in e27c6b4 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/55 | ||||
| * | | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-21 | 1 | -16/+20 |
| | | | | | | | | | | | | | This commit fixes the style issues introduced in b882bb0 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/84 | ||||
| * | | complete version of read from s3 subfolder | Ang Bel | 2024-08-21 | 1 | -32/+1 |
| | | | |||||
| * | | (tests) Read from s3 to df passes | Ang Bel | 2024-08-21 | 1 | -22/+4 |
| | | | |||||
| * | | function to write files from s3 into a list of dataframes. Current test is ↵ | Ang Bel | 2024-08-21 | 1 | -4/+30 |
| | | | | | | | | | failing due to AioClientCreator object has no attribute "_inject_s3_input_parameters" | ||||
| * | | wip: just more pseudocode | lian-manonog | 2024-08-20 | 1 | -12/+26 |
| | | | |||||
| * | | wip: writing pseudocode logic for the lambda_handler | lian-manonog | 2024-08-20 | 1 | -14/+22 |
| | | | |||||
| * | | wip: added read_csb functionailty to lambda_handler | lian-manonog | 2024-08-19 | 1 | -8/+32 |
| | | | |||||
| * | | wip: setting up test files for transform_lambda | lian-manonog | 2024-08-19 | 1 | -0/+9 |
| |/ | |||||
| * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-16 | 1 | -1/+3 |
| | | | | | | | This commit fixes the style issues introduced in 890ca04 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47 | ||||
| * | dummy comment to test checks | Ang Bel | 2024-08-16 | 1 | -1/+1 |
| | | |||||
| * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-16 | 1 | -26/+40 |
| | | | | | | | This commit fixes the style issues introduced in 4428b8d according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47 | ||||
| * | refactoring to be more in line with pythonic code practices and prevent sql ↵ | Ang Bel | 2024-08-16 | 1 | -6/+11 |
| | | | | | injection | ||||
| * | refactor following github actions major risk message | Ang Bel | 2024-08-16 | 1 | -34/+31 |
| | | |||||
| * | style: format code with Autopep8, Black and Ruff Formatter | deepsource-autofix[bot] | 2024-08-16 | 1 | -59/+71 |
| | | | | | | | This commit fixes the style issues introduced in dd68d94 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47 | ||||
| * | refactoring for extract lambda to filter by last updated and if not empty ↵ | Ang Bel | 2024-08-16 | 1 | -15/+11 |
| | | | | | write it s3 | ||||
| * | extract bucket name retrieval helper function and replace the bucket name ↵ | Ang Bel | 2024-08-16 | 1 | -4/+7 |
| | | | | | placeholders | ||||
| * | Secrets manager integration into the extract lambda reviewed. | Ang Bel | 2024-08-16 | 1 | -13/+16 |
| | | |||||
| * | procefss_and_upload_tables test in progress | Ang Bel | 2024-08-15 | 1 | -9/+21 |
| | | |||||
| * | Merge branch 'feature-extract-lambda-data-extraction' into tf-secrets-manager | lian-manonog | 2024-08-15 | 1 | -21/+132 |
| |\ | |||||
| | * | test: mid-through test for process_and_upload_tables | T-Aji | 2024-08-14 | 1 | -9/+7 |
| | | | |||||
 |
index : de-project-bentley | |
| Northcoders project - ETL pipeline with AWS | Alex Schofield |
| aboutsummaryrefslogtreecommitdiffstats |