aboutsummaryrefslogtreecommitdiffstats
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* fix: handle no buckets properlyAlex Schofield2024-08-221-0/+3
|
* fix: change default parameters for bucket_name and clientAlex Schofield2024-08-221-2/+9
|
* Merge branch 'development' into feature/transform_lambdabulve-ad2024-08-214-96/+86
|\
| * chore: tidy-up repository & remove unused filesAlex Schofield2024-08-201-49/+0
| |
| * feat: add more logging for debuggingAlex Schofield2024-08-201-2/+5
| |
| * refactor: change rows output to debug logger outputAlex Schofield2024-08-201-2/+1
| |
| * feat: add additional logging and exclude unnecessary tableAlex Schofield2024-08-201-5/+7
| |
| * feat: add advanced loggingAlex Schofield2024-08-201-5/+11
| |
| * feat: conditional logic for if bucket is emptyAlex Schofield2024-08-201-9/+23
| |
| * feat: revert s3 streaming to previous implementation for uploadingAlex Schofield2024-08-201-31/+25
| |
| * fix(extract_lambda): fix UnboundLocalError when db is called before it is ↵Alex Schofield2024-08-201-0/+1
| | | | | | | | assigned a value
| * refactor: optimise s3 streaming & file namingAlex Schofield2024-08-191-11/+6
| |
| * feat: use buffers for s3 upload instead of csv filesAlex Schofield2024-08-191-17/+33
| |
| * refactor: remove print statements in process_and_upload_tablesAlex Schofield2024-08-191-2/+0
| |
| * refactor: add timestamp function in extract_lambda.pyAlex Schofield2024-08-191-12/+17
| |
| * chore: reorganise imports in extract_lambdaAlex Schofield2024-08-191-5/+5
| |
| * refactor(retrieve_secrets): add logging for ClientErrorAlex Schofield2024-08-191-0/+1
| |
| * refactor(retrieve_secrets): improve error handling when retrieving secretsAlex Schofield2024-08-191-0/+3
| |
| * refactor(lambda_handler): remove unnecessary else statementAlex Schofield2024-08-191-8/+7
| |
| * refactor(retrieve_secrets): use aws recommended method for retrieving secretsAlex Schofield2024-08-191-8/+11
| |
| * refactor(log): implement logging ancestry - avoid using root loggerAlex Schofield2024-08-191-1/+1
| |
| * Merge branch 'development' into feature/test-process-upload-tables-v2Alex2024-08-191-0/+1
| |\
| | * Merge branch 'development' into extract-lambda-code-updateAlex2024-08-193-17/+18
| | |\
| | * | wip: fixed broken tests; hashed out test_error_retrieving_objectlian-manonog2024-08-191-0/+1
| | | |
| * | | style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-191-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | This commit fixes the style issues introduced in 4f629e5 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/65
| * | | add working process and upload tables testEllie2024-08-191-2/+2
| | | |
| * | | wip: fixing last testEllie2024-08-191-2/+3
| | |/ | |/|
| * | style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-193-17/+18
| |/ | | | | | | | | | | This commit fixes the style issues introduced in e27c6b4 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/55
* | style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-211-16/+20
| | | | | | | | | | | | This commit fixes the style issues introduced in b882bb0 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/84
* | complete version of read from s3 subfolderAng Bel2024-08-211-32/+1
| |
* | (tests) Read from s3 to df passesAng Bel2024-08-211-22/+4
| |
* | function to write files from s3 into a list of dataframes. Current test is ↵Ang Bel2024-08-211-4/+30
| | | | | | | | failing due to AioClientCreator object has no attribute "_inject_s3_input_parameters"
* | wip: just more pseudocodelian-manonog2024-08-201-12/+26
| |
* | wip: writing pseudocode logic for the lambda_handlerlian-manonog2024-08-201-14/+22
| |
* | wip: added read_csb functionailty to lambda_handlerlian-manonog2024-08-191-8/+32
| |
* | wip: setting up test files for transform_lambdalian-manonog2024-08-191-0/+9
|/
* style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-161-1/+3
| | | | | | This commit fixes the style issues introduced in 890ca04 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47
* dummy comment to test checksAng Bel2024-08-161-1/+1
|
* style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-161-26/+40
| | | | | | This commit fixes the style issues introduced in 4428b8d according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47
* refactoring to be more in line with pythonic code practices and prevent sql ↵Ang Bel2024-08-161-6/+11
| | | | injection
* refactor following github actions major risk messageAng Bel2024-08-161-34/+31
|
* style: format code with Autopep8, Black and Ruff Formatterdeepsource-autofix[bot]2024-08-161-59/+71
| | | | | | This commit fixes the style issues introduced in dd68d94 according to the output from Autopep8, Black and Ruff Formatter. Details: https://github.com/ajschofield/de-project-bentley/pull/47
* refactoring for extract lambda to filter by last updated and if not empty ↵Ang Bel2024-08-161-15/+11
| | | | write it s3
* extract bucket name retrieval helper function and replace the bucket name ↵Ang Bel2024-08-161-4/+7
| | | | placeholders
* Secrets manager integration into the extract lambda reviewed.Ang Bel2024-08-161-13/+16
|
* procefss_and_upload_tables test in progressAng Bel2024-08-151-9/+21
|
* Merge branch 'feature-extract-lambda-data-extraction' into tf-secrets-managerlian-manonog2024-08-151-21/+132
|\
| * test: mid-through test for process_and_upload_tablesT-Aji2024-08-141-9/+7
| |
| * test: passing test for function connect_to_databaseT-Aji2024-08-141-15/+25
| |
| * feat/passing tests to helper function list_existing_s3_filesT-Aji2024-08-141-5/+7
| |
git.ajschof.me — hosted by ajschofield — powered by cgit