Upload Module¶

Upload files to S3 bucket.

s3_tools.upload.upload_file_to_key(bucket: str, key: str, local_filename: str) → str[source]¶

Upload one file from local disk and store into AWS S3 bucket.

Parameters

bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
local_filename (str) – Local file from where the data will be uploaded.

Returns

The S3 full URL to the file.

Return type

str

Examples

>>> write_object_from_file(
...     bucket="myBucket",
...     key="myFiles/music.mp3",
...     local_filename="files/music.mp3"
... )
http://s3.amazonaws.com/myBucket/myFiles/music.mp3

s3_tools.upload.upload_files_to_keys(bucket: str, paths_keys: List[Tuple[str, str]], threads: int = 5) → List[Tuple[str, str, Any]][source]¶

Upload list of files to specific objects.

Parameters

bucket (str) – AWS S3 bucket where the objects will be stored.
paths_keys (List[Tuple[str, str]]) – List with a tuple of local path to be uploaded and S3 key destination. e.g. [(“Local_Path”, “S3_Key”), (“Local_Path”, “S3_Key”)]
threads (int, optional) – Number of parallel uploads, by default 5.

Returns

A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message. Attention, the output list may not follow the same input order.

Return type

List[Tuple[str, str, Any]]

Examples

>>> upload_files_to_keys(
...     bucket="myBucket",
...     paths_keys=[
...         ("MyFiles/myFile.data", "myData/myFile.data"),
...         ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3"),
...         ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc")
...     ]
... )
[
    ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3", True),
    ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc", True),
    ("MyFiles/myFile.data", "myData/myFile.data", True)
]

s3_tools.upload.upload_folder_to_prefix(bucket: str, prefix: str, folder: str, search_str: str = '*', threads: int = 5) → List[Tuple[str, str, Any]][source]¶

Upload local folder to a S3 prefix.

Function to upload all files for a given folder (recursive) and store them into a S3 bucket under a prefix. The local folder structure will be replicated into S3.

Parameters

bucket (str) – AWS S3 bucket where the object will be stored.
prefix (str) – Prefix where the objects will be under.
folder (str) – Local folder path where files are stored. Prefer to use the full path for the folder.
search_str (str.) – A match string to select all the files to upload, by default “*”. The string follows the rglob function pattern from the pathlib package.
threads (int, optional) – Number of parallel uploads, by default 5

Returns

A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message.

Return type

List[Tuple[str, str, Any]]

Examples

>>> upload_folder_to_prefix(
...     bucket="myBucket",
...     prefix="myFiles",
...     folder="/usr/files",
... )
[
    ("/usr/files/music.mp3", "myFiles/music.mp3", True),
    ("/usr/files/awesome.wav", "myFiles/awesome.wav", True),
    ("/usr/files/data/metadata.json", "myFiles/data/metadata.json", True)
]