S3 Objects

All functionalities to deal with AWS S3 Objects.

Check

Check objects on S3 bucket.

s3_tools.objects.check.object_exists(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → bool[source]

Check if an object exists for a given bucket and key.

Parameters:

bucket (str) – Bucket name where the object is stored.
key (Union[str, Path]) – Full key for the object.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

True if the object exists, otherwise False.

Return type:

bool

Raises:

Exception – Any problem with the request is raised.

Example

>>> object_exists("myBucket", "myFiles/music.mp3")
True

s3_tools.objects.check.object_metadata(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → Dict[str, Any][source]

Get metadata from an S3 object.

Parameters:

bucket (str) – Bucket name where the object is stored.
key (Union[str, Path]) – Full key for the object.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Metadata from the object.

Return type:

Dict[str, Any]

Raises:

Exception – Any problem with the request is raised.

Example

>>> object_metadata("myBucket", "myFiles/music.mp3")
{
    'ResponseMetadata': {},
    'AcceptRanges': 'bytes',
    'LastModified': datetime.datetime(2020, 10, 31, 20, 46, 13, tzinfo=tzutc()),
    'ContentLength': 123456,
    'ETag': '"1234567890abcdef1234567890abcdef"',
    'ContentType': 'audio/mpeg',
    'Metadata': {}
}

Copy

Copy S3 objects.

s3_tools.objects.copy.copy_keys(source_bucket: str, source_keys: List[Union[str, Path]], destination_bucket: str, destination_keys: List[Union[str, Path]], threads: int = 5, aws_auth: Dict[str, str] = {}) → None[source]

Copy a list of S3 objects from source bucket to destination.

Parameters:

source_bucket (str) – S3 bucket where the objects are stored.
source_keys (List[Union[str, Path]]) – S3 keys where the objects are referenced.
destination_bucket (str) – S3 destination bucket.
destination_keys (List[Union[str, Path]]) – S3 destination keys.
threads (int, optional) – Number of parallel uploads, by default 5.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Raises:

IndexError – When the source_keys and destination_keys have different length.
ValueError – When the keys list is empty.

Examples

>>> copy_keys(
...     source_bucket='bucket',
...     source_keys=[
...         'myFiles/song.mp3',
...         Path('myFiles/photo.jpg'),
...     ],
...     destination_bucket='bucket',
...     destination_keys=[
...         Path('myMusic/song.mp3'),
...         'myPhotos/photo.jpg',
...     ]
... )

s3_tools.objects.copy.copy_object(source_bucket: str, source_key: Union[str, Path], destination_bucket: str, destination_key: Union[str, Path], aws_auth: Dict[str, str] = {}) → None[source]

Copy S3 object from source bucket and key to destination.

Parameters:

source_bucket (str) – S3 bucket where the object is stored.
source_key (Union[str, Path]) – S3 key where the object is referenced.
destination_bucket (str) – S3 destination bucket.
destination_key (Union[str, Path]) – S3 destination key.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Examples

>>> copy_object(
...    source_bucket='bucket',
...    source_key='myFiles/song.mp3',
...    destination_bucket='bucket',
...    destination_key='myMusic/song.mp3',
... )

s3_tools.objects.copy.copy_prefix(source_bucket: str, source_prefix: Union[str, Path], destination_bucket: str, change_prefix: Optional[Tuple[Union[str, Path], Union[str, Path]]] = None, filter_keys: Optional[str] = None, threads: int = 5, aws_auth: Dict[str, str] = {}) → None[source]

Copy S3 objects from source bucket to destination based on prefix filter.

Parameters:

source_bucket (str) – S3 bucket where the objects are stored.
source_prefix (Union[str, Path]) – S3 prefix where the objects are referenced.
destination_bucket (str) – S3 destination bucket.
change_prefix (Tuple[Union[str, Path], Union[str, Path]], optional) – Text to be replaced in keys prefixes, by default is None. The first element is the text to be replaced, the second is the replacement text.
filter_keys (str, optional) – Basic search string to filter out keys on result (uses Unix shell-style wildcards), by default is None. For more about the search check “fnmatch” package.
threads (int, optional) – Number of parallel uploads, by default 5.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Examples

>>> copy_prefix(
...     source_bucket='MyBucket',
...     source_prefix='myFiles',
...     destination_bucket='OtherBucket',
...     filter_keys='*images*',
...     change_prefix=('myFiles', 'backup')
... )

Delete

Delete objects from S3 bucket.

s3_tools.objects.delete.delete_keys(bucket: str, keys: List[Union[str, Path]], dry_run: bool = True, aws_auth: Dict[str, str] = {}) → None[source]

Delete all objects in the keys list from S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the objects are stored.
keys (List[Union[str, Path]]) – List of object keys.
dry_run (bool) – If True will not delete the objects.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Examples

>>> delete_keys(
...     bucket="myBucket",
...     keys=[
...         "myData/myMusic/awesome.mp3",
...         Path("myData/myDocs/paper.doc")
...     ],
...     dry_run=False
... )

s3_tools.objects.delete.delete_object(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → None[source]

Delete a given object from S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key for the object that will be deleted.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Examples

>>> delete_object(bucket="myBucket", key="myData/myFile.data")

s3_tools.objects.delete.delete_prefix(bucket: str, prefix: Union[str, Path], dry_run: bool = True, aws_auth: Dict[str, str] = {}) → Optional[List[Union[str, Path]]][source]

Delete all objects under the given prefix from S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the objects are stored.
prefix (Union[str, Path]) – Prefix where the objects are under.
dry_run (bool) – If True will not delete the objects.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

List of S3 keys to be deleted if dry_run True, else None.

Return type:

List[Union[str, Path]]

Examples

>>> delete_prefix(bucket="myBucket", prefix="myData")
[
    "myData/myMusic/awesome.mp3",
    "myData/myDocs/paper.doc"
]

>>> delete_prefix(bucket="myBucket", prefix=Path("myData"), dry_run=False)

Download

Download S3 objects to files.

s3_tools.objects.download.download_key_to_file(bucket: str, key: Union[str, Path], local_filename: Union[str, Path], progress=None, task_id: int = -1, aws_auth: Dict[str, str] = {}, extra_args: Dict[str, str] = {}) → bool[source]

Retrieve one object from AWS S3 bucket and store into local disk.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key where the object is stored.
local_filename (Union[str, Path]) – Local file where the data will be downloaded to.
progress (rich.Progress) – Instance of a rich Progress bar, by default None.
task_id (int) – Task ID on the progress bar to be updated, by default -1.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 download_file method, by default is empty. Allowed download arguments: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_DOWNLOAD_ARGS

Returns:

True if the local file exists.

Return type:

bool

Examples

>>> download_key_to_file(
...     bucket="myBucket",
...     key="myData/myFile.data",
...     local_filename="theFile.data",
... )
True

s3_tools.objects.download.download_keys_to_files(bucket: str, keys_paths: List[Tuple[Union[str, Path], Union[str, Path]]], threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}, as_paths: bool = False, default_extra_args: Dict[str, str] = {}, extra_args_per_key: List[Dict[str, str]] = []) → List[Tuple[Union[str, Path], Union[str, Path], Any]][source]

Download list of objects to specific paths.

Parameters:

bucket (str) – AWS S3 bucket where the objects are stored.
keys_paths (List[Tuple[Union[str, Path], Union[str, Path]]]) –
List with a tuple of S3 key to be downloaded and local path to be stored. e.g. [

(“S3_Key”, “Local_Path”), (Path(“S3_Key”), “Local_Path”), (“S3_Key”, Path(“Local_Path”)), (Path(“S3_Key”), Path(“Local_Path”)),

]
threads (int) – Number of parallel downloads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
as_paths (bool) – If True, the keys are returned as Path objects, otherwise as strings, by default is False.
default_extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 download_file method, by default is empty. The extra arguments will be applied to all S3 keys.
extra_args_per_key (List[Dict[str, str]]) – Extra arguments to be passed for each S3 key to the boto3 download_file method, by default is empty. The default extra arguments will be merged with the extra arguments passed for each key.

Returns:

A list with tuples formed by the “S3_Key”, “Local_Path”, and the result of the download. If successful will have True, if not will contain the error message. Attention, the output list may not follow the same input order.

Return type:

List[Tuple]

Examples

>>> download_keys_to_files(
...     bucket="myBucket",
...     keys_paths=[
...         ("myData/myFile.data", "MyFiles/myFile.data"),
...         ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3"),
...         ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc"),
...     ]
... )
[
    ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3", True),
    ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc", True),
    ("myData/myFile.data", "MyFiles/myFile.data", True),
]

s3_tools.objects.download.download_prefix_to_folder(bucket: str, prefix: Union[str, Path], folder: Union[str, Path], search_str: Optional[str] = None, remove_prefix: bool = True, threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}, as_paths: bool = False, default_extra_args: Dict[str, str] = {}) → List[Tuple[Union[str, Path], Union[str, Path], Any]][source]

Download objects to local folder.

Function to retrieve all files under a prefix on S3 and store them into local folder.

Parameters:

bucket (str) – AWS S3 bucket where the objects are stored.
prefix (Union[str, Path]) – Prefix where the objects are under.
folder (Union[str, Path]) – Local folder path where files will be stored.
search_str (str) – Basic search string to filter out keys on result (uses Unix shell-style wildcards), by default is None. For more about the search check “fnmatch” package.
remove_prefix (bool) – If True will remove the the prefix when writing to local folder. The remaining “folders” on the key will be created on the local folder.
threads (int) – Number of parallel downloads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
as_paths (bool) – If True, the keys are returned as Path objects, otherwise as strings, by default is False.
default_extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 download_file method, by default is empty. The extra arguments will be applied to all S3 keys.

Returns:

A list with tuples formed by the “S3_Key”, “Local_Path”, and the result of the download. If successful will have True, if not will contain the error message.

Return type:

List[Tuple]

Examples

>>> download_prefix_to_folder(
...     bucket="myBucket",
...     prefix="myData",
...     folder="myFiles",
... )
[
    ("myData/myFile.data", "MyFiles/myFile.data", True),
    ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3", True),
    ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc", True),
]

List

List S3 bucket objects.

s3_tools.objects.list.list_objects(bucket: str, prefix: Union[str, Path] = '', search_str: Optional[str] = None, max_keys: int = 1000, aws_auth: Dict[str, str] = {}, as_paths: bool = False) → List[Union[str, Path]][source]

Retrieve the list of objects from AWS S3 bucket under a given prefix and search string.

Parameters:

bucket (str) – AWS S3 bucket where the objects are stored.
prefix (Union[str, Path]) – Prefix where the objects are under.
search_str (str) – Basic search string to filter out keys on result (uses Unix shell-style wildcards), by default is None. For more about the search check “fnmatch” package.
max_keys (int) – Max number of keys to have pagination.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
as_paths (bool) – If True, the keys are returned as Path objects, otherwise as strings, by default is False.

Returns:

List of keys inside the bucket, under the path, and filtered.

Return type:

List[Union[str, Path]]

Examples

>>> list_objects(bucket="myBucket", prefix="myData")
[
    "myData/myFile.data",
    "myData/myMusic/awesome.mp3",
    "myData/myDocs/paper.doc"
]

>>> list_objects(bucket="myBucket", prefix="myData", search_str="*paper*", as_paths=True)
[
    Path("myData/myDocs/paper.doc")
]

Move

Move S3 objects.

s3_tools.objects.move.move_keys(source_bucket: str, source_keys: List[Union[str, Path]], destination_bucket: str, destination_keys: List[Union[str, Path]], threads: int = 5, aws_auth: Dict[str, str] = {}) → None[source]

Move a list of S3 objects from source bucket to destination.

Parameters:

source_bucket (str) – S3 bucket where the objects are stored.
source_keys (List[Union[str, Path]]) – S3 keys where the objects are referenced.
destination_bucket (str) – S3 destination bucket.
destination_keys (List[Union[str, Path]]) – S3 destination keys.
threads (int, optional) – Number of parallel uploads, by default 5.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Raises:

IndexError – When the source_keys and destination_keys have different length.
ValueError – When the keys list is empty.

Examples

>>> move_keys(
...     source_bucket='bucket',
...     source_keys=[
...         'myFiles/song.mp3',
...         'myFiles/photo.jpg',
...     ],
...     destination_bucket='bucket',
...     destination_keys=[
...         'myMusic/song.mp3',
...         'myPhotos/photo.jpg',
...     ],
... )

s3_tools.objects.move.move_object(source_bucket: str, source_key: Union[str, Path], destination_bucket: str, destination_key: Union[str, Path], aws_auth: Dict[str, str] = {}) → None[source]

Move S3 object from source bucket and key to destination.

Parameters:

source_bucket (str) – S3 bucket where the object is stored.
source_key (Union[str, Path]) – S3 key where the object is referenced.
destination_bucket (str) – S3 destination bucket.
destination_key (Union[str, Path]) – S3 destination key.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Examples

>>> move_object(
...    source_bucket='bucket',
...    source_key='myFiles/song.mp3',
...    destination_bucket='bucket',
...    destination_key='myMusic/song.mp3',
... )

Presigned URL

Create presigned URL for S3 bucket objects.

s3_tools.objects.presigned_url.get_presigned_download_url(bucket: str, key: Union[str, Path], expiration: int = 300, aws_auth: Dict[str, str] = {}) → str[source]

Generate a presigned URL to download an S3 object.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key for the object that will be downloaded.
expiration (int) – Time in seconds for the presigned URL to remain valid, default 5 minutes.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Presigned URL.

Return type:

str

Raises:

Exception – Any problem with the request is raised.

Examples

>>> import requests     # To install: pip install requests
>>> url = get_presigned_download_url(
...    bucket='myBucket',
...    key='myData/myFile.data',
... )
>>> response = requests.get(url)

s3_tools.objects.presigned_url.get_presigned_upload_url(bucket: str, key: Union[str, Path], fields: Optional[dict] = None, conditions: Optional[list] = None, expiration: int = 300, aws_auth: Dict[str, str] = {}) → dict[source]

Generate a presigned URL S3 POST request to upload a file.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
key (Union[str, Path]) – Key for the object that will will be stored.
fields (Optional[dict]) – Dictionary of prefilled form fields.
conditions (Optional[list]) – List of conditions to include in the policy.
expiration (int) – Time in seconds for the presigned URL to remain valid, default 5 minutes.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

A dictionary with two elements: url and fields. Url is the url to post to. Fields is a dictionary filled with the form fields and respective values to use when submitting the post.

Return type:

dict

Raises:

Exception – Any problem with the request is raised.

Examples

>>> import requests     # To install: pip install requests
>>> response = get_presigned_upload_url(
...    bucket='myBucket',
...    key='myData/myFile.data',
... )
>>> with open('myFile.data', 'rb') as f:
...    files = {'file': ('myFile.data', f)}
...    http_response = requests.post(response['url'], data=response['fields'], files=files)

s3_tools.objects.presigned_url.get_presigned_url(client_method: str, method_parameters: Optional[dict] = None, http_method: Optional[str] = None, expiration: int = 300, aws_auth: Dict[str, str] = {}) → str[source]

Generate a presigned URL to invoke an S3.Client method.

Parameters:

client_method (str) – Name of the S3.Client method, e.g., ‘list_buckets’.
method_parameters (Optional[dict]) – Dictionary of parameters to send to the method.
expiration (int) – Time in seconds for the presigned URL to remain valid, default 5 minutes.
http_method (Optional[str]) – HTTP method to use, e.g., GET, POST. If not specified, will automatically be select the appropriate method.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Presigned URL.

Return type:

str

Raises:

Exception – Any problem with the request is raised.

Examples

>>> get_presigned_url(
...    client_method='list_objects',
...    method_parameters={'Bucket': 'myBucket'},
... )
https://myBucket.s3.amazonaws.com/?encoding-type=url&AWSAccessKeyId=ASI&Signature=5JLAcSKQ%3D&x-amz-security-token=FwoGZXIvY%&Expires=1646759818

Read

Read S3 objects into variables.

s3_tools.objects.read.read_object_to_bytes(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → bytes[source]

Retrieve one object from AWS S3 bucket as a byte array.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Object content as bytes.

Return type:

bytes

Examples

>>> read_object_to_bytes(
...     bucket="myBucket",
...     key="myData/myFile.data",
... )
b"The file content"

s3_tools.objects.read.read_object_to_dict(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → Dict[Any, Any][source]

Retrieve one object from AWS S3 bucket as a dictionary.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Object content as dictionary.

Return type:

Dict[Any, Any]

Examples

>>> read_object_to_dict(
...     bucket="myBucket",
...     key="myData/myFile.json",
... )
{"key": "value", "1": "text"}

s3_tools.objects.read.read_object_to_text(bucket: str, key: Union[str, Path], aws_auth: Dict[str, str] = {}) → str[source]

Retrieve one object from AWS S3 bucket as a string.

Parameters:

bucket (str) – AWS S3 bucket where the object is stored.
key (Union[str, Path]) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

Object content as string.

Return type:

str

Examples

>>> read_object_to_text(
...     bucket="myBucket",
...     key="myData/myFile.data"
... )
"The file content"

Upload

Upload files to S3 bucket.

s3_tools.objects.upload.upload_file_to_key(bucket: str, key: Union[str, Path], local_filename: Union[str, Path], progress=None, task_id: int = -1, aws_auth: Dict[str, str] = {}, extra_args: Dict[str, str] = {}) → str[source]

Upload one file from local disk and store into AWS S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
key (Union[str, Path]) – Key where the object will be stored.
local_filename (Union[str, Path]) – Local file from where the data will be uploaded.
progress (rich.Progress) – Instance of a rich Progress bar, by default None.
task_id (int) – Task ID on the progress bar to be updated, by default -1.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 upload_file method, by default is empty. Allowed upload arguments: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS

Returns:

The S3 full URL to the file.

Return type:

str

Examples

>>> upload_file_to_key(
...     bucket="myBucket",
...     key="myFiles/music.mp3",
...     local_filename="files/music.mp3",
... )
http://s3.amazonaws.com/myBucket/myFiles/music.mp3

s3_tools.objects.upload.upload_files_to_keys(bucket: str, paths_keys: List[Tuple[Union[str, Path], Union[str, Path]]], threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}, as_paths: bool = False, default_extra_args: Dict[str, str] = {}, extra_args_per_key: List[Dict[str, str]] = []) → List[Tuple[Union[str, Path], Union[str, Path], Any]][source]

Upload list of files to specific objects.

Parameters:

bucket (str) – AWS S3 bucket where the objects will be stored.
paths_keys (List[Tuple[Union[str, Path], Union[str, Path]]]) – List with a tuple of local path to be uploaded and S3 key destination. e.g. [(“Local_Path”, “S3_Key”), (“Local_Path”, “S3_Key”)]
threads (int, optional) – Number of parallel uploads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
as_paths (bool) – If True, the keys are returned as Path objects, otherwise as strings, by default is False.
default_extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 upload_file method, by default is empty. The extra arguments will be applied to all S3 keys.
extra_args_per_key (List[Dict[str, str]]) – Extra arguments to be passed for each S3 key to the boto3 upload_file method, by default is empty. The default extra arguments will be merged with the extra arguments passed for each key.

Returns:

A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message. Attention, the output list may not follow the same input order.

Return type:

List[Tuple[Union[str, Path], Union[str, Path], Any]]

Raises:

ValueError – extra_args_per_key when used must have the same length of paths_keys.

Examples

>>> upload_files_to_keys(
...     bucket="myBucket",
...     paths_keys=[
...         ("MyFiles/myFile.data", "myData/myFile.data"),
...         ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3"),
...         ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc"),
...     ],
... )
[
    ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3", True),
    ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc", True),
    ("MyFiles/myFile.data", "myData/myFile.data", True),
]

s3_tools.objects.upload.upload_folder_to_prefix(bucket: str, prefix: Union[str, Path], folder: Union[str, Path], search_str: str = '*', threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}, as_paths: bool = False, default_extra_args: Dict[str, str] = {}) → List[Tuple[Union[str, Path], Union[str, Path], Any]][source]

Upload local folder to a S3 prefix.

Function to upload all files for a given folder (recursive) and store them into a S3 bucket under a prefix. The local folder structure will be replicated into S3.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
prefix (Union[str, Path]) – Prefix where the objects will be under.
folder (Union[str, Path]) – Local folder path where files are stored. Prefer to use the full path for the folder.
search_str (str.) – A match string to select all the files to upload, by default “*”. The string follows the rglob function pattern from the pathlib package.
threads (int, optional) – Number of parallel uploads, by default 5
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
as_paths (bool) – If True, the keys are returned as Path objects, otherwise as strings, by default is False.
default_extra_args (Dict[str, str]) – Extra arguments to be passed to the boto3 upload_file method, by default is empty. The extra arguments will be applied to all S3 keys.

Returns:

A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message.

Return type:

List[Tuple[Union[str, Path], Union[str, Path], Any]]

Examples

>>> upload_folder_to_prefix(
...     bucket="myBucket",
...     prefix="myFiles",
...     folder="/usr/files",
... )
[
    ("/usr/files/music.mp3", "myFiles/music.mp3", True),
    ("/usr/files/awesome.wav", "myFiles/awesome.wav", True),
    ("/usr/files/data/metadata.json", "myFiles/data/metadata.json", True),
]

Write

Write variables into S3 objects.

s3_tools.objects.write.write_object_from_bytes(bucket: str, key: str, data: bytes, aws_auth: Dict[str, str] = {}) → str[source]

Upload a bytes object to an object into AWS S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (bytes) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

The S3 full URL to the file.

Return type:

str

Raises:

TypeError – If data is not a bytes type.

Examples

>>> data = bytes("String to bytes", "utf-8")
>>> write_object_from_bytes(
...     bucket="myBucket",
...     key="myFiles/file.data",
...     data=data
... )
http://s3.amazonaws.com/myBucket/myFiles/file.data

s3_tools.objects.write.write_object_from_dict(bucket: str, key: str, data: Dict, aws_auth: Dict[str, str] = {}) → str[source]

Upload a dictionary to an object into AWS S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (dict) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

The S3 full URL to the file.

Return type:

str

Raises:

TypeError – If data is not a dict type.

Examples

>>> data = {"key": "value", "1": "text"}
>>> write_object_from_dict(
...     bucket="myBucket",
...     key="myFiles/file.json",
...     data=data
... )
http://s3.amazonaws.com/myBucket/myFiles/file.json

s3_tools.objects.write.write_object_from_text(bucket: str, key: str, data: str, aws_auth: Dict[str, str] = {}) → str[source]

Upload a string to an object into AWS S3 bucket.

Parameters:

bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (str) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.

Returns:

The S3 full URL to the file.

Return type:

str

Raises:

TypeError – If data is not a str type.

Examples

>>> data = "A very very not so long text"
>>> write_object_from_text(
...     bucket="myBucket",
...     key="myFiles/file.txt",
...     data=data
... )
http://s3.amazonaws.com/myBucket/myFiles/file.txt