S3 Objects
All functionalities to deal with AWS S3 Objects.
Check
Check objects on S3 bucket.
- s3_tools.objects.check.object_exists(bucket: str, key: str, aws_auth: Dict[str, str] = {}) bool [source]
Check if an object exists for a given bucket and key.
- Parameters
bucket (str) – Bucket name where the object is stored.
key (str) – Full key for the object.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
True if the object exists, otherwise False.
- Return type
bool
- Raises
Exception – Any problem with the request is raised.
Example
>>> object_exists("myBucket", "myFiles/music.mp3") True
Delete
Delete objects from S3 bucket.
- s3_tools.objects.delete.delete_keys(bucket: str, keys: List[str], dry_run: bool = True, aws_auth: Dict[str, str] = {}) None [source]
Delete all objects in the keys list from S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the objects are stored.
keys (List[str]) – List of object keys.
dry_run (bool) – If True will not delete the objects.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
Examples
>>> delete_keys( ... bucket="myBucket", ... keys=[ ... "myData/myMusic/awesome.mp3", ... "myData/myDocs/paper.doc" ... ], ... dry_run=False ... )
- s3_tools.objects.delete.delete_object(bucket: str, key: str, aws_auth: Dict[str, str] = {}) None [source]
Delete a given object from S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the object is stored.
key (str) – Key for the object that will be deleted.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
Examples
>>> delete_object(bucket="myBucket", key="myData/myFile.data")
- s3_tools.objects.delete.delete_prefix(bucket: str, prefix: str, dry_run: bool = True, aws_auth: Dict[str, str] = {}) Optional[List[str]] [source]
Delete all objects under the given prefix from S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the objects are stored.
prefix (str) – Prefix where the objects are under.
dry_run (bool) – If True will not delete the objects.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
List of S3 keys to be deleted if dry_run True, else None.
- Return type
List[str]
Examples
>>> delete_prefix(bucket="myBucket", prefix="myData") [ "myData/myMusic/awesome.mp3", "myData/myDocs/paper.doc" ]
>>> delete_prefix(bucket="myBucket", prefix="myData", dry_run=False)
Download
Download S3 objects to files.
- s3_tools.objects.download.download_key_to_file(bucket: str, key: str, local_filename: str, progress=None, task_id: int = - 1, aws_auth: Dict[str, str] = {}) bool [source]
Retrieve one object from AWS S3 bucket and store into local disk.
- Parameters
bucket (str) – AWS S3 bucket where the object is stored.
key (str) – Key where the object is stored.
local_filename (str) – Local file where the data will be downloaded to.
progress (rich.Progress) – Instance of a rich Progress bar, by default None.
task_id (int) – Task ID on the progress bar to be updated, by default -1.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
True if the local file exists.
- Return type
bool
Examples
>>> read_object_to_file( ... bucket="myBucket", ... key="myData/myFile.data", ... local_filename="theFile.data" ... ) True
- s3_tools.objects.download.download_keys_to_files(bucket: str, keys_paths: List[Tuple[str, str]], threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}) List[Tuple[str, str, Any]] [source]
Download list of objects to specific paths.
- Parameters
bucket (str) – AWS S3 bucket where the objects are stored.
keys_paths (List[Tuple[str, str]]) – List with a tuple of S3 key to be downloaded and local path to be stored. e.g. [(“S3_Key”, “Local_Path”), (“S3_Key”, “Local_Path”)]
threads (int) – Number of parallel downloads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
A list with tuples formed by the “S3_Key”, “Local_Path”, and the result of the download. If successful will have True, if not will contain the error message. Attention, the output list may not follow the same input order.
- Return type
list of tuples
Examples
>>> download_keys_to_files( ... bucket="myBucket", ... keys_paths=[ ... ("myData/myFile.data", "MyFiles/myFile.data"), ... ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3"), ... ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc") ... ] ... ) [ ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3", True), ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc", True), ("myData/myFile.data", "MyFiles/myFile.data", True) ]
- s3_tools.objects.download.download_prefix_to_folder(bucket: str, prefix: str, folder: str, search_str: Optional[str] = None, remove_prefix: bool = True, threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}) List[Tuple[str, str, Any]] [source]
Download objects to local folder.
Function to retrieve all files under a prefix on S3 and store them into local folder.
- Parameters
bucket (str) – AWS S3 bucket where the objects are stored.
prefix (str) – Prefix where the objects are under.
folder (str) – Local folder path where files will be stored.
search_str (str) – Basic search string to filter out keys on result (uses Unix shell-style wildcards), by default is None. For more about the search check “fnmatch” package.
remove_prefix (bool) – If True will remove the the prefix when writing to local folder. The remaining “folders” on the key will be created on the local folder.
threads (int) – Number of parallel downloads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
A list with tuples formed by the “S3_Key”, “Local_Path”, and the result of the download. If successful will have True, if not will contain the error message.
- Return type
List[Tuples]
Examples
>>> download_prefix_to_folder( ... bucket="myBucket", ... prefix="myData", ... folder="myFiles" ... ) [ ("myData/myFile.data", "MyFiles/myFile.data", True), ("myData/myMusic/awesome.mp3", "MyFiles/myMusic/awesome.mp3", True), ("myData/myDocs/paper.doc", "MyFiles/myDocs/paper.doc", True) ]
List
List S3 bucket objects.
- s3_tools.objects.list.list_objects(bucket: str, prefix: str = '', search_str: Optional[str] = None, max_keys: int = 1000, aws_auth: Dict[str, str] = {}) List[str] [source]
Retrieve the list of objects from AWS S3 bucket under a given prefix and search string.
- Parameters
bucket (str) – AWS S3 bucket where the objects are stored.
prefix (str) – Prefix where the objects are under.
search_str (str) – Basic search string to filter out keys on result (uses Unix shell-style wildcards), by default is None. For more about the search check “fnmatch” package.
max_keys (int) – Max number of keys to have pagination.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
List of keys inside the bucket, under the path, and filtered.
- Return type
List[str]
Examples
>>> list_objects(bucket="myBucket", prefix="myData") [ "myData/myFile.data", "myData/myMusic/awesome.mp3", "myData/myDocs/paper.doc" ]
>>> list_objects(bucket="myBucket", prefix="myData", search_str="*paper*") [ "myData/myDocs/paper.doc" ]
Move
Move S3 objects.
- s3_tools.objects.move.move_keys(source_bucket: str, source_keys: List[str], destination_bucket: str, destination_keys: List[str], threads: int = 5, aws_auth: Dict[str, str] = {}) None [source]
Move a list of S3 objects from source bucket to destination.
- Parameters
source_bucket (str) – S3 bucket where the objects are stored.
source_keys (List[str]) – S3 keys where the objects are referenced.
destination_bucket (str) – S3 destination bucket.
destination_keys (List[str]) – S3 destination keys.
threads (int, optional) – Number of parallel uploads, by default 5.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Raises
IndexError – When the source_keys and destination_keys have different length.
ValueError – When the keys list is empty.
Examples
>>> move_keys( ... source_bucket='bucket', ... source_keys=[ ... 'myFiles/song.mp3', ... 'myFiles/photo.jpg' ... ], ... destination_bucket='bucket', ... destination_keys=[ ... 'myMusic/song.mp3', ... 'myPhotos/photo.jpg' ... ] ... )
- s3_tools.objects.move.move_object(source_bucket: str, source_key: str, destination_bucket: str, destination_key: str, aws_auth: Dict[str, str] = {}) None [source]
Move S3 object from source bucket and key to destination.
- Parameters
source_bucket (str) – S3 bucket where the object is stored.
source_key (str) – S3 key where the object is referenced.
destination_bucket (str) – S3 destination bucket.
destination_key (str) – S3 destination key.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
Examples
>>> move_object( ... source_bucket='bucket', ... source_key='myFiles/song.mp3', ... destination_bucket='bucket', ... destination_key='myMusic/song.mp3' ... )
Read
Read S3 objects into variables.
- s3_tools.objects.read.read_object_to_bytes(bucket: str, key: str, aws_auth: Dict[str, str] = {}) bytes [source]
Retrieve one object from AWS S3 bucket as a byte array.
- Parameters
bucket (str) – AWS S3 bucket where the object is stored.
key (str) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
Object content as bytes.
- Return type
bytes
Examples
>>> read_object_to_bytes( ... bucket="myBucket", ... key="myData/myFile.data" ... ) b"The file content"
- s3_tools.objects.read.read_object_to_dict(bucket: str, key: str, aws_auth: Dict[str, str] = {}) Dict[Any, Any] [source]
Retrieve one object from AWS S3 bucket as a dictionary.
- Parameters
bucket (str) – AWS S3 bucket where the object is stored.
key (str) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
Object content as dictionary.
- Return type
Dict[Any, Any]
Examples
>>> read_object_to_dict( ... bucket="myBucket", ... key="myData/myFile.json" ... ) {"key": "value", "1": "text"}
- s3_tools.objects.read.read_object_to_text(bucket: str, key: str, aws_auth: Dict[str, str] = {}) str [source]
Retrieve one object from AWS S3 bucket as a string.
- Parameters
bucket (str) – AWS S3 bucket where the object is stored.
key (str) – Key where the object is stored.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
Object content as string.
- Return type
str
Examples
>>> read_object_to_text( ... bucket="myBucket", ... key="myData/myFile.data" ... ) "The file content"
Upload
Upload files to S3 bucket.
- s3_tools.objects.upload.upload_file_to_key(bucket: str, key: str, local_filename: str, progress=None, task_id: int = - 1, aws_auth: Dict[str, str] = {}) str [source]
Upload one file from local disk and store into AWS S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
local_filename (str) – Local file from where the data will be uploaded.
progress (rich.Progress) – Instance of a rich Progress bar, by default None.
task_id (int) – Task ID on the progress bar to be updated, by default -1.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
The S3 full URL to the file.
- Return type
str
Examples
>>> write_object_from_file( ... bucket="myBucket", ... key="myFiles/music.mp3", ... local_filename="files/music.mp3" ... ) http://s3.amazonaws.com/myBucket/myFiles/music.mp3
- s3_tools.objects.upload.upload_files_to_keys(bucket: str, paths_keys: List[Tuple[str, str]], threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}) List[Tuple[str, str, Any]] [source]
Upload list of files to specific objects.
- Parameters
bucket (str) – AWS S3 bucket where the objects will be stored.
paths_keys (List[Tuple[str, str]]) – List with a tuple of local path to be uploaded and S3 key destination. e.g. [(“Local_Path”, “S3_Key”), (“Local_Path”, “S3_Key”)]
threads (int, optional) – Number of parallel uploads, by default 5.
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message. Attention, the output list may not follow the same input order.
- Return type
List[Tuple[str, str, Any]]
Examples
>>> upload_files_to_keys( ... bucket="myBucket", ... paths_keys=[ ... ("MyFiles/myFile.data", "myData/myFile.data"), ... ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3"), ... ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc") ... ] ... ) [ ("MyFiles/myMusic/awesome.mp3", "myData/myMusic/awesome.mp3", True), ("MyFiles/myDocs/paper.doc", "myData/myDocs/paper.doc", True), ("MyFiles/myFile.data", "myData/myFile.data", True) ]
- s3_tools.objects.upload.upload_folder_to_prefix(bucket: str, prefix: str, folder: str, search_str: str = '*', threads: int = 5, show_progress: bool = False, aws_auth: Dict[str, str] = {}) List[Tuple[str, str, Any]] [source]
Upload local folder to a S3 prefix.
Function to upload all files for a given folder (recursive) and store them into a S3 bucket under a prefix. The local folder structure will be replicated into S3.
- Parameters
bucket (str) – AWS S3 bucket where the object will be stored.
prefix (str) – Prefix where the objects will be under.
folder (str) – Local folder path where files are stored. Prefer to use the full path for the folder.
search_str (str.) – A match string to select all the files to upload, by default “*”. The string follows the rglob function pattern from the pathlib package.
threads (int, optional) – Number of parallel uploads, by default 5
show_progress (bool) – Show progress bar on console, by default False. (Need to install extra [progress] to be used)
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
A list with tuples formed by the “Local_Path”, “S3_Key”, and the result of the upload. If successful will have True, if not will contain the error message.
- Return type
List[Tuple[str, str, Any]]
Examples
>>> upload_folder_to_prefix( ... bucket="myBucket", ... prefix="myFiles", ... folder="/usr/files", ... ) [ ("/usr/files/music.mp3", "myFiles/music.mp3", True), ("/usr/files/awesome.wav", "myFiles/awesome.wav", True), ("/usr/files/data/metadata.json", "myFiles/data/metadata.json", True) ]
Write
Write variables into S3 objects.
- s3_tools.objects.write.write_object_from_bytes(bucket: str, key: str, data: bytes, aws_auth: Dict[str, str] = {}) str [source]
Upload a bytes object to an object into AWS S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (bytes) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
The S3 full URL to the file.
- Return type
str
- Raises
TypeError – If data is not a bytes type.
Examples
>>> data = bytes("String to bytes", "utf-8") >>> write_object_from_bytes( ... bucket="myBucket", ... key="myFiles/file.data", ... data=data ... ) http://s3.amazonaws.com/myBucket/myFiles/file.data
- s3_tools.objects.write.write_object_from_dict(bucket: str, key: str, data: Dict, aws_auth: Dict[str, str] = {}) str [source]
Upload a dictionary to an object into AWS S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (dict) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
The S3 full URL to the file.
- Return type
str
- Raises
TypeError – If data is not a dict type.
Examples
>>> data = {"key": "value", "1": "text"} >>> write_object_from_dict( ... bucket="myBucket", ... key="myFiles/file.json", ... data=data ... ) http://s3.amazonaws.com/myBucket/myFiles/file.json
- s3_tools.objects.write.write_object_from_text(bucket: str, key: str, data: str, aws_auth: Dict[str, str] = {}) str [source]
Upload a string to an object into AWS S3 bucket.
- Parameters
bucket (str) – AWS S3 bucket where the object will be stored.
key (str) – Key where the object will be stored.
data (str) – The object data to be uploaded to AWS S3.
aws_auth (Dict[str, str]) – Contains AWS credentials, by default is empty.
- Returns
The S3 full URL to the file.
- Return type
str
- Raises
TypeError – If data is not a str type.
Examples
>>> data = "A very very not so long text" >>> write_object_from_text( ... bucket="myBucket", ... key="myFiles/file.txt", ... data=data ... ) http://s3.amazonaws.com/myBucket/myFiles/file.txt