mdsisclienttools.datastore.ReadWriteHelper

Module Contents

Functions

_fetch_all_datasets(→ List[Dict[str, Any]])

fetch_datasets

_fetch_dataset(→ Optional[Dict[str, Any]])

_fetch_dataset

_read_dataset(→ Dict[str, Any])

read_dataset

_write_dataset(→ Dict[str, Any])

_write_dataset

print_creds(→ None)

print_creds

print_command_suggestion(→ None)

print_command_suggestion

_download_files(→ None)

download_files

_upload_files(→ None)

upload(→ None)

Given a source path, handle and authorisation information, will

download(→ None)

Given a download path, handle and authorisation information, will

Attributes

DEFAULT_DATA_STORE_ENDPOINT

mdsisclienttools.datastore.ReadWriteHelper.DEFAULT_DATA_STORE_ENDPOINT = https://data-api.mds.gbrrestoration.org
mdsisclienttools.datastore.ReadWriteHelper._fetch_all_datasets(auth: mdsisclienttools.auth.TokenManager.BearerAuth, endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → List[Dict[str, Any]][source]

fetch_datasets Given basic bearer auth will call the fetch datasets API endpoint to find an example dataset.

Parameters

auth (BearerAuth) – The bearer auth object

Returns

The list of registry items

Return type

List[Dict[str,Any]]

See Also (optional)

Examples (optional)

mdsisclienttools.datastore.ReadWriteHelper._fetch_dataset(handle_id: str, auth: mdsisclienttools.auth.TokenManager.BearerAuth, endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → Optional[Dict[str, Any]][source]

_fetch_dataset Given the handle ID to lookup and the basic auth will call the fetch dataset data store API to get more information about the dataset.

Parameters

  • handle_id (str) – The handle ID to lookup

  • auth (BearerAuth) – The bearer token auth

Returns

A dictionary for the item if found or None if not

Return type

Optional[Dict[str, Any]]

Raises

  • Exception – If a non status code 200 response.

  • See Also (optional) –

  • -------- –

  • Examples (optional) –

  • -------- –

mdsisclienttools.datastore.ReadWriteHelper._read_dataset(dataset_id: str, auth: mdsisclienttools.auth.TokenManager.BearerAuth, endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → Dict[str, Any][source]

read_dataset Gets AWS read credentials using the data store API.

Parameters

  • dataset_id (Dict[str, Any]) – The datasets handle ID

  • auth (BearerAuth) – The bearer token auth

Returns

AWS credentials

Return type

Dict[str,Any]

See Also (optional)

Examples (optional)

mdsisclienttools.datastore.ReadWriteHelper._write_dataset(dataset_id: str, auth: mdsisclienttools.auth.TokenManager.BearerAuth, endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → Dict[str, Any][source]

_write_dataset Gets AWS write credentials using the data store API.

Parameters

  • dataset_id (Dict[str, Any]) – The datasets handle ID

  • auth (BearerAuth) – The bearer token auth

  • (optional) (Examples) –

  • -------- –

  • (optional) –

  • -------- –

mdsisclienttools.datastore.ReadWriteHelper.print_creds(creds: Dict[str, Any]) → None[source]

print_creds Given the creds objects, displays an AWS export format.

Parameters

  • creds (Dict[str, Any]) – The AWS creds

  • (optional) (Examples) –

  • -------- –

  • (optional) –

  • -------- –

mdsisclienttools.datastore.ReadWriteHelper.print_command_suggestion(s3_loc: Dict[str, Any]) → None[source]

print_command_suggestion Prints a suggestion of some basic S3 commands.

Parameters

  • s3_loc (Dict[str, Any]) – The s3 location object

  • (optional) (Examples) –

  • -------- –

  • (optional) –

  • -------- –

mdsisclienttools.datastore.ReadWriteHelper._download_files(s3_loc: Dict[str, str], s3_creds: Dict[str, Any], destination_dir: str) → None[source]

download_files Uses the cloudpathlib library to download all the files and sub dirs/files from the specified s3 location (s3_uri in s3_loc) into the specified destination directory.

Parameters

  • s3_loc (Dict[str, str]) – S3 location object

  • s3_creds (Dict[str, Any]) – S3 creds which match input format (i.e. expiry attribute dropped)

  • destination_dir (str) – The relative destination directory - will be created if required

  • (optional) (Examples) –

  • -------- –

  • (optional) –

  • -------- –

mdsisclienttools.datastore.ReadWriteHelper._upload_files(s3_loc: Dict[str, str], s3_creds: Dict[str, Any], source_dir: str) → None[source]
mdsisclienttools.datastore.ReadWriteHelper.upload(handle: str, auth: mdsisclienttools.auth.TokenManager.BearerAuth, source_dir: str, data_store_api_endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → None[source]

Given a source path, handle and authorisation information, will retrieve read only credentials from the data store API, fetch the dataset information, then upload all the dataset files to the handle id specified location.

Parameters

  • handle (str) – The handle ID of the dataset to download.

  • auth (BearerAuth) – The bearer auth object. See TokenManager library.

  • source_dir (str) – The path of to folder of files to upload.

Raises

ValueError – Raises a value error if the handle id is invalid.

mdsisclienttools.datastore.ReadWriteHelper.download(download_path: str, handle: str, auth: mdsisclienttools.auth.TokenManager.BearerAuth, data_store_api_endpoint: str = DEFAULT_DATA_STORE_ENDPOINT) → None[source]

Given a download path, handle and authorisation information, will retrieve read only credentials from the data store API, fetch the dataset information, then download all the dataset files to the specified location.

Parameters

  • download_path (str) – The path of to download the files to. If the folder/path does not exist, this will create the folder. If there are nested folders, all folders/files will be downloaded and paths created.

  • handle (str) – The handle ID of the dataset to download.

  • auth (BearerAuth) – The bearer auth object. See TokenManager library.

Raises

ValueError – Raises a value error if the dataset cannot be found.