skyplane.api.transfer_job#
Classes
|
class that chunks the original files and makes the chunk requests |
|
copy job that copies the source objects to the destination |
|
|
|
sync job that copies the source objects that does not exist in the destination bucket to the destination |
|
transfer job with transfer configurations |
|
Represents transfer pair between source and destination |
- class skyplane.api.transfer_job.Chunker(src_iface, dst_ifaces, transfer_config, concurrent_multipart_chunk_threads=64, num_partitions=1)[source]#
Bases:
object
class that chunks the original files and makes the chunk requests
- Parameters:
src_iface (StorageInterface) – source object store interface
dst_iface – destination object store interface
transfer_config (TransferConfig) – the configuration during the transfer
concurrent_multipart_chunk_threads (int) – the maximum number of concurrent threads that dispatch multipart chunk requests (default: 64)
- chunk(transfer_pair_generator)[source]#
Break transfer list into chunks.
- Parameters:
transfer_pair_generator (Generator) – generator of pairs of objects to transfer
- Return type:
- static map_object_key_prefix(source_prefix, source_key, dest_prefix, recursive=False)[source]#
map_object_key_prefix computes the mapping of a source key in a bucket prefix to the destination. Users invoke a transfer via the CLI; aws s3 cp s3://bucket/source_prefix s3://bucket/dest_prefix. The CLI will query the object store for all objects in the source prefix and map them to the destination prefix using this function.
- Parameters:
source_prefix (string) – source bucket folder prefix
source_key (string) – source file key to map in the folder prefix
destination_prefix (string) – destination bucket folder prefix
recursive (bool) – whether to copy all the objects matching the pattern (default: False)
- static prefetch_generator(gen_in, buffer_size)[source]#
Prefetches from generator while handing StopIteration to ensure items yield immediately. Start a thread to prefetch items from the generator and put them in a queue. Upon StopIteration, the thread will add a sentinel value to the queue.
- transfer_pair_generator(src_prefix, dst_prefixes, recursive, prefilter_fn=None)[source]#
Query source region and return list of objects to transfer.
- Parameters:
src_prefix (string) – source bucket folder prefix
dst_prefix (string) – destination bucket folder prefix
recursive (bool) – if true, will copy objects at folder prefix recursively
prefilter_fn (Callable[[ObjectStoreObject], bool]) – filters out objects whose prefixes do not match the filter function (default: None)
- Return type:
- class skyplane.api.transfer_job.CopyJob(src_path, dst_paths, recursive=False, requester_pays=False, job_id=None)[source]#
Bases:
TransferJob
copy job that copies the source objects to the destination
- Parameters:
- dispatch(dataplane, dispatch_batch_size=100, transfer_config=Field(name=None, type=None, default=<dataclasses._MISSING_TYPE object>, default_factory=<function CopyJob.<lambda>>, init=False, repr=True, hash=None, compare=True, metadata=mappingproxy({}), _field_type=None))[source]#
Dispatch transfer job to specified gateways.
- gen_transfer_pairs(chunker=None, transfer_config=Field(name=None, type=None, default=<dataclasses._MISSING_TYPE object>, default_factory=<function CopyJob.<lambda>>, init=False, repr=True, hash=None, compare=True, metadata=mappingproxy({}), _field_type=None))[source]#
Generate transfer pairs for the transfer job.
- Parameters:
chunker (Chunker) – chunker that makes the chunk requests
- Return type:
- property http_pool#
http connection pool
- property src_iface: StorageInterface#
Return the source object store interface
- class skyplane.api.transfer_job.GatewayMessage(chunk=None, upload_id_mapping=None)[source]#
Bases:
object
- class skyplane.api.transfer_job.SyncJob(src_path, dst_paths, requester_pays=False, job_id=None)[source]#
Bases:
CopyJob
sync job that copies the source objects that does not exist in the destination bucket to the destination
- dispatch(dataplane, dispatch_batch_size=100, transfer_config=Field(name=None, type=None, default=<dataclasses._MISSING_TYPE object>, default_factory=<function CopyJob.<lambda>>, init=False, repr=True, hash=None, compare=True, metadata=mappingproxy({}), _field_type=None))#
Dispatch transfer job to specified gateways.
- finalize()#
Complete the multipart upload requests
- gen_transfer_pairs(chunker=None, transfer_config=Field(name=None, type=None, default=<dataclasses._MISSING_TYPE object>, default_factory=<function SyncJob.<lambda>>, init=False, repr=True, hash=None, compare=True, metadata=mappingproxy({}), _field_type=None))[source]#
Generate transfer pairs for the transfer job.
- Parameters:
chunker (Chunker) – chunker that makes the chunk requests
- Return type:
- property http_pool#
http connection pool
- size_gb()#
Return the size of the transfer in GB
- property src_iface: StorageInterface#
Return the source object store interface
- verify()#
Verify the integrity of the transfered destination objects
- class skyplane.api.transfer_job.TransferJob(src_path, dst_paths, recursive=False, requester_pays=False, job_id=None)[source]#
Bases:
ABC
transfer job with transfer configurations
- Parameters:
src_path (str) – source full path
dst_paths (str) – destination full path
recursive (bool) – if true, will transfer objects at folder prefix recursively (default: False)
requester_pays (bool) – if set, will support requester pays buckets. (default: False)
uuid (str) – the uuid of one single transfer job
- property src_iface: StorageInterface#
Return the source object store interface