Configuration#
Skyplane comes with a variety of knobs to tune to adjust performance or change how VMs are provisioned. You can configure these options using the skyplane config set <key> <value>
command.
Increasing performance of transfers via paralllelism#
Skyplane transfers files in parallel across a set of gateway VMs. The total bandwidth of a transfer is mostly determined by the number of VMs used. Skyplane will automatically partition and distribute the transfer across the set of gateway VMs.
To increase the parallelism of a transfer, run:
$ skyplane config set max_instances <number>
By default, Skyplane will use a maximum of 1 VM in each region. This limit is conservative since too many VMs can potentially result failed transfers due to VCPU limits and other resource constraints.
If you encounter VCPU limits, increase your VCPU limits following this guide:
Reusing Gateways#
It can take 45s to 2m to provision gateway VMs for a transfer. If you are repeatedly transferring data between the same pair of regions, you can reuse gateway VMs. By default, Skyplane terminates these VMs to avoid unnecessary VM costs.
When running a cp
or sync
command, pass the --reuse-gateways
flag to Skyplane to reuse gateway VMs:
$ skyplane cp --reuse-gateways <source> <destination>
$ skyplane sync --reuse-gateways <source> <destination>
We will attempt to automatically deprovision these gateways after 15 minutes by default. Change this interval via skyplane config set autoshutdown_minutes <minutes>
. With --reuse-gateways
, Skyplane will start a background job on each gateway VM that triggers a VM shutdown after the specified delay. Note you will still pay for associated VM costs such as storage and network IPs even if VMs are shut down.
To ensure that all gateways are stopped and no longer incur charges, run:
$ skyplane deprovision
Spot Instances to reduce instance costs#
Spot instances reduce the cost of provisioning VMs. These instances are charged at a lower price than on-demand instances but can be preempted at any time. If this occurs, the transfer will fail.
To use spot instances, run:
$ skyplane config set aws_use_spot_instances True
$ skyplane config set azure_use_spot_instances True
$ skyplane config set gcp_use_spot_instances True
Configuring networking between gateways#
Skyplane supports encrypting data end-to-end. This is useful for encrypting data that is stored on a local disk. We enable end-to-end encryption by default. To disable it, run:
$ skyplane config set encrypt_e2e false
Skyplane automatically compresses data at the source region to reduce egress fees from data transfer. We use the LZ4 compression algorithm by default as it can compress data at line rate. To disable compression, run:
$ skyplane config set compress false
Skyplane continually computes checksums at the source region to verify data integrity. We use the MD5 checksum algorithm by default. To disable checksum verification, run:
$ skyplane config set verify_checksums false
Optionally and in addition to end-to-end encryption, Skyplane supports sending data over a TLS encrypted socket. We don’t generally recommend using this unless you have a specific security requirement as it can increase performance variablity of transfers. To enable further TLS socket encryption, run:
$ skyplane config set encrypt_socket_tls true