Installation#
We’re ready to install Skyplane. It’s as easy as:
---> Install skyplane from PyPI:
$ pip install "skyplane[aws]"
# install support for other clouds as needed:
# $ pip install "skyplane[azure]"
# $ pip install "skyplane[gcp]"
# $ pip install "skyplane[all]"
Note
If you are using an M1 Macbook with the arm64 architecture, you will need to install skyplane as follows:
$ GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install skyplane[all]
Setting up Cloud Credentials#
Skyplane needs access to cloud credentials to perform transfers. To get started with setting up credentials, make sure you have cloud provider CLI tools installed:
---> For AWS:
$ pip install awscli
---> For Google Cloud:
$ pip install gcloud
---> For Azure:
$ pip install azure
Once you have the CLI tools setup, log into each cloud provider’s CLI:
---> For AWS:
$ aws configure
---> For Google Cloud:
$ gcloud auth application-default login
---> For Azure:
$ az login
Now, you can initialize Skyplane with your desired cloud providers. Skyplane autodetects cloud credentials and valid regions from your CLI environment.
---> Setup cloud provider connectors:
$ skyplane init
Transferring Data#
We’re ready to use Skyplane! Let’s use skyplane cp to copy files from AWS to GCP:
---> 🎸 Ready to rock and roll! Copy some files:
$ skyplane cp -r s3://... gs://...
To transfer only new objects, you can instead use skyplane sync:
---> Copy only diff
$ skyplane sync s3://... gs://...
To transfer from local disk or HDFS cluster, you can use skyplane cp as well:
(Note: On-Prem require additional setup. Please navigate to the On-Prem section for more details)
---> Copy from local disk
$ skyplane cp -r /path/to/local/file gs://...
---> Copy from HDFS
$ skyplane cp -r hdfs://... gs://...