S3cmd : A commandline tool to manage data on Amazon S3

S3cmd is a tool for managing data on Amazon S3 storage. It is a simple and easy to use command line tool through which user can easily access s3 data. It is also ideal for scripts, automated backups triggered from cron, sync etc. s3cmd is an open source project and is free for both commercial and private use. You will only have to pay Amazon for using the s3 storage, and no money is needed to pay for using s3cmd tool. Installation: –The installation of s3cmd is very simple and comprises of single command. sudo apt-get update && sudo apt-get install s3cmd Configure s3cmd: – To access s3 storage using s3cmd tool you need to configure your s3cmd tool with S3 ACCESSKEYID, SECRETACCESSKEY. So, to do this type the command
s3cmd --configure
And just supply your s3 creds when prompt to do so. You do not need to configure other options just leave it blank for if you don’t know about it, and save your changes when prompted. This will create a .s3cfg file in your root directory containing the information to access the AWS storage. The disadvantage of using s3cmd is that at one time you can use only one account. The disadvantage of using s3cmd is that at one time you can use only one account and to change the account you have to reconfigure your s3cmd by using the “s3cmd –configure” command. Access: To test whether s3cmd is working fine or not, just type the following command.
s3cmd ls
It will show all the buckets you are having on the s3. And to access a particular bucket just use the path of that bucket. For example you have a bucket with path ‘s3://fh-assets’. Just use:
s3cmd ls s3://fh-assets
This will list the content of this bucket only You can easily read from or write to s3 storage using s3cmd get/put options. GET is for reading and PUT is for writing. For example we have a file abc.pdf on s3 and we want to retrieve that to local system. Just use the following command.
s3cmd get s3://fh-assets/abc.pdf 123.pdf
This will copy abc.pdf in the current directory with a name 123.pdf And to put a file into s3 use the following command
s3cmd put 123.pdf s3://fh-assets/abc.pdf
Similarly if you want to copy a folder just use -r . SYNC:This command is used to sync local directory with buckets.for example:
s3cmd sync local/directory bucket_name
Here local/directory is the source and bucket_name is destination in which the modifications occur. We can use sync command to both upload and download from s3. Just we need to change the source and the destination. There are various options available with sync option, but before moving forward we need to see that what types of transfers s2cmd performs, as this will help us to understand s3cmd sync better. In s3cmd there are basically two modes of transfer i.e.
  • Unconditional transfer — In this all matching files are uploaded to S3 or downloaded back from S3. This is similar to a standard unix cp command.
  • Conditional transfer — In this, only those files that don’t exist at the destination in the same version are transferred by the s3cmd sync command. By default a md5 checksum and file size is compared. This is similar to a unix rsync command Now continuing with options that are available with s3cmd are:-
  • –dry-run – > In this run we’ll first check with —dry-run to see what would be uploaded.
  • –skip-existing – > if you don’t want to compare checksums and sizes of the remote vs local files and only want to upload those that are new.
  • –delete-removed -> to get a list of files that exist remotely but are no longer present locally.
Enjoy the handy new tool in your kitty !!
  • maheepsingh717

    Nice Blog!!

    • maheepsingh717

      Good!!!

    • maheepsingh717

      well explained!!

      • maheepsingh717

        true!!

        • maheepsingh717

          test

          • Guest

            testxc

          • Guest

            dfjei

          • maheepsingh717

            Great

    • maheepsingh717

      Well done!!

    • maheepsingh717

      well explained!!

  • maheepsingh717

    Nice Blog!!!

  • Rohini Chaudhary

    Very Informative blog. Here is one another informative blog I found which is also well explained. Ubuntu lover might be like it.

    • Ajeet Khan

      Great post. Thank you for sharing

      • Rohini Chaudhary

        Welcome. And yes it is useful post.