Archive for the ‘Amazon S3’ Category

S3cmd : A commandline tool to manage data on Amazon S3

S3cmd is a tool for managing data on Amazon S3 storage. It is a simple and easy to use command line tool through which user can easily access s3 data. It is also ideal for scripts, automated backups triggered from cron, sync etc. s3cmd is an open source project and is free for both commercial and private use. You will only have to pay Amazon for using the s3 storage, and no money is needed to pay for using s3cmd tool. Installation: –The installation of s3cmd is very simple and comprises of single command. sudo apt-get update && sudo apt-get install s3cmd Configure s3cmd: – To access s3 storage using s3cmd tool you need to configure your s3cmd tool with S3 ACCESSKEYID, SECRETACCESSKEY. So, to do this type the command
s3cmd --configure
And just supply your s3 creds when prompt to do so. You do not need to configure other options just leave it blank for if you don’t know about it, and save your changes when prompted. This will create a .s3cfg file in your root directory containing the information to access the AWS storage. The disadvantage of using s3cmd is that at one time you can use only one account. The disadvantage of using s3cmd is that at one time you can use only one account and to change the account you have to reconfigure your s3cmd by using the “s3cmd –configure” command. Access: To test whether s3cmd is working fine or not, just type the following command.
s3cmd ls
It will show all the buckets you are having on the s3. And to access a particular bucket just use the path of that bucket. For example you have a bucket with path ‘s3://fh-assets’. Just use:
s3cmd ls s3://fh-assets
This will list the content of this bucket only You can easily read from or write to s3 storage using s3cmd get/put options. GET is for reading and PUT is for writing. For example we have a file abc.pdf on s3 and we want to retrieve that to local system. Just use the following command.
s3cmd get s3://fh-assets/abc.pdf 123.pdf
This will copy abc.pdf in the current directory with a name 123.pdf And to put a file into s3 use the following command
s3cmd put 123.pdf s3://fh-assets/abc.pdf
Similarly if you want to copy a folder just use -r . SYNC:This command is used to sync local directory with buckets.for example:
s3cmd sync local/directory bucket_name
Here local/directory is the source and bucket_name is destination in which the modifications occur. We can use sync command to both upload and download from s3. Just we need to change the source and the destination. There are various options available with sync option, but before moving forward we need to see that what types of transfers s2cmd performs, as this will help us to understand s3cmd sync better. In s3cmd there are basically two modes of transfer i.e.
  • Unconditional transfer — In this all matching files are uploaded to S3 or downloaded back from S3. This is similar to a standard unix cp command.
  • Conditional transfer — In this, only those files that don’t exist at the destination in the same version are transferred by the s3cmd sync command. By default a md5 checksum and file size is compared. This is similar to a unix rsync command Now continuing with options that are available with s3cmd are:-
  • –dry-run – > In this run we’ll first check with —dry-run to see what would be uploaded.
  • –skip-existing – > if you don’t want to compare checksums and sizes of the remote vs local files and only want to upload those that are new.
  • –delete-removed -> to get a list of files that exist remotely but are no longer present locally.
Enjoy the handy new tool in your kitty !!

S3FS : Mounting Amazon S3 as a filesystem

S3FS (Simple Storage Services File System) is a FUSE based file system backed by Amazon S3 storage buckets, and if mounted once, S3 can be used just like a drive in our local system. This helps a lot if you have an app which interacts very frequently with Amazon S3. Follow these steps to mount a bucket from s3 to local system using S3fs in UBUNTU 1. First of all you need to install the libraries that are needed by s3fs to run on the system. To install those libraries run the following command in console sudo aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs 2. After that Download the appropriate version of s3fs you want to use by the following link In this example we are using s3fs-1.68, and can be downloaded from this link 3. After downloading the s3fs go to that path in your terminal and run the following commands
  • tar xvzf s3fs-1.68.tar.gz
  • cd s3fs-1.68/
  • ./configure –prefix=/usr
  • make
  • make install (as root)
4. After this you need to supply your s3 credentials to s3fs to mount the bucket. There are different ways of doing that, you can choose whichever is suitable to you
  • By setting the AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables by using export command in UBUNTU. export AWSACCESSKEYID=******************* export AWSSECRETACCESSKEY=xxxxxxxxxxxx
  • By using a .passwd-s3fs file in your home directory. In this you just need to create the .passwd-s3fs file in home directory, and add the creds for s3 into it. The s3fs password file has this format (use this format if you have only one set of credentials): accessKeyId:secretAccessKey If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well: bucketName:accessKeyId:secretAccessKey
  • By using the system-wide /etc/passwd-s3fs file. In this you just need to create the passwd-s3fs file in /etc directory, and add the creds for s3 into it similar to the last point.
5. After that change the permissions of the password file
  • If you are using ~/.passwd-s3fs then permission should be 600
  • If you are using /etc/passwd-s3fs then permission should be 640
6. And last, to mount bucket(mybucket) to specific directory(/path/to/directory/) s3fs mybucket /path/to/directory/ 7. To unmount the mounted drive use this command. fusermount -u /path/to/directory/ 8. To automount s3 bucket to every time to local system after boot, edit the /etc/fstab file and add this at the end of the file s3fs#bucket_name /path/to/directory/ passwd_file=/path/to/passwd/file And after this edit /etc/rc.local and add the following line before exit 0. mount -a 9. If you get any error/warning during system boot mounting just add the option in the current command s3fs#bucket_name /path/to/directory/ fuse _netdev,passwd_file=/path/to/passwd/file This command will omit the warnings and mount the bucket to your local system. Now you can access Amazon S3 just like another folder on your file system.