Single Sign-On In Rails Application

Single sign-on (SSO) is a property of access control of multiple related, but independent software systems. With this property a user logs in once and gains access to all systems without being prompted to log in again at each of them. Such as we once log in to google, we are permitted to access the Youtube, Docs, Google Drive etc. dnd1999110501 We may think about sharing the session cookies among the all domains but its generally not a good idea. One can steal cookies to sign in from other system or browser. Single SignOn Helps us in getting around this problem and implement a much robust system. How do we implement Single Sign-On in Rails App? We will try to get our hand around on how to build a prototype around a rails app to implement single sign-on. We will be looking a sample rails application powered by CASino gem Why we are using CASino ?
  1. Distributed under MIT LIcense
  2. Active Development since
  3. External authentication sources (LDAP, SQL)
  4. Two-factor authentication
  5. Session-overview for logged-in users
  6. Full localization support
  7. REST-API
So what is CASino exactly? CASino is a simple Single sign-on server application. It supports the CAS protocol and can therefore be used in combination with almost every web programming language out there. CAS is an authentication system originally created by Yale University to provide a trusted way for an application to authenticate a user. CAS centralizes authentication: It allows all your applications to ask users to login to a single sign-on server The implementation. Here we have two apps client and server. We assume that the app is using devise for the demonstration purposes. The Logic can easily be extended to other authentication methods though. The app is as simple as configuring a few variables in the CASino gem.
  1. user request to client domain .
  2. it redirects to the server if not authorised.
  3. Start with adding the gems client app
  4. redirects to back url to whichever client request url is
Step #1 Implementing the client application. Gemfile # If You face some compatability issues then use ruby 2.1

gem 'devise_cas_authenticatable', git: '<a href="https://github.com/jpamaya/devise_cas_authenticatable">https://github.com/jpamaya/devise_cas_authenticatable</a>'

routes.rb

devise_for :users, skip: [:sessions], controllers: { cas_sessions: 'sso_cas' }

devise_scope :user do

get "sign_in", to: "devise/cas_sessions#new"

delete "sign_out", to: "devise/cas_sessions#destroy"

end

devise.rb # ==> Configuration for SSO server authentication

config.cas_base_url = "http://localhost:4000/"

config.cas_create_user = false

config.cas_destination_logout_param_name = 'service'

config.cas_enable_single_sign_out = true

User.rb

devise :cas_authenticatable

Now run client application on localhost:3000 and begin with typing users/sign_in Since we overwrote devise’s sign in mechanism, so instead of presenting devise’s normal sign in page, we are redirected to localhost:4000 which is the backend app to authenticate the users. Lets prepare the backend app to authenticate the users.  Step #2 Start with implementing code in server app limitation use ruby ruby “2.1.5” and rails “3.2.11” check rails -v ‘~>(‘ 3.2.11′). for server app Gemfile #It users these db to store sessions and some tokens. casino will take care all about these

gem 'sqlite3'   # for sqlite support

gem 'mysql2'    # for mysql support

gem 'pg'        # for postgresql support

gem 'casino'

Run bundle install and rails g casino:install The CAS configuration is stored under config/cas.yml. This is where you configure how your SSO handles logins. An example configuration can be found in the file config/cas.yml.example provide the valid db credentials and run the migrations. Run this app on localhost:4000 , Just follow the old steps
  1. Goto localhost:3000/users/sign_in
  2. You will redirected to localhost:4000 enter credentials click sign in
  3. You will be redirected back on successful login
In the links below you can find some ready made examples for you to clone these  apps and have better understanding of SSO.
Server exampleruby “2.1.5”git clone https://github.com/codescrum/casino-sso-server-example.gitRails 4 client examplegit clone https://github.com/codescrum/casino-sso-client-rails4-example.gitRails 3 client example git clone https://github.com/codescrum/casino-sso-client-rails3-example.git
Let us know in comments if you face any troubles in the implementation !! PS: Single Sign-out has been left as a thought for another day and we will discuss that in a follow up blog.

The technology stack powering the on demand applications

Uber and Ola have given rise to a slew of Ondemand applications which claim themselves to be Uber/Ola of X. Some of the verticals where these applications have found their usability include but not limited to taxi, transportation, Logistics, Home Services, Health apps and so on. In this blog we will go deeper into the various components and technology involved in on Demand Apps.
Components Involved in Building and On Demand App
In most of the cases there are 3 components involved in building an On-Demand App:
  • A cloud backend for Data Interaction APIs and live dispatch of Jobs
  • An Android/iOS App user to make a job(e.g book a ride with Uber/Ola)
  • An app with the drivers to receive the jobs created by the users and attend to them.

Lets analyze each of these components in detail and how one can go around building each of them.
Web Backend
Web Backend is the most critical part of the On Demand applications because backend is what powers both the consumer and the driver apps. The web backend must meet the following requirements for a successful implementation:
  • A Light footprint to dispatch the jobs on fly.
  • A Good dispatch components to maximize the efficiency of the field workers(drivers in case of uber).
  • A Socket connectivity to enable the jobs dispatch and job information to the user.
  • A Good structure for a flexible and structured development

We have recently worked on Pikk which is an Ondemand Application for Logistics Industry(The carrier being the mini trucks for now).

jQuery Validate

jQuery Validation plugin is popular and simple plugin for client side validations. It can be customized with plenty of options. It can be easily integrated into your existing forms. The plugin already has set of useful validation methods including URL and email validation that come out of the box. The plugin provides an API to write your own methods. All in built methods comes with default error messages in english. In the blog, I will try to explore a few methods and usage of jQuery Validate Plugin. Plugin Methods This library adds three jQuery plugin methods:
  • validate() – Validates the selected form.
  • valid() – Checks whether the selected form or selected elements are valid.
  •  rules() – Read, add and remove rules for an element
Setting up: You need to download the files and then place dist/jquery.validate.min.js file in #= require jquery.validate.min Applying to the form:
  • You just need to add a class ‘jqvalidate’ to the form that you want to validate. In rails, you will generally do it like following
    <%=  form_for(@account, :html =&gt; { :class =&gt; "jqvalidate" }) do |f| %>
    .........
    .........
    <% end %>
    
  • Next you need to create a function that is to be called before the form is loaded:
    $.validate_form = () -&gt;
    $.validator.addClassRules({
    cc_number: { required: true, digits: true, minlength: 15, maxlength: 16},
    cc_cvv: { required: true, digits: true, minlength: 3, maxlength: 4}
    })
    $('.jqvalidate').validate({
    ignore: ".jqvalidate_ignore"
    errorClass: "invalid"
    })
    
  1. addClassRules is used to specify validations on the fields with its in built methods. cc_number and cc_cvv are the classes given to two different fields respectively on which the validations are applied.
  2. ‘.jqvalidate_validate’ class is given to those fields that are not subject to the validations by this plugin.
  3. ‘invalid’ is the class given to the error messages.
The blog post just details the start for the plugin. We encourage you to dig more and explore this great plugin. For more details, refer to the documentation.

Capybara Basics For Automated testing of Ruby on Rails Application

Automated Testing is an integral part of large scale production Ruby on Rails applications. A solid Integration test suite ensures that the integration between various components of the application is smooth and seamless. An integration test generally spans multiple controllers and actions, tying them all together to ensure they work together as expected. It tests more completely than either unit or functional tests do, exercising the entire stack, from the dispatcher to the database. Capybara comes in handy in simulating the testing in real user environment. In this blog we will explore various ways by which we can strengthen our apps with Capybara.

    • Capybara is an integration testing tool for rack based web applications. It simulates how a user would interact with a website.
    • Capybara helps in testing your web application as a real user would do. It is independent of the driver that is running your tests and comes with Rack::Test and Selenium support built in. For other drivers, you need to include their respective gem.
Capybara requires Ruby 1.9.3 or later and can be installed as a gem by listing in your gem file and then installing bundle or it may be installed directly by typing:
gem install capybara
  • Include capybara in your helper file:
    require 'capybara/rails'
Using Capybara with Rspec
Integrating capybara with rspec is fairly simple. Lets go through a step by step process to understand the integration. The following test simulates a login in a real user environment.
  • Include capybara’s rspec directory
    require 'capybara/rspec'
  • Capybara specs are required to be placed in spec/features
  • Here is a simple test to test the login page working.
    describe "Login" do
    before :each do
    user = User.create(:email =&gt; 'RoR@enbake.com'”,:password =&gt; 'password')
    end
    it "logs the user in" do
    visit login_path
    fill_in "Email", with: user.email
    fill_in "Password", with: 'kanwal_enbake'
    click_button "Sign in"
    page.should have_content "success"
    end
    end
    
    The test source code is fairly simple to understand and implement so I will not be going into the details here.
Testing Javascript with Capybara
  • By default Capybara uses the :rack_test driver, which is fast but limited as it does not support JavaScript. So you need to switch to Capybara.javascript_driver(:selenium by default) by using :js => true. For instance: describe “example of testing js”, :js => true do it “tests js with default js driver” do end it “tests js with one specific driver” , :driver => :webkit do end end
  • To make Capybara available in all test cases deriving fromActionDispatch::IntegrationTest : describe ActionDispatch::IntegrationTest include Capybara::DSL end
Drivers
  • You may sometimes, want to switch default driver and want to run everything in selenium, you could do:Capybara.default_driver = :selenium
  • In any case – provided you are using Rspec or Cucumber – if you want use faster :rack_test as your default_driver, and use JavaScript-capable driver only for selective test cases, you can use :js => true for Rspec and @javascript for Cucumber. By default, JavaScript tests are run using the :selenium driver. You can change this setting by Capybara.javascript_driver.
  • You can also, particularly in before or after blocks, change driver temporarily: Capybara.current_driver = :webkit
  • Capybara.use_default_driver would switch back to default driver
Note: switching the driver creates a new session, so you may not be able to switch in the middle of a test.
Techniques and Trivial Testing Tasks
Navigating
  • visit(‘/projects’)
  • visit(post_comments_path(post))
Clicking links and buttons
  • click_link(‘id-of-link’)
  • click_link(‘Link Text’)
  • click_button(‘Save’)
  • click(‘Link Text’) # Click either a link or a button
  • click(‘Button Value’)
Interacting with forms
  • fill_in(‘First Name’, :with => ‘John’)
  • fill_in(‘Password’, :with => ‘Seekrit’)
  • fill_in(‘Description’, :with => ‘Really Long Text…’)
  • choose(‘A Radio Button’)
  • check(‘A Checkbox’)
  • uncheck(‘A Checkbox’)
  • attach_file(‘Image’, ‘/path/to/image.jpg’)
  • select(‘Option’, :from => ‘Select Box’)
Scoping
  • within(“//li[@id=’employee’]”) do fill_in ‘Name’, :with => ‘Jimmy’ end
  • within(:css, “li#employee”) do fill_in ‘Name’, :with => ‘Jimmy’ end
  • within_fieldset(‘Employee’) do fill_in ‘Name’, :with => ‘Jimmy’ end
  • within_table(‘Employee’) do fill_in ‘Name’, :with => ‘Jimmy’ end
Querying
  • page.has_xpath?(‘//table/tr’)
  • page.has_css?(‘table tr.foo’)
  • page.has_content?(‘foo’)
  • page.should have_xpath(‘//table/tr’)
  • page.should have_css(‘table tr.foo’)
  • page.should have_content(‘foo’)
  • page.should have_no_content(‘foo’)
  • find_field(‘First Name’).value
  • find_link(‘Hello’).visible?
  • find_button(‘Send’).click
  • find(‘//table/tr’).click
  • locate(“//*[@id=’overlay'”).find(“//h1″).click
  • all(‘a’).each { |a| a[:href] }
  • current_path.should == post_comments_path(post)
Scripting
    • result = page.evaluate_script(‘4 + 4′)
    • page.execute_script(“$(‘#id_of_anything’)”).click
Note: unlike page.evaluate_script, page.execute_script do not return anything. You have to explicitly perform action to make it return a value Debugging
  • save_and_open_page
  • page.save_screenshot(‘screenshot.png’) *driver specfic
Asynchronous JavaScript
  • click_link(‘foo’)
  • click_link(‘bar’)
  • page.should have_content(‘baz’)
  • page.should_not have_xpath(‘//a’)
  • page.should have_no_xpath(‘//a’)
XPath and CSS
  • within(:css, ‘ul li’) { … }
  • find(:css, ‘ul li’).text
  • locate(:css, ‘input#name’).value
  • Capybara.default_selector = :css
  • within(‘ul li’) { … }
  • find(‘ul li’).text
  • locate(‘input#name’).value
Calling remote servers
  • Normally Capybara expects to be testing an in-process Rack application, but you can also use it to talk to a web server running anywhere on the internet, by setting app_host: Capybara.current_driver = :selenium Capybara.app_host =https://www.google.com
I hope the blog gives a good introduction of adding capybara tests to your rails apps. Its just the tip of the iceberg and the possibilities are endless. Lets make our apps more robust and flexible .. :).

Embedly Integration in Rails

Embedly is a powerful tool to embed multimedia content or images or even blogs in your application. It comes in handy especially to make your content easier to share.
Embedly provides an easy to use official ruby gem to integrate Embedly into your RoR app. The usage of the gem is fairly straight forward. I will try to go step by step on how to integrate embedly API into an RoR app.
  • Add the gem in your gem file
    gem "embedly"
    Bundle the Gems
  • Get the API key from “https://app.embed.ly/login”. For the purpose of the blog we assume that the API key is accessible through the config variable AppConfig.embedly[:api_key]
  • Mostly the embeds will stay the view side logic so lets create a small helper to generate an embed from a Youtube Video. The call to the API is pretty straight forward and intuitive as we can see from the following code.
  • require 'embedly'
    require 'json'def display
    embedly_api =Embedly::API.new AppConfig.embedly[:api_key]
    obj = embedly_api.oembed :url => 'http://www.youtube.com/watch?v=sPbJ4Z5D-n4&feature=topvideos'
    obj.first.html
    end
  • The code calls the embedly API to generate an ombed for the youtube url and returns the html for the embed. This html can be used directly to embed the content into your blog or app.
  • Thats it !!

Enjoy supercharged content with your app and feel free to post any comments in the comments section.

Rapportive Alternatives

As LinkedIn pulls the plug on the features that once made Rapportive such a popular extension, we analyzed the various similar extensions in the chrome store and how closely they fulfill the void that Rapportive has left :

Rapporto: One of the fastest dropins for Rapportive, the plugin almost emulates the original Rapportive functionality and matches the UI as well seamlessly.  The chrome extension might not be exactly same as rapportive but it gets sufficiently close to fill the void. However, if you would to see some trusted names behind the product, then Rapporto might not be the way for you. Moreover, the future of Rapporto is not yet certain as its founders are yet to comment on the roadmap for the extension.

Vibe: We have been using Vibe before Rapportive as well but the current changes in Rapportive make it much more viable. Vibe is just not a gmail extension as it is available on a plenty of platforms including Mobile platforms(Android, iPhone etc). You can just hover over any email on any website and Vibe will bring up a nice looking widget presenting the information about the user. The UI is eye candy though it might not be as sleek as Rapportive’s. The application is free for personal use and you will need to contact their sales team if you wish to use the application for business purposes.

Ark Browser Plugin : Ark is a browser plugin which enables you to see contact information from not only gmail but yahoo, hotmail and AOL as well. The plugin goes quite close to the Rapportive in terms of UI and seems to have quite a lot of contacts indexed. The plugin is available on firefox as well if you are not a Chrome Junkie.

360social.me : 360social.me is an extremely robust browser plugin which presents the information quite similar to Rapportive did in its original role, the UI however is not as sleek as Rapportive. The extension is free to use but it comes with a professional plan as well for which they charge nominal subscription fees.

We are sure that one of the above apps will replace your rapportive workflow. If you are already using some other plugin, we look forward to your comments.

A Step by Step Guide to Setup Rails application on Ec2 instance (Ubuntu Server)

Sometimes the Bitnami or other Rails AMIs doesn’t fit your needs directly and you will feel the need of building the Server yourself.Here I go step by step in building such a stack on top of Amaon EC2 Ubuntu Server.
  • Rails applications are a little bit different to install on servers but the process is very easy.Rails application needs a web server and an application server to run with. For development, it comes with default Webrick server that serve as application server on local machine. For setting it up on production server, we have the following choices on Web and application servers :-

    • Web servers

      1. Apache

      2. Nginx

    • Application Servers

      1. Passenger

      2. Thin

      3. Puma

      4. Unicorn

  • The simplest and best combination consists of Nginx + Passenger. It allows greater flexibility for configuration and also allows good speed over other combinations. So we are going to setup an Rails application using Nginx + passenger configuration on a bare Ubuntu server. Here are the steps :-

  1. Launch an Ec2 instance with ubuntu AMI. Make sure you have HTTP and SSH access to the server.

  2. SSH into the server by using private key (.pem) used while launching the instance and install the available updates by running :-

    sudo apt-get install updates
  3. Now you need to setup ruby on your server, so install the single user rvm ruby by following this blog.
  4. Load the rvm and make the installed ruby as default by running the following commands :-

    source ~/.rvm/scripts/rvm
    rvm use 2.1.0 –default
  5. Install the version control to clone your rails application to server. We generally use Git with rails application which can be installed by running the following command :-

    sudo apt-get install git
  6. Now clone your application on the server :-

    git clone yourepo.git

    Note:- In case of private git repository, you need to add public key of server to deploy keys of your repository, otherwise you will be promped with an permission denied error.

    OR

    Deploy using application to this server using Capistrano script. Please read this blog for more details on deploying your application using Capistrano.

  7. Now go to your application and install the gems by running bundle install command. If you want to setup your database on the same server, you can do the same by using the following commands 😐

    • In case of MYSQL

      sudo apt-get install mysql-server mysql-client
      sudo apt-get install libmysql++-dev
    • In case of POSTGRESQL, follow this blog for installation and then install the development headers

      sudo apt-get install libpq-dev

      After setting this up, migrate your databases in whichever environment you want to launch the server.

  8. Now install the Passenger gem by running :-

    gem install passenger

  9. Next step is to install the Nginx server, but we have some pre-requisits for this.

      1. It needs curl development headers which can be installed by :-

        sudo apt-get install libcurl4-openssl-dev

      2. It will be installed under /opt directory and your user should have permissions to that folder, so make your user as user owner for /opt directory by :-

        sudo chown -R ubuntu /opt

  10. Now install the Nginx server with passenger extension by running the following command :-

    passenger-install-nginx-module

  11. Set your Nginx server as service in init script by using the following commands :-

    wget -O init-deb.sh http://library.linode.com/assets/660-init-deb.sh
    sudo mv init-deb.sh /etc/init.d/nginx
    sudo chmod  +x /etc/init.d/nginx
    sudo /usr/sbin/update-rc.d -f nginx defaults
  12. Setup your application path in the nginx configuration file i.e. /opt/nginx/conf/nginx.conf.

    server {

    listen 80;

    server_name localhost;

    root /home/ubuntu/my_application/public #<-- be sure to point to 'public'

    passenger_enabled on;

    rails_env production;

    }

  13. Lastly start your server by running the following command :-

    sudo service nginx start

     


Braintree Integration In Rails Application

Braintree is one gateway that is getting a lot of traction and user love these days due to its ease of use and getting started easily. I have indicated here a few things that should get you started with braintree in rails quickly. In this Blog, we’ll securely process a credit card transaction using the official braintree gem and the Client-side encryption method, utilizing Braintree.js library. The blog covers the case where the form to accept the payments is rendered on the client side only.     Note: Before starting you’ll need to sign up for a Sandbox account. That will provide us with the required braintree API keys. There are two parts to the problem. First one is setting up the form and securing the data to be sent to the server. The next is to process the transaction the server side. Payment Form A simple payment form captures the credit card information along with amount of payment to be made. Here goes a very simple HTML form which asks for all the required info:
</pre>
<h1>Braintree Credit Card Transaction Form</h1>
<div><form id="braintree- payment-form" action="/create_transaction" method="POST"><label>Card Number</label>
<input type="text" autocomplete="off" size="20" data-encrypted-name="number" />

<label>CVV</label>
<input type="text" autocomplete="off" size="4" data-encrypted-name="cvv" />

<label>Expiration (MM/YYYY)</label>
<input type="text" name="month" size="2" /> / <input type="text" name="year" size="4" />

<input id="submit" type="submit" /></form></div>
<pre>
Next up, we need to encrypt the payment that is being sent from client side to our server for processing. Braintree provides a nice JS library to encrypt the data that is being sent from your form. The integration is as simple as including the library and calling a do it all method.
<script type="text/javascript" src="https://js.braintreegateway.com/v1/braintree.js"></script><script type="text/javascript">// <![CDATA[
var braintree = Braintree.create("YourClientSideEncryptionKey"); braintree.onSubmitEncryptForm('braintree-payment-form');
// ]]></script>
As you can see, we just did couple of steps to secure the data being sent to the server.
  • We initialized the Braintree.js with client-side encryption key.
  • We then called the Braintree.js onSubmitEncryptForm method with the id of the form to encrypt the form before sending it to the server.
Thats it, your data is being securely sent to your server now for processing. Lets move on to the server side of the things. First up install the official braintree gem in your Gemfile. Setting up Rails Controller to commit a transaction with Braintree
  • Every Braintree user receives a unique set of API keys. These need to available in your application.
  • To retrieve the keys, first sign in your Sandbox account.
  • On the sandbox home page, you can get the API keys required to execute transactions with BrainTree.
  • The code will look like this(for sandbox environment)
  • 	Braintree::Configuration.environment = :sandbox
    	Braintree::Configuration.merchant_id = “use_your_merchant_id”
    	Braintree::Configuration.public_key = “use your public key”
    	Braintree::Configuration.private_key = “use your private key”
  • Next Up, lets call up the API call to make the transaction. Braintree gem comes with exhaustive set of method calls to interact with the gateway. Here we initiate a sale with Braintree.
  • 	result = Braintree::Transaction.sale(:amount => “1000.00”,
    :credit_card => {:number => params[:number], :cvv => params[:cvv], :expiration_month => params[:month], :expiration_year => params[:year]},
    :options => {:submit_for_settlement => params[:settlement]})
  • Thats it !! The result object will tell us whether initiating the payment succeeded or not. The object comes loaded with methods to test the result like result.success? and result.failed?
This is not the end to the whole payment process in Braintree. Braintree will send calls/hooks informing your application about the various payment states but that is out of scope for the purpose of this blog and will better taken up as a post for separate. I hope the blog helps you get up with Braintree Quickly. Please feel free to drop in comments if any.

S3cmd : A commandline tool to manage data on Amazon S3

S3cmd is a tool for managing data on Amazon S3 storage. It is a simple and easy to use command line tool through which user can easily access s3 data. It is also ideal for scripts, automated backups triggered from cron, sync etc. s3cmd is an open source project and is free for both commercial and private use. You will only have to pay Amazon for using the s3 storage, and no money is needed to pay for using s3cmd tool. Installation: –The installation of s3cmd is very simple and comprises of single command. sudo apt-get update && sudo apt-get install s3cmd Configure s3cmd: – To access s3 storage using s3cmd tool you need to configure your s3cmd tool with S3 ACCESSKEYID, SECRETACCESSKEY. So, to do this type the command
s3cmd --configure
And just supply your s3 creds when prompt to do so. You do not need to configure other options just leave it blank for if you don’t know about it, and save your changes when prompted. This will create a .s3cfg file in your root directory containing the information to access the AWS storage. The disadvantage of using s3cmd is that at one time you can use only one account. The disadvantage of using s3cmd is that at one time you can use only one account and to change the account you have to reconfigure your s3cmd by using the “s3cmd –configure” command. Access: To test whether s3cmd is working fine or not, just type the following command.
s3cmd ls
It will show all the buckets you are having on the s3. And to access a particular bucket just use the path of that bucket. For example you have a bucket with path ‘s3://fh-assets’. Just use:
s3cmd ls s3://fh-assets
This will list the content of this bucket only You can easily read from or write to s3 storage using s3cmd get/put options. GET is for reading and PUT is for writing. For example we have a file abc.pdf on s3 and we want to retrieve that to local system. Just use the following command.
s3cmd get s3://fh-assets/abc.pdf 123.pdf
This will copy abc.pdf in the current directory with a name 123.pdf And to put a file into s3 use the following command
s3cmd put 123.pdf s3://fh-assets/abc.pdf
Similarly if you want to copy a folder just use -r . SYNC:This command is used to sync local directory with buckets.for example:
s3cmd sync local/directory bucket_name
Here local/directory is the source and bucket_name is destination in which the modifications occur. We can use sync command to both upload and download from s3. Just we need to change the source and the destination. There are various options available with sync option, but before moving forward we need to see that what types of transfers s2cmd performs, as this will help us to understand s3cmd sync better. In s3cmd there are basically two modes of transfer i.e.
  • Unconditional transfer — In this all matching files are uploaded to S3 or downloaded back from S3. This is similar to a standard unix cp command.
  • Conditional transfer — In this, only those files that don’t exist at the destination in the same version are transferred by the s3cmd sync command. By default a md5 checksum and file size is compared. This is similar to a unix rsync command Now continuing with options that are available with s3cmd are:-
  • –dry-run – > In this run we’ll first check with —dry-run to see what would be uploaded.
  • –skip-existing – > if you don’t want to compare checksums and sizes of the remote vs local files and only want to upload those that are new.
  • –delete-removed -> to get a list of files that exist remotely but are no longer present locally.
Enjoy the handy new tool in your kitty !!

S3FS : Mounting Amazon S3 as a filesystem

S3FS (Simple Storage Services File System) is a FUSE based file system backed by Amazon S3 storage buckets, and if mounted once, S3 can be used just like a drive in our local system. This helps a lot if you have an app which interacts very frequently with Amazon S3. Follow these steps to mount a bucket from s3 to local system using S3fs in UBUNTU 1. First of all you need to install the libraries that are needed by s3fs to run on the system. To install those libraries run the following command in console sudo aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs 2. After that Download the appropriate version of s3fs you want to use by the following link In this example we are using s3fs-1.68, and can be downloaded from this link 3. After downloading the s3fs go to that path in your terminal and run the following commands
  • tar xvzf s3fs-1.68.tar.gz
  • cd s3fs-1.68/
  • ./configure –prefix=/usr
  • make
  • make install (as root)
4. After this you need to supply your s3 credentials to s3fs to mount the bucket. There are different ways of doing that, you can choose whichever is suitable to you
  • By setting the AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables by using export command in UBUNTU. export AWSACCESSKEYID=******************* export AWSSECRETACCESSKEY=xxxxxxxxxxxx
  • By using a .passwd-s3fs file in your home directory. In this you just need to create the .passwd-s3fs file in home directory, and add the creds for s3 into it. The s3fs password file has this format (use this format if you have only one set of credentials): accessKeyId:secretAccessKey If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well: bucketName:accessKeyId:secretAccessKey
  • By using the system-wide /etc/passwd-s3fs file. In this you just need to create the passwd-s3fs file in /etc directory, and add the creds for s3 into it similar to the last point.
5. After that change the permissions of the password file
  • If you are using ~/.passwd-s3fs then permission should be 600
  • If you are using /etc/passwd-s3fs then permission should be 640
6. And last, to mount bucket(mybucket) to specific directory(/path/to/directory/) s3fs mybucket /path/to/directory/ 7. To unmount the mounted drive use this command. fusermount -u /path/to/directory/ 8. To automount s3 bucket to every time to local system after boot, edit the /etc/fstab file and add this at the end of the file s3fs#bucket_name /path/to/directory/ passwd_file=/path/to/passwd/file And after this edit /etc/rc.local and add the following line before exit 0. mount -a 9. If you get any error/warning during system boot mounting just add the option in the current command s3fs#bucket_name /path/to/directory/ fuse _netdev,passwd_file=/path/to/passwd/file This command will omit the warnings and mount the bucket to your local system. Now you can access Amazon S3 just like another folder on your file system.