Saving Time and Money with Fake S3
Amazon S3 is a extremely powerful service at the core of Amazon Web Services. However, outside of a production environment, S3 can be challenging to work with. It involves passing keys around, provisioning user accounts, and a reliable network connection — not to mention it costs money.
At Spool, we built Fake S3 to make working with S3 in development and testing environments much easier. Our goal was to make a self contained executable that can mimic the majority of S3 Rest API with few external dependencies.
For development, each engineer runs her own instance of Fake S3 where she can put gigabytes of images and video to develop and test against, and her setup will work offline because it is all local. We also have a continuous integration setup that is running tests 24/7 (often against large video files). Fake S3 saves us $1000 a month in bandwidth alone for our tests. In both development and testing, the time saved in not waiting for assets to go back and forth into AWS, especially our larger media files, makes Fake S3 very useful.
We’re releasing Fake S3 as a gem on github. It’s an early release and we’ll keep improving it. If you have ideas or issues, please contribute to the project!
Installation
gem install fakes3
Running
fakes3 -r ~/fakes3_root -p 10001
Example Client Code
require 'rubygems'
require 'aws/s3'
include AWS::S3
AWS::S3::Base.establish_connection!(:access_key_id => "123",
:secret_access_key => "abc",
:server => "localhost",
:port => "10001")
Bucket.create('mystuff')
('a'..'z').each do |filename|
S3Object.store(filename, 'Hello World', 'mystuff')
end
bucket = Bucket.find('mystuff')
bucket.objects.each do |s3_obj|
puts "#{s3_obj.key}:#{s3_obj.value}"
end
Bucket.delete("mystuff",:force => true) # Delete your bucket and all its keys
In general clients can work if you can specify a host and port to connect them, as well as forcing path style requests (instead of subdomains.) Subdomain style S3 requests can work, but it involves adding your bucket names into /etc/hosts (ie. s3.localhost or mybucket.localhost) or using dnsmasq if you have a large number of buckets.
Simulating Network Conditions
Another useful feature is the support for simulating network conditions. You can run fakes3 with bandwidth limiting enabled like so:
fakes3 -r ~/fakes3_root -p 10001 --limit=50K
This will limit your GET request bandwidth to 50K/s per request instead of instantly off of your local machine. This is very convenient, for example, to simulate how mobile devices would behave in the real world.
Related Tools
Fake S3 is great for development and testing due to its simplicity, but is not intended to replace S3 in production. If you want to replace S3, there are other tools such as Ceph, ParkPlace (supports bitorrent), Boardwalk (S3 interface in front of MongoDB), and RiakCS that you can check out.
spoolblog.wordpress.com 