Tag Archive for 's3'

Installing FUSE + s3fs and sshfs on Ubuntu

Fuse is a program to mount ‘foreign’ filesystems to your computer. This is how I got an Amazon S3 (simple storage service) bucket mounted on my Ubuntu server as if it were a local folder:

Install the necessary libraries:
sudo aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs

We also installed the sshfs (SSH File System) in order to test out FUSE. Go ahead and create a mount folder and mount a remote drive to it:
sudo mkdir -p /mnt/sshfs
sudo chown user:fuse /mnt/sshfs
sudo sshfs user@hostname:/path/on/remote/server /mnt/sshfs

You should now be able to browse files on /path/on/remote/server, locally from /mnt/sshfs. Please note that you will need to set the ownership of the mount before you’ll be able to use it. The fuse group was created automatically and is a good place to start.

To unmount the drive:
fusermount -u /mnt/sshfs

Download the s3fs source (Revision 177 as of this writing) from the Google Code project:
wget http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz

Untar and install the s3fs binary:
tar xzvf s3fs-r177-source.tar.gz
cd ./s3fs
sudo configure
sudo make
sudo make install

In order to use the allow_other option (see below) you will need to modify the fuse configuration:
sudo vi /etc/fuse.conf

And uncomment the following line in the conf file:
...
#user_allow_other

Now you can mount an S3 bucket like this:
sudo mkdir -p /mnt/s3
s3fs bucketname -o accessKeyId=XXX -o secretAccessKey=YYY -o use_cache=/tmp -o allow_other /mnt/s3

You will need to replace the XXX above with your real Amazon Access Key and YYY with your real Secret Key. I’ve also told it to cache the bucket’s files locally (in /tmp) and to Allow other users to be able to manipulate files in the mount. Check the wiki documentation for more options available to s3fs, including how to save your Access Key and Secret Key in /etc/passwd-s3fs.

Now any files written to /mnt/s3 will be replicated to your Amazon S3 bucket.

Amazon announces SimpleDB

This is going to be huge! Combine it with their other services, and you have maximum scalability for any project. Media Temple (my current host) may lose me after all of this comes out of beta. I am already using S3 for backups and will begin rolling it into all of my upcoming podcast projects. I’ve been flirting with this Ubuntu AMI (Amazon Machine Image) on their EC2 platform, with good results. Keep ‘em coming, Amazon.

Amazon SimpleDB is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud. These services are designed to make web-scale computing easier and more cost-effective for developers.

Learn more and sign up here.