Fuse is a program to mount ‘foreign’filesystems to your computer. This is how I got an Amazon S3 (simple storage service) bucket mounted on my Ubuntu server as if it were a local folder:
Install the necessary libraries:
sudo aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs
We also installed the sshfs (SSH File System) in order to test out FUSE. Go ahead and create a mount folder and mount a remote drive to it:
sudo mkdir -p /mnt/sshfs
sudo chown user:fuse /mnt/sshfs
sudo sshfs user@hostname:/path/on/remote/server /mnt/sshfs
You should now be able to browse files on /path/on/remote/server, locally from /mnt/sshfs. Please note that you will need to set the ownership of the mount before you’ll be able to use it. The fuse group was created automatically and is a good place to start.
To unmount the drive:
fusermount -u /mnt/sshfs
Download the s3fs source (Revision 177 as of this writing) from the Google Code project:
wget http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz
Untar and install the s3fs binary:
tar xzvf s3fs-r177-source.tar.gz
cd ./s3fs
sudo configure
sudo make
sudo make install
In order to use the allow_other option (see below) you will need to modify the fuse configuration:
sudo vi /etc/fuse.conf
And uncomment the following line in the conf file:
...
#user_allow_other
Now you can mount an S3 bucket like this:
sudo mkdir -p /mnt/s3
s3fs bucketname -o accessKeyId=XXX -o secretAccessKey=YYY -o use_cache=/tmp -o allow_other /mnt/s3
You will need to replace the XXX above with your real Amazon Access Key and YYY with your real Secret Key. I’ve also told it to cache the bucket’s files locally (in /tmp) and to Allow other users to be able to manipulate files in the mount. Check the wiki documentation for more options available to s3fs, including how to save your Access Key and Secret Key in /etc/passwd-s3fs.
Now any files written to /mnt/s3 will be replicated to your Amazon S3 bucket.
Thank you, Eric. I will be experimenting with this shortly.
Pingback: Use Amazon S3 online storage as an extra harddisk with Googles s3fs « Rforge
Pingback: Using Shopp’s API to create non-standard solutions « xentek
Hello Eric, and thank you for your instructions.
I am having problems with both sshfs and s3fs and would like to know if you’ve encountered similar problems.
Everyting is fine and dandy until I attempt to mount something using FUSE, which is when I get an error message:
# sshfs user@server:/path/ /tmp/fuse/
fuse: device not found, try ‘modprobe fuse’first
Running “modprobe fuse” does nothing whatsoever, no error messages, no output at all.
Ideas?
Did you try
sudo modprobe fuse?Thank, yes, I was root.
mans,
I was running into this problem on Ubuntu 9.04 EC2 instance, apparently there is a defect of sorts wrt fuse module. Only solution I found was upgrading to 9.10, where the problem disappears.
I’ve used fuse+s3fs successfully on 9.04, but this was a Slicehost image. Thanks for the heads up.
What’s interesting is that with the 9.04 ec2 image I had the same problem but with local 9.04 installs and internal servers it works fine. I’m assuming it’s a problem with the ec2 9.04 ami. Something to look into.
Also, you may want to try mounting it in your /mnt folder, which is intended for such things: http://www.linfo.org/mnt.html
Thank you,
unfortunately there was no difference.
Did you create an empty folder in the spot you want to mount first, and chown it to fuse:fuse before trying to mount it?
Nifty trick, thanks for sharing it!
It works amazingly well.
That it does. However you don’t need to mount and unmount using sudo.
Yeah, probably not. I just realized I’ve been overusing sudo and have started to curb my own use of it. Haven’t gone back and updated any of HOWTOs to reflect that.
Pingback: Installing FUSE + s3fs and sshfs on Ubuntu › ec2base
Pingback: Is there a way to mount a SSH connection in Linux/Ubuntu? - QuestionBin::Answer
Good job! It may be helpful in the process of studying ubuntu so I’ve scrape it using scrapebook(firefox addon). Can you recommend any books about ubuntu for me? yeah, I am a noob of ubuntu.
thanks in advance!
Great blog I’ve actually bookmarked it for learning ubuntu. Can you recommend any books about ubuntu for me? yep, I am a beginner of ubuntu. :] thanks in advance.
The sshfs works fine on (ami-714ba518) LTS 10.04 but the s3fs mounts does not. The s3 drive mounts, but nothing shows up in the directory (s3 chown user:fuse // ~# /usr/bin/s3fs bucket -o accessKeyId=xxx -o secretAccessKey=yyy /mnt/s3)
Curious if how to insert the sshfs into fstab, if possible –
Thanks!!
I haven’t upgraded to 10.04 on any of my servers yet (just started the upgrade on my desktop today). However, here’s my fstab recipe:
s3fs#BUCKET /mnt/s3 fuse allow_other,accessKeyId=XXX,secretAccessKey=YYY,use_cache=/tmp 0 0As always, replace BUCKET, XXX and YYY with your actual S3 Bucketname, access key and secret key.
I’ll revisit this article when I’ve upgraded one of my servers.
Eric –
Just a follow up –
Working Great!
Using sshfs I mounted my entire godaddy VPS to my EC2 instance. Made transferring my server a breeze.
S3 works fine now.
Thanks!
Awesome!
Hello, Thanks for the great post. I just did this and when I try to copy a file into the new share I get “No such file or directory”. However, I can cd into that directory with no problem, just can’t create or copy files there. I am root, and root is the owner and group, however I can’t change the perms because I again get “No such file or directory”. I’m running Ubuntu 9.10 on EC2. Any help greatly appreciated!
Thanks,
T
I got the mount working by first creating a s3 bucket with http://s3tools.org/s3cmd
Now all is well, although I am unable to change the owner and group of the mount. I’ll continue testing and see if my web app can write to the mount as is.
Thanks!
You have to set the permissions on the mount point (the folder you created before mounting s3). Unmount the drive, then set the permissions, then remount the drive and the permissions should stick. The docs on Fuse may help you sort this out: http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FAQ
Frequently when I copy files to S3 with S3FS it fails and says that there is not enough space on the device for me.
Does anybody have an idea what the problem is?
The default configuration shown here caches data to /tmp to speed up the access. You might be able to turn that off and then retry your copy. However, for best results you should leave a bit of space to perform the copy.
Pingback: Script for Backing up directories to Amazon S3 | FutureNode
Hi,
I’ve been using this guide for a while now and wanted to say thanks.
After having a bunch of data stuck in JungleDisk recently and working in a remote location with little bandwidth, firing up an EC2 instance with s3fs and jungledisk client got it all back into S3 relatively quickly.
Cheers!
Glad I could be of Service!
s3fs: option accessKeyId is no longer supported
outdated?
ok it was easy
1) SIMPLY store you credential in passwd-s3fs (chmod 640) in this form
assessID:secretKEY
2) mount your drive with this syntax: s3fs bucketname /mnt/localmountpoint
thanks
You’ve missed out ./configure before compiling and installing s3fs.
Thanks, Dean! I’ve corrected the article.
You mentioned over-use of sudo earlier in the comments, but a protip: When building/installing from source, you only need (and therefore, should only use) sudo for `make install` since it’s the only one that needs to write to root-owned directories. The `./configure` and `make` commands should only make writes in or below the current directory, which is likely owned by you, and only needs to read root-owned stuff out on the file system with permissions 644 or higher.
Don’t want to run an untrusted ./configure as root lest some jerk slipped an `rm -rf /` in there somewhere.
absolutely. all good points.
Pingback: Installing FUSE + s3fs and sshfs on Ubuntu « xentek - Записная книжка
Pingback: Installing FUSE + s3fs and sshfs on Ubuntu - Записная книжка