Amazon S3 Sync files from Linux to Bucket
Thanks to https://tecadmin.net/
Install s3cmd Package
s3cmd is available in default rpm repositories for CentOS,RHEL and Ubuntu systems, You can install it using simply executing following commands on your system.
On CentOS/RHEL:
# yum install s3cmd
On Ubuntu/Debian:
$ sudo apt-get install s3cmd
On SUSE Linux Enterprise Server 11:
# zypper addrepo http://s3tools.org/repo/SLE_11/s3tools.repo
# zypper install s3cmd
Install Latest s3cmd using Source
If you are not getting latest version of s3cmd using above package managers, You can install last s3cmd version on your system using source code. Visit this url or use below command to download latest version of s3cmd.
$ wget http://ufpr.dl.sourceforge.net/project/s3tools/s3cmd/1.6.1/s3cmd-1.6.1.tar.gz
$ tar xzf s3cmd-1.6.1.tar.gz
Now install it using below command with source files.
$ cd s3cmd-1.6.1
$ sudo python setup.py install
Configure s3cmd Environment
In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.
After getting key files, use below command to configure s3cmd.
# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes
New settings:
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Encryption password: xxxxxxxxxx
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
Uses of s3cmd Command Line
Once configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.
1. List All S3 Bucket
Use following command to list all s3 buckets in your aws account.
# s3cmd ls
2. Creating New Bucket
To create a new bucket in Amazon s3 use below command. It will create bucket named tecadmin in S3 account.
# s3cmd mb s3://tecadmin
Bucket 's3://tecadmin/' created
3. Uploading file in Bucket
Below command will upload file file.txt to s3 bucket using s3cmd command.
# s3cmd put file.txt s3://tecadmin/
file.txt -> s3://tecadmin/file.txt [1 of 1]
190216 of 190216 100% in 0s 1668.35 kB/s done
4. Uploading Directory in Bucket
If we need to upload entire directory use -r to upload it recursively like below.
# s3cmd put -r backup s3://tecadmin/
backup/file1.txt -> s3://tecadmin/backup/file1.txt [1 of 2]
9984 of 9984 100% in 0s 18.78 kB/s done
backup/file2.txt -> s3://tecadmin/backup/file2.txt [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
Make sure you are not adding trailing slash in upload directory named backup (eg: backup/), else it will upload only content of backup directory only.
# s3cmd put -r backup/ s3://tecadmin/
backup/file1.txt -> s3://tecadmin/file1.txt [1 of 2]
9984 of 9984 100% in 0s 21.78 kB/s done
backup/file2.txt -> s3://tecadmin/file2.txt [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
5. List Data of S3 Bucket
List the objects of s3 bucket using ls switch with s3cmd.
# s3cmd ls s3://tecadmin/
DIR s3://tecadmin/backup/
2013-09-03 10:58 190216 s3://tecadmin/file.txt
6. Download Files from Bucket
Some times if we need to download files from s3 bucket, Use following commands to download it.
# s3cmd get s3://tecadmin/file.txt
s3://tecadmin/file.txt -> ./file.txt [1 of 1]
4 of 4 100% in 0s 10.84 B/s done
7. Remove Data of S3 Bucket
To remove files are folder from s3 bucket use following commands.
Removing file from s3 bucket
# s3cmd del s3://tecadmin/file.txt
File s3://tecadmin/file.txt deleted
Removing directory from s3 bucket
# s3cmd del s3://tecadmin/backup
File s3://tecadmin/backup deleted
8. Remove S3 Bucket
If we don’t need s3 bucket any more, we can simply delete it using following command. Before removing bucket make sure its empty.
# s3cmd rb s3://tecadmin
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty
Above command failed because of s3 bucket was not empty
To remove bucket first remove all objects inside bucket and then use command again.
# s3cmd rb s3://tecadmin
Bucket 's3://tecadmin/' removed
1. Syncing Files from Local => S3 Bucket
For example I want to sync my local directory /root/mydir/ to S3 bucket directory s3://tecadmin/mydir/ where tecadmin is bucket name. I have created some new files in /root/mydir/ and sync to s3 bucket using following command.
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
[code language=”plain”][/code]
2. Syncing Files from S3 Bucket => Local Directory
[code language=”plain”][/code]
[code language=”plain”][/code]
We can also used –preserve, –skip-existing and –delete-removed parameters during syncing files from S3 bucket to Local directory as followings.
# s3cmd sync s3://tecadmin/mydir/ --preserve /root/mydir/
# s3cmd sync s3://tecadmin/mydir/ --skip-existing /root/mydir/
# s3cmd sync s3://tecadmin/mydir/ --delete-removed /root/mydir/
Read more about s3cmd sync http://s3tools.org/s3cmd-sync
Some other useful commands:
<
p dir=”ltr”>Directory Size
du -m /some/path | sort -nr | head -n 20
Overall size:
du -sh /*
Add user to a folder
useraddd user -d /var -d /sbin/nologin
Leave a Reply