Ludo's notes

To content | To menu | To search

Tag - ubuntu

Entries feed - Comments feed

Thursday, February 9 2012

How to create and use an Amazon Web Services S3 bucket with s3cmd tool

AWS S3 crendentials

I had to create and use an AWS S3 bucket. I decided to use s3cmd tool to access this bucket.

You can find this tool at: • http://s3tools.org/s3cmd.

Install s3cmd
root@blackbox:~/ec2# apt-get install s3cmd
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following NEW packages will be installed:
  s3cmd
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 58.8 kB of archives.
After this operation, 303 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ oneiric/universe s3cmd all 1.0.0-1 [58.8 kB]
Fetched 58.8 kB in 1s (55.5 kB/s)
Selecting previously deselected package s3cmd.
(Reading database ... 154045 files and directories currently installed.)
Unpacking s3cmd (from .../archives/s3cmd_1.0.0-1_all.deb) ...
Processing triggers for man-db ...
Setting up s3cmd (1.0.0-1) ...
root@blackbox:~/ec2#
Check the command is there
root@blackbox:~/ec2# s3cmd
ERROR: /root/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.
root@blackbox:~/ec2#

Configure s3cmd

Get the information

To configure s3cmd, you need two information: • Access Key • Secret Key

You can get these credentials from: • https://aws-portal.amazon.com/gp/aws/securityCredentials

Run “s3cmd –configure” command

root@blackbox:~/ec2# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: __[Your Access Key]__
Secret Key: __[Your Secret Key]__

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: __[Your passphrase to protect your Secret Key]__
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: yes

New settings:
  Access Key: [Your Access Key]
  Secret Key: [Your Secret Key]
  Encryption password: [Your passphrase to protect your Secret Key]
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
root@blackbox:~/ec2#
Create the S3 bucket
root@blackbox:~/ec2# s3cmd mb s3://ludo$$
Bucket 's3://ludo9166/' created
root@blackbox:~/ec2# 
Put a public file on this bucket
root@blackbox:/tmp# s3cmd --acl-public put first-boot.sh s3://ludo9166
first-boot.sh -> s3://ludo9166/first-boot.sh  [1 of 1]
 1685 of 1685   100% in    0s     4.12 kB/s  done
Public URL of the object is: http://ludo9166.s3.amazonaws.com/first-boot.sh
root@blackbox:/tmp# 
Check the file is there
root@blackbox:/tmp# s3cmd ls s3://ludo9166
2012-02-09 03:26      1685   s3://ludo9166/first-boot.sh
root@blackbox:/tmp#
Get the file through s3cmd
root@blackbox:/tmp# s3cmd get s3://ludo9166/first-boot.sh
Get the file through basic http request
root@blackbox:/tmp# curl -O http://ludo9166.s3.amazonaws.com/first-boot.sh

Thursday, December 15 2011

Why does btrfs show up twice the capacity available?

If you decide to use the btrfs data (or/and metadata) replication, you will see the df command reports you your storage get used twice as fast as you expect. Btrfs doesn't act as for example mdadm. Each file will get striped at the block level across all devices available depending on the replication policy selected.

Create a new test file system

root@blackbox:~# mkfs.btrfs -m raid1 -d raid1 -L btrfs01 /dev/sdb /dev/sdc

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

adding device /dev/sdc id 2
fs created label btrfs01 on /dev/sdb
     nodesize 4096 leafsize 4096 sectorsize 4096 size 298.02GB
Btrfs Btrfs v0.19
root@blackbox:~#

Mount it

root@blackbox:~# mkdir /mnt/btrfs01

root@blackbox:~# mkdir /mnt/btrfs01
root@blackbox:~# mount /dev/sdb /mnt/btrfs01/

Check the capacity available

root@blackbox:~# df -h -F btrfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb              299G   56K  296G   1% /mnt/btrfs01
root@blackbox:~#

Create a new file (dd)

root@blackbox:~# dd if=/dev/zero of=/mnt/btrfs01/file.10G bs=1M count=$((10*1024))
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 176.188 s, 60.9 MB/s
root@blackbox:~#

Check how much space this file uses on the file system

root@blackbox:~# df -h -F btrfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb              299G   21G  276G   7% /mnt/btrfs01
root@blackbox:~#

root@blackbox:~# btrfs-show
Label: btrfs01  uuid: d2b77569-a8be-4ec7-862a-d0fc1e6c8511
     Total devices 2 FS bytes used 10.01GB
     devid    1 size 149.01GB used 12.03GB path /dev/sdb
     devid    2 size 149.01GB used 12.01GB path /dev/sdc

Btrfs Btrfs v0.19
root@blackbox:~#

Get the usage explanation from btrfs tool

root@blackbox:~# btrfs filesystem df /mnt/btrfs01/
Data, RAID1: total=11.00GB, used=10.00GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=14.11MB
Metadata: total=8.00MB, used=0.00
root@blackbox:~#

Thursday, December 8 2011

Ubuntu 11.10 XDMCP remote access from MAC OS X (Lion)

XDCMP Session from OS X to Ubuntu

Enable XDMCP on Ubuntu 11.10

root@blackbox:~# cp /etc/lightdm/lightdm.conf /etc/lightdm/lightdm.conf.$(date -I)
root@blackbox:~# cat /etc/lightdm/lightdm.conf

[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu

[XDMCPServer]
enabled=true
root@blackbox:~#

Restart X display manager

root@blackbox:~# restart lightdm
lightdm start/running, process 5684
root@blackbox:~#

Open X session from OS X to Ubuntu

ludo@idefix:~ % uname -a                                                                                                
Darwin idefix 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug  9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
ludo@idefix:~ %

ludo@idefix:~ % Xephyr :1 -screen 1024x768 -query blackbox