Ludo's notes

To content | To menu | To search

Tag - aws

Entries feed - Comments feed

Thursday, February 9 2012

How to create and use an Amazon Web Services S3 bucket with s3cmd tool

By Ludovic Francois on Thursday, February 9 2012, 05:11 - unix

AWS S3 crendentials

I had to create and use an AWS S3 bucket. I decided to use s3cmd tool to access this bucket.

You can find this tool at: • http://s3tools.org/s3cmd.

Install s3cmd
root@blackbox:~/ec2# apt-get install s3cmd
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following NEW packages will be installed:
  s3cmd
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 58.8 kB of archives.
After this operation, 303 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ oneiric/universe s3cmd all 1.0.0-1 [58.8 kB]
Fetched 58.8 kB in 1s (55.5 kB/s)
Selecting previously deselected package s3cmd.
(Reading database ... 154045 files and directories currently installed.)
Unpacking s3cmd (from .../archives/s3cmd_1.0.0-1_all.deb) ...
Processing triggers for man-db ...
Setting up s3cmd (1.0.0-1) ...
root@blackbox:~/ec2#
Check the command is there
root@blackbox:~/ec2# s3cmd
ERROR: /root/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.
root@blackbox:~/ec2#

 

Configure s3cmd

Get the information

To configure s3cmd, you need two information: • Access Key • Secret Key

You can get these credentials from: • https://aws-portal.amazon.com/gp/aws/securityCredentials

Run “s3cmd –configure” command

root@blackbox:~/ec2# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: __[Your Access Key]__
Secret Key: __[Your Secret Key]__

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: __[Your passphrase to protect your Secret Key]__
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: yes

New settings:
  Access Key: [Your Access Key]
  Secret Key: [Your Secret Key]
  Encryption password: [Your passphrase to protect your Secret Key]
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
root@blackbox:~/ec2#
Create the S3 bucket
root@blackbox:~/ec2# s3cmd mb s3://ludo$$
Bucket 's3://ludo9166/' created
root@blackbox:~/ec2#
Put a public file on this bucket
root@blackbox:/tmp# s3cmd --acl-public put first-boot.sh s3://ludo9166
first-boot.sh -> s3://ludo9166/first-boot.sh  [1 of 1]
 1685 of 1685   100% in    0s     4.12 kB/s  done
Public URL of the object is: http://ludo9166.s3.amazonaws.com/first-boot.sh
root@blackbox:/tmp#
Check the file is there
root@blackbox:/tmp# s3cmd ls s3://ludo9166
2012-02-09 03:26      1685   s3://ludo9166/first-boot.sh
root@blackbox:/tmp#
Get the file through s3cmd
root@blackbox:/tmp# s3cmd get s3://ludo9166/first-boot.sh
Get the file through basic http request
root@blackbox:/tmp# curl -O http://ludo9166.s3.amazonaws.com/first-boot.sh

no comment no trackback

Wednesday, January 18 2012

FreeBSD on EC2 Amazon Web Services

By Ludovic Francois on Wednesday, January 18 2012, 09:54 - unix

AWS Console with FreeBSD instance running

As now the only way to have FreeBSD running on Amazon is to have it running with the Windows AMIs.

Find FreeBSD instance

root@blackbox:~/ec2# ec2-describe-images -a | grep -i freebsd | grep -iv windows
IMAGE   ami-0d08cc64    118940168514/FreeBSD/EC2 7.4b-RELEASE Cluster Compute   118940168514    available       public          x86_64  machine                             ebs     hvm     xen
IMAGE   ami-d29b6abb    118940168514/FreeBSD/EC2 8.2-RC1        118940168514    available       public          i386    machine aki-407d9529
                        ebs     paravirtual     xen
IMAGE   ami-4a15e423    118940168514/FreeBSD/EC2 8.2-RC1.5      118940168514    available       public          i386    machine aki-407d9529
                        ebs     paravirtual     xen
IMAGE   ami-c003f3a9    118940168514/FreeBSD/EC2 8.2-RC3        118940168514    available       public          i386    machine aki-407d9529
                        ebs     paravirtual     xen
IMAGE   ami-423bc82b    118940168514/FreeBSD/EC2 8.2-RELEASE    118940168514    available       public          i386    machine aki-407d9529
                        ebs     paravirtual     xen
IMAGE   ami-646c9e0d    118940168514/FreeBSD/EC2 8.2a-RELEASE amd64/HVM 118940168514    available       public          x86_64  machine    ebs      hvm     xen
IMAGE   ami-290eca40    118940168514/FreeBSD/EC2 8.2b-RELEASE Cluster Compute   118940168514    available       public          x86_64  machine                             ebs     hvm     xen
IMAGE   ami-b55f99dc    118940168514/FreeBSD/EC2 8.2b-RELEASE i386/XEN  118940168514    available       public          i386    machine aki-407d9529                        ebs     paravirtual     xen
IMAGE   ami-c01aeca9    118940168514/FreeBSD/EC2 9.0-CURRENT 2010-12-12 118940168514    available       public          i386    machine aki-407d9529                        ebs     paravirtual     xen
IMAGE   ami-a0fc0dc9    118940168514/FreeBSD/EC2 9.0-CURRENT 2010-12-29 118940168514    available       public          i386    machine aki-407d9529                        ebs     paravirtual     xen
IMAGE   ami-f4db2a9d    118940168514/FreeBSD/EC2 9.0-CURRENT 2011-01-01 118940168514    available       public          i386    machine aki-407d9529                        ebs     paravirtual     xen
IMAGE   ami-8cce3fe5    118940168514/FreeBSD/EC2 9.0-CURRENT 2011-01-04 118940168514    available       public          i386    machine aki-407d9529                        ebs     paravirtual     xen
IMAGE   ami-479f482e    118940168514/FreeBSD/EC2 9.0-RELEASE Cluster Compute    118940168514    available       public          x86_64  machine                             ebs     hvm     xen
IMAGE   ami-7a41b713    844886073610/FreeBSD-9 Base     844886073610    available       public          i386    machine aki-407d9529       ebs      paravirtual     xen
IMAGE   ami-c841b7a1    844886073610/FreeBSD-9 Python   844886073610    available       public          i386    machine aki-407d9529       ebs      paravirtual     xen
root@blackbox:~/ec2#

Start your FreeBSD instance

root@blackbox:~/ec2# ec2-run-instances ami-7a41b713 -k ${EC2_KEYPAIR} -t t1.micro
RESERVATION     r-79e5d618      955701963229    default
INSTANCE        i-cec7c0ac      ami-7a41b713                    pending ec2-keypair     0               t1.micro        2012-01-18T18:17:21+0000    us-east-1a      aki-407d9529                    monitoring-disabled                                     ebs                        paravirtual      xen             sg-7ac33e12     default
root@blackbox:~/ec2#

Check the instance is running

root@blackbox:~/ec2# ec2-describe-instances i-cec7c0ac
RESERVATION     r-79e5d618      955701963229    default
INSTANCE        i-cec7c0ac      ami-7a41b713    ec2-23-20-17-21.compute-1.amazonaws.com ip-10-244-135-190.ec2.internal  running ec2-keypair0
                t1.micro        2012-01-18T18:17:21+0000        us-east-1a      aki-407d9529                    monitoring-disabled     23.20.17.21 10.244.135.190                  ebs                                     paravirtual     xen             sg-7ac33e12     default
BLOCKDEVICE     /dev/sda1       vol-094d7364    2012-01-18T18:18:05.000Z
BLOCKDEVICE     /dev/sdb        vol-0b4d7366    2012-01-18T18:18:05.000Z
root@blackbox:~/ec2# 
0*$ zsh

Wait few minutes to get the instance fully loaded and ssh

root@blackbox:~/ec2# ssh -i ec2-keypair ec2-23-20-17-21.compute-1.amazonaws.com

If you are not able to login check what is going on in the console

root@blackbox:~/ec2# ec2-get-console-output i-cec7c0ac
i-cec7c0ac
2012-01-18T18:20:55+0000
Xen Minimal OS!
  start_info: 0xa01000(VA)
    nr_pages: 0x26700
  shared_inf: 0xdf617000(MA)
     pt_base: 0xa04000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x967000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x946780-0x966780
MM: Init
      _text: 0x0(VA)
     _etext: 0x621f5(VA)
   _erodata: 0x76000(VA)
     _edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
       _end: 0x966d34(VA)
  start_pfn: a10
    max_pfn: 26700
Mapping memory range 0xc00000 - 0x26700000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for b3e000(b3e000)-0(26700000)
MM: done
Demand map pfns at 26701000-36701000.
Heap resides at 36702000-76702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x26701000.
Initialising scheduler
Thread "Idle": pointer: 0x36702008, stack: 0xbf0000
Initialising xenbus
Thread "xenstore": pointer: 0x36702478, stack: 0x26600000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x367028e8, stack: 0x26610000
"main" "root=/dev/sda1" "ro" "4" 
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/524/2049
Failed to read /local/domain/0/backend/vbd/524/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/524/2049/feature-flush-cache.
2097152 sectors of 0 bytes
**************************
vbd 2064 is hd1
******************* BLKFRONT for device/vbd/2064 **********


backend at /local/domain/0/backend/vbd/524/2064
Failed to read /local/domain/0/backend/vbd/524/2064/feature-barrier.
Failed to read /local/domain/0/backend/vbd/524/2064/feature-flush-cache.
18874368 sectors of 0 bytes
**************************
[H[J  Booting 'FreeBSD kernel'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/kernel/kernel vfs.root.mountfrom=ufs:da1s1,machdep.idle_mwait=0,bo
ot_verbose=1

xc_dom_probe_bzimage_kernel: kernel is not a bzImage
close blk: backend at /local/domain/0/backend/vbd/524/2049
close blk: backend at /local/domain/0/backend/vbd/524/2064
WARNING: loader(8) metadata is missing!
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
APIC: Using the MPTable enumerator.
SMP: Added CPU 0 (BSP)
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #68: Sun Dec 12 03:52:29 UTC 2010
    root@chch.daemonology.net:/usr/obj/i386.i386/usr/src/sys/XEN i386
WARNING: WITNESS option enabled, expect reduced performance.
Xen reported: 2659.998 MHz processor.
Timecounter "ixen" frequency 1953125 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz (2660.00-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x1067a  Family = 6  Model = 17  Stepping = 10
  Features=0xbfe3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>

Instruction TLB: 4 KB Pages, 4-way set associative, 128 entries
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 6144 kbytes, 16-way associative, 64 bytes/line
real memory  = 644874240 (615 MB)
Physical memory chunk(s):
0x00000000007d0000 - 0x0000000025b4dfff, 624418816 bytes (152446 pages)
avail memory = 621105152 (592 MB)
[XEN] IPI cpu=0 irq=128 vector=RESCHEDULE_VECTOR (0)
[XEN] IPI cpu=0 irq=129 vector=CALL_FUNCTION_VECTOR (1)
Event-channel device installed.
random: <entropy source, Software, Yarrow>
io: <I/O>
mem: <memory>
Pentium Pro MTRR support enabled
null: <null device, zero device>
nfslock: pseudo-device
[XEN] xen_rtc_probe: probing Hypervisor RTC clock
rtc0: <Xen Hypervisor Clock> on motherboard
[XEN] xen_rtc_attach: attaching Hypervisor RTC clock
rtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s)
xs_probe: Probe retuns 0
xenstore0: <XenStore> on motherboard
Grant table initialized
xc0: <Xen Console> on motherboard
Device configuration finished.
procfs registered
Timecounters tick every 10.000 msec
[XEN] hypervisor wallclock nudged; nudging TOD.
lo0: bpf attached
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: bpf attached
xn0: Ethernet address: 12:31:3d:00:80:50
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xbd0: 1024MB <Virtual Block Device> at device/vbd/2049 on xenbusb_front0
xbd0: attaching as da0
GEOM: new disk da0
xbd1: 9216MB <Virtual Block Device> at device/vbd/2064 on xenbusb_front0
xbd1: attaching as da1
WARNING: WITNESS option enabled, expect reduced performance.
GEOM: new disk da1
command 0xc2ab1800 not in queue, flags = 0xdeadc0de, bit = 0x10
panic: command not in queue
cpuid = 0
KDB: enter: panic
[ thread pid 12 tid 100024 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why
db> 
root@blackbox:~/ec2#

no comment no trackback