31 July, 2012

setting up live resize on ec2

ec2 doesn't let you do a live resize on an attached elastic block store; and the procedure for resizeng offline is a bit awkward - make a snapshot and restore that snapshot into a bigger EBS volume (here's a stack overflow article about that).

LVM lets you add space to a volume dynamically, and ext2 can cope with live resizing of a filesystem now. So if I was using LVM, I think I'd be able to do this live.

So what I'm going to do is:

  • firstly move this volume to LVM without resizing. This will involve downtime as it will be roughly a variant of the above-mentioned "go offline and restore to a different volume"
  • secondly use LVM to add more space: by adding another EBS to use in addition (rather than as a replacement) for my existing space; adding that to LVM; and live resizing the ext2 partition.

First, move this volume to LVM without resizing.

The configuration at the start is that I have a large data volume mounted at /backup, directly on an attached EBS device, /dev/xvdf.

$ df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdf              99G   48G   52G  49% /backup

in AWS web console, create a volume that is a little bit bigger than the volume i already have. so 105 gb. no snapshot. make sure its in same availability zone as the instance/other volume.

attach volume to instance, in the aws console.

on the linux instance, it should now appear:

$ dmesg | tail
[15755792.707506] blkfront: regular deviceid=0x860 major,minor=8,96, assuming parts/disk=16
[15755792.708148]  xvdg: unknown partition table
$ cat /proc/partitions 
major minor  #blocks  name
 202        1    8388608 xvda1
 202       80  104857600 xvdf
 202       96  110100480 xvdg
xvdg is the new EBS device.

Despite that dmesg warning, screw having a partition table - I'm using this as a raw device. It might suit your tastes at this moment to create partitions though, but it really doesn't matter.

Now I'm going to make that 105Gb on xvdg into some LVM space: (there's a nice LVM tutorial here if you want someone else's more detailed take)

 # pvcreate /dev/xvdg
  Physical volume "/dev/xvdg" successfully created
# vgcreate backups /dev/xvdg
  Volume group "backups" successfully created

Now we've created a volume group backups which contains one physical volume - /dev/xvdg. Later on we'll add more space into this backups volume group, but for now we'll make it into some space that we can put a file system onto:

# vgdisplay | grep 'VG Size'
  VG Size               105.00 GiB
so we have 105.00 GiB available - the size of the whole new EBS volume created earlier. It turns out not quite, so I'll create a logical volume with only 104Gb of space. What's a wasted partial-gigabyte in the 21st century?
# lvcreate --name backup backups --size 105g
  Volume group "backups" has insufficient free space (26879 extents): 26880 required.
# lvcreate --name backup backups --size 104g
  Logical volume "backup" created

Now that new logical volume has appeared and can be used for a file system:

$ cat /proc/partitions 
major minor  #blocks  name

 202        1    8388608 xvda1
 202       80  104857600 xvdf
 202       96  110100480 xvdg
 253        0  109051904 dm-0
# ls -l /dev/backups/backup
lrwxrwxrwx 1 root root 7 Jul 25 20:35 /dev/backups/backup -> ../dm-0
It appears both as /dev/dm-0 and as /dev/backups/backup - this second name based on the parameters we supplied to vgcreate and lvcreate.

Now we'll do the bit that involves offline-ness: I'm going to take the /backup volume (which is /dev/xvdf at the moment) offline and copy it into this new space, /dev/dm-0.

# umount /backup
# dd if=/dev/xvdf of=/dev/dm-0
This dd takes quite while (hours) - its copying 100gb of data. While I was waiting, I discovered that you can SIGUSR1 a dd process on linux to get IO stats: (thanks mdm)
$ sudo killall -USR1 dd
$ 41304+0 records in
41303+0 records out
43309334528 bytes (43 GB) copied, 4303.97 s, 10.1 MB/s

Once that is finished, we can mount the copied volume:

# mount /dev/backups/backup /backup
# df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/backups-backup
                       99G   68G   32G  69% /backup
Now we have the same sized volume, with the same data on it, but now inside LVM.

Second, add more space

Now we've got our filesystem inside LVM, we can start doing interesting things.

The first thing I'm going to do is reuse the old space on /dev/xvdf as additional space.

To do that, add it as a physical volume; add that physical volume to the volume group; allocate that new space to the logical volume; and then resize the ext2 filesystem.

These commands add the old space into the volume group:

# pvcreate /dev/xvdf
  Physical volume "/dev/xvdf" successfully created
# vgextend backups /dev/xvdf
  Volume group "backups" successfully extended

... and these commands show you how much space is available (by trying to allocate too much) and then add to the space:

# lvresize /dev/backups/backup -L+500G
  Extending logical volume backup to 604.00 GiB
  Insufficient free space: 128000 extents needed, but only 25854 available
# lvresize /dev/backups/backup -l+25854
  Rounding up size to full physical extent 25.25 GiB
  Extending logical volume backup to 129.25 GiB
  Logical volume backup successfully resized

Even though we've now made the dm-0 / /dev/backups/backup device much bigger, the filesystem on it is still the same size:

 df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/backups-backup
                       99G   68G   32G  69% /backup

But not for long...

Unfortunately:

# resize2fs /dev/backups/backup
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/backups/backup is mounted on /backup; on-line resizing required
old desc_blocks = 7, new_desc_blocks = 9
resize2fs: Kernel does not support online resizing
the version of the kernel on this host doesn't allow online resizing (some do). So I'll have to unmount it briefly to resize:
# umount /backup
# resize2fs /dev/backups/backup
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/backups/backup to 33882112 (4k) blocks.
The filesystem on /dev/backups/backup is now 33882112 blocks long.

# mount /dev/backups/backup /backup
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/backups-backup
                      128G   68G   60G  53% /backup
So there's the bigger fs. (though not as big as I had expected... I only seem to have got 30G extra worth of storage, not 100 as I was expecting...

Well it turns out that all the space wasn't allocated to this LV even though I thought I'd done that:

# vgdisplay
...
  Alloc PE / Size       33088 / 129.25 GiB
  Free  PE / Size       19390 / 75.74 GiB
...
but no matter. I can repeat this procedure a second time without too much trouble (indeed doing this procedure easily is the whole reason I want LVM installed...

Having done that, I end up with the expected bigger filesystem:

# df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/backups-backup
                      202G   68G  135G  34% /backup

Now whenever I want to add more space, I can repeat step 2 with just a tiny bit of downtime for that particular filesystem; and if I get round to putting on a kernel with online resizing (my raspberry pi has it, why doesn't this?) then I won't need downtime at all...

23 July, 2012

autogenerating reverse DNS for ipv6

I was getting annoyed by manually configuring an IPv6 reverse domain.

For reverse DNS, you need to break the IP address up into pieces (bytes for IPv4, nibbles for IPv6), reverse them, put dots between pieces, to get a domain name. Then at that domain name, you put a reference to the hostname for that IP.

So an IP address like 2001:8b0:7c:1:216:76ff:fe16:755a turns into a domain name a.5.5.7.6.1.e.f.f.f.6.7.6.1.2.0.1.0.0.0.c.7.0.0.0.b.8.0.1.0.0.2.ip6.arpa., and there you can find a PTR record pointing to the hostname dildano.hawaga.org.uk

Forming those long domain names was/is quite awkward, and its a task well suited to automation. All of the hosts already have forward DNS entries, so there's not even much additional information needed to generate the reverse zone.

I wrote a tool (in an unholy alliance of Haskell and dig) which queries a bunch of forward zones and outputs the appropriate reverse DNS records ready for pasting into a zone file.

You specify zones (and appropriate servers) that will be asked for AAAA records; then all of the AAAA records which refer to IPv6 addresses on the specified network will be converted into PTR records and sent to stdout, ready to paste into a zone file.

$ dnsrz hawaga.org.uk@dildano.hawaga.org.uk clifford.ac@malander.clifford.ac charlottevrinten.org@dildano.hawaga.org.uk mrsclifford.eu@malander.clifford.ac --prefix=200108b0007c0001
 
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR clifford.ac.
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR malander.clifford.ac.
3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR malander.mrsclifford.eu.
4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR fecolith.clifford.ac.
4.1.2.0.0.f.e.f.f.f.3.9.d.0.2.0 PTR pomade.clifford.ac.
a.5.5.7.6.1.e.f.f.f.6.7.6.1.2.0 PTR dildano.hawaga.org.uk.
c.0.2.a.4.c.e.f.f.f.3.6.b.1.2.0 PTR newsnowdrop.mrsclifford.eu.
0.a.0.c.b.a.e.f.f.f.3.6.1.2.2.0 PTR tenesmus.clifford.ac.
7.2.f.0.1.9.e.f.f.f.b.4.5.2.2.0 PTR coprolith.clifford.ac.
c.2.5.d.b.f.e.f.f.f.b.e.7.2.a.b PTR pygar.hawaga.org.uk.
c.2.5.d.b.f.e.f.f.f.b.e.7.2.a.b PTR pygar-6.hawaga.org.uk.
b.6.5.8.2.f.e.f.f.f.8.c.c.b.a.c PTR laptop.hawaga.org.uk.

I wanted to use the Haskell dns package which I've used a bit before; but it didn't have enough features: no zone transfer capability, for a start... so I invoke dig and parse that out.

The commandline syntax is: <zonename>@<DNS server> where zonename is a forward zone, and the specified server will answer AXFRs for that zone. Thats quite icky but it gets around needing a full Haskell DNS implementation.

The code is on github under benclifford/dnsrz.

(later: as fits my tradition of writing a tool and then finding someone has done something similar first, bind comes with a tool arpaname which will convert an IP address into a reverse name, though it doesn't do all the other stuff above, but does work for ipv4 too: http://ftp.isc.org/isc/bind9/cur/9.9/doc/arm/man.arpaname.html

18 July, 2012

first impressions of raspberry pi

I just got my Raspberry Pi, left on doorstep from Farnell in a package labelled "tshirt"; indeed inside the package was a (free) element14/pi tshirt, and wrapped inside that was my actual raspberry pi. Its taken a while to arrive (like most people's) - it was ordered on the 9th of May.

The first afternoon I spent playing with it came with an excited nostalgic feeling to my teenage years, scrabbling to put random pieces of hardware together scrounged around the house in order to get a working system: a keyboard off one host, a usb hub stolen from someone, a long ethernet wire borrowed from someone because the TV is in one room and the ethernet ports are in another. It was made more nostalgic because my best friend from secondary school who used to be my main partner in this kind of activity turned up to have a play too.

I fairly easily got the example debian distribution downloaded (my first linux distro downloaded by bittorrent) and running on a 4gb SD card; and once that was in place, it was easily to get a browser up and watch the bunnycam.

After giving up on getting flash running in order to get BBC iPlayer, I discovered get-iplayer (on the way getting annoyed by this blog post by a BBC dude about how people shouldn't use 3rd party clients). I can just about play BBC radio audio at the same time as the bunny webcam is streaming video. But it didn't seem to have enough power to decode an iplayer video stream instead of the bunny webcam, which was a bit disappointing.

There seem to be a few funny presumably hardware related problems: the device doesn't reboot at all, and sometimes when I boot, sound does not work at all. I haven't had time or inclination to track these down at all. Most concerning for me is that the experience of different devices seems slightly different - for example, other people seem to be able to reboot just fine.

After downloading the recommended Debian distro, I discovered that there was a beta release of a much more recent version, wheezy. This was a lot smoother experience than the recommended distro - it resized the image automatically to fill the whole 4g disk space, and just seemed more polished. Either way, those distributions are debian based, so the amazingness (if you're being nostalgic about 80s/90s era hacking) of apt comes into play to get a big range of software installed.

It was cool that the wheezy distribution came with Scratch - a visual programming language that is sort of a year-2012 version of Logo. (That's cool - one of the first things I thought about when I heard of the goals of the raspberry pi was to think: why don't you get people programming scratch instead?)

Something interesting I discovered was that the ARM architecture does floating point emulations completely differently to intel: there are different calling conventions for software floating point and hardware floating point, and it seems that all the packages in your distro need recompiling to make use of hardware floating point, if they were originally built for software floating point. I'll have a play with that later.

The GPIO port had pins soldered on, which i wasn't expecting. I'm not sure what to use those for. I've had a user port on a computer before, but never used them much - just for flashing LEDs and making model trains drive around. maybe some geeky weather station thing?

So what do you need to get started, in addition to the pi? Well a keyboard and video (composite or HDMI) and a power supply - I'm using the USB-based power supply I stole from my mifi. Probably you'd like a mouse and an ethernet connection.

If I get up to anything cool, I'll write about it on this blog, no doubt.

10 July, 2012

eye in the wrong place.

lights turn on automatically in shitter when you walk in. great! lights turn off after a while taking a dump. wave arms. its all dark. get out phone for light. realise sensor is *outside* the toilet room in sink area. wait for someone to walk by for light. wonder if I should submit technical fault for mislocated sensor?

03 July, 2012

dnstracer

I started getting into writing a program for checking all paths of a DNS query. Then I discovered that there are two (1, 2) tools, both called dnstracer around for doing that... no new ideas under the sun. I guess I'll have to make mine cooler.