EC2 EBS persistent storage

EBS are EC2's persistent attachable storage block devices. Other then providing persistent storage, they have the benefit of being much faster than instance's internal storage.

EBS provide the following important features:

• Differential snapshots on S3
• Mirrored on two physical underlying storages
• Can be reattached to any other instance in the same availability zone
• EBS can be created with size ranging from 1GB to 1TB
• Striping across several EBS volumes with dm_raid or lvm can boost performance or give more space than 1TB
• Snapshots can be shared with other AWS users

EBS volumes have the following limitations:

• Can only be used on the same availability zone, to move to another EBS you must snapshot and spawn a new EBS
• Can't be resized directly, to resize you must snapshot and spawn another bigger EBS
• Can't be shared between instances
• A Snapshot can't be mounted without spawning another EBS

Although it's possible to create a filesystem directly on the EBS without creating a partition, most filesystem tools expect the filesystem to be inside a partition.

Snapshots to S3

The snapshotting process of EBS volumes to S3 is low level, external and independent of the instance attached to the volume. This means all properties and metadata (like UUID and LVM metadata) of the volume will be the same on any new volumes spawned from the snapshot

After the snapshot commenced, writes are separated from the current volume and the EBS is copied to S3 while the volume is operational in what appears to be copy-on-write mode.

Performance

When snapshotting the volume is usable but serious performance degradation is inflected. This should be taken into account when designing backup schemes because snapshot processes can last several hours.

Read performance: Write performance:

Filesystem consistency

Unlike LVM snapshots, the snapshotting mechanism in EC2 knows nothing about the filesystem and therefor cannot force IO buffers to flush and verify that the filesystem is in a consistent state. To maintain consistency, the filesystem must be put into a consistent state manually. There are several ways to achieve consistency:

• unmount the filesystem for the duration of the ec2-create-snapshot command or API call
• freeze the filesystem with xfs_freeze if it's an XFS filesystem
• freeze the filesystem with dmsetup or LVM

For most systems, unmounting is not a viable option because it requires programs using the filesystem to release files - which usually means to shut down. Freezing the filesystem is probably the best approach but requires either XFS or devicemapper/LVM.

• XFS freeze - The filesystem must be created as XFS. To freeze and snapshot use the following commands:

  xfs_freeze -f /EBS/vol-to-snapshot
  ec2-create-snapshot vol-id
  xfs_freeze -u /EBS/vol-to-snapshot
  

• dmsetup/LVM - The filesystem must be created over device mapper or LVM. To create over device mapper use:

  echo "0 $(blockdev --getsize /dev/sdh) linear /dev/sdh 0" |dmsetup create mapped_name
  mkfs -t ext3 /dev/mapper/mapped_name
  

  To suspend the filesystem use dmsetup suspend /dev/mapper/mapped_name and to unfreeze dmsetup resume /dev/mapper/mapped_named

Automount EBS volumes

Usually EBS volumes are attached to an instance with the ec2-attach-volume command and then mounted with mount. However, on elastic systems incorporating autoscaling and spawning of instances manual mounts can be cumbersome. Auto mounting with UDEV provides an easy solution to this situation:

First add the following UDEV rule

# cat /etc/udev/rules.d/61-ebs.rules 
SUBSYSTEM=="block", DRIVERS=="vbd", ATTR{start}=="?*", ACTION=="add", RUN="/bin/mount -U %E{ID_FS_UUID}"

Now define the mount on /etc/fstab with the volume's UUID:

UUID=afcf6b48-3335-4c76-45fc-4b7fb59edb72   /mnt/EBS1   ext3   noauto,defaults   0   0

don't forget to reload udev.

The UUID of the volume can be extracted with blkid or vol_id

-- AvishaiIshShalom - 05 Nov 2009