Archive for linux

A conundrum – ESXi and solaris zfs

Despite the change to a full hypervisor I wanted to keep using zfs to manage the storage of datafiles. Partly because it seems like a great system and the reporting and management is excellent and simple and partly because that’s what I started out with and I’m too bloody minded to change.

One of my inspirations for this came from solarisinternals.com who have moved to a similar setup.  There they have setup zfs to access the disks on esx server using ‘raw disk mode’ which is obviously exactly what I want isn’t it? This would be my preferred route. I know it goes against the virtual ethos but I could whack the drives into any solaris box and ‘zfs import’ them in the event of a hardware failure. I gives me options to keep the drives without having to offload all the data and re-format; if I run into problems with ESXi in the future.

Well NOT SO FAST…..it’s a case of needed to do the research and read the small print. ESXi 4.0 doesn’t support virtual machines accessing raw disk devices (‘raw disk mode’). It seems that older versions (or maybe other vmware products do or did). RDM or ‘Raw disk Mapping’ is a supoprted option in ESXi 4 but that refers to mapping onto raw disks over a SAN (NOT LOCALLY).

I have created an opensolaris 2009.06 virtual machine running on the hypervisor. The root pool or system disk of this is infact a .vmdk file sitting on the internal mirrored pair of drives sitting in the server. My intention was to add additional drives that would be managed directly under opensolaris. BUT this just doesn’t seem possible…ESXi 4.0 doesn’t allow raw device access or direct disk access.

Research is ongoing I have two choices it seems.

1. Use the hardware raid capabilities of the SAS/SATA RAID cards – Then just use ZFS to manage quotas/snapshots and management stuff. BUT I’m nervous about recovering these should a controller fail (I’m left with a situation where in order to recover the data I’d need to buy a very specific and very expensive RAID controller – or wait ages until the right thing came up on ebay). Also RAID-Z in ZFS removes write-hole errors.
2. Create virtual disks on the actual disks and use zfs to manage these as if they were actual disks. I guess I can see a disaster recover route for this option. The disk could be hooked out and connected to any sata controller and then read from within ESXi (I think) but I need to check that. This would have to be slower wouldn’t it?
3. Forget zfs completely. Use hardware RAID and create another virtual machine which is a small footprint freenas or similar box let that take care of all the file serving work. Still need to think about a possible route for recovering the data in the event of a hardware failure.
4. Find work around – there is always a work around!
5. Sod it – switch to hyper-v which does seem to support it!

ESXi access local disks as a raw device – workarounds
1. Use vmkfstools: There do seem to be worked examples
here………http://www.hardforum.com/showthread.php?t=1441318
and here…http://www.hardforum.com/showthread.php?t=1429640
2. Edit configuration files by hand
discussed here… http://communities.vmware.com/thread/145589?tstart=0&start=0

Decisions, decisions…

Comments (140)

LINUX HOME SERVER – Storage setup

The main job of the home server is to secure the storage of family photos and other files I really don’t want to loose and to share them over the network to any PC in the house. The storage in the server is held as identical 250GB drives in a 4-way ICYDOCK disk carrier.

This is connected to a SATALINK Sil3114 4-port SATA PCI – it’s only SATA-I (1.5Gbps) so I don’t expect blistering performance. In addition although it 4-port apparently it’s only a 2 channel card with a port multiplier. What can I say the card including postage was less than £15 (including postage)!

I decided that although it’s expensive on disks I should make the storage RAID-1. Although the Sil3114 card does allow you to create ‘HARDWARE’ RAID devices in the SATA card’s BIOS this is not true RAID but a ‘fakeraid’ (apparently).

After some research the recommend solution in LINUX is to use the very mature software raid capability – since I want to manage a number of filesystems for different items (videos,music etc.) possibly resize them and modify the allocation of space – I decided to go the route of LVM on top of RAID. There are loads of howto guides and blog entries explaining how to do this. I’ve really created this entry to document the setup for myself.

HARDWARE:

Trying to be organised….I fiddled around with the wiring so that the top drive is drive 0 on the controlled, – 1,2 & 3 go down the carrier (4 at the base). Also, to try to get maximum performance, I arranged the RAID drives so that there is one drive on each channels in a mirrored pair. So drive 0 & 2 are a pair as are drives 1 & 3.

SOFTWARE RAID

This guide gave me all I needed to know about creating a RAID-1 device from two underlying physical drives (http://www.devil-linux.org/documentation/1.0.x/ch01s05.html).

1. First fdisk the drives to prepare them;
I set a single primary partition up on each disk to occupy the whole space available and set the partition type to ‘Linux RAID autodetect’ (fd).

2. Prepare the mdmadm.conf file
echo ‘DEVICE /dev/sd[bcde]1’ > /etc/mdadm.conf

3. Use mdadm to create a RAID-1 device:
mdadm –create /dev/md0 –chunk=64 –level=raid1 –raid-devices=2 /dev/sdb1 /dev/sdc1

4. Save the information about the just created array(s)
# mdadm –detail –scan >> /etc/mdadm.conf

So in the end my mdadm.conf looks like this…
DEVICE /dev/sd[bcde]1
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=d016aaee:80f4911b:acb5d753:fd2968f0

5. Waited for the RAID device to be built..don’t reboot until it’s done!
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
245111616 blocks [2/2] [UU]
[>………………..]  resync =  2.1% (5344192/245111616) finish=11256.6min speed=353K/sec

Christ this doesn’t look good 353k/sec in SLOW and 11256.6Min is a LONG TIME.

Oh hang on 2 minutes later…
# tail /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
245111616 blocks [2/2] [UU]
[==>………………]  resync = 13.8% (33990784/245111616) finish=72.0min speed=48838K/sec

6. Worry about the disk performance
The second /proc/mdstat doesn’t look so bad but it’s not 1.5 Gb/sec is it – I must do some benchmarking to see if there is an underlying problem with the disk/controller setup!

7. Do some Benchmarking
Once the RAID device (/dev/md0) was built I rebooted to check it would come back up as a RAID array – everything seems OK. Then I tried testing the performance of the raid device /dev/md0 and the underlying drives (sdb and sdc). It’s not very thorough but I used ‘hdparm -tT’ to get and estimate of the disk performance;

Disk                                          Device         Cached Reads    Buffered Reads
(system disk 5400rpm 2.5″)     /dev/sda      2190                 35
(storage disk 1 7200rpm 3.5″) /dev/sdb      2120                 66
(storage disk 2 7200rpm 3.5″) /dev/sdc      2115                 68
(raid device software raid)       /dev/md0     2060                 66

It turns out that the storage drives as a RAID aren’t too bad – from what I can gather through the magic of google that kind of value for a SATA-I drive of that vintage is OK.
The system drive result is disappointing – I guess it reflects trade-off in performance that you suffer when you choose a fairly old-style mobile hard drive to decrease power consumption. It still seems low to me but not low enough to start exploring AHCI versus IDE and all that mularky.

SOFTWARE – LVM setup

Firstly according to this guy’s blog “As I learnt the hard way, the LVM detection happens before RAID”. His instructions call for you to modify the filter setting in the /etc/lvm/lvm.conf file. I’ve created a filter for my specific setup:

filter = [ "r|/dev/cdrom|", "r|/dev/sd[bcde]1|", "a|/dev/md[01]|" ] Now go through the LVM set-up of the /dev/md0 raid array 1. Create physical volume

#pvcreate /dev/md0
Physical volume “/dev/md0” successfully created

2.  Create the Volume group

#vgcreate array1 /dev/md0
Volume group “array1” successfully created

3.  Create the logical volumes (the actual ‘drives/devices’ visible to the user)

I couldn’t really decide on what volumes to create I guess the thing about using LVM is that I don’t have to. To help manage I wanted to have separate volume for different items (one for picture, one for videos, one for music and one for ‘general’). Given the total storage available of 250GB I decided on an initial layout of;
General – 25GB
Pictures – 50GB
Music    – 50GB
Videos  – 125GB

So I set them up like this…

#lvcreate –name general –size 25G array1
Logical volume “general” created
#lvcreate –name pictures –size 50G array1
Logical volume “pictures” created
#lvcreate –name music –size 50G array1
Logical volume “music” created
#lvcreate –name videos –size 108G array1
Logical volume “videos” created

Ah yes the difference between the inflated way disk salesmen measure capacity and the miserly way linux does have lost me nearly 20GB somewhere (nevermind).

4. Finally create the filesystems
Now the logical volumes exist the filesystems need to be created . Sod researching to very best type of filesystem I just set them up as ext3….

#mkfs.ext3 -m 0 -T largefile /dev/array1/videos
#mkfs.ext3 -m 0 /dev/array1/pictures
#mkfs.ext3 -m 0 /dev/array1/music
#mkfs.ext3 -m 0 /dev/array1/general

They all have no space allocated for root (hence -m 0) and the video filesystem is optimised for large files. All seems well (no error messages anyway). Ok that’s it should be done – when I reboot it will all come back OK (fingers crossed).

When I rebooted it all came back and I ran the LVM tool that comes with CENTOS – it all seems OK (not that I’m at all suprised!)

 

OK so far so good! Using the LVM tool I created mount points for the volumes and got them to be mounted at boot.

Comments (147)

LINUX HOME SERVER – power consumption measurements

With the set-up of my LINUX HOME SERVER I choose components in the hope to minimise the power consumption (at the obvious expense of performance) of the system. The cost of electricity isn’t going down and I wanted to try and avoid an enormous bill each year just for having the server idel for long periods of time. As well as choosing power saving components I disabled anything in the BIOS that wasn’t going to be used (e.g sound, serial and parallel devices) in an attempt to reduce power consumption still further.

First I made a sort of back of the envelope calculation at what the maximum power consumption should be;
31W CPU (published TDP of the chip)
25W Motherboard/chipset (can’t remember where I saw this! – oh yes here…http://www.silentpcreview.com/forums/viewtopic.php?p=285356)
4W System disk
2W RAM
16W 2x Storage disks
78W TOTAL POWER CONSUMPTION BY COMPONENTS
104W Allowing for PSU 75% efficiency

That’s my target but I don’t have to guess because I borrowed a plug-in energy monitor that tells me the power consumption of any appliance.
 

So what the verdict?   Actually when I booted the machine for the first time the power consumption spiked at 105W (blimey that’s uncanny!) for a second or so then subsided to a steady 60W when the machine is totally idle (just sitting at the login screen). I should note that the server used 4W when turned off.

I wanted to dig a bit deeper and estimate the contribution of individual components the overall idle power consumption. so I started to systematically remove all the components I could
1. Turned off – 4W
2. Maximum recorded – 105W 
3. At Idle – 60W
4. As 3 but with the two storage disk removed – 42W
5. As 4 but with the ICYDOCK cage unplugged – 42W
6. As 5 but with the PCI SATA card removed –  39.5W*
7. As 6 but with a Hauppauge Nova-T tuner card installed  – 44.5W*
8. As 3 but with a Hauppauge Nova-T tuner card installed  –  65.5W*
The only other things I could unplug would be the cooling fans and I didn’t fancy doing that. 

*The meter measures to the nearest Watt – but in some readings the value oscillated up and down by 1W.

Overall I guess I’m pretty happy with 60W at idle. I saw a review of Windows Home Server Machines where they measured power consumption and it’s pretty comparable (the range was 40 to 55W) if a little on the high side. This is acceptable considering my server is a much more capable machine that the average WHS offering. I guess the most striking results are;
— Just how much power Hard Drives consume…my estimate is that each drive uses 9W of power just sitting idle.
— Having a PCI card sitting in the machine that I’m not using (the Hauppauge card) is undoing alot of savings made elsewhere – the card uses about 5 or 6W of power and is just sitting there doing nothing at the moment.
— Although not all PCI cards are the same (obvious I guess) they all consume a few Watts – the PCI SATA card is estimated to use about 2.5W while the PCI DVB-T tuner card about 5/6W. It occurs to me that a PCI-express based card (which operate at lower voltages) should consume less power – it would be interesting to do a comparison using a Sil3114 PCI-e card (if such a thing exists) or a PCI-e based TV tuner.

Cost implications…
Well if it runs 24 hours a day at 60W (idle) then my bill will be increased by about £65.

I’ve based this on my simple calculation;
0.06Kw * 24 = 1.44 kWh (kiloWatt Hour) per day
1.44 * 365 = 525.60 Total kWh per year
£0.1236  * = £64.96 Total cost of running the computer

Next steps….
I guess I must see if I can make use of the mobile style power saving capabilities of the setup yet more by using power management to power down most if not all of the system if it sits idel for long periods of time. I guess the obvious place to start is with the hard disks since they use so much juice.

Comments (149)

LINUX HOME SERVER

I bought this old rack-mount chassis off ebay for a song (about a tenner to you and me) and started to think about setting up a server for general duties in the Ultimate Wired Home[TM] – you know sitting there doing nothing for days on end then serving up a couple of kb in old family pictures once in a blue moon.
 

Since it will almost certainly be sitting idle for long periods I thought I’d treat myself and go for a low-power MODT (mobile on desktop) system.

Motherboard – Gigabyte 8I945GMMFY-RH
CPU – Intel Core Duo T2500
RAM – 1GB (single channel)
HDD – 80GB (mobile 2.5″) SAMSUNG HM080HI drive (SATA 1 / 1.5Gbps)
PSU – X-POWER ATX-400TD

I hoped that by choosing these components; 
A Mobile chipset – with all the power saving features that should give (~30W).
A Mobile T2500 CPU – the rated TDP is just 31W
Mobile 2.5″ Harddisk – should use less than 1W at idle (maybe 4W under use)
A ‘fairly’ decent PSU – >75% efficiency (according to the specs)
A single RAM module – Half the power of a dual-channel setup!! (Each DDR2 module is consumes about 2W)

To this heady mix I added a ICYDOCK 4-way disk carrier (another ebay bargain) and a Sil3114 4-port SATA PCI controller. I did some research and I guess that each hard drive will add about 8W to the mix.
 

Initially I toyed with Windows Home Server. I got hold of a trial verison with a resonable trial period and installed it on a spare PC but I found myself somehow underwhelmed. Maybe I’ve overspecified the server and I guess I’m not the sort of user they’re targetting with the WHS concept but I just thought that you could acheive the same with a NAS box – all the potential of having a computer server in the house and it wasn’t really used.

Eventually I went for CENTOS linux for the O/S it’s a mature distribution targeted at the server (rather than having all the stuff that comes with a desktop O/S) and it seems that the maintainers of the distribution put an emphasis on stablility and reliablity rather than early adoption.

Comments (1)