Situation:
1. I'm using vsphere 5.1 with thin VMs.
2. I will only use VMs for backup targets.
3. I'm currently using Windows 2003 Server VM, projecting an NFS volume, to back up to. One big advantage of this is NTFS compression. The problem is 2003 and 2008 cannot handle VMs without frequently running into file size issues. Example: I have a 1TB NTFS compressed volume that I export as an NFS volume. It has 667GB of free space. If I clone a VM to it that has a declared size of 80GB, and a thin size of 42GB, this is the command and the result:
# vmkfstools -i inh.vmdk -d thin /vmfs/volumes/nas-2/backups/server1/inh/inh-2014-03-19_05-22-41/inh.vmdk
Destination disk format: VMFS thin-provisioned
Cloning disk 'inh.vmdk'...
Clone: 78% done.Failed to clone disk: There is not enough space on the file system for the selected operation (13)
A few days later after some copying and deleting it might start working again, or I can run chkdsk on it, and it will back up again. You can run chkdsk 10 times in a row and get errors every time. Microsoft knows about the problem, didn't commit to fixing it, it hasn't been fixed, and if they haven't fixed it by now, I doubt they ever will, since these are not their current products. Back then virtualization, and multi-gigabyte files were not a market reality.
4. I read that ESXi defaults to thin when copying to NFS, and I specify it in vmkfstools when doing the copy anyway.
5. To fix my backup file size issues, I was thinking about using a ?NIX to replace 2003 as the NFS VM for the backup. It seems ?NIX doesn't have good compression options. ZFS looks like its about as good as it gets, and from what I read, it is great for managing a lot of data and disks, but slow, and a resource hog for this type of work.
6. I decided that just backing up thin would be good enough, and that I could get away without compression. However, when I tested the backups on the NFS volumes, they used their full declared size. The following are the results of these measurements.
As backed up to a Windows Server 2003 NFS volume
# ls –l fstop-flat.vmdk
-rw------- 1 root 42949672 85899345920 Mar 14 06:53 fstop-flat.vmdk
# stat fstop-flat.vmdk
File: fstop-flat.vmdk
Size: 85899345920 Blocks: 167772160 IO Block: 131072 regular file
Calculation:
167772160 * 512 = 85,899,345,920
85,899,345,920 /1024/1024/1024 = 80GB (Full size, not thin at all)
Same VM running on VMFS Volume:
# ls -l fstop-flat.vmdk
-rw------- 1 root root 85899345920 Mar 15 23:11 fstop-flat.vmdk
# stat fstop-flat.vmdk
File: fstop-flat.vmdk
Size: 85899345920 Blocks: 124723200 IO Block: 131072 regular file
Calculation:
124723200 * 512 = 63,858,278,400
63,858,278,400 /1024/1024/1024 = 59.47265625GB (Thin)
Another VM backed up to a Windows Server 2003 compressed NFS volume
- inh-flat.vmdk has a declared size of 80GB
- As thin on VMFS, it has a size of 38.72 GB
- As backed up to a Windows thin .vmdk, with NTFS compression, projected as an NFS share it is 19.26GB
Analysis/Thoughts:
1. The VMware docs state that, by default, vmdks made on an NFS volume are thin. Indeed, when I backup my thin VMs to an NFS share, and restore back to a the vmfs file system, the .vmdk comes back thin, so there is no doubt that it is stored as thin.
2. Even though a .vmdk backed up to an NFS share is thin, unlike under vmfs, a thin .vmdk residing on an NFS volume consumes all of the blocks required by its declared size. E.G. We have a VM declared thin with a size of 80GB, which uses 59.47GB on a vmfs volume, but when backed up to a an NFS store, it requires 80GB of disk space every time. Therefore, on a 250GB NFS volume, there is room for not more than 3 backups.
3. The only way to efficiently store backups on an NFS volume is if that volume supports compression. The only solid choices in that realm seem to be NTFS and ZFS. ZFS is resource intensive because of all of the other things it’s designed to do. If I went ZFS, I would probably just add it to an existing FreeBSD VM. I don’t need all of the stuff FreeNAS does. Windows through 2008 have problems with large files. I haven’t tried Windows 2012.
Question:
At this point I’m thinking about trying a Windows 2012 VM to take the place of the Windows 2003 VM to see if 2012 has beaten the volume file size issues. By 2012, virtualization was hot, and thus I would guess it might not have the problem with large files. Anyone out there using a ?NIX VM for this duty that is efficient, reliable, and supports compression?
Thanks!