IBM FlashSystem 9100 Comprestimator V1.0.1 

Quick Start Guide

 

Read this first!

IBM FlashSystem 9100 Comprestimator  can be installed only on supported Windows operating systems (see list below). After installation completes, the binary files for other supported operating systems are available in the Windows installation folder.

By default, the files are copied to:
In Windows 64-bit: c:\Program Files (x86)\IBM\IBM FlashSystem 9100 Comprestimator
In Windows 32-bit: 'c:\Program Files\IBM\IBM FlashSystem 9100 Comprestimator

 Installing IBM FlashSystem 9100 Comprestimator

To install IBM FlashSystem 9100 Comprestimator on a Linux:

1.     Log into the host using the root account.

2.     Copy the Comprestimator binary file from the installation folder to any folder on the host using an SCP utility such as WinSCP or PuTTY.

3.     Login to the UNIX host, and confirm that the file has executed permissions, to add execute permissions to the binary file, type chmod +x comprestimator.

To install IBM FlashSystem 9100 Comprestimator on ESXi:

1.     Enable SSH on the ESXi server. Check the following URL for instructions or skip this step if SSH is already enabled: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017910

2.     Copy the ESX/comprestimator_esx binary file to any folder on the ESXi server. You can copy the file to the ESXi server by using the secure copy protocol (SCP), for example ‘scp’ on Linux or ‘WinSCP’ on Microsoft Windows

To install IBM FlashSystem 9100 Comprestimator on a Windows host:

1.     Log into the host.

2.     Copy the appropriate Comprestimator binary file from the corresponding folder to any folder on the host.

 

Introduction to IBM FlashSystem 9100 Comprestimator

IBM FlashSystem 9100 Comprestimator is a command line host-based utility that can be used to estimate expected compression rate for block-devices. The IBM FlashSystem 9100 Comprestimator utility uses advanced mathematical and statistical formulas to perform the sampling and analysis process in a very short and efficient way. The utility also displays its accuracy level by showing the maximum error range of the results achieved based on the formulas it uses. The utility runs on a host that has access to the devices that will be analyzed, and performs only read operations so it has no effect whatsoever on the data stored on the device.

The following section provides useful information on installing IBM FlashSystem 9100 Comprestimator on a host and using it to analyze devices on that host. Depending on the environment configuration, in many cases Comprestimator will be used on more than one host, in order to analyze additional data types.

It is important to understand block-device behavior when analyzing traditional (fully-allocated) volumes. Traditional volumes that were created without initially zeroing the device may contain traces of old data in the block-device level. Such data will not be accessible or viewable in the file-system level. When using Comprestimator to analyze such volumes, the expected compression results will reflect the compression rate that will be achieved for all the data in the block-device level, including the traces of old data. This simulates the volume mirroring process of the analyzed device into a compressed volume. Later, when volume mirroring is actually used to compress the data on the storage system, it will process all data on the device (including both active data and traces of old data) and get it compressed. After that when storing more active data on the compressed volume, traces of old data will start getting deleted by new data that is written into the volume. As more active data accumulates in the device the compression rate achieved will be adjusted to reflect the accurate savings achieved for the active data. This block-device behavior is limited to traditional volumes and will not occur when analyzing thinly provisioned volumes. 

Regardless of the type of block-device being scanned, it is also important to understand a few characteristics of common file-systems space management. When files are deleted from a file-system, the space they occupied before being deleted will be freed and available to the file-system even though the data on disk was not actually deleted but rather the file-system index and pointers were updated to reflect this change. When using Comprestimator to analyze a block-device used by a file-system – all underlying data in the device will be analyzed, regardless of whether this data belongs to files that were already deleted from the file-system. For example – you can fill a 100GB file-system and make it 100% used, then delete all the files in the file-system making it 0% used. When scanning the block-device used for storing the file-system in this example, Comprestimator (or any other utility for that matter) will access the data that belongs to the files that were already deleted.

In order to reduce the impact of block-device and file-system behavior mentioned above it is highly recommended to use Comprestimator to analyze volumes that contain as much active data as possible rather than volumes that are mostly empty of data. This increases accuracy level and reduces the risk of analyzing old data that is already deleted but may still have traces on the device.


Using Comprestimator:

To use Comprestimator on a Linux, ESXi server:

1.     Log into the server using the root account.

2.     Obtain the list of device names:
In Linux: Use the “fdisk –l” command.
In ESXi 4.0: Use the “esxcli corestorage device list | grep Dev” command.
In ESXi 5.0: Use the “esxcli storage core device list | grep Dev” command.

3.     Run Comprestimator with the -d <device_name> flags to analyze a device or a partition.

 

Comprestimator is designed to scan any block device that is readable by the OS itself. This typically includes devices managed by logical volume managers (LVMs) or partitioned by the OS. However, for practical reasons, since compression is applied to physical volumes, it is recommended to estimate compression by running Comprestimator on the same block device/physical volume that will be compressed, and not on a logical volume, which may be spanning on those volumes. It is thereby highly recommended to always analyze the native block-device when using Comprestimator.

Some volume managers "reserve" some of the LUN capacity for internal use. Since Comprestimator reads directly from the block device, some of these IOs may fail. The tool will tolerate up to 1% failed IOs and a scan will be aborted if this threshold is reached.

Note that for Flashsystem the Maximum effective capacity per enclosure is limited based on the number of modules configured in the system.  This configuration will limit the total physical and effective capacity on the enclosure.

 

To use Comprestimator on a Windows server:

1.     Log into the server using an account with administrator privileges.

2.     Open an elevated Command Prompt with administrator rights (Run as Administrator).

3.     Run Comprestimator with the -l flag to list the disk.

4.     Run Comprestimator with the –n flag to analyze the specific disk.

5.     Alternatively, run Comprestimator with –d and the device name. Note that when using –d option with the long device name, the device name must be wrapped in double quotation marks.

 

Syntax

Using Linux, ESXi:

 

comprestimator -s DRP|FCM [ -h | -d device ] [ -c filename ] [-v] [ -p number_of_threads ] [-P] [-I] [ --config=task_file ]  --stripewidth

 

 

Using Windows:

 

 

comprestimator -s DRP|FCM [ -h | -l | -d device | -n disk_number ] [ -c filename ] [-v] [ -p number_of_threads ] [-P] [-I] [ --config=task_file ]  --stripewidth

 

-s, --storageSys

Specify storage system type. Supported values are: DRP, FCM

-d

Specifies the device name. Note that the name must be wrapped in double quotation

device

Path of device to analyze (e.g. /dev/sda  in Linux).

--stripewidth       

Stripe width for the planned FCM array.  Allowable values: 5 - 16.  Default 12. 

Stripe width is the width of a single unit of redundancy in the array.  Arrays with smaller stripe width are quick to complete rebuild operations.  But such arrays also use more capacity for parity, and parity is usually less compressible than user data. 

-n disk_number

Only in Windows – The disk number. Identify this number by running Comprestimator with the –l flag first.

-l

List the disk numbers and names available to use by Comprestimator. The drive numbers reported match the “Disk #” reported in Windows Disk Management.

-p

Specifies the number of processes (or threads in Windows) between 1 and 50. The default value is 10.

number_of_processes

The number of processes (or threads in Windows) used by Comprestimator.

-P

Displays the results using a paragraph format.

-c

Export the results to a CSV-formatted output file.

file name

File name of CSV-formatted output file.

-h

Display usage information.

-v

Display the results of every few samples. Note that the output is extended to a few thousands of lines as a result.

--config

Configuration file that contains list of devices to analyze. Used for automation.

file

Configuration file that contains list of devices to analyze. The file should contain a list of devices in the same format of the –d option, one device in each line.
For example:

# cat devlist_file
/dev/sda1
/dev/sda2
…

 

Explanation of scan results

Sample#

The number of the current sample reported.

Device

The device name used in the scan.

Size(GB)

The total size of the device as reported by the operating system, in gigabytes.

Compressed Size(GB)

The estimated size of the device if it will be in gigabytes.

Total Savings(GB)

The total estimated savings from thin-provisioning and compression, in gigabytes.

Total Savings(%)

The estimated savings from thin-provisioning and compression, in percentage of the size of the device.
This value is calculated in the following method:
Total Savings(%) = 1-( Compressed Size(GB) / Size(GB) )

Thin Provision Savings (%)

The estimated savings from thin-provisioning.  Shows how much of the volume is user data vs unmapped/zero.

Compression Savings(%)

The estimated savings from compression.

Compression Accuracy

Range(%)

The accuracy of the estimate provided by Comprestimator. The results provided are estimated based on samples from the device and therefore may be lower or higher than the actual compression that would be achieved. The approximate accuracy of the results is represented as a percentage of the total size of the device.
For example, if the estimated Compression Savings (%) is 67%, and the Compression Accuracy Range is 5%, the actual compression savings (in percentage) if this device will be compressed is between 62% and 72%.

Comprestimator can also be used without any flag to get help and usage information. Following is an example of the output you should expect for each analyzed device/partition:

Sample#

Device      

Size (GB)

Compressed Size (GB)

Total Savings (GB)

Total Savings (%)

Thin Provision Savings (%)

Compression Savings (%)

Compression Accuracy Range %

3425

/dev/dm-2 

500

112.8

387.1

77.4%

39.6%

62.6%

 4.7%

Examples

Linux

To display the usage information:

# ./comprestimator

To list all the available device names in Linux:

# fdisk -l

Disk /dev/sda: 78.3 GB, 78383153152 bytes
255 heads, 63 sectors/track, 9529 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000021
 Device Boot Start  End  Blocks   Id System
/dev/sda1 *  1      8555 68718006 83 Linux
/dev/sda2    8556   8924 2963992+ 5  Extended
/dev/sda5    8556   8924 2963961  82 Linux swap / Solaris

Use the device names listed in the output, in this example: /dev/sda1, /dev/sda2, /dev/sda5.

To run a standard scan of /dev/sdb device (Linux version):

# ./comprestimator –d /dev/sdb -s DRP

Version: 1.5.3.1 (Build w0117)
Start time: 13/07/2013 15:54:02.918234
Device name: /dev/dm-2
Device size: 500.0 GB
Number of processes: 10
Exhaustive: no
Sample# | Device | size (GB) | Compressed Size(GB) | Total Savings(GB) | Total Savings(%) |Thin Provision Savings(%) | Compression Savings (%) | Compression Accuracy Range(%)
3425 | /dev/dm-2 |500.0|112.8| 387.1 |77.4% |39.6% |62.6% |4.7%

VMWare vSphere ESXi 4.0

The device names that can be scanned using Comprestimator on an ESXi 4 Server can be obtained by using the following command when logged into the ESXi server with root account using SSH:

~ # esxcli corestorage device list

To display a shorter, more readable list, you can use the following command to filter out unnecessary information:

~ # esxcli corestorage device list | grep Dev
    Device Type: Direct-Access
    Devfs Path:
/vmfs/devices/disks/naa.600605b0042ff75017a86e2e531227e7
    Device Type: Direct-Access
    Devfs Path: /vmfs/devices/disks/naa.6005076801ac861ef800000000000007
    Device Type: Direct-Access
    Devfs Path: /vmfs/devices/disks/naa.6005076801ac861ef800000000000008
    Device Type: Direct-Access
    Devfs Path: /vmfs/devices/disks/naa.6005076801ac861ef80000000000000a
    Device Type: Direct-Access
    Devfs Path: /vmfs/devices/disks/naa.6005076801ac861ef80000000000000b
    Device Type: Direct-Access
    Devfs Path: /vmfs/devices/disks/naa.6005076801ac861ef80000000000000d
    Device Type: CD-ROM
    Devfs Path: /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0

This will list only the device types and their respective path names. The device path names can then be used as arguments for Comprestimator.

To scan a device in ESXi 4:

#. /comprestimator -d /vmfs/devices/disks/naa.600605b0042ff75017a86e2e531227e7

VMWare vSphere ESXi 5.0

The device names that can be scanned using Comprestimator on an ESXi 5 Server can be obtained by using the following command when logged into the ESXi server with root account using SSH:

~ # esxcli storage core device list

To display a shorter, more readable list, you can use the following command to filter out unnecessary information:

~ # esxcli storage core device list | grep Dev

This will list only the device types and their respective path names. The device path names can then be used as arguments for Comprestimator.

To scan a device in ESXi 5:

#. /comprestimator -d /vmfs/devices/disks/naa.60050768029c00e8140000000000002d

Windows

To scan a device in Windows, first identify the disk number using the -l argument:

# comprestimator –l

Drive number [0]        \\?\ide#diskhitachi_hts725032a9a364pc3zc70f#5&1e544866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

The drive numbers reported match the “Disk x” reported in Windows Disk Management. Now run a standard scan of disk 0:

# comprestimator –n 0 -v
Version: 1.5.1.1 (Build w0082)
Start time: 17/07/2013 17:33:08.728421
Device name: \\?\ide#diskhitachi_hts725032a9a364pc3zc70f#5&1e54866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}
Device size: 298 GB
Number of processes: 10
Exhaustive: no
...
...

Alternatively, the drive numbers reported match the “Disk #” reported in Windows Disk Management. Now run a standard scan of disk 0:

# comprestimator –d "\\?\ide#diskhitachi_hts725032a9a364_________________pc3zc70f#5&1e544866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}"

Known issues

The following is a list of known issues in the software current release. These issues are due to a combination of environment constraints and current version limitations.

 

Table 1: Known issues

ID

Details

12599

Comprestimator reports incorrect results (i.e. -1.$%) for devices smaller than 1GB. Use Comprestimator only with devices larger than 1GB.

Troubleshooting

Linux, ESXi

1.     Run the tool with –v option – check output.

2.     Check the contents of the log file – comprestimator.log in the current working directory.

3.     Confirm that the device is readable using standard UNIX commands:

bash-4.2# dd bs=262144 if=/dev/rdsk/c1t0d0s2 of=/dev/null

4.     If the device cannot be read, verify that it is indeed a raw device path name.

5.     If the utility crashes or hangs, rerun the utility in verbose mode, and with trace information:

Linux:

# strace –o /tmp/strace.txt –f ./comprestimator –v –d /dev/sdb

Check the contents of the log file – "comprestimator.log" in the current working directory.

 

Comprestimator is supported and can be used on the following client operating system versions:

·         Windows 2003 Server, Windows 7, Windows 2008 Server , Windows 8, Windows 2012

·         ESXi 4, 5

·         Red Hat Enterprise Linux Version 5.x, 6

·         SUSE SLES 11

·         Ubuntu 12

·         CentOS 5.x