I/O Token Allocations (IOTAs) are used in Teradata Database 14.10 with SLES 11 systems to enforce the I/O side of Workload Management Capacity on Demand (WM COD) and other SLES 11 hard limits.
This posting describes what IOTAs are and what role they play in enforcing I/O limits with WM COD. In addition, and maybe more importantly, we’ll be looking at how IOTA numbers being reported in ResUsage can help you assess the I/O utilization of your workloads, and the I/O utilization of your entire platform, whether or not you have WM COD enabled.
Background on IOTAs and WM COD
The I/O side of WM COD is enforced at the individual disk drive level. Incoming I/O requests are prioritized and their cost is calculated before they are issued. Costing takes place in the TDMeter module shown below, which itself is part of a larger piece of code that resides on each disk drive called TDSched .
I/O costing is based on the physical bandwidth that is expected to be transferred when the I/O is executed. Different types of I/O (read vs. write) and differently-sized data blocks have different costs assigned to them.
In order to enforce COD at the prescribed level, the TDMeter module accumulates and disperses tokens. Each token represents unit of I/O bandwidth and a set number of these tokens are made available to the disk drive at intervals of a few milliseconds. Only when enough tokens have been accumulated to match the cost a particular I/O request, will that request be executed.
When WM COD is enabled, tokens (or groups of tokens referred to as IOTAs) are accumulated on a periodic time basis into a WM COD bucket that exists for each drive. Once there are enough tokens in the bucket for an I/O, the required tokens are removed from the bucket as payment for the I/O, and the I/O is performed.
The number of tokens issued per interval is determined based on the WM COD setting. Tokens are only issued to a disk drive if there is actually an I/O that is ready to run. Consequently, the number of tokens does not build up over time in the absence of I/O demand. You use them at the time they are issued or you lose them.
When there is an I/O waiting to run it will wait until enough tokens have arrived in the bucket to allow that I/O to be released. When that happens, the number of tokens that reflect the expected bandwidth of the I/O are taken out of the bucket, and any remaining tokens are discarded, unless there are additional I/Os waiting to run on that device.
If the number of tokens distributed to the bucket on a given device has been exhausted, and there are still I/Os waiting to run, those I/O requests will have to wait until the next interval for more tokens to be provided. Tokens are issued in such a way that total I/O bandwidth from that disk drive will conform to the COD settings that have been established.
IOTAs and ResUsageSPS
Whether or not you have WM COD enabled, tokens are available for viewing in some of the ResUsage tables. And they can provide a very helpful function when it comes to assessing I/O usage.
The two most useful fields related to tokens that appear in ResUsageSPS are FullPotentialIota and UsedIota. When you manipulate them appropriately you can derive the percent of the total system I/O bandwidth each workload on a node consumed during that logging interval.
The FullPotentialIota field reports the maximum IOTAs available across all disks on the node and represents the total I/O bandwidth across all drives on that node. Since it is a node-level metric, be mindful when you look at ResUsageSPS output that the FullPotentialIota column will be reported as the same number for each workload on that node, similar to NodeID. It is actually a single value, but is repeated in each SPS table row to make it easy to perform calculations.
This single-node excerpt of output from ResUsageSPS was captured during a test where four workloads were active, one in each of the four Timeshare access levels. Notice that FullPotentialIota is the same value for all the workloads on the node.
UsedIota, another field shown above, reports the number of IOTAs consumed by the four different workloads in a single logging interval. To translate these two IOTA fields into a percent of I/O utilization for each workload, divide Used by Potential: UsedIota / FullPotentialIota.
The next graphic illustrates the result of that calculation
.jpg)
In this example, four workloads were actively performing very I/O-intensive work. Each of the four SLES 11 Timeshare access levels had a single workload active. The four access levels have a priority difference of 1:2:4:8, which correlates to the differences in the I/O utilization among the four workloads reported in the output above.
The ResSpsView does this calculation for you, in a field named Used_FullPotential_ByWd.
IOTAs and ResUsageSPMA
ResUsageSPMA is just as useful as ResUsageSPS is when it comes to IOTA data, with or without WM COD.
You can derive system I/O utilization numbers in a similar way using the ResUsageSPMA table or its views. In 14.10 the IOTA fields are held in Spare fields within the SPMA table. In the ResSpmaView view, the spare fields have been renamed in this way:
Spare07 AS FullPotentialIota,
Spare08 AS CodPotentialIota,
Spare09 AS UsedIota,
To assess system I/O utilization, you can use this calculation: UsedIota / FullPotentialIota. Below is some output from ResSpmaView that shows this calculation.
.jpg)
IOTAs in ResUsage With WM COD Enabled
There is a third IOTA-related metric that appears in both ResUsage tables, CodPotentialIota. This field represents the tokens available for use when COD is enabled. The tokens that get reported in this column are used for internal purposes in order to control the I/O requests in a way that honors the WM COD setting. This metric does not lend itself to general purpose reporting, nor it is easy for someone without special internal knowledge to gain insights from it. For general purpose reporting, it is recommended that the focus be on the Full and Used IOTA fields.
CodPotentialIota values may appear somewhat confusing to the casual observer. With WM COD enabled, token counts are accumulated only when an I/O is pending, so they are demand-driven. In addition, any unused IOTAs are discarded immediately if no further I/O’s are waiting to execute on the drive, so not all tokens that were initially made available to a drive are reported.
CodPotentialIota only plays a role when COD is enabled. As a result, CodPotentialIota will show up in ResUsage output as being equal to UsedIota if there is no COD on the system. This can be seen in the ResUsageSPS output below, collected when there were four active Timeshare workloads on a system with no COD.
100% WM COD
Because the ResSpsView view was used to produce this output, the calculated field Used_FullPotential_ByWd field is provided. That tells you the I/O utilization for each of the four workloads. Notice that on each node the individual workload I/O utilization numbers add up to almost 100% This may not always be the case, but here it is a consequence of the very I/O-intensive work that was being executed in this test.
Contrast that usage example to one with 50% WM COD defined on the platform. The same workloads are active and the same work is being executed.
50% WM COD
In this case the I/O utilization within each workload is approximately cut in half, and the sum of the I/O utilization values is almost 50%, reflecting the WM COD setting. This validates that the disk sub-system is appropriately limiting I/O requests so that only 50% of the platform’s disk bandwidth will be used when 50% WM COD has been enabled.
Conclusion
IOTAS were designed as an internal mechanism to enforce the I/O side of capacity on demand and other SLES 11 hard limits within the disk subsystem. However, some of the IOTA fields that appear in ResUsageSPS and ResUsageSPMA provide an easy and accurate method of determining workload I/O utilization, or system level I/O usage at the node level.
Using the simple formula of dividing used IOTAs by potential IOTAs, you now have a new tool in understanding your I/O usage, no matter how complex or how simple your workload.