Often in our tech industry there is a penchant to spout off performance numbers without qualifying the metrics and conditions under which these numbers are derived. The SOA Gateway community is not immune to this indulgence. I have to admit, even I am guilty of committing this sin sometimes.
In the SOA Gateway world, performance cannot simply be defined in terms of transactions per second (TPS) due to complexity of a message transaction and the task policy of the gateway. As a result, SOA Gateways today always specify a specific task (i.e XML transformation, WS-Encryption) and the associated TPS. However, this type of metric still falls short of fully expressing the true performance metric of a SOA Gateway. For example, a common task that is staple of every SOA Gateway is schema validation. This task validates the the structure of incoming and outgoing SOAP/XML messages. The performance of a SOA Gateway when performing validation is often expressed in terms of Schema Validation TPS.
This is simply not sufficient. Further qualifiers that should be applied to schema validation performance numbers are as follows:
- What is the size of the message?
- What transport protocol (HTTP 1.0, HTTP 1.1, MQ etc) was used to derive the numbers?
- Was the deployment in proxy mode or was it in service mode?
- How many clients were used in generation of load?
- Was the validation task performed on both inbound and out bond messages?
- How complex was the message structure and its associated schema (i.e n-dimensional arrays, abstract types).
The last bullet is a real challenge and it really affects the validation performance of a gateway.
Unless, these qualifiers are resolved, the numbers are subjective at best. Maybe one day we will learn some lessons from the automotive industry to really define a true metric in defining performance of each task in a SOA Gateway.
