intertek

NMSO - National Master Standing Offer
Quality Assurance

Intertek's Standalone System Performance Benchmark Suite

performs a series of tasks which replicate actual use, providing a realistic measurement of how each software package will perform on a particular system.
Results obtained from the various Performance Benchmark Suites reveal concrete, real-world differences between comparable systems.


Intertek's Standalone System Performance Benchmark Suite was specifically designed to assess overall performance.
  • Methodology:
  • Overview
  • Performance
  • Features
  • TCO
  • Usability
Each advance in hardware and software technology poses new challenges to evaluation techniques. In previous evaluations Intertek was able to apply common test methodologies across all categories of the NMSO. Except for some minor variations to reflect the category, desktop systems, and notebooks were each subjected to the same suite of tests for Performance, Compatibility and Features/Usability. Since it became increasingly difficult to differentiate between the products in the server categories using results produced by methodologies that were designed for standalone categories Intertek utilized a test suite specifically targeted at the server products. In addition, the methodologies for the standalone system categories have been upgraded to reflect environments that are current in the industry.
All test suites for standalone systems were executed with (currently in a Windows 7 OS environment). The network environment of Windows 2000 Advanced Server was utilized. All network related aspects of the benchmark methodologies were conducted in a 10BaseT Ethernet environment (Desktops and Notebooks).
One way of expressing a system's performance is through the analysis of the raw measurements of its various components, including the speed of the processor, the number of wait states on memory access, and the speed at which the hard disk reads randomly and sequentially stored data. These measurements allow for the comparison of the performance of the individual system components but in and of themselves do not provide a realistic indication of the performance of the system operating as a whole.
Prior to testing, questionnaires were sent to each system manufacturer. These questionnaires required detailed specification of more than 150 attributes of each system. Some items are informational (e.g., CPU manufacturer); whereas other features (e.g., Dual-Processor Capability on-board) are considered relevant features which contribute to the overall flexibility of the system.
The attributes for TCO consist of questions relating to the Environment (green capabilities), Security, Deployment, Certifications. TCO also consists of the verification of the System Restore, System migration, and Network management utilities. Value added services are also noted.
Intertek examines each system for usability by focusing on a few key areas; system design, documentation, and Vendor specific Web Site content. In looking at system design, testers start by rating system teardown. Once inside the system, testers assess how easy it is to perform normal system upgrading such as increasing the system's RAM and to add adding mass storage devices.
Outside the unit, testers look for non-mandatory enhancements such as a reset button and clearly marked and color coded I/O ports. Testers navigate through each systems Setup Utility (if supplied) to see if the integrated hard disk controller and built-in I/O ports can be disabled (if applicable). Documentation considerations consist of: manuals that are comprehensive, include easy-to-read diagrams, and offer up-to-date technical information. Additional points are given for refinements such as quality and detail of illustrations and ease and breadth of comprehension. The presence of a glossary and an index is a bonus. Web sites are rated based on ease of navigation and technical areas; as well as client (NMSO) specific areas. Restore CD's (including hidden partition restores) are evaluated for ease of use and completeness of driver installation, etc. (if supplied with system).