Certification Programs
Go


IPMG Best Value Grid

Introduction

 

NSTL technicians will evaluate each printer submitted to the benchmark as follows:

  1. Testing for all standalone printers will consist of:
    • Performance
    • Features
    • Usability
    • Quality
  2.   

  3. Network Printer testing will consist of:
    • Network Performance
    • Features
    • Usability
    • Quality
  4.   

  5. Multi-function printer testing will consist of:
    • Printer Performance, Copying Performance
    • Features
    • Usability
    • Quality

  

Nstl will separate all printers into the following four categories:

  1. Monochrome Page Printers
  2. Colour Page Printers
  3. Monochrome Specialty Page Printers
  4. Colour Specialty Page Printers

 

Test Setup

  • All standalone printer testing will be conducted in a Windows XP Professional environment
  • All network related aspects of the benchmark methodology was conducted in a 100BaseT Ethernet environment using static TCP/IP addressing on a Windows 2003 Server
  • Hardware Configuration:  Pentium 4 – 2.6 GHz, 80G HD, Colour VGA monitor, 512 MB RAM
  • Software Configuration: NSTL’s Printer Test Suite, Microsoft Office XP, CorelDraw! 9.0, AutoCAD 2004, Adobe Acrobat 6.0.

 

Overall Printer Score Methodology

 

An overall score for each printer was derived by averaging the weighted scores for each of the four major NSTL tests: speed, quality, features and usability. Weights for each of the tests are determined by individual client needs. The final overall scores are presented on a scale from 1 to 10, with 10 representing the best possible score in a single category. Assigned weights vary based on category, a sample follows – categories and weights yet to be assigned:

Printers:

Test Weight
Speed 20%
Quality 25%
Features 25%
Usability 30%

 

Speed Scoring Methodology

 

The Speed of each printer will be tested using a series of predetermined files. NSTL uses a number of real world application files in order to give a more realistic indication of the printer capabilities. The files were created using Microsoft office, Adobe Acrobat, and AutoCAD applications. Each file type contains one or more of the following elements:  graphics sparse text, dense text, fonts, bit-mapped images. The Microsoft Word document contains text, embedded Excel charts, and graphic images. The Microsoft Excel document contains a combination of simple spreadsheets, and a variety of charts. The PowerPoint file is a basic presentation including text and graphics. The Adobe Acrobat PDF file is similar in form to many PDF files in use today, it contains a combination of text and graphics. Please see the below list for a detailed breakdown of the individual files.

 

  1. Ms Word
    • Graphics
    • Sparse Text
    • Dense Text
    • Fonts
    • Bit-mapped Images
  2. PowerPoint
    • Graphics
    • Sparse Text
    • Fonts
    • Bit-mapped Images
  3. Excel
    • Sparse Text
    • Graphics
    • Fonts
  4. Adobe PDF
    • Graphics
    • Sparse Text
    • Fonts
    • Bit-mapped Images
  5. Auto CAD
    • Drawings utilizing lines and text

 

 A printer's raw score on any given test is the time it takes to generate one of the predetermined files. All tests will be performed at least twice to check for relative consistency. All raw scores will be recorded in number of seconds. If the speed times for two runs of the test vary more than 5%, NSTL will rerun the test until two scores obtained are within a 5% degree of variance.

 

Standalone Laser Printers:

The speed tests will be conducted in both PCL and PostScript modes, where both are available, at the standard and highest resolutions available. The overall Speed score is determined by the better performing emulation. The printer vendor / supplier will advise NSTL on the resolutions (high and low) that they would recommend for the speed tests. The elements tested are:

TEST Element

  • Text
  • Graphics
  • Sparse Text
  • Dense Text
  • Fonts
  • Bit-mapped Images

 

Network Laser Printers

The speed tests will be conducted in a network (Windows 2000 Server) environment. The tests will be conducted in both PCL and PostScript modes, where both are available, at the standard and highest resolutions available. The major part of the test will be conducted with Duplex enabled, with one test being run in Simplex mode. The overall Speed score is based upon the better performing emulation. The printer vendor / supplier will advise NSTL on the resolutions (high and low) that they would recommend for the speed tests. The elements tested are:

TEST Element

  • Dense Text
  • Sparse Text
  • Fonts
  • Bit-mapped Images
  • Graphics
  • MS Word document
  • Multi-job submissions (Text, Bitmap, Graphics)
  • Duplex Test – MS Word Document

 

Colour Page Printers

The speed tests will be conducted at the lowest resolution available. The networked products will be tested with Duplex enabled, with one test being run in Simplex mode. The elements tested are:

TEST Element

  • Dense Text
  • Sparse Text (Paper Handling)
  • Fonts
  • Bit-mapped Images
  • Graphics
  • MS Word document
  • Multi-job submissions (Text, Bitmap, Graphics)

 

Basic/CAD Plotter & Workgroup Plotters

The speed tests will be conducted at the lowest resolution available. The elements tested are:

TEST Element

  • Sparse Text
  • Bit-mapped Images
  • Graphics
  • Fonts
  • Wide graphic image
  • Plotter Specific image (AutoCAD)

 

To calculate the overall Speed score for each printer, a geometric mean will be taken of the results for all the speed tests for each printer. The geometric mean for each printer will then be converted to a 10-point scale. Thus, the overall Speed score for each printer is a value from 1 to 10, with 10 representing the best overall Speed. The formula for converting the geometric mean to a score is:

(minimum geometric mean for all page printers / printer's geometric mean) x 10

 

Quality Scoring Methodology

 

NSTL rates the quality of all printers using the following test categories, where applicable:

  • Monochrome objectives
  • Colour objectives

 

The quality of each printer is evaluated by NSTL using the output from several Postscript files. Quality testing will be conducted at the highest resolution and quality settings available for the product. In order to create comparable quality tests for printers which may not have PostScript capability, NSTL has developed its own PostScript interpreter, as a part of its’ test suite. The quality output contains both subjective and objective scoring. However, the subjective scoring applies only to the colour printers.

 

Objective tests:

The monochrome quality tests consist of a series of pages containing such elements as a photographic image, text in a wide range of sizes, and a variety of circles and horizontal and vertical lines. These tests also stress more esoteric features, such as the printer's paper-handling ability and how well the printer displays reversed (white on black) text and graphics. These tests demonstrate a printer's suitability for graphics-oriented applications. The colour quality test examines the ability of the printer to print a wide range of hues as well as to blend between the hues. It also examines ink placement problems - the ability of the printer to accurately place lines without misregistration or smearing. These tests are self-scoring and absolute in that each score is only a function of the particular printer's abilities, it does not compare one printer against another.

 

Subjective tests:

One output consists of a set of five photographs scanned on an HP IICX scanner.  They were scanned at 200x200 dpi, 24 bits per pixel.  Two other images are composites of photographs scanned at 600x600 dpi. The files are full-page images, a bit larger than 8.5x11 inches. The quality of these pages are judged relative to the other printers in the grouping. When quality output from all printers in a category is completed, a NSTL technician will group the output into different categories. Although output will be marked on the back to identify the printer from which it came, the tester evaluating the quality of the output on the printed side of each sheet will not know which printer produced any given sheet of output. The technician will judge each element of the output in a given category on a scale of 1 to 5, with 1 representing the relatively worst output and 5 representing the relatively best output.

Each quality output file will be generated on both paper provided by the printer vendor and transparencies.

To calculate the overall quality score for each printer, the ratings assigned to each major element will be averaged for each image printed; an average of all of the scores will be calculated to achieve a score for both vendor paper and transparencies. The scaled quality score will be calculated, by multiplying the scaled objective score by 0.8 and adding the result to 0.2 times the scaled subjective score.

The final score for each printer will be adjusted to a 10-point scale for easy comparison. An overall quality score of 10 thus represents the highest possible score for a printer in a category.

 

Features Scoring Methodology:

 

Prior to the beginning of benchmark testing, vendors will be issued a features questionnaire for each printer they are submitting. The questionnaire will consist of more than 70 items for which vendors specify the particular attributes for each printer. During the course of benchmark testing, NSTL technicians will review the features questionnaire and verify the responses against the printer and documentation which will be provided for the benchmark. Individual features for each printer are tabulated in the results section of the report.

The verified features for each printer will be scored according to the following methodology. A number of items are assigned weights and scored (e.g., auto switching between ports). Weighted items are assigned a weight from 0.5 to 2. Exact weights for each item will be found in the first column of the features table in the data section of the report. A printer receives a score of "1" for each weighted feature it includes. The printer receives a score of "0" for each weighted feature it does not include. If an item is listed as optional, the printer receives a score of ".5" for that item. All items for each printer are then multiplied by the corresponding weight and compared to the total possible number of points for the category. The final score for each printer will be adjusted to a 10-point scale for easy comparison. An overall features score of 10 thus represents the highest possible score for a printer on the features questionnaire.

Many of the items in the questionnaire are included for informational purposes only and are not used in the computation of the features score (e.g., voltages supported).

 

Usability Scoring Methodology:

 

During the course of benchmark testing, NSTL technicians will review and scored each printer on a number of predefined usability criteria. A list of those items is presented below. The NSTL technicians will work with the printer and documentation to perform some basic tasks such as setting up the printer, loading paper and changing some items on the printer's setup/control menu. Points will be assigned based on the functionality and ease of use of the printer.

An overall usability score will be derived by averaging the scores for all seven items. The final score for each printer is adjusted to a 10 point scale for easy comparison. An overall usability score of 10 thus represents the highest possible score for a printer on the usability questionnaire.

 

Usability Items

  • Overall Usability
  • Memory Expansion Accessibility
  • Paper Handling
  • Ink Medium Installation
  • Setup/Control Panel Functionality
  • Status Messages / Indicators
  • Documentation Organization and Clarity
  • Software Driver Installation

 

  • Network Category ONLY:
    • Network Functionality

 

  • Multifunction Products:
    • Faxing
    • Scanning
    • Copying