Datasheet

Section 14 A/D Converter (8 Analog Input Channel Version)
Rev.7.00 Feb. 14, 2007 page 550 of 1108
REJ09B0089-0700
A/D Conversion Precision Definitions: The chip’s A/D conversion precision definitions are
given below.
• Resolution
The number of A/D converter digital output codes.
• Offset error
The deviation of the analog input voltage value from the ideal A/D conversion characteristic
when the digital output changes from the minimum voltage value B'0000000000 to
B'0000000001 (see figure 14.9).
• Full-scale error
The deviation of the analog input voltage value from the ideal A/D conversion characteristic
when the digital output changes from B'1111111110 to B'1111111111 (see figure 14.9).
• Quantization error
The deviation inherent in the A/D converter, given by 1/2 LSB (see figure 14.8).
• Nonlinearity error
The error with respect to the ideal A/D conversion characteristic between the zero voltage and
the full-scale voltage. Does not include the offset error, full-scale error, or quantization error.
• Absolute precision
The deviation between the digital value and the analog input value. Includes the offset error,
full-scale error, quantization error, and nonlinearity error.