CD Notes

Lexical Errors and Error Recovery

Understanding lexical errors in compiler design, types of errors detected during scanning, and error recovery strategies used by lexical analyzers.

Introduction

Lexical errors occur when the lexical analyzer encounters character sequences that cannot be matched to any valid token pattern. Since the lexical analyzer has a limited view of the program (it sees only characters, not structure), it can detect only a small class of errors. More complex errors are left to the syntax and semantic analysis phases.

Types of Lexical Errors

1. Illegal Characters

Characters that don't belong to the language's alphabet:

c example

int x = 5 @ 3;    // '@' is not a valid operator in C
float π = 3.14;   // 'π' may not be valid in some languages

2. Malformed Numbers

c example

int a = 12.34.56;     // Multiple decimal points
int b = 0x3G5;        // Invalid hex digit 'G'
float c = 1.5e;       // Incomplete exponent
int d = 09;           // Invalid octal digit '9'

3. Unterminated Strings

c example

char *s = "hello;          // Missing closing quote
char *t = "line 1
line 2";                    // Unescaped newline in string

4. Unterminated Comments

c example

/* This comment
   never ends
int x = 5;    // This is inside the comment!

5. Invalid Escape Sequences

c example

char c = '\q';    // \q is not a valid escape sequence
char *s = "test\z"; // \z undefined

6. Identifier Length Exceeded

c example

// Some languages limit identifier length
int thisIsAnExtremelyLongVariableNameThatExceedsMaximumLength = 0;

Error Detection

The lexical analyzer detects an error when:

┌─────────────────────────────────────────────────┐ │ Error condition: │ │ │ │ The scanner is in a non-accepting state AND │ │ no valid transition exists for the current │ │ input character AND no prefix of the remaining │ │ input matches any token pattern. │ └─────────────────────────────────────────────────┘

Error Recovery Strategies

1. Panic Mode Recovery

The simplest strategy — delete characters from the input until a valid token can be formed:

Example

Token panicModeRecovery(char *position) { printf("Error at line %d: unexpected character '%c'\n", lineNum, *position); // Skip until we find a valid starting character while (*position && !isValidTokenStart(*position)) { position++; } // Try scanning again return getNextToken(); }

Example:

Input	$abc = 5;
Error	'$' is illegal
Action	Skip '$', continue scanning from 'a'
Result	IDENTIFIER "abc" ...

2. Single Character Deletion

Remove one character and retry:

Example

Input: intt x = 5; After: int x = 5; (delete extra 't')

3. Single Character Insertion

Insert a presumably missing character:

Example

Input: "hello (unterminated string) After: "hello" (insert closing quote at end of line)

4. Character Replacement

Replace an incorrect character:

Example

Input: 1O2 (letter O instead of zero) After: 102 (replace 'O' with '0')

5. Character Transposition

Swap two adjacent characters:

Example

Input: fi (x > 0) (fi instead of if) After: if (x > 0) (swap 'f' and 'i')

Error Reporting

Good error messages should include:

Error Report Format

─────────────────────

filename:line:column: error: description

source line content

^^^^ indicator

Example

─────────────────────

main.c:7:15: error: invalid character '@' in expression

int result = a @ b;

Implementation

c example

typedef struct {
    int line;
    int column;
    char *filename;
    char *message;
    char *sourceLine;
    ErrorSeverity severity;
} LexicalError;

void reportLexicalError(LexicalError *err) {
    fprintf(stderr, "%s:%d:%d: %s: %s\n",
            err->filename, err->line, err->column,
            severity_str(err->severity), err->message);
    fprintf(stderr, "    %s\n", err->sourceLine);
    fprintf(stderr, "    %*s^\n", err->column - 1, "");
}

Error Recovery in Practice

Handling Unterminated Strings

Example

Token scanString() { char buffer[MAX_STRING]; int i = 0; char c = getChar(); // Skip opening quote while (c != '"') { if (c == '\n' || c == EOF) { reportError("Unterminated string literal"); // Recovery: treat newline/EOF as end of string break; } if (c == '\\') { c = getChar(); // Get escape character c = processEscape(c); } buffer[i++] = c; c = getChar(); } buffer[i] = '\0'; return makeToken(STRING, buffer); }

Handling Unterminated Comments

Example

void skipBlockComment() { int depth = 1; // For nested comments while (depth > 0) { char c = getChar(); if (c == EOF) { reportError("Unterminated comment"); return; // Recovery: treat EOF as comment end } if (c == '/' && peekChar() == '*') { getChar(); depth++; } else if (c == '*' && peekChar() == '/') { getChar(); depth--; } } }

Errors the Lexical Analyzer CANNOT Detect

Error Type	Example	Detected By
Missing semicolon	`int x = 5`	Parser
Undeclared variable	`y = x + 1` (x undeclared)	Semantic analysis
Type mismatch	`int x = "hello"`	Semantic analysis
Missing operator	`x y` (meant `x + y`)	Parser
Logic errors	`if (x = 5)` (meant `==`)	None (valid code)

Best Practices for Error Handling

Never crash: The scanner should always recover and continue
Report precise location: Line and column numbers
Show context: Display the source line with the error
Limit cascading: One error shouldn't trigger many false errors
Suggest fixes: When possible, suggest what might be correct
Count errors: Stop after too many errors

Interview Questions

What types of errors can a lexical analyzer detect?

Illegal characters, malformed literals (numbers, strings), unterminated strings and comments, invalid escape sequences, and identifiers exceeding maximum length. It cannot detect syntax errors, type errors, or logic errors.

What is panic mode error recovery?

Panic mode recovery discards characters from the input until the scanner finds a character that could begin a valid token. It's simple to implement and guarantees forward progress, but may skip valid code.

Why can't the lexical analyzer detect all errors in a program?

Because it only sees individual characters and tokens without understanding program structure. It can't determine if tokens are in the right order (syntax) or if they make semantic sense (types, declarations).

How does a scanner handle unterminated strings?

Common strategies: (1) Report error and treat the newline/EOF as the string terminator, (2) Skip to the next quote mark, (3) Report error and continue scanning from the next line. The first approach is most common.

What information should a good error message include?

The filename, line number, column number, error description, the offending source line, and a caret (^) pointing to the exact error position. If possible, suggest a correction.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Lexical Errors and Error Recovery.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Compiler Design topic.

Search Terms

compiler-design, compiler design, compiler, design, lexical, analysis, errors, lexical errors and error recovery

Continue learning this concept

Lexical AnalysisFinite Automata in Lexical AnalysisHow finite automata (DFA and NFA) are used in lexical analysis to implement pattern matching for token recognition in compilers.Lexical AnalysisIntroduction to Lexical AnalysisComprehensive introduction to lexical analysis, the first phase of a compiler that reads source code and produces tokens for the parser.Lexical AnalysisLex Tool - Lexical Analyzer GeneratorComplete guide to the Lex tool for automatic generation of lexical analyzers from regular expression specifications, including syntax, rules, and examples.Lexical AnalysisLexemes in Compiler DesignUnderstanding lexemes, their role in lexical analysis, how they are identified from source code, and their relationship with tokens and patterns.Lexical AnalysisLexical Analyzer DesignComplete guide to designing and implementing a lexical analyzer including architecture, algorithms, buffering strategies, and practical implementation.Lexical AnalysisPatterns in Lexical AnalysisUnderstanding patterns that define valid lexemes for each token type, pattern specification using regular expressions, and pattern matching in scanners.

PreviousLexical Analyzer Design NextPatterns in Lexical Analysis

CD Notes

Lexical Errors and Error Recovery

Understanding lexical errors in compiler design, types of errors detected during scanning, and error recovery strategies used by lexical analyzers.

Introduction

Types of Lexical Errors

1. Illegal Characters

Characters that don't belong to the language's alphabet:

c example

int x = 5 @ 3;    // '@' is not a valid operator in C
float π = 3.14;   // 'π' may not be valid in some languages

2. Malformed Numbers

c example

int a = 12.34.56;     // Multiple decimal points
int b = 0x3G5;        // Invalid hex digit 'G'
float c = 1.5e;       // Incomplete exponent
int d = 09;           // Invalid octal digit '9'

3. Unterminated Strings

c example

char *s = "hello;          // Missing closing quote
char *t = "line 1
line 2";                    // Unescaped newline in string

4. Unterminated Comments

c example

/* This comment
   never ends
int x = 5;    // This is inside the comment!

5. Invalid Escape Sequences

c example

char c = '\q';    // \q is not a valid escape sequence
char *s = "test\z"; // \z undefined

6. Identifier Length Exceeded

c example

// Some languages limit identifier length
int thisIsAnExtremelyLongVariableNameThatExceedsMaximumLength = 0;

Error Detection

The lexical analyzer detects an error when:

Error Recovery Strategies

1. Panic Mode Recovery

The simplest strategy — delete characters from the input until a valid token can be formed:

Example

Example:

Input	$abc = 5;
Error	'$' is illegal
Action	Skip '$', continue scanning from 'a'
Result	IDENTIFIER "abc" ...

2. Single Character Deletion

Remove one character and retry:

Example

Input: intt x = 5; After: int x = 5; (delete extra 't')

3. Single Character Insertion

Insert a presumably missing character:

Example

Input: "hello (unterminated string) After: "hello" (insert closing quote at end of line)

4. Character Replacement

Replace an incorrect character:

Example

Input: 1O2 (letter O instead of zero) After: 102 (replace 'O' with '0')

5. Character Transposition

Swap two adjacent characters:

Example

Input: fi (x > 0) (fi instead of if) After: if (x > 0) (swap 'f' and 'i')

Error Reporting

Good error messages should include:

Error Report Format

─────────────────────

filename:line:column: error: description

source line content

^^^^ indicator

Example

─────────────────────

main.c:7:15: error: invalid character '@' in expression

int result = a @ b;

Implementation

c example

typedef struct {
    int line;
    int column;
    char *filename;
    char *message;
    char *sourceLine;
    ErrorSeverity severity;
} LexicalError;

void reportLexicalError(LexicalError *err) {
    fprintf(stderr, "%s:%d:%d: %s: %s\n",
            err->filename, err->line, err->column,
            severity_str(err->severity), err->message);
    fprintf(stderr, "    %s\n", err->sourceLine);
    fprintf(stderr, "    %*s^\n", err->column - 1, "");
}

Error Recovery in Practice

Handling Unterminated Strings

Example

Handling Unterminated Comments

Example

Errors the Lexical Analyzer CANNOT Detect

Error Type	Example	Detected By
Missing semicolon	`int x = 5`	Parser
Undeclared variable	`y = x + 1` (x undeclared)	Semantic analysis
Type mismatch	`int x = "hello"`	Semantic analysis
Missing operator	`x y` (meant `x + y`)	Parser
Logic errors	`if (x = 5)` (meant `==`)	None (valid code)

Best Practices for Error Handling

Never crash: The scanner should always recover and continue
Report precise location: Line and column numbers
Show context: Display the source line with the error
Limit cascading: One error shouldn't trigger many false errors
Suggest fixes: When possible, suggest what might be correct
Count errors: Stop after too many errors

Interview Questions

What types of errors can a lexical analyzer detect?

What is panic mode error recovery?

Why can't the lexical analyzer detect all errors in a program?

How does a scanner handle unterminated strings?

What information should a good error message include?

The filename, line number, column number, error description, the offending source line, and a caret (^) pointing to the exact error position. If possible, suggest a correction.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Lexical Errors and Error Recovery.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Compiler Design topic.

Search Terms

compiler-design, compiler design, compiler, design, lexical, analysis, errors, lexical errors and error recovery

Continue learning this concept

PreviousLexical Analyzer Design NextPatterns in Lexical Analysis