JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD B.Tech III Year PRINCIPLES OF COMPILER CONSTRUCTION Previous Question Paper (February/March - 2022 - R18)

The Lexical Analyzer, or scanner, is the first phase of a compiler. Its primary role is to read the stream of characters from the source program, group them into meaningful sequences called lexemes, and produce a stream of tokens as output for the syntax analyzer.

Key functions include:

• Tokenization: Identifying keywords, identifiers, operators, literals, and punctuation.
• Stripping: Removing white spaces, tabs, newlines, and comments that are not relevant to the syntax.
• Error Handling: Reporting lexical errors such as illegal characters or unterminated strings.
• Symbol Table Interaction: Entering identifiers into the symbol table to store attributes like type and scope.

The lexical analyzer often operates as a subroutine of the parser, providing the next token only when requested. This separation simplifies the compiler design by isolating the platform-dependent character-handling logic from the language-dependent syntax logic.

Input buffering is a technique used by the lexical analyzer to increase the efficiency of reading characters from the source file. Since reading from disk is expensive, the scanner uses a buffer to hold a block of characters, minimizing the number of system calls.

To handle the lookahead requirement where the scanner must look one or more characters ahead to distinguish between tokens (e.g., distinguishing '=' from '=='), a two-buffer scheme is typically employed:

• The buffer is divided into two halves, each of size N (the size of a disk block).
• Two pointers, lexeme_beginning and forward, move across the buffer.
• When the forward pointer reaches the end of one half, the scanner refills the other half with the next block of input.
• Sentinels (e.g., the EOF character) are placed at the end of each buffer half to detect the boundary, reducing the number of conditional checks required per character read.

This approach allows the scanner to perform efficient lookahead while maintaining a continuous stream of input, ensuring that the lexical analysis process remains fast and scalable.