Unicode Lexer Generator
This is a Unicode aware lexer generator for C. It reads a declarative lexer specification, typically written using regular expressions and C actions, and produces a C scanner that recognizes and classifies tokens from an input stream.
The generated scanner converts raw input text into tokens that can be consumed by a parser, evaluator, compiler, or domain-specific reader. Generating the scanner from a compact specification keeps its behavior tied to clearly defined rules instead of scattering token recognition across handwritten conditionals.
The scanner can be used directly by C programs or together with a parser to build configuration languages, command interpreters, data format parsers, domain specific languages, source code tools, and other text processing applications.
The lexer generator provides:
-
Generates a C scanner from a compact, declarative specification.
-
Flex compatible syntax for regular expressions, rules, and C actions.
-
Scans Unicode code points rather than raw encoded bytes.
-
Supports Unicode characters and character ranges directly in token rules.
-
Provides start conditions for activating different rule sets in different contexts.
-
Produces scanners that can be used independently or together with a parser.
Parser Generator
This parser generator for C turns a declarative grammar into a C parser for programming languages, data formats, configuration files, command languages, and other forms of structured input.
Grammars are written using familiar Bison compatible, BNF style production rules with C semantic actions. The generated parser consumes tokens from a lexer, verifies that they follow the grammar, and can evaluate expressions, build syntax trees, create application objects, or produce other values defined by the program.
Using a formal grammar keeps language syntax centralized, readable, and easier to test and extend than parsing logic built from handwritten conditionals. The parser can be paired with a Unicode aware lexer to support identifiers, operators, literals, and symbols beyond ASCII.
The parser generator provides:
-
Generates a C parser from a compact, declarative grammar.
-
Bison compatible grammar syntax and C semantic actions.
-
Supports context free grammars, precedence, optional elements, and nested structures.
-
Provides syntax error detection and recovery.
-
Supports direct evaluation and the construction of syntax trees, documents, and objects.
-
Keeps lexical analysis, grammatical structure, and semantic processing clearly separated.