refactor(compiler): refactor fory compiler into hierarchical architecture (#3179)

## Why?

The current Fory compiler mixes FDL-native and protobuf-compatible
syntax handling in a single parser, making it difficult to add support
for new IDL formats like .proto and .fbs files. The validation logic is
scattered across parsing and Schema.validate(), and there's no clear
separation between parsing, semantic analysis, and code generation.

## What does this PR do?

This PR refactors the Fory compiler into a hierarchical, multi-frontend
architecture that establishes the Fory IDL AST as the **canonical
intermediate representation (IR)**, with separate frontend parsers for
different IDL formats.

**Key changes:**

1. **New directory structure** with clear separation of concerns:
   - `ir/` - Intermediate Representation (canonical Fory AST)
- `ast.py` - Core AST node definitions with `SourceLocation` tracking
- `types.py` - Extended type system (primitives including varint, tagged
types, etc.)
     - `validator.py` - Centralized semantic validation
     - `emitter.py` - FDL text emitter for debugging translated schemas
   - `frontend/` - IDL Frontends
     - `base.py` - Base frontend interface
     - `fdl/` - FDL Frontend (lexer + parser)
- `proto/` - Protobuf Frontend (lexer + parser + translator to Fory IR)
     - `fbs/` - FlatBuffers Frontend (placeholder)

2. **Proto3 frontend** - Full support for parsing `.proto` files and
translating to Fory IR:
- Proto3 syntax parsing (messages, enums, nested types, maps, repeated
fields)
- Type mapping (int32→var_uint32, sint32→varint32, fixed32→uint32, etc.)
- Fory extension options (`(fory).id`, `(fory).ref`, `(fory).nullable`,
etc.)
   - Well-known types support (google.protobuf.Timestamp, Duration)

3. **Simplified FDL syntax** - Removed protobuf-style `(fory)` prefix
from options:
   - File options: `option use_record_for_java_message = true;`
   - Type options: `message Foo [id=100] { ... }`
   - Field options: `MyType data = 1 [ref=true, nullable=true];`

4. **Extended type system** with new primitive kinds:
   - Signed/unsigned variants: `int8`-`int64`, `uint8`-`uint64`
- Variable-length encoding: `varint32`, `varint64`, `var_uint32`,
`var_uint64`
   - Tagged types: `tagged_int64`, `tagged_uint64`
   - Additional types: `float16`, `duration`, `decimal`

5. **Improved code generators** for all target languages with better
type mapping

6. **CLI enhancements**:
   - Auto-detect input format by file extension (`.fdl`, `.proto`)
   - New `--emit-fdl` flag to output translated FDL for debugging

7. **Cross-language integration tests** for proto-based schemas

## Related issues

Closes #3178

## Does this PR introduce any user-facing change?

- CLI now accepts `.proto` files directly (in addition to `.fdl`)
- FDL syntax simplified: `option (fory).xxx` → `option xxx`
- New primitive types available in FDL

- [x] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?

## Benchmark

N/A - This is a compiler refactoring that doesn't affect runtime
performance.

Author: chaokunyang

.github/sync.yml

@@ -20,5 +20,7 @@ apache/fory-site@main:
     dest: docs/guide/
   - source: docs/specification/
     dest: docs/specification/
+  - source: docs/compiler/
+    dest: docs/compiler/
   - source: docs/benchmarks/
     dest: static/img/benchmarks/

benchmarks/cpp_benchmark/README.md

@@ -31,12 +31,12 @@ Note: Protobuf is fetched automatically via CMake FetchContent, so no manual ins
 
 | Datatype     | Operation   | Fory TPS   | Protobuf TPS | Faster      |
 | ------------ | ----------- | ---------- | ------------ | ----------- |
-| Mediacontent | Serialize   | 2,430,924  | 484,368      | Fory (5.0x) |
-| Mediacontent | Deserialize | 740,074    | 387,522      | Fory (1.9x) |
-| Sample       | Serialize   | 4,813,270  | 3,021,968    | Fory (1.6x) |
-| Sample       | Deserialize | 915,554    | 684,675      | Fory (1.3x) |
-| Struct       | Serialize   | 18,105,957 | 5,788,186    | Fory (3.1x) |
-| Struct       | Deserialize | 7,495,726  | 5,932,982    | Fory (1.3x) |
+| Mediacontent | Serialize   | 11,319,876 | 1,181,595    | Fory (9.6x) |
+| Mediacontent | Deserialize | 2,729,388  | 835,956      | Fory (3.3x) |
+| Sample       | Serialize   | 16,899,403 | 10,575,760   | Fory (1.6x) |
+| Sample       | Deserialize | 3,079,241  | 1,450,789    | Fory (2.1x) |
+| Struct       | Serialize   | 43,184,198 | 29,359,454   | Fory (1.5x) |
+| Struct       | Deserialize | 54,599,691 | 38,796,674   | Fory (1.4x) |
 
 ## Quick Start
 
@@ -47,6 +47,34 @@ cd benchmarks/cpp_benchmark
 ./run.sh
 ```
 
+### Run Options
+
+```bash
+./run.sh --help
+
+Options:
+  --data <struct|sample>       Filter benchmark by data type
+  --serializer <fory|protobuf> Filter benchmark by serializer
+  --duration <seconds>         Minimum time to run each benchmark (e.g., 10, 30)
+  --debug                      Build with debug symbols for profiling
+```
+
+Examples:
+
+```bash
+# Run only Struct benchmarks
+./run.sh --data struct
+
+# Run only Fory benchmarks
+./run.sh --serializer fory
+
+# Run each benchmark for at least 10 seconds (for more stable results)
+./run.sh --duration 10
+
+# Combine options
+./run.sh --data struct --serializer fory --duration 5
+```
+
 ## Building
 
 ```bash

benchmarks/cpp_benchmark/run.sh

@@ -32,6 +32,7 @@ JOBS=16
 DATA=""
 SERIALIZER=""
 DEBUG_BUILD=false
+DURATION=""
 
 # Parse arguments
 usage() {
@@ -42,6 +43,7 @@ usage() {
     echo "Options:"
     echo "  --data <struct|sample>       Filter benchmark by data type"
     echo "  --serializer <fory|protobuf> Filter benchmark by serializer"
+    echo "  --duration <seconds>         Minimum time to run each benchmark (e.g., 10, 30)"
     echo "  --debug                      Build with debug symbols and low optimization for profiling"
     echo "  --help                       Show this help message"
     echo ""
@@ -50,6 +52,7 @@ usage() {
     echo "  $0 --data struct            # Run only Struct benchmarks"
     echo "  $0 --serializer fory        # Run only Fory benchmarks"
     echo "  $0 --data struct --serializer fory"
+    echo "  $0 --duration 10            # Run each benchmark for at least 10 seconds"
     echo "  $0 --debug                  # Build for profiling (visible function names in flamegraph)"
     echo ""
     echo "For profiling/flamegraph, use: ./profile.sh"
@@ -66,6 +69,10 @@ while [[ $# -gt 0 ]]; do
             SERIALIZER="$2"
             shift 2
             ;;
+        --duration)
+            DURATION="$2"
+            shift 2
+            ;;
         --debug)
             DEBUG_BUILD=true
             shift
@@ -125,6 +132,10 @@ echo ""
 # Step 2: Run benchmark
 echo -e "${YELLOW}[2/3] Running benchmark...${NC}"
 BENCH_ARGS="--benchmark_format=json --benchmark_out=benchmark_results.json"
+if [[ -n "$DURATION" ]]; then
+    BENCH_ARGS="$BENCH_ARGS --benchmark_min_time=${DURATION}s"
+    echo -e "Duration: ${DURATION}s per benchmark"
+fi
 if [[ -n "$FILTER" ]]; then
     BENCH_ARGS="$BENCH_ARGS --benchmark_filter=$FILTER"
     echo -e "Filter: ${FILTER}"

compiler/README.md

@@ -220,29 +220,27 @@ message Example {
 }
 ```
 
-### Fory Extension Options
+### Fory Options
 
-FDL supports protobuf-style extension options using the `(fory)` prefix:
+FDL uses plain option keys without a `(fory)` prefix:
 
 **File-level options:**
 
 ```fdl
-option (fory).use_record_for_java_message = true;
-option (fory).polymorphism = true;
+option use_record_for_java_message = true;
+option polymorphism = true;
 ```
 
 **Message/Enum options:**
 
 ```fdl
-message MyMessage {
-    option (fory).id = 100;
-    option (fory).evolving = false;
-    option (fory).use_record_for_java = true;
+message MyMessage [id=100] {
+    option evolving = false;
+    option use_record_for_java = true;
     string name = 1;
 }
 
-enum Status {
-    option (fory).id = 101;
+enum Status [id=101] {
     UNKNOWN = 0;
     ACTIVE = 1;
 }
@@ -252,25 +250,29 @@ enum Status {
 
 ```fdl
 message Example {
-    MyType friend = 1 [(fory).ref = true];
-    string nickname = 2 [(fory).nullable = true];
-    MyType data = 3 [(fory).ref = true, (fory).nullable = true];
+    MyType friend = 1 [ref=true];
+    string nickname = 2 [nullable=true];
+    MyType data = 3 [ref=true, nullable=true];
 }
 ```
 
-See `extension/fory_options.proto` for the complete list of available options.
-
 ## Architecture
 
 ```
 fory_compiler/
 ├── __init__.py           # Package exports
 ├── __main__.py           # Module entry point
 ├── cli.py                # Command-line interface
-├── parser/
-│   ├── ast.py            # AST node definitions
-│   ├── lexer.py          # Hand-written tokenizer
-│   └── parser.py         # Recursive descent parser
+├── frontend/
+│   └── fdl/
+│       ├── __init__.py
+│       ├── lexer.py      # Hand-written tokenizer
+│       └── parser.py     # Recursive descent parser
+├── ir/
+│   ├── __init__.py
+│   ├── ast.py            # Canonical Fory IDL AST
+│   ├── validator.py      # Schema validation
+│   └── emitter.py        # Optional FDL emitter
 └── generators/
     ├── base.py           # Base generator class
     ├── java.py           # Java POJO generator
@@ -280,13 +282,13 @@ fory_compiler/
     └── cpp.py            # C++ struct generator
 ```
 
-### Parser
+### FDL Frontend
 
-The parser is a hand-written recursive descent parser that produces an AST:
+The FDL frontend is a hand-written lexer/parser that produces the Fory IDL AST:
 
-- **Lexer** (`lexer.py`): Tokenizes FDL source into tokens (keywords, identifiers, punctuation)
-- **AST** (`ast.py`): Defines node types - `Schema`, `Message`, `Enum`, `Field`, `FieldType`
-- **Parser** (`parser.py`): Builds AST from token stream with validation
+- **Lexer** (`frontend/fdl/lexer.py`): Tokenizes FDL source into tokens
+- **Parser** (`frontend/fdl/parser.py`): Builds the AST from the token stream
+- **AST** (`ir/ast.py`): Canonical node types - `Schema`, `Message`, `Enum`, `Field`, `FieldType`
 
 ### Generators
 

compiler/fory_compiler/__init__.py

@@ -15,13 +15,14 @@
 # specific language governing permissions and limitations
 # under the License.
 
-"""FDL (Fory Definition Language) compiler for Apache Fory."""
+"""Fory IDL compiler for Apache Fory."""
 
 __version__ = "0.1.0"
 
-from fory_compiler.parser.ast import Schema, Message, Enum, Field, EnumValue, Import
-from fory_compiler.parser.parser import Parser
-from fory_compiler.parser.lexer import Lexer
+from fory_compiler.ir.ast import Schema, Message, Enum, Field, EnumValue, Import
+from fory_compiler.frontend.fdl import FDLFrontend
+from fory_compiler.frontend.fbs import FBSFrontend
+from fory_compiler.frontend.proto import ProtoFrontend
 
 __all__ = [
     "Schema",
@@ -30,6 +31,7 @@
     "Field",
     "EnumValue",
     "Import",
-    "Parser",
-    "Lexer",
+    "FDLFrontend",
+    "FBSFrontend",
+    "ProtoFrontend",
 ]