Reference implementation of HERACLES Assembler tools

.gitignore

@@ -0,0 +1,205 @@
+#========================
+# Intermediate and Output files
+#========================
+
+*.tmp
+*.temp
+*.mem
+*.minst
+*.cinst
+*.xinst
+*.csv
+*.out
+#*#
+*~
+tmp/
+
+
+# Local files
+#========================
+
+*.yml
+*.pyc
+*.bak
+*.pkl
+*.lock
+*.swp
+tfedlrn.egg-info/
+bin/out
+
+#========================
+# Generated docs
+#========================
+
+*.htm
+*.html
+*.pdf
+html
+latex
+[Dd]ocs/
+
+
+#========================
+# Eclipse & PyDev intermediate files
+#========================
+
+.metadata/
+RemoteSystemsTempFiles/
+.settings
+.project
+.pydevproject
+
+#========================
+# Visual Studio
+#========================
+
+# User-specific files
+*.rsuser
+*.suo
+*.user
+*.userosscache
+*.sln.docstates
+
+# User-specific files (MonoDevelop/Xamarin Studio)
+*.userprefs
+
+# Mono auto generated files
+mono_crash.*
+
+# Visual Studio cache/options directory
+.vs/
+.vscode/
+
+#========================
+# Python general stuff
+#========================
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

.pre-commit-config.yaml

@@ -38,6 +38,10 @@ repos:
           - HEADER
           - --comment-style
           - //                   # defaults to: #
+        - id: remove-tabs
+          name: remove-tabs
+          files: \.(py)$
+          args: [--whitespaces-count, '4']
     - repo: https://github.com/psf/black-pre-commit-mirror
       rev: 24.4.0 # Updated 2024/04
       hooks:

assembler_tools/hec-assembler-tools/README.md

@@ -0,0 +1,139 @@
+# HERACLES Code Generation Framework (Reference Implementation)
+
+The tools in this directory are the reference implementation of and Assembler codegenerator that takes a pre-generated Polynomial Instruction Set Architecture (P-ISA) program containing instructions that use an abstract, flat memory model for polynomial operations, such as those applied in homomorphic encryption (HE), and maps them to a corresponding set of instructions compatible with the HERACLES architecture, accounting for hardware restrictions, including memory management for the HERACLES memory model.
+
+## Table of Contents
+1. [Dependencies](#dependencies)
+2. [Inputs](#as_inputs)
+3. [Outputs](#as_outputs)
+   1. [Assembler Instruction Specs](#asm_specs)
+4. [Executing the Assembler](#executing_asm)
+   1. [Running for a Pre-Generated Kernel](#executing_single)
+   2. [Running for a Batch of Operations](#executing_batch)
+5. [Debug Tools](./debug_tools/README.md)
+
+## Dependencies <a name="dependencies"></a>
+
+This project is Python based. Dependencies for the project are as follows:
+
+- Python 3.10 or newer (Tested with 3.10)
+- Python pip 22.0 or newer
+- Requirements for the Python environment listed in file `requirements.txt`.
+
+  Install with `pip`:
+
+  ```bash
+  pip3 install -r requirements.txt
+  ```
+
+It is recommended to install Python dependencies and run inside a [Python virtual environment](https://virtualenv.pypa.io/en/stable/index.html) to avoid polluting the global Python.
+
+## Inputs <a name="as_inputs"></a>
+
+The assembler framework requires two inputs:
+
+- **Abstract P-ISA kernel**: a kernel of instructions in abstract P-ISA where the memory model is abstracted with a flat, infinite structure.
+
+- **Memory mapping metadata**: metadata information indicating the location where variable identifiers from the kernel are stored in the HERACLES HBM.
+
+Kernels and metadata are structured in comma-separated value (csv) files.
+
+P-ISA kernels, along with corresponding memory metadata required as input to the assembler, are generated by Python script `HERACLES-SEAL-isa-mapping/kernels/run_he_op.py` in the repo [HERACLES-SEAL-isa-mapping](https://github.com/IntelLabs/HERACLES-SEAL-isa-mapping)
+
+## Outputs <a name="as_outputs"></a>
+
+On a successful run, given a P-ISA kernel in file `filename.csv` (and corresponding memory metadata file), the assmebler generates three files:
+
+- `filename.minst`: contains the list of instructions for the MINST queue.
+- `filename.cinst`: contains the list of instructions for the CINST queue.
+- `filename.xinst`: contains the list of instructions for the XINST queue.
+
+### Assembler Output Instruction Specs <a name="asm_specs"></a>
+
+The format for the output files and instruction set can be found at [HCGF Instruction Specification](docsrc/specs.md).
+
+## Executing the Assembler <a name="executing_asm"></a>
+
+There are two ways to execute the assembler:
+
+- [Running on a pre-generated kernel](#executing_single): uses the main interface of the assembler to assemble a single pre-existing kernel.
+
+  This method is intended for a production chain.
+
+or
+
+- [Running for a batch of kernels](#executing_batch): uses a provided script wrapper to generate a collection of kernels and runs them through the assembler.
+
+  This method is intended for testing purposes as it generates test kernels using external tools before assembling.
+
+### Running for a Pre-Generated Kernel <a name="executing_single"></a>
+
+Given a P-ISA kernel (`filename.csv`) and corresponding memory mapping file (`filename.mem`), there are three steps to assemble them into HERACLES code.
+
+1. Pre-process the P-ISA input kernel using `he_prep.py`.
+
+```bash
+# pre-process kernel: outputs filename.tw.csv
+python3 he_prep.py filename.csv
+```
+
+2. Assemble the pre-processed result using `he_as.py`.
+
+```bash
+# use output from pre-processing as input to asm:
+# outputs filename.tw.minst, filename.tw.cinst, filename.tw.xinst
+python3 he_as.py filename.tw.csv --input_mem_file filename.mem
+```
+
+3. Link the assembler output into a HERACLES program using `he_link.py`.
+
+```bash
+# link assembled output (input prefix: filename.tw)
+# outputs filename.minst, filename.cinst, filename.xinst
+python3 he_link.py filename.tw --input_mem_file filename.mem --output_prefix filename
+```
+
+This will generate the main three output files in the same directory as the input file:
+
+- `filename.minst`: contains the list of instructions for the MINST queue.
+- `filename.cinst`: contains the list of instructions for the CINST queue.
+- `filename.xinst`: contains the list of instructions for the XINST queue.
+
+Intermediate files, if any, are kept as well.
+
+The linker program is able to link several assembled kernels into a single HERACLES program, given a correct memory mapping for the resulting program.
+
+This version of executing is intended for the assembler to be usable as part of a compilation pipeline.
+
+Use commands below for more configuration information, including changing output directories, input and output filenames, etc.
+
+```bash
+python3 he_prep.py -h
+python3 he_as.py -h
+python3 he_link.py -h
+```
+
+### Running for a Batch of Operations <a name="executing_batch"></a>
+
+This project provides script `gen_he_ops.py` that allows for assembling a batch of P-ISA kernels generated for HE operations. It calls the generator script internally to generate a batch of kernels, and then runs them through the assembler.
+
+Since the script to generate P-ISA kernels resides in another repo (HERACLES-SEAL-isa-mapping), we must specify the location of the cloned external repo using the environment variable `HERACLES_MAPPING_PATH`. Correctly setting this variable should result in the following path being valid: `$HERACLES_MAPPING_PATH/kernels/run_he_ops.py` .
+
+Provided script, `gen_he_ops.py`, takes in a YAML configuration file that specifies parameters for operations to assemble. To obtain a template for the configuration file, use the script itself with the `--dump` command line flag. Use `-h` flag for more information.
+
+```bash
+# save template for configuration file to ./config.yaml
+python3 gen_he_ops.py config.yaml --dump
+```
+
+Set the parameters in the configuration file to match your needs and then execute the script as shown below (code for Linux terminal).
+
+```bash
+# env variable pointing to HERACLES-SEAL-isa-mapping
+export HERACLES_MAPPING_PATH=/path/to/HERACLES-SEAL-isa-mapping
+python3 gen_he_ops.py config.yaml
+```
+
+Based on your chosen configuration, this will generate kernels, run them through the assembler and place all the outputs (and intermediate files) in the output directory specified in the configuration file.
+
+This way of executing is mostly intended for testing purposes.

assembler_tools/hec-assembler-tools/assembler/common/__init__.py

@@ -0,0 +1,13 @@
+import os
+
+def makeUniquePath(path: str) -> str:
+    """
+    Returns a unique, normalized, and absolute version of the given file path.
+
+    Args:
+        path (str): The file path to be processed.
+
+    Returns:
+        str: A unique, normalized, and absolute version of the input path.
+    """
+    return os.path.normcase(os.path.realpath(os.path.expanduser(path)))