Skip to content

Executor: incorporate hotfixes into dev#16

Merged
johncmerfeld merged 4 commits intodevelopmentfrom
executor/em-error
Feb 12, 2026
Merged

Executor: incorporate hotfixes into dev#16
johncmerfeld merged 4 commits intodevelopmentfrom
executor/em-error

Conversation

@johncmerfeld
Copy link
Copy Markdown
Collaborator

@johncmerfeld johncmerfeld commented Feb 4, 2026

Incorporate #17 and #19 back into the main code history


Original:

This makes triaging earthmover_run errors much easier. Right now all the app receives is a Python error caused by the empty raise (which apparently doesn't work in a finally block). Now it receives the entire Earthmover logs as its "stacktrace", e.g.

Traceback (most recent call last):
  File "/executor/executor/executor.py", line 107, in execute
    self.earthmover_run()
  File "/executor/executor/executor.py", line 402, in earthmover_run
    raise Exception(em.stderr)
Exception: 2026-02-04 23:24:08.585 earthmover INFO starting...
2026-02-04 23:24:08.741 earthmover INFO skipping hashing and run-logging (no `state_file` defined in config)
2026-02-04 23:24:08.741 earthmover INFO creating output directory /executor/output
2026-02-04 23:24:51.360 earthmover ERROR (near line 64 of `/executor/bundles/packages/student_id_wrapper/packages/STAAR_Interim/earthmover.yaml` in `$transformations.staar_interim_student_assessment.operations:rename_columns`)  column ADMINISTRATION TYPE not present in the dataset)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/earthmover/__main__.py", line 256, in main
    em.generate(selector=args.selector)
  File "/usr/local/lib/python3.12/site-packages/earthmover/earthmover.py", line 378, in generate
    self.execute(active_graph)
  File "/usr/local/lib/python3.12/site-packages/earthmover/earthmover.py", line 275, in execute
    node.execute()
  File "/usr/local/lib/python3.12/site-packages/earthmover/nodes/transformation.py", line 48, in execute
    self.data = operation.execute(self.data, data_mapping=self.upstream_sources)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/earthmover/operations/column.py", line 188, in execute
    self.error_handler.throw(
  File "/usr/local/lib/python3.12/site-packages/earthmover/error_handler.py", line 117, in throw
    raise Exception(
Exception: (near line 64 of `/executor/bundles/packages/student_id_wrapper/packages/STAAR_Interim/earthmover.yaml` in `$transformations.staar_interim_student_assessment.operations:rename_columns`)  column ADMINISTRATION TYPE not present in the dataset)

Tested in my dev environment

@johncmerfeld johncmerfeld requested a review from jalvord1 February 4, 2026 23:28
@johncmerfeld johncmerfeld self-assigned this Feb 4, 2026
@johncmerfeld johncmerfeld changed the title Executor: send Earthmover logs in stacktrace (dev) Executor: send Earthmover logs in stacktrace Feb 5, 2026
@johncmerfeld johncmerfeld added the bug Something isn't working label Feb 5, 2026
@johncmerfeld johncmerfeld changed the title (dev) Executor: send Earthmover logs in stacktrace Executor: incorporate hotfixes into dev Feb 9, 2026
@johncmerfeld johncmerfeld merged commit 6d126c5 into development Feb 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants