feat: Add JSON, YAML, and Toon output options for zpm test results by AshokThangavel · Pull Request #979 · intersystems/ipm

AshokThangavel · 2025-12-04T13:36:40Z

Feature: Add Multi-Format Output (Toon, JSON, YAML) to `zpm test`

fixes #971

Summary

This Pull Request includes significant flexibility to the zpm test command by allowing users to define the output format for both terminal display and file generation. This enhancement supports Toon (default), JSON, and YAML formats, improving integration with external reporting tools and overall user experience.

It summarizes only the failed unit tests by default.

1. Terminal Output Control via `-f` / `-output` Flag

The new optional flag, -f (or -output), allows users to specify the desired output format of the test results displayed directly in the terminal (stdout).

Supported Formats: Toon, Json, Yaml
Default Behavior: If the flag is omitted, the output defaults to the Toon format.

Interaction with Verbose Flag (`-v`):

Explicit Output (No -v): If the -v (verbose) flag is not defined, the selected format will be displayed to the terminal.
- Example Command:
```
zpm "zpm test -only" 
# Output defaults to Toon format summary 
```
- Explicit Format Command:
```
zpm "zpm test -only -f Json"
# Output is displayed as JSON summary
```
Verbose Override: If the verbose flag (-v) is defined in the command, the -f flag is ignored, and no output summary is displayed (maintaining the existing verbose behavior).

Example Toon Output Structure (Terminal):

Test result summary

unitTest:
  id: 20
  namespace: USER
  duration: 63.107128
  testDateTime: 2025-12-04 13:01:56

results[1]{suiteName,testcaseName,methodName,status,assertAction,assertCounter,assertDescription,assertLocation}:
  (root),Test.PM.Unit.AAFailed,TestDatabaseResources,20,AssertStatusOK,1,"testing => ERROR #5001: test error","TestDatabaseResources+2^Test.PM.Unit.AAFailed.cls"

2. Dedicated Configuration Directives for File Generation (`-D`)

The existing configuration directive logic (-D) used for file output (e.g., -DUnitTest.JUnitOutput) has been extended to support the new formats to easily generate and store results files.

The following new configuration directives are now supported:

Directive	Output Format	Example Usage
`-DUnitTest.JsonOutput`	JSON	`/path/to/testresult.json`
`-DUnitTest.YamlOutput`	YAML	`/path/to/testresult.yaml`
`-DUnitTest.ToonOutput`	Toon	`/path/to/testresult.toon`

Example File Generation Command:

zpm "zpm test -only -DUnitTest.JsonOutput=/usr/irissys/mgr/user/testresult/testresult.json"

Unit Test Verification

A dedicated unit test class, Test.PM.Unit.TestResultsOPFormatAndFileGenTest, has been created to thoroughly test the new file generation functionality for all three formats.

The test results below confirm that files are successfully generated and then cleaned up:

Counter	Action	Status	Description	Location
3	AssertStatusOK	passed	yaml file generated successfully in /home/.../test-reports/	TestResultFileGeneration+12^...
4	AssertStatusOK	passed	Json file generated successfully in /home/.../test-reports/	TestResultFileGeneration+16^...
5	AssertStatusOK	passed	Toon file generated successfully in /home/.../test-reports/	TestResultFileGeneration+20^...
11	AssertStatusOK	passed	File 'test.Json' has been deleted successfully	ShowGeneratedFilesAndCleaup+14^...
12	AssertStatusOK	passed	File 'test.toon' has been deleted successfully	ShowGeneratedFilesAndCleaup+14^...
13	AssertStatusOK	passed	File 'test.yaml' has been deleted successfully	ShowGeneratedFilesAndCleaup+14^...

isc-tleavitt · 2025-12-04T14:16:56Z

@AshokThangavel thank you for this! And for your other recent contributions.

Just as a heads up / to set expectations, we have a serious backlog of large PRs right now and the people who most need to review (@isc-kiyer and I) also have a bunch of other non-IPM stuff on our plates. Getting community contribution on a project like this is like a dog chasing it's tail, and now that we've caught it we need to figure out what to do. :) There's a history of outstanding community contribution on IPM, but as part of inclusion in IRIS we've necessarily taken more control.

On this particular PR, big picture (without a thorough review):

The motivating use case is LLM processing of test results - as such, it's great that TOON is the default, but we don't want to have a list of all the passed assertions. A high level summary of the # of passed/failed assertions/tests and details on the failures would be ideal, with -verbose controlling inclusion of passed assertions in the specified output format.
The storage of errors that happen at various levels - e.g., an exception thrown during a test, an error %Status or exception thrown from On(Before|After)(AllTests|OneTest) - is nonintuitive. We need to be sure, and unit test, that errors at these levels are captured in output; otherwise you can have tests fail silently and not notice.

AshokThangavel · 2025-12-04T16:36:59Z

Hi @isc-tleavitt
Thanks again for taking the time to provide feedback despite your heavy workload. I completely understand the challenge of balancing community contributions with internal control, and I appreciate the dedication you both put into managing this project.

Regarding LLM-Friendly Output

The current implementation of the non-verbose terminal output for all formats (Toon, JSON, YAML) already excludes all passed assertions by default. It only displays or serializes the summary counts and the detailed information for failed assertions
When Interaction with the -v (verbose) flag is defined, the default behavior of displaying verbose output is maintained. so user must explicitly define -f flag (e.g., zpm "zpm test -v -f Toon"). If -v is present and -f is omitted by default

isc-dchui

Seconding @isc-tleavitt's comment about needing extensive testing. Could you please create a test IPM module in ipm\tests\integration_tests\Test\PM\Integration\_data and a slew of tests that check that failures in the various places (in a test method, in On(Before|After)(AllTests|OneTest)) are caught and outputted properly?

isc-dchui · 2026-02-02T14:55:58Z

src/cls/IPM/ResourceProcessor/Test.cls

+            write !
+            if $data(pParams("outputformat"),outputFormat)||('tVerbose) {
+                write !,"Test result summary",!
+                // TODO: Move this default format to ^IPM.Config.Test("outputFormat") rather than keeping it hardcoded.


%IPM.Repo.UniversalSettings would also be a good place for this. In either case, you still want to have some default if the globals aren't populated and some validation if they are.

Hi @isc-dchui
Thank you. I've created TestReportFormat in %IPM.Repo.UniversalSettings settings class and add description as well.

isc-dchui · 2026-02-02T14:59:26Z

src/cls/IPM/Test/JsonOutput.cls

+	pTestStatus As %String = "",
+	pTestIndex As %Integer = {$order(^UnitTest.Result(""),-1)}) As %Status
+{
+    set tSC = $$$OK


Nit: in new code, avoid t- and p- prefixes

removed the t and p prefix Infront of variables

isc-dchui · 2026-02-02T15:00:33Z

src/cls/IPM/Test/JsonOutput.cls

+        set fileStream.TranslateTable="UTF8"
+        do fileStream.LinkToFile(pFileName)
+        set responseJson = ..JSON(pTestIndex, pTestStatus)
+        do fileStream.Write(responseJson.%ToJSON())


Make sure to ThrowOnError for LinkToFile and Write here and do the same throughout for methods that return statuses

Thank you. Updated the code!

isc-dchui · 2026-02-02T15:04:22Z

src/cls/IPM/Test/JsonOutput.cls

+    try {
+        set responseJson= ..JSON(pTestIndex, pCaseStatus)
+        write !
+        do responseJson.%ToJSON()


Not fully sure what's happening here. Doesn't %ToJSON return the string of the dynamic object? But it's not being written here.

Thank you. Updated the code!

isc-dchui · 2026-02-02T15:05:59Z

src/cls/IPM/Test/YamlOutput.cls

+            do yamlStream.WriteLine("          methods:")
+        }
+        do yamlStream.WriteLine("            - methodName: """_tResult.methodName_"""")
+        //do yamlStream.WriteLine("              testMethod: """_tResult.testMethod_"""")


Any particular reason why these lines are commented out?

No Specific reason. Updated the code!

isc-dchui · 2026-02-02T15:08:44Z

src/cls/IPM/Test/YamlOutput.cls

+            do yamlStream.WriteLine("  namespace: """_tResult.namespace_"""")
+            do yamlStream.WriteLine("  duration: "_tResult.duration)
+            do yamlStream.WriteLine("  testDateTime: """_tResult.testDateTime_"""")
+            do yamlStream.WriteLine( "")


Nit: extra space before the ""

removed the space. thank you!

isc-dchui · 2026-02-02T15:09:32Z

src/cls/IPM/Test/ToonOutput.cls

+        while tResult.%Next() {
+            if currentID = "" {
+                set currentID = pTestIndex
+                do fileStream.WriteLine("unitTest:")


Lots of statuses that need checking here

Thank you. Updated the code!

Still need to check the %Status returned by the WriteLine() calls

isc-dchui · 2026-02-02T15:10:10Z

src/cls/IPM/Main.cls

 <modifier name="dev" dataAlias="DeveloperMode" dataValue="1" description="Sets the DeveloperMode flag for the module's lifecycle. Key consequences of this are that ^Sources will be configured for resources in the module, and installer methods will be called with the dev mode flag set." />
 <modifier name="quiet" aliases="q" dataAlias="Verbose" dataValue="0" description="Produces minimal output from the command." />
 <modifier name="verbose" aliases="v" dataAlias="Verbose" dataValue="1" description="Produces verbose output from the command." />
+<modifier name="output-format" aliases="output,f" dataAlias="outputformat" value="true" description="Specifies the desired output format for the command result (e.g., json, yaml, toon)." />


Might as well include "format" as an alias if "f" is an alias.

Also this looks like it only affects the test output, so you should specify that.

Added format alias as well. Thank you!

isc-dchui · 2026-02-02T15:13:14Z

CHANGELOG.md

 - #938 Added flag -export-python-deps to package command
 - #462: The `repo` command for repository configuration now supports secret input terminal mode for passwords with the `-password-stdin` flag
 - #935: Adding a generic JFrog Artifactory tarball resource processor for bundling artifact with a package and deploying it to a final location on install.
+- #971: Adds support for JSON, YAML, and Toon formats via the -f flag and new -DUnitTest.*Output directives.


Move this to 0.10.7 please

moved to 0.10.7 thank you!

isc-dchui · 2026-02-02T15:14:40Z

tests/unit_tests/Test/PM/Unit/TestResultsOPFormatAndFileGenTest.cls

@@ -0,0 +1,60 @@
+/// Unit Test Class to validate the ZPM 'test' command output configuration.


Would also be good to test the verify command

test on zpm "zpm verify" as well. Thank you!

Wait sorry, where is this test on zpm verify?

isc-dchui · 2026-03-13T14:39:00Z

@AshokThangavel I believe this one is still in your court

AshokThangavel · 2026-03-13T15:05:52Z

Hi @isc-dchui Thanks for the reminder. I'll review the comments and push the updates.

isc-dchui

@AshokThangavel Made another pass!

isc-dchui · 2026-03-18T14:10:58Z

src/cls/IPM/Test/Abstract.cls

-	pFileName As %String,
-	pCaseStatus As %String = "",
-	pTestIndex As %Integer = {$order(^UnitTest.Result(""),-1)}) As %Status [ Abstract ]
+	FileName As %String,


Nit: method parameters should be in camelCase, not PascalCase

isc-dchui · 2026-03-18T14:15:24Z

src/cls/IPM/ResourceProcessor/Test.cls

+                if defaultOutputFormat'="" {
+                    set outputFormat = defaultOutputFormat
+                }
+                else {


Nit: should be one line, i.e. } else {

isc-dchui · 2026-03-18T14:16:14Z

src/cls/IPM/Test/Abstract.cls

+{
+SELECT
+    count(*) as TotalCounts,
+    tinstance.Namespace AS namespace,


Nit: no need for t- prefixes here either

isc-dchui · 2026-03-18T14:18:24Z

src/cls/IPM/Test/JsonOutput.cls

+
+ClassMethod OutputToDevice(
+	TestIndex As %Integer = {$order(^UnitTest.Result(""),-1)},
+	pCaseStatus As %String = "") As %Status


Nit: remove p- prefix and change to camelCase

isc-dchui · 2026-03-18T14:20:08Z

src/cls/IPM/Test/JsonOutput.cls

+	TestStatus As %String) As %DynamicObject
+{
+    if TestStatus'="" {
+        set tResult = ..FilteredTestResultsFunc(TestIndex, TestStatus)


Nit: remove the t-prefixes

isc-dchui · 2026-03-18T14:23:22Z

src/cls/IPM/Test/ToonOutput.cls

+        while tResult.%Next() {
+            if currentID = "" {
+                set currentID = pTestIndex
+                do fileStream.WriteLine("unitTest:")


Still need to check the %Status returned by the WriteLine() calls

isc-dchui · 2026-03-18T14:26:06Z

tests/unit_tests/Test/PM/Unit/TestResultsOPFormatAndFileGenTest.cls

@@ -0,0 +1,60 @@
+/// Unit Test Class to validate the ZPM 'test' command output configuration.


Wait sorry, where is this test on zpm verify?

feat: Add JSON, YAML, and Toon output options for zpm test results

4307599

AshokThangavel requested review from isc-dchui, isc-egabhart, isc-eneil, isc-jili, isc-jlechtne, isc-kiyer, isc-pbarton and isc-tleavitt as code owners December 4, 2025 13:36

docs: Add changelog entry

dbb7bed

isc-dchui requested changes Feb 2, 2026

View reviewed changes

Merge branch 'main' into feat/llm-friendly-output-for-ipm-test

cd77c52

AshokThangavel requested a review from isc-cborbonm as a code owner March 13, 2026 15:08

AshokThangavel added 2 commits March 16, 2026 22:30

refactor: updated the code based on comments

aef4c32

Refactor: add format alias and clarify output-format description

d1c0dce

isc-dchui reviewed Mar 18, 2026

View reviewed changes

		@@ -0,0 +1,60 @@
		/// Unit Test Class to validate the ZPM 'test' command output configuration.

Conversation

AshokThangavel commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Feature: Add Multi-Format Output (Toon, JSON, YAML) to zpm test

Summary

1. Terminal Output Control via -f / -output Flag

Interaction with Verbose Flag (-v):

Example Toon Output Structure (Terminal):

2. Dedicated Configuration Directives for File Generation (-D)

Example File Generation Command:

Unit Test Verification

Uh oh!

isc-tleavitt commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AshokThangavel commented Dec 4, 2025

Regarding LLM-Friendly Output

Uh oh!

isc-dchui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isc-dchui Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isc-dchui commented Mar 13, 2026

Uh oh!

AshokThangavel commented Mar 13, 2026

Uh oh!

isc-dchui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshokThangavel commented Dec 4, 2025 •

edited

Loading

Feature: Add Multi-Format Output (Toon, JSON, YAML) to `zpm test`

1. Terminal Output Control via `-f` / `-output` Flag

Interaction with Verbose Flag (`-v`):

2. Dedicated Configuration Directives for File Generation (`-D`)

isc-tleavitt commented Dec 4, 2025 •

edited

Loading

isc-dchui Feb 2, 2026 •

edited

Loading