Wiki Navigation

Unit Testing (Pro): Baselines and Exports

This page covers regression snapshot baselines, test run result exports, and portable test-definition transfer bundles.

Baselines (regression snapshots)

Regression snapshot tests capture “expected” outputs and alert on drift. Treat baselines like code:

  • update baselines intentionally (after expected data changes),
  • review baseline diffs before accepting,
  • keep tolerances realistic (avoid flaky tests).

Prompt:

“Create a regression snapshot baseline for our KPI suite. Explain when we should update the baseline and how to review changes safely.”

Masking note (common in governed environments):

  • If your environment has PII/numeric masking enabled, some snapshot modes are intentionally restricted. If “full row capture” or “top-N capture” is blocked, ask the assistant to switch to a safe baseline mode (hash/aggregate) or to explain what governance setting is enforcing the restriction.

What baseline management supports today:

  • list saved baselines for a test,
  • capture a new baseline,
  • delete an old baseline.

Prompt:

“List baselines for monthly-sales-snapshot, capture a new one, then keep only the current baseline.”

Exporting results (shareable formats)

export is for run results and reports. Use export_tests / import_tests when you want to move test definitions between models.

Ask for the format that fits your workflow:

  • Markdown (PR comments / tickets)
  • JUnit (CI systems)
  • HTML (readable reports)
  • JSON (automation)

Prompts:

“Run all enabled tests and export results as Markdown suitable for a PR comment.” “Export the last test run as JUnit for CI.”

Where exports go (and what “content omitted” means)

Test results can be returned inline in chat, or written to a file on the machine running the MCP server.

Two common cases write to disk:

  1. You explicitly ask to save the export to a file (recommended for PR artifacts).
  2. The export is too large to return inline; the server automatically saves it and returns a file path instead of the content.

Important:

  • If the MCP server runs remotely (or in a container), the file is saved on the server host, not your laptop.
  • Returned saved_to paths are server-local absolute paths.
  • Relative paths are interpreted on the server. They resolve under MCP_ENGINE_TESTS_EXPORT_ROOT when that is configured; otherwise SemanticOps MCP uses its managed tests directory (typically ~/.mcp-engine/tests).

Copy/paste prompts:

“Export the last test run as Markdown and save it to kpi-suite.md. Then summarize and tell me the absolute saved_to path.” “If the export is too large to return inline, save it to a file and tell me the path.”

Transferring test definitions

Definition bundles contain tests only. They do not include run history, snapshot baselines, or stored model identity, so imports can target the currently connected model. Bundles include full spec, assert, and context payloads; when masking is enabled, export requires an explicit include_sensitive opt-in.

Prompts:

“Export all tests tagged smoke as a portable test-definition bundle.” “Preview this bundle import, show add/update/skip/error/warning counts, and warn me about missing model references without blocking the import.” “Apply the bundle with add_skip mode so existing tests are left unchanged.”

Conflict modes:

  • add_update: add new tests and update existing ids.
  • add_skip: add new tests and skip existing ids.
  • fail_on_conflict: report existing ids as conflicts and do not apply until resolved.

See also