SemanticOps Test Runner (Pro)

SemanticOps Test Runner is a desktop app for defining, running, and reviewing tests for Power BI semantic models. It is designed for Power BI developers who want a guided testing workflow without hand-authoring every manage_tests payload.

What you can use it for

• Create semantic model tests with a guided wizard.
• Apply starter packs for metadata, documentation, relationship, referential-integrity, presentation, and time-intelligence checks.
• Filter tests by folder, type, status, or search text, then run all tests or selected tests.
• Review pass/fail results, recent runs, flaky tests, and degrading tests.
• Export results as Markdown, JUnit XML, HTML, or JSON.
• Validate model changes before publishing, merging, or handing work to another developer.

What it tests

The Test Runner is built on the same Pro unit-testing engine as manage_tests.

App workflow

1. Open a PBIX in Power BI Desktop.
2. Launch Test Runner from the External Tools ribbon.
3. Confirm the connected model.
4. Create tests with the wizard, or preview and apply starter packs.
5. Run the full suite or a targeted subset.
6. Review failures, history, trends, exported reports, and portable test bundles.

Starter packs

Starter packs generate useful baseline coverage so you do not have to start from a blank suite.

• Metadata Quality: documentation, naming, formatting, display-folder, and relationship-metadata checks.
• Documentation Baseline: table, column, and measure description coverage.
• Presentation Hygiene: format-string and display-folder checks for visible fields.
• Referential Integrity: orphan-key checks plus relationship sanity checks.
• Relationship Governance: directionality, cardinality, and key-column visibility checks.
• Time Intelligence: date-table checks, marked-date-table validation, and continuity heuristics.

Recommended adoption pattern:

1. Start with metadata-quality.
2. Add referential-integrity or relationship-governance for relationship risk.
3. Add a few measure assertions for core KPIs.
4. Add one performance budget for an important query.
5. Export a Markdown report after each meaningful model change.

Dashboard and history

The app includes a dashboard for quick suite health:

• total, passing, and failing test counts,
• recent runs,
• flaky or degrading test insights,
• test type breakdown,
• folder breakdown.

Run history is persisted for the connected model. The history view helps you inspect previous runs, filter by status, and export reports for review.

Storage and model identity

Tests, baselines, and run history are stored by SemanticOps MCP, not by the desktop UI. By default, that state lives under the server's managed storage directory, typically ~/.mcp-engine/tests.

When a model is connected, SemanticOps MCP uses a stable model identity so tests remain associated with the same model even if a PBIX file is renamed or moved. The first persisted testing action for a model may write a stable model-id annotation into the model metadata; save the PBIX if you want that identity to persist.

Exports

Use the format that matches your workflow:

• Markdown: pull requests, tickets, and review notes.
• JUnit XML: CI systems and build artifacts.
• HTML: readable reports for non-technical reviewers.
• JSON: automation and downstream tooling.

Exports can be returned inline or saved to disk. Saved export paths are local to the machine running the SemanticOps MCP server. The Transfer view uses portable test-definition bundles for import/export, while result exports remain focused on run reports.

Installation

SemanticOps Test Runner is intended for 64-bit Power BI Desktop on Windows.

Typical installation flow:

1. Download and extract the Test Runner package.
2. Run the registration helper included in the package.
3. Accept the elevation prompt when Windows asks for administrator access.
4. Restart Power BI Desktop.
5. Open a PBIX and launch Test Runner from the External Tools ribbon.

The registration helper writes the Power BI External Tool registration into:

%CommonProgramFiles(x86)%\Microsoft Shared\Power BI Desktop\External Tools\

App vs MCP tool

SemanticOps offers two related testing surfaces:

Both use the same underlying managed test definitions, baselines, and run history.

Privacy and masking

The bundled Test Runner sidecar force-disables numeric and PII masking for its own local process so the UI can display raw expected and actual values. This override is process-local to the Test Runner sidecar. Other SemanticOps MCP clients continue to honor their configured masking preferences.

For sensitive datasets, be deliberate about regression snapshot modes. Hash and aggregate snapshots are safer defaults than full-row or top-N captures when sensitive data may be present.

Prompts to use with SemanticOps MCP

Current boundaries

• It is not a DAX IDE; use tools like DAX Studio for deep exploratory DAX authoring.
• It is not a separate .pbitest.json file workflow; tests live in SemanticOps MCP managed state.
• It is not a standalone CI runner; export JUnit XML when you want CI-oriented artifacts.
• Full Power BI Desktop External Tool behavior is Windows-focused because Power BI Desktop is Windows-only.

See also

• Unit testing (manage_tests)
• Packs and running tests
• Baselines and exports
• Server state and isolation
• Security and privacy
• Pricing