Skip to main content

Eval Tests

The Tableau MCP project uses Vitest for eval tests. Eval tests are located in the tests/eval directory and are named *.test.ts.

What is an Eval test?​

Eval tests (aka evals) are tests used to evaluate MCP tool implementations using LLM-based scoring. The tests provide assessments for accuracy, completeness, relevance, clarity, and reasoning and help answer questions like:

  • Can the model consistently choose the correct tools to answer the user prompt?
  • Can the model generate the correct tool inputs based on the user prompt?
  • Does the tool implementation accurately answer the user prompt?
  • Is the tone suitable for the target audience?

Running​

The eval tests can only be run:

  1. Locally.
  2. If you have access to a site the tests understand. Currently, that's only https://10ax.online.tableau.com/#/site/mcp-test/.
  3. If you have an OpenAI API key or access to an OpenAI-compatible gateway.

To run them locally:

  1. Ensure you do not have a .env file in the root of the project.
  2. Create a tests/.env file with contents:
SERVER=https://10ax.online.tableau.com
SITE_NAME=mcp-test
AUTH=direct-trust
JWT_SUB_CLAIM=<your email address>
CONNECTED_APP_CLIENT_ID=<redacted>
CONNECTED_APP_SECRET_ID=<redacted>
CONNECTED_APP_SECRET_VALUE=<redacted>

  1. Create a tests/.env.reset file with the same contents except all the env var values are empty. (Environment variables get set at the beginning of each test and cleared at the end of each test.)

  2. Create a tests/eval/.env file with contents:

OPENAI_API_KEY=<your OpenAI API key>
  1. Run npm run test:eval or select the vitest.config.eval.ts config in the Vitest extension and run them from your IDE.

Desktop remove-field recovery evals​

tests/eval/removeField.test.ts covers W-24252809 with eight model-driven cases: stale Rows and Columns assumptions, recovery after a wrong-shelf error, an already-removed field, missing Size and Angle encodings, preservation of pending edits, and Detail/lod removal. The model receives the built Desktop tool schemas and the current implementation's error diagnostics and success response. Tool execution uses in-memory worksheet fixtures; no Tableau session, site, or workbook changes are needed.

Successful removals report the removed reference and resulting draft placements. The fixture uses the same result formatter as production and matches the SDK's single-content tool output; offline checks cover that parity and reject invalid combinations of read selectors.

The deterministic grader checks inspection before removal, correct final placements, preservation of unrelated fields, and absence of redundant retries or live refreshes of pending drafts. It grades tool behavior rather than the wording of the final answer. These are focused recovery evals, not end-to-end Desktop or full-tool-catalog discovery tests.

Build Desktop and run only this model suite with OPENAI_API_KEY, OPENAI_BASE_URL (for a gateway), and optionally EVAL_TEST_MODEL configured in your environment or tests/eval/.env:

npm run build:desktop
npm run test:eval -- run tests/eval/removeField.test.ts

Use a Desktop bundle built from the same checkout as the tests: schemas come from the bundle, while removal diagnostics and the shared success formatter come from the source. Comparing revisions requires both to match.

Set ENABLE_LOGGING=true to log each scenario's tool sequence, model-turn boundaries, call IDs, and final response. Call IDs are also logged as tools start, so a timeout retains partial execution evidence. A failed grade includes the tool sequence in its assertion output. Every case is bounded to eight agent turns. Identical call IDs across repeated runs can indicate gateway response caching; do not count cached repeats as independent model samples. This suite disables SDK trace export; model requests use the configured gateway.

The fixture, grading, and tool-isolation checks run without a model credential, Desktop build, or network access:

npm run test:eval -- run tests/eval/removeFieldScenario.test.ts tests/eval/base.test.ts

Desktop dashboard apply evals​

tests/eval/dashboardApply.test.ts covers W-24366364 with ten model-driven cases: adding an already registered worksheet to an empty dashboard, replacing membership while retaining view settings, removing the final worksheet, ordinary layout edits, recovery from unsupported worksheet zone types in desktop and Phone layouts, rejection of a genuinely blank worksheet, and safe refusal of missing registrations in both empty and populated dashboards, preserving concurrent Desktop edits, and repairing unsafe zone types already present in the live dashboard. Known crash hazards cannot be grandfathered from the live baseline. Existing-dashboard edits use the surgical dashboard endpoint. Desktop builds advertising External Client API 0.2.22 or later create missing worksheet views during surgical dashboard apply. The MCP planner permits automatic completion on those builds, after every worksheet has been populated, and apply verifies the registrations in live readback. Older or unknown API versions still require manual registration in Desktop. A whole-workbook replacement is never a fallback for missing registrations.

The model uses the built Desktop tool schemas. Cache files and the External API are in-memory fixtures, while applies run the production loadDashboardXml implementation, validation, and response formatters. The fake API preserves the distinction between dashboard layout updates and workbook view registrations, and models additive workbook actions. The independent grader checks the final layouts, required registrations (allowing retained unused viewpoints), retained view settings, unrelated worksheets/data sources/actions, apply route, readback, and redundant retries. It grades tool behavior, not final-answer wording.

No Tableau session or workbook is accessed. These focused evals do not replace live rendering or native zone-removal validation. Build the Desktop bundle from the same checkout as the tests and configure model credentials as described above, then run:

npm run build:desktop
npm run test:eval -- run tests/eval/dashboardApply.test.ts

Each case allows ten agent turns and two minutes. ENABLE_LOGGING=true records tool calls, call IDs, model turns, and final responses; SDK trace export is disabled. The fixture and grader checks run offline, including negative controls for the original missing-registration bug, duplicate actions, incomplete Phone repairs, discarded draft changes, and omitted readback:

npm run test:eval -- run tests/eval/dashboardApplyScenario.test.ts tests/eval/base.test.ts

Environment Variables​

The following environment variables are used by the Eval tests:

OPENAI_API_KEY​

The OpenAI API key.

ENABLE_LOGGING​

When true, LLMs will stream their output to the console and tool call information will also be logged.

OPENAI_BASE_URL​

The base URL for the OpenAI-compatible gateway.

EVAL_TEST_MODEL​

The model to use for the Eval tests. If not set, the default model is used.

ALLOW_MUTATING_FLOW_EVALS​

When true, enables evals that can invoke the content-mutating Tableau Prep flow tools. These evals are skipped by default because the eval harness executes real MCP tool calls against the configured Tableau site. Only enable them on a disposable site.

This is a test-safety gate, separate from the product feature gates. Mutating flow evals also require FLOW_WRITE_TOOLS_ENABLED=true.

FLOW_WRITE_TOOLS_ENABLED​

Required, together with ALLOW_MUTATING_FLOW_EVALS=true, to expose the mutating flow tools during the mutating flow evals. Leave unset for the default eval run.

Running the Eval tests against a different site​

To run the Eval tests locally against a different site, you need to:

  1. Have a site that has the Superstore sample datasource and workbook (which exist with every new site). The tests query this datasource and workbook.
  2. Create and enable a Direct Trust Connected App in the site.
  3. Create a Pulse Metric Definition named Tableau MCP. Its details don't matter.
  4. Update the environmentData object in tests/constants.ts with the new site details.
  5. Follow the steps in the Running section, providing these new site details in the tests/.env file.

Debugging​

If you are using VS Code or a fork, you can use the Vitest extension to run and debug the Eval tests.