> ## Documentation Index > Fetch the complete documentation index at: https://docs.textql.com/llms.txt > Use this file to discover all available pages before exploring further. # Evaluations > Ask Ana the same questions over time and grade the answers, so you can see whether a model, a tool setting or an Ontology change makes analysis better or worse before you rely on it. > [!NOTE] > > **Availability:** Evaluations is in **private preview**. Contact your TextQL team to turn it on for your organization. ![The Evaluations page listing a folder of questions with their status, last eight results and Run buttons](https://mintcdn.com/textql/owZjkF_IfOcmI7PE/images/admin/evaluations/questions.png?fit=max\&auto=format\&n=owZjkF_IfOcmI7PE\&q=85\&s=93b5078712eca024568b21f60c250cd6) ## Know Ana Is Right Before Anything Changes An **evaluation question** is a question you want Ana to keep answering well, together with what a right answer looks like. Evaluations asks your questions under any change you want to try and puts the results next to today's on one scorecard: | Your questions | Any change | The scorecard, against today | | - | - | - | | Your top requests, questions with known answers, questions from Threads that went wrong | An Ontology Patch or Branch, a model, a harness, the tools Ana may use | Right answers, cost, time and tokens | Use it to: * **Make big changes safely** by measuring an Ontology change before it merges * **Catch regressions** when a model, a connector or the Ontology changes * **Pick the best setup** by trying models and harnesses on the same questions * **Make answers faster and cheaper** without making them less right * **Turn failures into fixes** by asking again the questions Ana once got wrong Open it from the left sidebar under **Evaluations**. ## Ask Ana to Run the Loop You can do everything on this page from a Thread. Ask Ana and she runs the loop for you: 1. **Define** what right looks like, as evaluation questions 2. **Change** something: the Ontology, the model, the harness or the tools 3. **Run** the questions under the change and today's setup, side by side 4. **Keep** the change if the scorecard is better. If it is worse, try again. For example: * "Redo the Ontology on a branch and prove it is better." * "Introduce a few mistakes into the Ontology and check the evals catch them." * "Try each harness with three models on my top questions and find the best." * "Make the queries for my top requests faster without changing their answers." * "Find where you failed in last week's Threads, add evals for them, fix it and re-check." Ana can only change what you could, and changes to the Ontology go through review as they always do. ## Questions Each question has a name, the text Ana is asked, the connectors it answers from, and one of two ways of grading. | Type | How it is graded | Connectors | | - | - | - | | **Accuracy** | Ana's final query must return the same rows as a reference query, or its answer must match a single expected value | Exactly one connector that runs SQL | | **Quality** | A judge model grades Ana's answer against criteria you write | One or more connectors, including Power BI and Tableau | For an accuracy question with an expected value, choose the value's type (text, number, whole number or true/false). Numbers are compared with a tolerance of 0.1 unless you set another one; set it to 0 to require an exact match. For a quality question, write criteria a reviewer could check against the answer, such as "names the quarter" or "uses the revenue table, not the CRM export". For a Power BI or Tableau connector you can narrow the question to specific workspaces, datasets, datasources or workbooks. ![Editing a quality question with judge criteria, two connectors and schedules](https://mintcdn.com/textql/owZjkF_IfOcmI7PE/images/admin/evaluations/question-editor.png?fit=max\&auto=format\&n=owZjkF_IfOcmI7PE\&q=85\&s=8ec3ae965b1d2495ec7a2c954256c7f1) ### Questions are Ontology files Every question is an `.eval` file in your Ontology, one question per file. Folders group related questions, and a question's folder decides who can see it, as it does for other Ontology files. In an organization that manages its objects as files, saving a question in the editor writes its file. ### Who a question runs as Each question runs as its **runner**: a member or service account set on the question. Every run asks the question with the runner's access, so results don't change depending on who starts them. You can change what a question asks, or run it, only if you are allowed to act as its runner. If the runner can't use one of the question's connectors, the question says so when you save or run it. ## Running questions Select one or more questions and choose **Run**. Each question is asked in a fresh thread, and a question can be asked several times in one run so that one lucky or unlucky answer doesn't decide the result. ### Comparing setups To compare, run the same questions under two setups in one launch. A setup can change the model, the [harness](https://docs.textql.com/core/how-it-works/ana/harnesses.md), the tools Ana may use, the number of attempts, or the Ontology version the questions read. The launch page shows each setup's right answers, cost, time and tokens, and each question's result under each setup side by side, with a **Why** row explaining each result: the grader's reasoning, the expected and actual values, or why a question could not be graded. ![A comparison of three models showing pass rate, duration, cost and tokens per setup, and the questions that changed against the baseline](https://mintcdn.com/textql/owZjkF_IfOcmI7PE/images/admin/evaluations/compare.png?fit=max\&auto=format\&n=owZjkF_IfOcmI7PE\&q=85\&s=5fe026a17f09593a7dcad6f5f20c295d) ### Testing an Ontology change A Patch or a [Branch](https://docs.textql.com/core/ontology/branches.md) can be run before it merges. A question the change edits or adds is asked in the change's version, on that change. A comparison between Ontology versions only asks questions that are the same in every version it compares, so the scores measure the Ontology change rather than a change to the questions themselves. ### Schedules A question can run on up to 10 schedules, each a cron expression and a timezone. Questions that share a schedule and a runner run together as one launch, so a weekly run reads as a single result you can compare week over week. When a scheduled run starts failing, Evaluations sends a notification. ## Results The Evaluations page shows each question's status, its last eight results and when it last ran. Open a question to see its full history, the latest answer and how it was graded. Runs started only to compare setups count in a question's history but not in its pass or fail status, so experiments don't change how a question reads. ## Permissions | Permission | Lets a member | | - | - | | `evaluation:read` | Open Evaluations and see the questions and runs they have access to | | `evaluation:write` | Create, edit, run and delete questions they have editor access to | | `evaluation:write_private` | Manage who can access any question | Seeing a run requires access to every question it asked. Running or editing a question also requires being allowed to act as its runner. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.