Overview
Observability is a retrospective tool for organization administrators. It gives you a structured view of past threads and playbook runs across your workspace.
After threads complete, TextQL automatically analyzes them for quality signals: gaps in context, execution errors, signs of user frustration, potential inaccuracies. These appear as warnings attached to individual threads, which you can drill into to understand exactly what happened and where things broke down.
The goal is to give administrators a systematic way to:
- Monitor analysis quality across your organization over time
- Identify patterns — recurring warning types, problematic connectors, or topic areas where Ana consistently struggles
- Apply targeted fixes — improving ontology coverage, enriching the context library, or adjusting connector configuration based on what you find
- Drive user education — spotting where users are asking questions Ana cannot yet answer well, and helping them prompt more effectively in the meantime
Improvements made in response to what you find here — better context, clearer ontology, stronger metric definitions — will take effect in future threads. Observability does not retroactively change past results, but it is the primary tool for closing the feedback loop between what Ana does today and how well she performs tomorrow.
Access it from the left sidebar under Observability.
If Observability is not visible in your sidebar, ask your admin to enable it under Settings → Features & Tools.
Who Can Access Observability
Observability is available to organization administrators only. Members without admin privileges will not see it in the sidebar.
Stats Bar
At the top of the page, four cards show aggregate metrics for the selected time range: Total Runs, Playbook Runs, Feed Agents, and Warnings (with a warn rate percentage). Each card shows a delta compared to the previous equivalent period.
Warning Breakdown
Next to the run volume chart, the Warning Breakdown panel ranks the most frequent warning types by count, color-coded by category — blue for Causes, amber for Symptoms, red for Outcomes. Click any warning type to automatically filter the thread table below to show only threads with that warning.
Warning Types
Warnings are grouped into three categories that reflect where in the agent’s process something went wrong. Each warning also has a severity level (high, medium, or low).
Causes (blue) — Configuration issues you can fix
These warnings indicate that the agent lacked the information it needed. Fixing them usually means improving your ontology or semantic layer.
| Warning | Severity | What it means | Suggested action |
|---|
| Missing context | High | The agent lacked ontology or semantic layer information needed to answer accurately | Add missing table/column descriptions to the semantic layer, or onboard the relevant data source |
| Ignored instruction | Medium | Ana skipped steps specified in the prompt or failed to execute required tool calls | Review the conversation to see which instruction was missed; consider improving prompt instructions or adding context |
Symptoms (amber) — Execution problems
These warnings indicate something went wrong while the agent was running.
| Warning | Severity | What it means | Suggested action |
|---|
| Error loop | High | The agent hit 5+ consecutive execution errors | Review the error messages and check if the agent is retrying a fundamentally broken approach |
| Connector errors | Medium | 5+ SQL execution errors occurred | Review SQL errors for common patterns; check connector permissions and schema accuracy |
| Excessive tool calls | Medium | The agent made 57+ tool calls | Review the conversation for unnecessary retries or redundant tool usage; consider improving prompt instructions |
| Slow query | Medium | A SQL query took more than 25 minutes | Review the query plan for missing indexes or expensive joins; consider materializing frequently-queried aggregations |
| Long conversation | Low | The conversation has 74+ cells | Check if the user’s needs were met; long conversations may indicate missing context or unclear ontology |
| No results | Low | A query returned an empty result set unexpectedly | Verify the data exists for the queried time range; check filters and join conditions in the generated SQL |
Outcomes (red) — User-facing impact
These warnings indicate the user had a negative experience.
| Warning | Severity | What it means | Suggested action |
|---|
| User frustration | Medium | The user expressed frustration or had to repeat their question | Review the agent’s responses for accuracy; consider adding example queries or refining prompt instructions |
| Potential hallucination | High | The agent may have presented fabricated or inaccurate data | Verify the data accuracy; review the SQL query logic and ensure the ontology mappings are correct |
Time Range
Use the time range selector (top right) to control the window of data shown across the stats, charts, and table. Options are Last 7 days, Last 14 days, Last 30 days, and Last 90 days.
Filters
The filter bar appears above the table. From left to right, the controls are:
- Search — A text input to search threads by summary content
- Source — Filter by thread source: Thread, Playbook, or Slack
- Users — A multiselect dropdown to filter by one or more team members
- Warning — Filter by warning status. Modes include:
- All threads — Show all threads regardless of warnings
- Any warning — Show only threads that have at least one warning
- Specific warning — Show only threads with a particular warning type
Click Clear filters to reset all active filters at once.
Backfilling Warnings
New threads are scanned for warnings automatically. However, threads created before Observability was enabled will not have warnings attached. To analyze those older threads, use the Scan for Warnings feature.
If unanalyzed threads are detected, a Scan for Warnings button appears in the page header. Clicking it opens a modal where you can configure the scan:
- Time range — Choose how far back to scan: Last 7 days, Last 14 days, Last 30 days, or Last 90 days
- Re-analyze all threads in range — When enabled, every thread in the range is re-evaluated, including threads that have already been analyzed. This incurs additional LLM costs. When disabled (the default), only threads that have never been analyzed are processed.
Click Check threads to preview how many threads will be scanned, then Analyze N threads to start the backfill. A progress banner appears at the top of the page while the scan runs, showing how many threads have been processed. Once complete, the dashboard refreshes automatically with the new warning data.
The Observability Table
The main table lists every thread and playbook run within the selected time range. Each row shows:
- Timestamp — When the thread was created
- Source — Thread, Playbook, or Slack
- Summary — A short description of what the user asked
- User — The email of the member who created the thread
- Warnings — Any quality warnings detected on the thread
Rows with warnings are highlighted so they stand out at a glance. Click any row to open the Thread Detail Panel on the right.
Thread Detail Panel
The Thread Insights panel lets you identify which message or query to investigate further and understand how to fix it.
| Tab | What it shows |
|---|
| Warnings | All warnings detected on this thread, with description and a suggested action |
| Timeline | Total duration, LLM vs. execution time split, and a turn-by-turn breakdown — each turn shows LLM time, tool calls, queries, and total time |
| Overview | High-level stats: total duration, turns, steps, LLM time, and execution time |
Further Reading
Need Help?
If you’re seeing warnings you don’t know how to resolve, or Observability isn’t behaving as expected, contact support at support@textql.com. 
