Sandbox Storage - TextQL Docs

Every Python sandbox in a self-hosted deployment writes to one shared volume, sandbox-files-pvc, which the chart mounts into compute-engine at /mnt/sandbox-data. This guide covers how much space that volume needs, how retention keeps it bounded, and how to recover when it fills up. Chart values below are set in the values file you pass to helm upgrade (see Installation).

What lives on the volume

Path	Contents	Count
`sandbox-worker-*`	One directory per sandbox: the Python workspace snapshot (`.internal_state.pkl`), files Ana writes, chat attachments, and dashboard data	One per thread, dashboard, and Data App. Each scheduled playbook run is its own thread
`library`	The org’s context library and ontology files	One shared copy
`packages`	Custom Python packages installed under Settings → Packages (see Settings)	One shared copy per org

The volume is shared by every sandbox in the deployment, so when it fills up, every thread fails to write, not only the one that pushed it over. Threads report No space left on device, and dashboards fail with NameError: name 'dashboard_<source>' is not defined because their data sources could not be saved.

Capacity planning

Ontology size has little effect on storage. The library directory is one small shared copy. What drives growth is the number of active sandboxes and how much data each one pulls into Python. Every dataframe in a thread’s Python session is saved into its workspace snapshot, so a thread that only queries aggregates uses a few MB, while one that loads millions of rows can use several GB. Usage levels off instead of growing forever, because retention deletes idle sandbox directories (see Retention). At steady state the volume holds roughly:

(sandboxes touched per day) × (directoryRetentionDays) × (average sandbox size)

To measure your own numbers, run this from the compute-engine pod. On a large volume du can take several minutes.

kubectl exec -it deploy/compute-engine -c compute-engine -n <namespace> -- /bin/bash
cd /mnt/sandbox-data
df -h .
find . -maxdepth 1 -name 'sandbox-worker-*' -mtime -15 | wc -l   # use your directoryRetentionDays
du -sm sandbox-worker-* | sort -n | awk '{a[NR]=$1; s+=$1} END {print "dirs", NR, "total_GB", s/1024, "p50_MB", a[int(NR/2)], "p95_MB", a[int(NR*0.95)], "max_MB", a[NR]}'
du -sh library packages

Size the volume for two to three times the current steady state so growth in users and scheduled playbooks does not force another resize soon. Prefer growing the volume over shortening retention: a larger volume costs disk, while shorter retention makes more old threads restore from backup and drops the files they wrote. Whatever size you pick, alert on free space so you hear about it before users do. On GKE, alert on the Filestore metric file.googleapis.com/nfs/server/free_bytes_percent, for example below 20%. On AKS, alert on the file share’s used capacity against its quota.

Retention

Two settings control how long sandbox data is kept. compute.worker.directoryRetentionDays (Helm, default 15, or 30 in chart 1.3.25 and earlier) controls the shared volume. A daily job deletes any sandbox-worker-* directory whose last activity is older than this. Activity is measured from the workspace snapshot’s last write (or the directory’s, when there is no snapshot), not from when the sandbox was created, so a long-running thread that is still in use is kept. Workspace Retention (Settings → Security, default 30 days) controls a backup copy of each thread’s Python workspace in object storage, up to 5 GiB per thread. See Settings. When someone reopens a thread whose directory was deleted, the sandbox restores the Python workspace from the backup copy (if it is still within Workspace Retention) and re-attaches chat attachments, so the thread picks up where it left off after a slightly slower start. Files Ana wrote to disk that were not kept as artifacts, and workspaces larger than 5 GiB, are not restored. Dashboards reload their data sources the next time they start.

compute:
  worker:
    directoryRetentionDays: "15"

Lower directoryRetentionDays only when you cannot grow the volume. Keep Workspace Retention at or above it so a thread’s backup outlives its directory.

Platform differences

Platform	Storage	Fills up?	How to grow it
GKE	Filestore, Basic HDD tier	Yes, at the provisioned size	Expand the volume (see Expanding on GKE)
EKS	Amazon EFS	No, EFS grows automatically and `filesPvcSize` is not enforced	Nothing to resize. Retention still controls cost
AKS	Azure Files Premium (NFS)	Yes, at the share quota set by `filesPvcSize`	Raise `sandbox.filesPvcSize` and run `helm upgrade`. Expansion is enabled by default
Single node	Local path on the node disk	Yes, at the node disk size	Grow the node disk

Expanding on GKE

Filestore Basic HDD has a 1 TiB minimum, and the Filestore driver rounds smaller requests up to it. With the default filesPvcSize of 128Gi you already have 1 TiB, so a new size only helps if it is larger than the volume’s current capacity. Check it first:

kubectl get pvc sandbox-files-pvc -n <namespace>

The chart creates the Filestore StorageClass with expansion turned off. To grow the volume:

Allow expansion on the StorageClass. Its name is filestore-<namespace> unless you set sandbox.storageClassName.
kubectl patch storageclass filestore-<namespace> -p '{"allowVolumeExpansion": true}'
Set the new size and the expansion flag in your values file, then run helm upgrade.
sandbox: allowVolumeExpansion: true filesPvcSize: "2Ti"
Confirm the new capacity with kubectl get pvc sandbox-files-pvc -n <namespace>. Filestore expands in place without downtime. It cannot shrink, so increase in steps you are confident you will use.

Keep the volume on uninstall

Dynamically provisioned volumes default to the Delete reclaim policy, so deleting the PVC (for example by uninstalling the release) can delete the underlying storage along with every sandbox and the context library. Set the policy to Retain on every platform:

PV=$(kubectl get pvc sandbox-files-pvc -n <namespace> -o jsonpath='{.spec.volumeName}')
kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Recovering from a full volume

Work through these in order. Freeing space ends the outage. Expanding the volume and checking retention keep it from coming back.

Back up the volume. On GKE, find the Filestore instance backing the volume (kubectl get pvc sandbox-files-pvc -n <namespace> shows its name under VOLUME) in the GCP console (Filestore → Instances) and create an on-demand backup under Backups. On EKS, use AWS Backup for the EFS file system. On AKS, take a snapshot of the file share.

Optionally keep a local copy of the context library.

kubectl exec deploy/compute-engine -c compute-engine -n <namespace> -- tar czf - -C /mnt/sandbox-data library > library.tar.gz

Record usage before deleting anything. Run the measurement commands from Capacity planning and save the output. These numbers are your best sizing input, because they show the volume at its peak.

List idle sandbox directories. This uses the same activity rule as the daily cleanup job. Replace 14 with the number of idle days you are comfortable deleting.

cd /mnt/sandbox-data
for d in sandbox-worker-*/; do
  f="$d.internal_state.pkl"; [ -e "$f" ] || f="$d"
  [ -n "$(find "$f" -maxdepth 0 -mtime +14)" ] && echo "$d"
done > /tmp/idle-sandboxes.txt
wc -l /tmp/idle-sandboxes.txt

Delete only the listed directories.
while read -r d; do rm -rf "/mnt/sandbox-data/$d"; done < /tmp/idle-sandboxes.txt df -h /mnt/sandbox-data
Never delete library, packages, or anything that does not start with sandbox-worker-. Deleting a sandbox that is in use breaks that thread or dashboard until it restarts.
Reload affected work. Dashboards recover once space is free and the page is reloaded. Files that failed to write while the volume was full are gone, so rerun those steps in their threads.
Prevent a repeat. Set the reclaim policy to Retain, grow the volume using the numbers from step 3 (see Platform differences), add a free-space alert, and review Retention.

Contact TextQL with the output from step 3 if you want help choosing a size.

​What lives on the volume

​Capacity planning

​Retention

​Platform differences

​Expanding on GKE

​Keep the volume on uninstall

​Recovering from a full volume