Usage persistence sanitization
OpenAI Responses usage metadata persists token and cost information used by message footers, status totals, ChatGPT cached-token estimation, and later native-history reconstruction. Persisted usage retains only the supported OpenAI field paths documented in Cost estimation. Other root or nested fields are processed only as part of the current response and are not persisted.
Older monolithic JSON sessions may still contain unsupported usage fields in:
- assistant message metadata;
- nested phase-message metadata;
- retained provider-native metadata;
- stale private raw-usage metadata.
The package provides a maintenance command that removes unsupported usage
members while retaining the explicit root and nested-field allowlist and all
derived usage metadata. The scan includes canonical live sessions and
immutable snapshots under <session-root>/.snapshots.
Dry run
Dry run is the default:
plugins/openai_responses/scripts/sanitize_session_usage.py \
--session-root ~/.crystal/sessions \
--report-path /tmp/openai-usage-sanitizer-dry-run.json
Select individual sessions by repeating --session-id:
plugins/openai_responses/scripts/sanitize_session_usage.py \
--session-root ~/.crystal/sessions \
--session-id <session-id> \
--report-path /tmp/openai-usage-sanitizer-session.json
The JSON report contains affected occurrence counts, removed-field counts
grouped by persisted path, and projected byte savings. Nested removed fields
use dotted names such as
input_tokens_details.future_nested_detail. The legacy report property is
named removed_usage_root_keys. Reports do not include message content.
Apply
Stop or otherwise prevent writers from changing the selected session files before applying the migration. The current application does not participate in the maintenance command's file coordination, so a file-state recheck cannot replace a no-writer window.
Apply mode requires a recovery-backup directory outside the live session root:
plugins/openai_responses/scripts/sanitize_session_usage.py \
--session-root ~/.crystal/sessions \
--apply \
--backup-dir ~/.crystal/session-migration-backups/openai-usage-<run-id> \
--report-path /tmp/openai-usage-sanitizer-apply.json
For every affected canonical live session, the command:
- validates the original session;
- creates and validates a temporary sanitized replacement;
- rechecks the source content before publication;
- writes a gzip recovery copy;
- atomically replaces the source;
- validates the published session.
The command skips top-level conflict-backup filename patterns and snapshot
selection records ending in .selected.json. It stores live-session and
snapshot recovery copies in separate sessions/ and snapshots/ directories
under the selected backup root and refuses to overwrite an existing recovery
backup.
Run dry mode again after apply. A completed migration reports zero files that would change.
Recovery
Each changed session has a corresponding <session-id>.json.gz file in the
selected backup directory. To recover, keep the application stopped, decompress
the required file to a temporary path, validate that its embedded session ID
matches the destination filename, and atomically replace the live session file.