# DATAtokeniz

> DATAtokeniz converts files you already have — PDFs, Word and PowerPoint documents,
> spreadsheets, web pages, images, audio and video — into machine-learning-ready
> datasets. It is operated by Numen Labs, LLC.

## What it does

DATAtokeniz is a training-data preparation tool. You give it source material; it
returns a structured JSONL dataset in one of four shapes:

- **Evaluation** — grounded question/answer pairs for testing whether a model is right.
- **Retrieval (RAG)** — passages with citations, for grounding a model at query time.
- **Instruction tuning (SFT)** — instruction/response pairs for fine-tuning.
- **Multimodal** — image and media examples paired with text.

Every run also returns a quality report (duplicates, personal data, weakly grounded
rows, language mismatches, scored 0-100), a data card describing how the set was
built, and a checksum manifest.

## What makes it different

Provenance is recorded per row, not per file. Each segment traces back to the exact
source file and the exact page, timestamp or character range it came from, and each
field records whether it was extracted from the source, computed deterministically,
generated by a model, or verified by a human.

Extracted text and model-generated text are never silently mixed. Anything a model
produced is labelled as generated and keeps a reference to the source segment that
grounds it.

Segments containing detected secrets are excluded from exports entirely, and the
synthetic-expansion pass re-scans its own output before returning, because
model-generated text has never been scanned before that point.

DATAtokeniz can produce an EU AI Act Article 10 data-governance record for a
dataset: origin with per-source checksum and licence, preparation operations,
assumptions and limitations, suitability counts, bias checks, and personal-data
findings under Article 10(5).

## Facts

- Canonical dataset schema version: 2.0.0. Machine-readable at https://datatokeniz.com/api/schema
- Accepted input: PDF, DOCX, PPTX, CSV, JSON, HTML and URLs, PNG/JPEG images, audio and video, pasted text
- Output: JSONL (profile-specific, plus ShareGPT and Alpaca shapes where eligible), canonical JSON, CSV, Markdown data card, Parquet
- Usable with no account and no API key; the demo lane runs the deterministic pipeline in the browser session and stores nothing
- Google Drive access uses the drive.file scope only, which grants access solely to files the user picks or that DATAtokeniz creates
- Source files are processed in the request that carries them and are never written to DATAtokeniz storage; results go to the user's own Google Drive or a browser download
- Video face-landmark extraction runs client-side, so the video never leaves the user's machine

## Limits

DATAtokeniz does not make anyone compliant with any regulation. It produces
evidence and records; the obligations remain the operator's.

Audio and video are transcribed from the audio track only — visual content in a
video is not analysed. Scanned PDFs require a vision model to be configured; where
a capability is unavailable the run emits a warning and continues rather than
failing or silently substituting.

## Pages

- [Home](https://datatokeniz.com/): what it does, and a live demo that runs without an account
- [Pricing](https://datatokeniz.com/pricing): plans and enforced limits
- [Output schema](https://datatokeniz.com/api/schema): the canonical dataset contract as JSON Schema
- [Privacy](https://datatokeniz.com/privacy)
- [Terms](https://datatokeniz.com/terms)