Skip to content
Beta · version 0.8.1 · commands and the JSON format may still change.

Python API: scans

tabalyst.scan() scans one source in memory and writes nothing. tabalyst.generate_scans() scans several sources and writes their scan documents. tabalyst.plan_analyses(), tabalyst.plan_document() and tabalyst.plan_second_pass() plan the analyses a scan could still receive, with their estimated time and whether running them would require a confirmation (see Plan the analyses). tabalyst.generate_deep(), tabalyst.prepare_deep(), tabalyst.execute_deep() and tabalyst.run_second_pass() run the second pass of a scan and never prompt (see Add histograms and exact medians). The command is tabalyst scan; the result is described in the scan format.

import tabalyst
scan = tabalyst.scan("orders.json")
scans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
sheet = tabalyst.scan(
"sales.xlsx",
config=tabalyst.ScanConfig(excel={"dataset_path": "$.Orders"}),
)

Reads one source completely and returns its scan result. It does not read an Inspect file: choose the collection or the table with config.

tabalyst.scan(
source,
*,
config=None,
registry=None,
on_progress=None,
on_record=None,
workers=None,
counters=False,
)

Returns a tabalyst.ScanResult.

  • source: the file to scan. A file ending in .json is read as JSON, .jsonl or .ndjson as JSONL, .xlsx or .xlsm as an Excel workbook, and any other as CSV.
  • config: a tabalyst.ScanConfig that changes the settings, such as ScanConfig(excel={"dataset_path": "$.Orders"}) for a table or ScanConfig(json={"collections": ["$.customers[]"]}) for JSON collections. See Configuration.
  • workers: number of processes that analyze values; 1 scans in this process. By default, large sources use one per spare processor. The result does not depend on it.
  • counters: True adds run.counters, the work of the samplers (see Scan format).
  • on_progress: see the conventions.
  • registry, on_record: advanced. registry replaces the detectors (and scans in this process); on_record receives each analyzed record, in reading order, and must not modify it.

Scans several sources and writes one scan document each. It applies Inspect to JSON, JSONL and Excel sources, as the command does.

tabalyst.generate_scans(
input_specs,
*,
output=None,
output_dir=None,
config_path=None,
delimiter=None,
encoding=None,
collections=None,
force=False,
on_progress=None,
workers=None,
counters=False,
project_storage=False,
refresh=True,
html=False,
)

Returns a batch result whose successes hold the ScanResult in result. A successful job also has warnings.

  • input_specs: files or non-recursive glob patterns.
  • output: scan filename ending in .json, for a single source.
  • output_dir: folder for standalone scans named <stem>.scan.json.
  • delimiter, encoding: CSV delimiter and text encoding.
  • collections: JSON collections as absolute paths such as ["$.customers[]"], or the one Excel table such as ["$.Sales"].
  • project_storage: True stores the scans in the reusable storage of the command, as tabalyst scan does without -o or -d, instead of writing standalone documents.
  • html: also write the web page of each scan beside its document, or beside the source for a scan in the storage. A page that cannot be written is a warning in warnings. Off by default: only the command asks for it.
  • refresh: with project_storage, False reuses the stored scan when it is current for the source, the settings and the version, instead of scanning again. A successful job then has reused.
  • config_path, force, on_progress: see the conventions.
  • workers, counters: as for scan().

A source whose collection or table cannot be decided fails alone with a ConfigurationError; an option that a source cannot use is listed in batch.plan.warnings.