Scan format
tabalyst scan data.csv writes a reusable scan.json by default;
-o and -d export a standalone <stem>.scan.json. Both are JSON documents that
describe every field of the source. This page describes format
tabalyst.scan, version 0.2.0, revision 3. The format is
experimental: it can change incompatibly between releases. Always check
format, format_version and format_revision first; changes are listed in
the scan format changelog.
Top level
Section titled “Top level”{ "format": "tabalyst.scan", "format_version": "0.2.0", "format_revision": 4, "engine": { "version": "0.7.0", "normalization_version": 1, "detectors": {"number": 1, "date": 1, "email": 1, "...": 1}, "classification": {"catalogue": 4, "names": 2} }, "status": "complete", "run": { "started_at": "2026-10-06T12:00:00Z", "duration_seconds": 0.0023, "timings": { "reading": 0.0009, "analysis": 0.001, "finalize": 0.0003, "derived": 0.0, "other": 0.0001 }, "workers": 1, "peak_memory_bytes": 58720256 }, "source": { "format": "csv", "name": "orders.csv", "size_bytes": 63, "modified_at": "2026-09-27T05:06:40.930099Z", "sha256": "fe7a56e5...", "encoding": "utf-8-sig", "shape": { "lines": 4, "records": 3, "columns": 3, "fields": 3, "datasets": 1, "cells": 9, "missing_cells": 0, "records_with_missing": 0, "empty_records": 0, "duplicate_records": 0 }, "csv": { "delimiter": ",", "quote": "\"", "header": ["id", "amount", "email"], "bom": true, "line_ending": "lf" } }, "config": {"...": "effective scan configuration"}, "config_sha256": "030617af...", "scope": { "collections": null, "records_read": 3, "records_analyzed": 3, "records_excluded": 0, "exclusions": {} }, "datasets": [], "diagnostics": []}engine: the Tabalyst version, the normalization version and the version of each detector that ran.status:completewhen every record in scope was analyzed,partialwhen records were excluded under thetoleranterror policy. A file that cannot be read produces no document.run: what varies from one run to the next on the same source, kept apart from the description of the data.started_atandduration_secondsgive the time and the processing time.timingssplits the duration in seconds:reading(reading and parsing the source, hashing included),analysis(analysis of the values; with several workers, the time the main process spends on it, waiting included),finalize(final computation of the datasets),derived(the inferred patterns and the observations) andother(setup and assembly of the result); the five parts add up toduration_seconds.workersis the number of processes that analyzed values (1 in the main process).peak_memory_bytesis the peak memory of the main process, workers excluded, when the scan ended: it covers the whole process and isnullwhen the platform does not report it.source: the file as read.size_bytesandsha256cover every byte read.encodingis the CSV encoding, orutf-8orutf-8-sigfor JSON, andnullfor an Excel workbook.shapegives its global size:lines(physical lines of a CSV file, header and multi-line cells included,nullfor other formats),recordsanalyzed,columnsof a CSV or Excel table (nullfor JSON and JSONL),fieldsanddatasetsfound, andcells(recordstimescolumns,nullfor JSON and JSONL),missing_cells,records_with_missing,empty_recordsandduplicate_records(nullwhen a count is limited or disabled), summed over the datasets.csvisnullfor sources that are not CSV.csvalso gives thequotecharacter, whether the file starts with a UTF-8 byte order mark (bom) and itsline_ending(lf,crlf,cr,mixed, ornullwithout any line break).excelisnullunless the source is a workbook: it gives the table read, as described below.config: the complete effective configuration, with every default, as in thescansection of a configuration file.config_sha256is the SHA-256 of its canonical JSON (sorted keys, no whitespace, UTF-8): two scans with the same value used the same settings.scope: records read, analyzed and excluded, with exclusions counted by reason (width_mismatch,duplicate_key,record_too_large, and for JSONLinvalid_line,not_objectandline_too_long).collectionsisnullfor CSV, JSONL and Excel; for JSON it gives themode(autoorexplicit) and therequestedcollection paths.
Excel sources
Section titled “Excel sources”A workbook is read one table at a time, the one config.excel.dataset_path
names. source.excel records which:
"excel": { "dataset_path": "$.Orders", "sheet": "Orders", "table": null, "range": "A4:I124", "header_row": 4, "header": ["order_id", "order_date", "customer"]}dataset_path is $.Sheet or $.Sheet.Table; table is the name of a named
table, or null for a sheet. range is the A1 range of the header and the data
actually read, header_row the 1-based sheet row of the header, and header the
header cells, which name the fields. The records are the filled rows under the
header, with location.line the 1-based sheet row. Values keep their Excel
type: whole numbers are integer, other numbers number, booleans boolean;
dates and times are string values in ISO 8601 that the date detector
recognizes. config.excel holds dataset_path and header_row, and takes part
in config_sha256.
Datasets
Section titled “Datasets”A CSV source has one dataset, rows, and so has an Excel source: its table.
A JSON source has one dataset per
collection of records, and one document dataset when its root is not an
array and the collections were discovered automatically (mode auto). A
JSONL source has one dataset, $[]. The commands choose the collection with
Inspect and name it explicitly, so a scan
made by tabalyst scan or tabalyst report has mode explicit and no
document dataset; only tabalyst.scan() without json.collections discovers
collections.
{ "id": "$.customers[]", "kind": "collection", "collection_path": [{"key": "customers"}, {"items": true}], "record_count": 4, "record_types": {"object": 4}, "structure": { "paths": {"status": "complete", "value": 5}, "untracked_observations": 0, "depth_truncated_observations": 0, "max_depth_seen": 3 }, "records": {"...": "record facts, described below"}, "fields": []}kind:table(CSV and Excel),collectionordocument(JSON).record_count: analyzed records;record_types: records per JSON type.structure: the number of distinct field paths, and what themax_fieldsandmax_depthlimits left out.records: facts about whole records, described in the next section.
Records
Section titled “Records”Each dataset describes its records as a whole: records with missing values, empty records, duplicate records and a sample of whole rows.
{ "with_missing": {"count": 2, "records": [2, 5]}, "empty": {"count": 1, "records": [5]}, "duplicates": { "count": {"status": "complete", "value": 1}, "records": [3] }}- The values of a record are its strings, numbers, booleans and nulls: every cell of a CSV row. A JSON field absent from a record is not a value of it.
with_missing: records with at least one missing value, under thevalues.missingsetting.empty: records whose values are all missing.duplicates: records equal to an earlier record of the same dataset, beyond its first occurrence. Records are equal when every value, its field and its native type are equal, raw values compared, before normalization.countis a measure envelope:limitedwith reasonrecord_budgetwhen more distinct records thanlimits.max_tracked_recordswere read, itslower_boundbeing the duplicates found;disabledwhenrecords.duplicatesisfalse.recordslists record numbers, at mostlimits.max_listed_recordsof them. Record numbers are those offirst_recordand diagnostics.sample: the row sample, absent whenrecords.sample.rowsis 0. It holds the first records, with their values per field identifier, as text.
Row sample
Section titled “Row sample”records.sample keeps whole rows of the dataset, taken in the same pass: the first
edge records, the last edge records and, between them, a random but reproducible
choice of the others. Every row has the cells of the same record for all the projected
fields, so a later analysis can relate the values of two fields.
{ "selection": "start_middle_end", "hash": {"algorithm": "blake2b", "version": 1, "bits": 64, "seed": 42}, "requested": {"rows": 1000, "edge": 50, "middle": 900}, "middle_k": 900, "population": {"records": 3000, "interior": 2900, "scope": "interior_rows"}, "complete": false, "threshold": "4e361452e3b67ee0", "fields": ["column_1", "column_2"], "fields_limited": false, "cell_chars": 200, "truncated_cells": 0, "bytes": 2477121, "max_bytes": 16777216, "kept": 1000, "of": 1000, "rows": [ {"record": 1, "origins": ["start"], "values": {"column_1": ["a"], "column_2": ["1"]}} ]}requested: the sizes asked for.rowsis the total,edgethe number of rows at each end andmiddlethe rest,rows- 2 *edge.- A row has the
recordnumber, itsorigins(start,middle,end; a record that is both at the start and at the end of a short dataset is listed once with["start", "end"]) and itsvalues: text per field identifier, the values of a repeated path listed in order, a JSON null asnull,nullfor a hidden cell, no key for a field absent from the record. Empty strings, blanks and null markers are shown as read. Rows are in source order within each group: start, middle, end. - The middle is drawn from the interior only, records
edge+ 1 to the number of records minusedge(population.interior,scopeinterior_rows). It is not a uniform sample of the whole dataset: do not add the edges to an estimate over the whole dataset. The choice depends on the position of a record, never on its content (two identical records are two observations), on therandom_seedand on the version of the hash, so the same source and configuration give the same rows with any number of workers. completeis true when every record is a row, which is the case for a dataset of at mostrowsrecords.thresholdis the largest priority kept in the middle, written when some records of the interior were not kept.- A cell is cut at
cell_charscharacters (all the values of a repeated path together). The row names the cut fields intruncatedandtruncated_cellscounts them: a row with a cut or hidden cell cannot serve a relation on that field.fieldslists the projected fields;fields_limitedis true whenlimits.max_fieldsleft some paths untracked. - Sensitive values go through the exposure of their field: a cell holding a hidden
value is
null. bytesis the size of the rows by a cost model, andmax_bytesthe budget of the whole scan (records.sample.max_bytes). When the budget is exceeded, the middle shrinks first (middle_kbelowrequested.middle), then the start and the end keep what fits, and the block saysreasonmemory_budget.- A Report keeps this block with the rows
report.row_sampleasks for (10 at the start, 10 sampled and 10 at the end by default):keptsays how many it lists andofhow many the scan sampled.
Fields
Section titled “Fields”Fields are listed in order of discovery. A field is a path inside the records:
a CSV column, a JSON key, or the elements [] of an array.
| Key | Content |
|---|---|
id, path, display, name, parent | Identity: an identifier such as f4, the path as segments, its readable form such as orders[].amount (keys joined with json.flatten.separator), the last segment and the parent field identifier. |
collection | For a JSON array promoted to its own dataset, the identifier of that dataset. |
first_record, occurrences | The first record with a value, and the number of values at this path. |
presence | parent_count, present and absent: how often the field exists where it could exist. |
native_types | Values per native type: string, integer, number, boolean, null, object, array. |
strings | Strings per category: empty, blank (whitespace only), marker (a configured null marker) and content. |
missing | The missing count, the categories included in it and each component. |
arrays | For array fields: count, empty arrays, minimum, maximum and total length. |
values | Analyzable values: count, cardinality, frequencies, samples, first and last. |
string_characteristics, string_lengths | The character profile: case, character classes, composition, whitespace, invisible characters, special characters and distinct characters; length statistics and histogram. |
numeric, booleans, temporal | Exact numeric statistics, boolean counts, date range and formats. Numbers that a column proves to use one decimal convention include its ambiguous ones (see Detectors). numeric.coverage gives the share of the analyzable values the statistics describe; a field that a code detector reads (postal_code, phone, uuid, ip_address) has no numeric block (not_applicable, reason code_interpretation). |
normalization | For each normalization stage, how many values it changed and the distinct values left, plus groups of variants that normalize to the same value (see Normalization stages). |
technical_type | integer, number, date, boolean, text, mixed or empty, with confidence, counts and outside_count. |
detectors | One result per detector, described below. |
interpretations | candidates, the detectors that matched at least 95% of the values, and primary, the only candidate when there is exactly one. |
classification | The family and type of the field, with what decided them: see Classification. |
sensitive, exposure | Whether the field holds sensitive values, and how they are exposed: mask, hide, show, or null for other fields. |
Values are listed as their text with their native type, for example
{"value": "12.50", "type": "string", "count": 1}. Listings use the analytical
value: surrounding whitespace is trimmed and repeated spaces are collapsed.
Classification
Section titled “Classification”Each field has a classification that names its family and type. It is decided
from the blocks of the field alone, in the first pass, so a Scan gives the same
classification with one or several workers and without the source.
{ "family": "text", "type": "text_short", "basis": "inferred", "state": "established", "traits": [], "evidence": [ {"source": "technical_type", "id": "text", "role": "supports", "scope": {"population": "present_values", "of": 3, "partial": false, "accuracy": "exact"}}, {"source": "measure", "id": "string_lengths.median_length", "role": "supports", "scope": {"population": "present_values", "of": 3, "partial": false, "accuracy": "exact"}} ], "alternatives": [], "rule_version": 4002}| Key | Content |
|---|---|
family, type | Codes in lower snake case: text and text_short, numeric and integer. One pair per field; secondary properties are in traits. |
basis | inferred when the evidence decided, fallback when the technical type or a fallback did, declared for a constraint the user declares (not produced yet). |
state | established when the values themselves support the type, candidate when it rests on a name plus compatible values, undetermined when the evidence did not allow more. |
evidence | The facts used: source (detector, technical_type, measure, name_rule, position), their id (a detector id, a measure name or a name rule id), their role (supports, contradicts, context) and their scope (what population the fact covers, its size, whether it is partial and its accuracy). |
alternatives | Types considered and not retained, with a reason. Never a conclusion. |
traits | Secondary properties that never change the pair: single_pattern or multiple_patterns (from Inferred patterns), constant (one distinct value on a typed column), observed_unique (the exact count of distinct values equals the present values) and first_column (first column of a CSV or Excel table). The last two are facts about the column and never prove a role. |
rule_version | The version of the rules that decided, as one integer: the catalogue version times 1,000 plus the version of the name rules (4002 is catalogue 4, name rules 2). engine.classification records both separately. |
There is no score and no percentage of confidence: the evidence and the state say how firm the result is.
This version decides:
- Other / Empty for a field without present value, including a JSON object or array, whose values are not analyzed.
- Types carried by a detector, when it reaches
detection.minimum_shareof its eligible values (any detector listed ininterpretations.candidates, not only the primary interpretation):numeric/currency,numeric/percentageandnumeric/quantity;contact/emailandcontact/phone;geographic/postal_code;web_network/url,web_network/ip_addressandweb_network/uuid. A detector type outranks Enum, Text and the numeric types, so a column of long email addresses stayscontact/email. A bare number is nevercurrencyorpercentage: only their detector, which needs the symbol or the%, gives them. - Categorical:
categorical/booleanfrom thebooleandetector or abooleantechnical type (two categories such asMandFnever make a Boolean);categorical/enumfrom theenumerationdetector over a text field;categorical/constantfor a text field with a single exact analytical value in at least two present occurrences. - Temporal:
temporal/datefrom adatetechnical type;temporal/datetime(with or without a time zone) andtemporal/timefrom thedatedetector and the kinds oftemporal. A column that mixes natures keeps the one with the most values and lists the others inalternativeswith the reasonminority_kind. - Numeric:
numeric/integerandnumeric/numberfrom the technical type when no detector type competes (basisfallback). An Enum candidate over a numeric column is an alternative with the reasonnumeric_type_kept. - Identifier, from a name and compatible values:
identifier/idandidentifier/unique_id, alwayscandidate(see Identifiers). - Other / Mixed for a
mixedtechnical type that no detector refines. - Text, from a
texttechnical type, in five lengths:text_very_short,text_short,text_medium,text_longandtext_very_long. The length type is decided on the exact median of the lengths of the present values (string_lengths), with the bounds ofscan.classification; a median of 5.5 is Short with the defaults. The maximum is never a replacement: whenstring_lengthsis not complete the type istextalone,stateisundeterminedandalternativeslists the five lengths with the reasonstring_lengths_<status>.
When two credible types cannot be told apart, the result is an explicit
fallback and never the first one in the order of the detectors: state is
undetermined, both candidates are in alternatives with the reason
conflicting_candidates and their evidence has the role context. Two
detector types, or two kinds of dates of the same weight, give other/unknown.
A detector type that reads a numeric column as well, such as postal_code or
phone over whole numbers of five or ten digits, keeps numeric/integer
undetermined: the values cannot say whether the digits are a code or a quantity.
The name of a field is read after its values, by versioned rules in French and
English (email, courriel, code_postal, date_naissance, is_active…).
A name never creates a type. It can only:
- Reinforce a type the values already support: the rule is added to
evidencewith the rolesupports. The type and thestatedo not change. - Break a tie between types the values support equally. Five-digit whole
numbers named
postal_codebecomegeographic/postal_code, and the same numbers namedpostal_code_statusstaynumeric/integer. A conflict settled by a name hasstatecandidate, because the result rests on a name plus compatible values; the other type is inalternativeswith the reasonoutranked_by_nameorweakened_by_name. - Weaken a type: a name such as
email_statusorcustomer_id_lengthdescribes a property of the thing it names, not the thing. If the values still support the type, it is kept and the rulename.derived_measureis added with the rolecontradicts.
A type that only the name suggests is listed in alternatives with the reason
name_only when the technical type allows it, and is never the result: a column
named email that holds free text is Text.
Names are compared as whole words, never as pieces of words, after Unicode
normalization, the split of CamelCase and acronyms (CustomerID, ZIPCode),
the removal of accents and the harmonization of separators, so CustomerID,
customer_id and CUSTOMER-ID read alike. The header or the key is not
changed. For a JSON field the key of the field is the name, and the keys above it
are only context: postal.code as one key is a postal code, postal containing
code is not. The identifier names (id, customer_id, id_client, order_ref…)
give the identifier role.
The String analysis section of the report names the length of a column from
its maximum with the same bounds, so a column can be Short as a type and
medium there.
Identifiers
Section titled “Identifiers”identifier/id is the one type a name and the values create together. It needs:
- an identifier name rule (
id,identifiant, a name that starts or ends withid, ends withreforkey, or hascustomerororderwithid) and no weakening of it:customer_id_lengthandorder_id_statusdescribe a property of an identifier, not an identifier; - compatible values: a
textorintegertechnical type with no present value outside its family, and at least two present values.
The role replaces a Text of any length or a plain Integer, and nothing else. The
replaced type stays in alternatives with the reason replaced_by_role. An
Integer that postal_code or phone also reads (five or ten digits) is
undetermined and the identifier name settles it. A detector type, Enum, Constant,
Number, a date or time type, Boolean and Other keep their type: an identifier
never overrides a more specific type, and the role is then an alternative with
the reason name_only. A UUID keeps web_network/uuid and lists the role with the
reason uuid_name. The state is candidate, never established: the role rests
on a name, not on the values alone.
identifier/unique_id is an ID for which all of these hold:
- the field is not under a JSON array;
- no position is missing (
missing.countis 0, and no record lacks the field, even ifvalues.missingdoes not countabsent); - the number of distinct analytical values is exact and complete, and equals the
number of present values. The analytical values follow the NFC, trim and
whitespace stages, so
AandAare the same value. A limited or estimated cardinality never gives a Unique ID in the Scan. The Report can refine an ID whose cardinality was limited with the exact distinct count of the second pass.
The evidence of a Unique ID adds the measures values.cardinality and
missing.count, with the scope they cover: the present values and the positions
of the dataset, and partial set when the scan excluded records of that dataset.
A uniqueness seen in the scanned scope does not guarantee a business key,
uniqueness in the future or uniqueness in the records the scan excluded.
A column without an identifier name stays outside Identifier whatever its
position or its uniqueness: a first column of distinct dates, timestamps or
amounts keeps its type with the traits first_column and observed_unique.
observed_unique needs at least two present values. The trait high_cardinality
is not emitted: no threshold comes from the limits the Scan tracks.
Measure envelopes
Section titled “Measure envelopes”A measure that can be limited, disabled or inapplicable is wrapped in an
envelope with a status:
{"status": "complete", "value": 42}{"status": "limited", "reason": "distinct_limit", "limit": 100000, "lower_bound": 100001}{"status": "not_applicable", "reason": "no_values"}{"status": "disabled"}{"status": "failed", "reason": "detector_error", "diagnostic": 3}complete: exact for its population.limited: a configured limit stopped the measure.lower_bound, when present, is a proven minimum; no estimate is ever given as a value.not_applicable: meaningless here, for example without values.disabled: turned off by configuration, or hidden because the field is sensitive.failed: a technical failure;diagnosticis its index indiagnostics.
A scan can be complete while some of its measures are limited: the scan
status is about records, the envelope about one measure.
Distinct values and samples
Section titled “Distinct values and samples”values.cardinality counts the distinct values of a field in the original
domain: the values as decoded, before any normalization stage, so 00123,
123 and 123 are three values. It is an envelope with more members than
the others:
{"status": "complete", "value": 3, "accuracy": "exact", "domain": "original", "identity_accuracy": "exact"}{"status": "limited", "reason": "distinct_limit", "limit": 100000, "lower_bound": 100001, "domain": "original", "identity_accuracy": "exact", "upper_bound": 250000, "threshold_reached_at_record": 100000, "overflow_detected_at_record": 100001}domain: what is counted.originalis the only domain a scan writes.accuracyof acompletecount isexact. Acompleteenvelope withoutaccuracy, as in the other measures, is exact for its declared population.identity_accuracy: how two values are told apart.exactmeans that equal values are compared whole, never by a digest.upper_bound: the certain maximum of alimitedcount, the number of analyzed values of the field (values.count).threshold_reached_at_recordandoverflow_detected_at_record: for the reasondistinct_limit, the records (numbered likefirst_record) where the table reachedscan.limits.max_distinct_per_fielddistinct values and where one more was proven. A record with several values of the field counts once. They are absent for the other reasons and when a member is unknown: an absent member is nevernulland never zero.
When the table of distinct values is released, values.cardinality may carry
an estimate of the distinct count, from the sample described below:
"estimate": {"accuracy": "estimated", "method": "kmv", "value": 187500.0, "relative_standard_error": 0.01, "k": 10000, "theta_hex": "0000a3f1..."}accuracyisestimated: the value is a statistical estimate, never an exact count.relative_standard_erroris its expected relative error (about 1% atk10,000).valueis held within the certain boundslower_boundandupper_bound;unclamped_valueis the raw estimate, present only when a bound changed it.theta_hexis the threshold of the sample (seethresholdbelow). It is absent for a sensitive field.- A field with long texts has two domains, the stored values and
the long texts, which the estimate adds.
partsgives each one as{domain, accuracy, value, relative_standard_error, k, theta_hex}:accuracyisexactfor a domain whose sample dropped nothing (nokortheta_hex) andestimatedotherwise.relative_standard_erroris that of the sum, andkandtheta_hexare then in the parts, not in the estimate itself. - There is no estimate when a value fitted no sample because the memory budget granted a size of 0.
values.samples lists distinct original values with their exact counts, each
entry as {id, value, type, count, count_accuracy, identity_accuracy}:
selectionisallwhen the sample lists every distinct value of the field (at mostscan.limits.max_samplesof them, 10,000 by default), in the order they were first seen.- Otherwise
selectionisbottom_k: thekvalues of smallest priority, in ascending priority. The priority of a value is a stable hash of its type and its text (hash:algorithm,version,bitsandseed, which isscan.random_seed), so the sample depends on the set of distinct values and onk, not on the order of the records, on the reading in batches or on the workers.thresholdis the largest priority kept, as hexadecimal digits, present once a value was dropped. - Counts are exact for the whole field (
count_accuracyisexact): a value of the sample was counted from its first occurrence. They are not a ranking: the most frequent value can be absent, and the counts of a sample do not describe the other values. idnumbers the entry in the listing of the scan. A report that keeps some of the entries keeps theirid.- Values are in the original domain (
00123,123and123are three entries), unlikevalues.frequencies, which lists analytical values. keptis the number of entries listed andofthe number of original values sampled. A scan writes both equal, except for a sensitive field: exposure applies after the calculation,maskmerges the values with the same mask andhidelists none, sokeptis belowofand the originals cannot be recovered from the document. A report that holds the scan as its base also keeps fewer values and says so withkeptbelowof.
max_samples cannot exceed max_distinct_per_field; when it is not set it
follows a lower max_distinct_per_field.
Long texts
Section titled “Long texts”A value longer than scan.limits.max_stored_value_length (1,000 characters by
default) is never stored. It is counted and measured like any value, and a small
sample of such values is written in values.samples.long_texts, absent for a
field that has none:
"long_texts": {"selection": "bottom_k", "k": 100, "preview_chars": 200, "occurrences": 36143, "kept": 100, "of": 100, "digest": {"algorithm": "blake2b", "bits": 128}, "hash": {"algorithm": "blake2b", "version": 1, "bits": 128, "seed": 0}, "threshold": "00a1f3...", "listed": [{"id": 0, "digest": "c2f3e568...", "type": "string", "preview": "data customer contract...", "truncated": true, "original_length_chars": 1520, "original_length_bytes": 1520, "count": 2, "count_accuracy": "exact", "identity_accuracy": "fingerprint_based"}]}occurrencescounts every occurrence of a long value, exactly. The minimum, maximum, total and mean lengths are instring_lengths.- An entry keeps a
preview(the firstscan.limits.long_preview_charscharacters, 200 by default, cut on a character), the original lengths in characters and bytes,truncated(the preview is shorter than the text) and thedigestof the complete text. Two texts with the same beginning are two entries. The preview says nothing certain about the pattern or the alphabet of the whole text. identity_accuracyisfingerprint_based: two texts are the same entry when their 128-bit digests are equal, not when the full texts were compared.selection,k,hashandthresholdwork as for the values:allwhile no text was dropped, thenbottom_k. The sample holds at mostscan.limits.max_long_samplestexts (100 by default; 0 keeps none, and the occurrences are still counted).- The listing of the values (
selection,kept,of) describes the values that fit the stored length only. - For a sensitive field the block lists nothing (
keptis 0,ofis not): a preview and a digest are the value itself.
Samples by length
Section titled “Samples by length”A bottom_k sample favors the lengths that most values have: a length that few
values share can be absent from it. values.samples.by_length completes it with
a sample for each length, so a rare length stays visible. It is written for a
field whose sample is bottom_k and absent when the sample lists every value
(selection all), or when scan.limits.max_samples_per_length is 0:
"by_length": {"domain": "original", "unit": "characters", "per_length_k": 100, "max_lengths": 64, "limited": false, "tracked": 3, "strata": [ {"length": 4, "selection": "all", "kept": 2, "of": 2, "occurrences": 5, "count_accuracy": "exact", "ids": [10210, 10211]}, {"length": 8, "selection": "bottom_k", "kept": 100, "of": 100, "occurrences": 999990, "count_accuracy": "exact", "threshold": "0004c1...", "ids": [3, 8, 11, "..."]} ], "listed": [{"id": 10210, "value": "abcd", "type": "string", "count": 3, "count_accuracy": "exact", "identity_accuracy": "exact"}]}domainisoriginalandunitischaracters(code points of the original text): the length of00123is 5, whatever the normalization.- A stratum is a bottom-k of the values of one length, with at most
per_length_kvalues (scan.limits.max_samples_per_length, 100 by default). It uses the same priority as the sample, so a value that is in both is the same entry.occurrencesis the exact number of occurrences of that length,ofthe values sampled andkeptthose listed;selectionisallwhile the stratum dropped no value, andthresholdworks as for the sample. - A stratum lists the
idof its values. A value that is already insamples.listedis not copied: only the others are inby_length.listed, whose ids continue the numbering ofsamples.listed. - Counts are exact (
count_accuracyexact) for a length tracked from its first occurrence, as for the sample. A length admitted later hascount_accuracypartialandtracking_started_at_occurrence: its counts cover the occurrences from that one on, and no frequency of it is exact. A scan tracks every length from its first occurrence, so it writesexacttoday. - At most
scan.limits.max_length_stratalengths (64 by default) are tracked per field, the first met. The others are summed inother:{"lengths": 32, "occurrences": 96}, the number of untracked lengths and their occurrences, andlimitedistrue. The strata andotheradd up to the stored values of the field. The minimum, maximum and mean lengths stay exact instring_lengths. - Values longer than
scan.limits.max_stored_value_lengthare not in the strata: they have their own sample. - For a sensitive field the strata keep their
length,ofandoccurrencesand list nothing (keptis 0,idsis empty, nothreshold). - A report keeps the strata
without their values:
keptis 0. scan.limits.max_samples_per_lengthandmax_length_strataat 0 turn the samples by length off. Both are in the configuration.
Cases of the observed rules
Section titled “Cases of the observed rules”A rare case, such as two cells out of 100,000 that end with a space, is often
missing from a sample of distinct values. values.samples.cases keeps it: while
it reads the values, the scan counts the cells of each case and keeps a small
bottom-k of their original texts with exact counts. The member is absent when
the field has no case.
"cases": {"per_class_k": 5, "whitespace": [ {"kind": "trailing", "occurrences": 2, "selection": "all", "kept": 2, "of": 2, "listed": [{"value": "CLI-000017 ", "count": 1}, {"value": "CLI-004000 ", "count": 1}]} ], "lengths": {"max_lengths": 32, "classes": [ {"length": 6, "occurrences": 2, "selection": "all", "kept": 1, "of": 1, "listed": [{"value": "CLI-12", "count": 2}]} ]}}whitespacehas a class for eachkindthat has a cell:leadingandtrailing(thetrimstage changed the text) andrepeated(thecollapse_whitespacestage did). A text with two kinds is a case of both.occurrencesis the exact number of cells:repeatedequals thechangedcount of the stagecollapse_whitespace, andleadingandtrailingbound that oftrim.lengthsis written only when the field has a minority length (the observationminority_length) and has a class for each analytical length (the one ofstring_lengths) except the dominant one. The firstscan.limits.max_case_lengthslengths (32 by default) are tracked; the cells of the others are summed inother, as for the samples by length.- A class lists at most
per_class_ktexts (scan.limits.max_cases_per_class, 5 by default), chosen by the priority of the sample, so the choice does not depend on the order of the values or on the workers.countis the exact number of occurrences of that original text in the whole field.selectionisallwhile the class dropped no text: the list is then every distinct text of the case. Otherwise it isbottom_kandthresholdworks as for the sample. - A case holds its own text: it is not an
idofsamples.listed. A text longer thanscan.limits.max_stored_value_lengthis counted inoccurrencesand never listed. - A sensitive field follows its exposure:
maskmerges the texts with the same mask and adds their counts,hidelists none, and neither writes athreshold.occurrencesstay. - A report keeps the counts of the cases and empties their lists.
scan.limits.max_cases_per_classat 0 turns the cases off,max_case_lengthsat 0 turns off those by length. Both are in the configuration.
Sample sizes and the memory budget
Section titled “Sample sizes and the memory budget”The scan limits the memory its samples may take with a model of their cost, not
by measuring (scan.limits.max_sample_memory, 512 MiB by default). Sizes are
reserved in the order the source is read, so they are the same for any number of
workers. A field never takes more than an equal share of the budget, so a source
with very many columns cuts every column alike. A long text sample and the
samples by length each take their reservation from an eighth of the budget, and
the cases of the observed rules from another eighth.
The samples by length reserve the cost of a full stratum for each length they
may track, in the same order. The cases reserve the cost of all their classes
(the kinds of whitespace and max_case_lengths lengths) at per_class_k texts
each; when the budget grants fewer texts, per_class_k is the size in force and
requested_per_class_k and reason memory_budget say what was asked (the
warning sample_budget does not count them).
When the budget grants less than scan.limits.max_samples (or
max_long_samples), the block writes the size in force in k, the size asked
for in requested_k and reason memory_budget (the long texts block has
requested_k only), and a warning sample_budget names how many fields were
cut. The samples by length do the same with max_lengths,
requested_max_lengths and reason, only for a field that has more lengths
than the budget granted: a field that fits the lengths granted lost nothing and
says nothing. A grant of 0 lengths writes the block with tracked 0 and
limited true. A smaller sample is still a bottom-k: its estimate has the larger error of
its k. A size of 0 samples nothing for that field (kept and of are 0) and
the field has no estimate. Counts, lengths, statistics and the exact tables are
not affected. A sample that holds every value of its field (selection all)
is never cut.
Counters of a run
Section titled “Counters of a run”tabalyst scan --counters (tabalyst.scan(..., counters=True)) adds
run.counters.sampling, the work of the samplers summed over the fields:
hashed_keys, admitted_keys, hit_keys (occurrences of values already in a
sample), rejected_keys, evicted_keys, long_texts_hashed,
long_bytes_hashed, bottom_k_fields, reduced_fields, strata_fields
(fields with samples by length), strata_limited_fields (those with values in
other), k_effective_min, k_effective_max, sample_model_bytes,
long_model_bytes and length_model_bytes. Without the
option the member is absent. A figure that the scan did not measure is not
written. Counters describe the run, not the data.
Writing the document
Section titled “Writing the document”tabalyst scan writes the document without indentation by default, since the
sample entries make it large (about 10 MB for 90,000 entries). Set
output.json_indent to true in the
configuration to indent it,
with each sample entry on one line. The content is the same.
Inferred patterns
Section titled “Inferred patterns”After a scan reads the whole source, it infers the structural patterns of each
eligible field from the samples it kept, without reading the source again. They are
written in the top-level derived.patterns member, one entry per field, and
they are not the configured patterns,
which are detectors. A pattern here is a shape, such as two capitals followed by
four digits, found among the distinct values of a field: the discovery of a
structure, not a rule that the data must follow.
{ "derived": { "patterns": { "grammar": {"name": "tabalyst.pattern", "version": 2}, "algorithm": {"version": 1}, "domain": "original", "fields": [ { "dataset": "rows", "field": "column_1", "status": "complete", "eligibility": { "state": "eligible", "score": 1.0, "reasons": ["regular_structure"] }, "population": { "source": "samples", "selection": "all", "complete": true, "available": 3000, "distinct": 3000, "occurrences": 3000, "occurrences_basis": "exact", "excluded": { "long_text_occurrences": 0, "non_string_values": 0, "empty_values": 0, "too_long_values": 0, "too_complex_values": 0 } }, "families": [ { "id": "f1", "key": "UP-D", "d": 3000, "o": 3000, "patterns": [ { "id": "f1.p1", "signature": [ {"kind": "literal", "text": "CLI", "min": 1, "max": 1}, {"kind": "literal", "text": "-", "min": 1, "max": 1}, {"kind": "digit_ascii", "min": 7, "max": 7} ], "render": "'CLI-'9{7}", "format": "AAA-9{7}", "lengths": [3, 7], "d": 2178, "o": 2178, "examples": [0, 1, 2], "nodes": [] } ], "tables": [], "examples": [0, 1, 2] } ], "other": {"structures": 0, "d": 0, "o": 0}, "limits": [] } ] } }}-
grammarnames the version of the rules that cut a value into tokens and write a signature, and of the notation ofrender. A change of the rules raises itsversion.algorithm.versionis the version of the discovery itself (its scores and constants). -
Each field has an entry.
statusiscomplete,failedornot_applicablewith areason.failedmeans the inference raised on that field (reasoninference_error):diagnosticis the index of apattern_inference_failederror indiagnostics, which names the field and the exception type, and the other fields and the rest of the scan are unaffected. -
Every entry has an
eligibility: whether the field is worth a search for patterns, decided from the facts and the samples of the scan before any pattern is built. Only aneligiblefield is inferred; the others arenot_applicable, and theirreasonis the first reason that kept them out.stateis one of:eligible: the values have a regular structure.low_relevance: the field can be assessed, but a pattern would describe it poorly (free text, a handful of words, decimal numbers).excluded:exposure(the values of a sensitive field are masked or hidden, so no pattern is inferred),booleans_only,structures_only(only objects or arrays),long_texts_onlyorno_values.not_assessable: information is missing,samples_withheldortoo_few_distinct.
scoreruns from 0 to 1 and ranks the fields (0 for a field that is excluded or not assessable); a field is eligible from 0.35.reasonsexplain it:regular_structure(at most 6 forms of fixed length, such asAA-9999, cover at least 90% of the values) ormany_fixed_lengths(they do not, as for hyphenated ISBNs or free text),integer_digits(a column of whole numbers of 4 digits or more, read as text) orsmall_integers(shorter ones, read as counts or measures),decimal_numbers,no_structural_marks(no digit and no separator),few_distinct(fewer than 8 distinct values) andsemantic_detector(a detector already says what the values are, such as dates, which lowers the score without excluding the field). -
populationis what the inference read: the distinct values of the field’s sample and its samples by length, once each.completeis true when the population is every distinct value of the field: the sample isalland nothing was excluded or cut.availablecounts the usable values anddistinctthose that took part.excludedcounts what did not: the occurrences of long texts, whose preview never gives the pattern of the whole text (long_text_occurrences), and the values that are neither strings nor integers (an integer is read as its decimal text), are empty, are longer thanscan.pattern_inference.max_value_lengthor have more than 32 tokens. -
familiesgather the values of one structure: the classes of their tokens (digits, capitals, lowercase letters, other letters) and their separators, without the lengths (key:UP-Dis capitals, a hyphen, digits). Two structures that differ only at the start or at the end of a block, by one or two positions, are one family when the part they share is at least as long:AK-12345andAK-1234X(compositionlists their keys). Each family has anid(f1,f2, …), ranked by the larger of its shares of the distinct values and of the occurrences, and its counts:ddistinct values andooccurrences. A structure with a single value is a family of its own. -
A family lists its
patterns, one for each combination of lengths found in the values, withidf1.p1:'INV-'999,'INV-'9999and'INV-'99999are three patterns of one family. A pattern has a fixed length at every position and never a range. Itssignatureis the list of its tokens: a class of characters (digit_ascii,upper_ascii,lower_ascii,letter_unicodefor letters of any script,upper_digit,lower_digit,alphanumeric) repeatedmintomaxtimes (always the same number), or aliteraltext. Separators, spaces and signs are always literals. The characters that all the values of a pattern have at the start of a token become a literal:'AK-'9999; a fixed value and the separators that touch it are written as one block of text, and a single space between two elements as it is:'AK-' 9999. A pattern of a single value folds nothing. -
renderis a short reading of the signature:9for a digit,Afor a capital,afor a lowercase letter,@for any letter,Xfor a capital or a digit,xfor a lowercase letter or a digit,#for a letter or a digit, a count in braces for longer repeats, and literal letters and digits between quotes:AA9999'A'is two capitals, four digits and the letter A. Thesignatureis the reference.formatis the exact form of the commonest value of the pattern, andlengthsthe lengths of its tokens (a merged block counts as one). -
When a pattern holds several merged structures, its last positions are of the classes of all of them (
'AK-'9999X), andnodeslists the structures with their own counts ('AK-'99999,'AK-'9999A), withfacetcomposition. The counts of these nodes add up to the pattern, those of the patterns to the family. -
A pattern also has nodes of content when a part of its values is set apart by the first or last characters of a token, or by a short token.
facetsays what the node reads, counting the tokens of the structurekeyfrom 1 (UP-Dhas three: the capitals, the hyphen and the digits):head:1the first characters of the capitals,tail:3the last characters of the digits,token:3the whole token (at most 4 characters). A node has asignatureand arenderlike a pattern ('AK-'9999; a tail is written after the class:9{4}'00'), itsvalue(the text it fixes at its facet, spelled from the left, such asAK; for a node ofcomposition, the structure it reads),alsowhen a node merged into it (what that node fixed:facetandvaluepairs), its countsdando, itslevel(2 under a pattern) and again:bits_dandbits_o, the bits it saves when the values are described by it rather than by the pattern (a description length, on the distinct values and on the occurrences; one is negative when the node is kept for the other measure only), andsigma, the share of the bits of the characters it reads that its facet saves. A facet is published from asigmaof 0.10, and only when the characters it reads depart from chance: letters drawn at random make no node. The nodes of a facet are a part of the pattern, the other values being its rest. A longer prefix of the same facet ('AK'then'AK1'), or a facet that depends on the node (the year, once the prefix is known), is nested in itsnodes, down toscan.pattern_inference.max_depthlevels (the pattern is the first). Each value counts for the deepest node that holds it. -
dandocount distinct values and occurrences. The distinct values are those of the population.oisexactwhen the population is every distinct value (occurrences_basis), and an indicator of discovery on a sample otherwise: it is not the share of the rows of the file, which only a later pass can give. Each value counts for one family and one pattern. -
examplesare ids of entries of the field’s sample (see Distinct values and samples). A report keeps fewer sample values, so some ids may not be found in it. -
tablesof a family are of two kinds.facetlengthslists the combinations of lengths thatscan.pattern_inference.max_patterns_per_familykept out of its patterns. A facet of content (head:N,tail:N,token:N) that is published but that no node reads, because the values it splits do not depend on the nodes, is a table of the family:cellsofvalue(the characters, spelled from the left),dando,othersfor the rest, and thegainof the facet. The cells andothersadd up to the family. -
othercounts the structures that no published family reads, with their distinct values (d) and occurrences (o).limitsnames the budgets that acted:max_values(the population was cut toscan.pattern_inference.max_values),max_families,max_patterns_per_family,max_patterns(the patterns and nodes of the field, the nodes of the lowest gain going first) andmax_depth(a node was belowscan.pattern_inference.max_depthlevels and was left out). -
scan.pattern_inference.enabledset tofalseleavesderivedout of the document. A report carries the block as it is.
Character profile
Section titled “Character profile”string_characteristics counts the strings of a field by what their characters
are. It works on the raw values, before normalization, and counts occurrences of
values (a value found three times counts three times). Shares are not stored.
| Group | Members |
|---|---|
| Case | uppercase, lowercase, mixed_case, no_letters, starts_uppercase, starts_lowercase |
| Classes | with_ascii_lowercase, with_ascii_uppercase, with_other_latin_letters, with_non_latin_letters, with_ascii_digits, with_other_digits |
| Composition | ascii_only, non_ascii, letters_only, digits_only, alphanumeric_only, letters_and_digits |
| Whitespace | with_surrounding_whitespace, with_repeated_whitespace, with_line_breaks, with_tabs, with_non_breaking_spaces |
| Problem characters | with_control_characters, with_invisible_characters (U+200B to U+200D, U+FEFF, U+2060), with_replacement_character (U+FFFD) |
| Totals | characters (the characters of all the values), distinct_characters |
| Special characters | special_characters: the inventory of the punctuation and symbols |
distinct_characters is the exact number of distinct characters of the field. It
is limited (reason character_limit) above limits.max_distinct_characters.
special_characters lists, for each character that is neither a letter, a digit,
whitespace nor a control or format character, its char, its codepoint, its
value_count (distinct values that contain it) and its occurrence_count
(occurrences of values that contain it), most frequent first. The measure says
value_count_accuracy: exact, or upper_bound for a field whose table of
distinct values was released, since a value that comes back in a later batch is
counted again (occurrence_count stays exact). Above
limits.max_special_characters distinct characters the measure is limited and
lists none, so a partial list is never taken for the whole. On a sensitive field
exposed with mask or hide the counters stay and both distributions are
disabled.
Character map
Section titled “Character map”derived.characters.fields has one entry per field, in the order of the datasets and
fields, each a measure envelope of the character map of its strings
(status complete, limited, not_applicable, disabled or failed). It is computed
after the scan from the samples, never from the source.
{ "dataset": "rows", "field": "name", "characters": { "status": "complete", "value": { "basis": "exact", "values": 4, "characters": 20, "distinct": 11, "top": [{"char": "a", "codepoint": 97, "occurrence_count": 3}], "scripts": [{"name": "LATIN", "occurrence_count": 20}], "categories": [{"name": "Ll", "occurrence_count": 17}], "ends_uppercase": 0, "ends_lowercase": 4, "title_case": 3, "with_emoji": 0, "emoji_sequences": 0, "emoji": [] } }}basisisexactwhen the scan listed every distinct value, andsamplewhen the map counts the stored samples only. A second pass makes it exact.toplists thelimits.max_top_charactersmost frequent characters (20 by default), letters and digits included, from the most to the least frequent, then by code point.Aandaare two characters.scriptsgives the characters of the letters by script (the first word of the Unicode name:LATIN,CYRILLIC,CJK…).categoriesgives all the characters by Unicode general category (Lu,Nd,Po…) and adds up tocharacters.ends_uppercaseandends_lowercasecount the values whose last character is a capital or a lowercase letter, andtitle_casethe values written in title case (Jean-Pierre, notMcDonald). All three count occurrences of values.emojilists the most frequent emoji sequences (an emoji with its skin tone and joiners, a flag or a keycap is one sequence). The emoji are recognized by a table of ranges, an approximation of the Unicode emoji.- Above 65,536 distinct characters or 10,000 distinct emoji the entry is
limitedwith the reasoncharacter_limitand lists nothing. A field exposed withmaskorhideisdisabled.limits.max_top_characters0 removes the block.
Normalization stages
Section titled “Normalization stages”normalization.stages lists raw and then each stage in order (nfc, trim,
collapse_whitespace, casefold, strip_accents). Each has enabled,
changed (the values the stage modified, given the output of the previous
enabled stage; null for raw and disabled stages) and cardinality (the
distinct values after the stage, in a measure envelope).
A stage with changed 0 modified nothing, so it leaves the distinct values as
it found them. It then has equals, the name of the stage it takes the count
from (the previous enabled stage, or raw), and its cardinality is that
stage’s: exact when that one is, and limited with the same reason, limit and
lower_bound when the table of distinct values was released. The estimate and
the upper bound of the raw count are only in values.cardinality. changed is
counted over the whole scan, so this holds even when the table was dropped
before the end.
A stage that changed something after the table was released has a limited
count without a bound and no equals: its distinct values cannot be rebuilt
from values that were dropped. A later stage that changed nothing takes that
count. When no enabled stage changed any string, variant_groups is complete
and empty even if the table was released, since no spelling can be grouped with
another.
Observations and limits
Section titled “Observations and limits”After the scan, Tabalyst turns its facts into findings, written in the
top-level observations member. They are computed from the counts and the
inferred patterns, without reading the source, and they hold no value, only
counts and references (fields, records, diagnostics, patterns).
{ "observations": { "version": 1, "truncated": false, "summary": { "informational": 19, "candidate_anomalies": 24, "confirmed_issues": 0, "limits": 7, "has_observations": true, "has_quality_issues": true, "has_limits": true }, "datasets": [{"dataset": "rows", "...": "same as summary"}], "items": [ { "id": "o1", "rule": "duplicate_rows", "category": "records", "status": "candidate_anomaly", "dataset": "rows", "population": "records", "count": 8, "of": 3000, "accuracy": "exact", "records": [1521, 1814] } ], "limits": [ { "id": "l1", "dataset": "rows", "field": "column_19", "measure": "values.samples", "reason": "exposure" } ] }}- An observation has a
rule, acategoryand astatus.informationalis a remark that can be perfectly valid, such as a column that is mostly empty.candidate_anomalyis a signal that no rule can confirm, such as a rare pattern or duplicate rows.confirmed_issueis a defect by the structure of the source, such as excluded records or blank column names.populationsays whatcountcounts out ofof;accuracyisexact,lower_bound(a limit stops the count from being exact, andlimited_bynames it) orsample. - The rules of version 2 are
excluded_records,ambiguous_header,duplicate_rows,mixed_types,type_outliers,ambiguous_dates,variant_groups,minority_pattern,minority_length(one length holds at least 95% of at least 20 strings and the others at most 5%: a malformed value that a pattern absorbed),missing_values,empty_column,constant_column,trimmed_cellsandcollapsed_whitespace. A rule applies to a field, a dataset or the whole scan (nodataset). Their thresholds are constants of the version. - A limit is what the scan could not establish: a measure stopped by a
scan limit (
values.cardinality,records.duplicates.count), a sample cut by the memory budget or withheld by exposure, a truncated structure, or an inference that failed. It is never a defect of the data, so it is listed apart from the observations. A stage that repeats the limit of its input is not listed again. summaryanddatasets(one per dataset) count the findings and give three flags:has_observations(any observation),has_quality_issues(a candidate anomaly or a confirmed issue) andhas_limits. They are per scan and per dataset; for a field, read the items.itemsare sorted by status (confirmed issues first), then by dataset and field, and at mostscan.observations.max_items(1,000) of them and of the limits are listed.truncatedsays a list is shorter than its count; the counts ofsummaryare complete.scan.observations.enabledset tofalseleaves the member out. If the computation fails, the member is left out too and anobservations_failederror indiagnosticssays so; the scan is complete. A report carries the member as it is; itsissuesare unchanged.
JSON Schema
Section titled “JSON Schema”The package contains the JSON Schema of the scan document,
tabalyst/schemas/tabalyst.scan.schema.json, also available from
tabalyst.scanner.schema.scan_schema(). It describes the base document and
describes derived (the inferred patterns) and
observations (the observations) and accepts
the namespaces analyses and report, which extensions of the base may add. The package also has the schema of the
report, tabalyst/schemas/tabalyst.report.schema.json
(tabalyst.report_schema.report_schema()). Rules that link several members, such as
a count that must equal the sum of its parts, are checked by
tabalyst.scanner.validation.check_scan(), which returns the violated
invariants.
Detectors
Section titled “Detectors”{ "id": "number", "version": 1, "status": "complete", "coverage": { "eligible": 2, "tested": 2, "matched": 2, "ambiguous": 0, "invalid": 0, "not_matched": 0, "not_tested": 0, "share_tested": 1.0, "share_eligible": 1.0 }, "formats": [{"format": "0", "count": 1}, {"format": "0.0", "count": 1}], "evidence": { "matched": ["12.50", "7"], "ambiguous": [], "invalid": [], "not_matched": [] }, "details": {}, "adaptive": null}Every detector is listed for every field, with status complete,
not_applicable, disabled or failed.
coverage:eligiblevalues of an accepted type, split intomatched,ambiguous(several readings, such as01/02/2026),invalid(the right shape with a wrong content, such as a 13th month),not_matchedandnot_tested. Shares are rounded to four decimals.formats: the forms matched, such asYYYY-MM-DDor#,##0.0, by count.evidence: the first distinct values in each state.details: detector-specific counts, such as email domains or ambiguity evidence. Theemail,phoneandpostal_codedetectors addkeys: the extension and domain of an address, the region and area code of a number, the first character and area of a postal code, counted exactly with their occurrences (key,occurrences, then thekeysandothersbelow) and published even on a masked field; seescan.exposure.detector_keys.adaptive:nullwhen the detector tested every value of the field. When it recognized none of the firstdetection.warmup_valuesdistinct values, or at mostdetection.rare_shareof them and none in the second half of that warm-up, it is skipped for the other values except probes, andadaptiveis{"skipped_after": 10000, "warmup_reactions": 0, "not_tested": 1200, "diagnostic": null}: the warm-up size, the warm-up values it recognized, the values it did not test (included incoverage.not_tested), and the index of adetector_skipped_reactedwarning when probes were recognized more often than the warm-up allowed, meaning its counts are incomplete.numberanddateare never skipped.
The built-in detectors are number, date, boolean, enumeration,
email, url, phone, postal_code, currency, percentage, quantity,
uuid and ip_address. Patterns of the configuration appear as
pattern:<id>. Detectors check syntax, never real-world existence.
The evidence is published, and the configuration resolves an ambiguity,
for example with detectors.date.ambiguous_order. Numbers are also read by
the evidence of their own column, never by a guess:
-
When at least
detectors.number.min_evidence(3) unambiguous numbers of a field use one convention and none uses the other, the ambiguous numbers (1,234) are read in that convention and counted innumeric. With a decimal comma proven (2,5,1.234,56), a value such as1.250is read as 1250, a thousands group, rather than as 1.25. -
A column that uses both conventions, or too few of them, is read as before: its ambiguous numbers stay out of
numeric. Spaces between thousands (1 234) prove nothing on their own. -
The
numberdetector names what was done indetails.ambiguity.resolution:{"convention": "comma", "source": "evidence","evidence": {"comma": 490, "dot": 0}, "resolved": 110}resolvedcounts the numbers read by the evidence. Itscoverageandformatsstill describe how the values are written, so1,234staysambiguousthere andnumeric.countcan exceedcoverage.matched. -
detectors.number.infer_convention: falseturns the rule off, anddetectors.number.ambiguous_conventionalways wins over it. -
JSON and Excel numbers are numbers already: they are kept as they are, never read in a convention. Only text values are.
The second pass reads the numbers of such a field as the scan counted them.
Dates whose two readings are the same day (06-06-2020) need no order. The
date detector counts them as matched, with no format, and publishes their
number in details.ambiguity.same_day. They are neither in count and
candidates (the ambiguous dates) nor in evidence, since they prove no
order.
Sensitive values
Section titled “Sensitive values”A field is sensitive when a sensitive detector (email, phone,
ip_address, or a pattern declared sensitive) matched one of its values.
With the mask exposure, its listed values, evidence and variant
groups show masks: uppercase letters become A, other letters a and digits
9. Equal masks are merged. Under mask and hide, the numeric and
temporal blocks of a sensitive field are disabled, since a minimum or a
date range is itself a value. Counts are never masked.
Diagnostics
Section titled “Diagnostics”{ "code": "csv_width_mismatch", "level": "error", "message": "...", "count": 2, "dataset": "rows", "field": null, "detector": null, "details": {}, "locations": [{"record": 2, "line": 3}, {"record": 3, "line": 4}]}measures_limited has one diagnostic per field (field is the field
identifier), with details.measures mapping each limited measure, such as
values.cardinality, to its reason; its count is the number of limited
measures. details is empty for the other codes.
Diagnostics describe technical events, not data quality: excluded records,
reached limits (field_limit, depth_limit, measures_limited,
global_budget, record_budget, sample_budget), collections not found and
detector failures, and the failure of a post-scan analysis
(pattern_inference_failed, observations_failed). level is
error or warning. count is always complete; locations lists at most
errors.max_locations places, with record and line for CSV and JSONL,
record and element for JSON. For JSONL, line is the physical line number,
blank lines included, and record counts the non-blank lines.