Scan format changelog
A scan document written by tabalyst scan identifies its experimental
contract with three fields:
{ "format": "tabalyst.scan", "format_version": "0.2.0", "format_revision": 4}format identifies the document kind. format_version names the compatibility
family, and format_revision increases for meaningful structural or semantic
changes within it. The JSON profile has its
own version and changelog. Beta revisions may be incompatible; automatic
migration is not provided. tabalyst report --scan reads the current Scan
version only: scan the source again to use an older document.
Version 0.2.0, revision 1 — 2026-10-06
Section titled “Version 0.2.0, revision 1 — 2026-10-06”This is the starting point for the current Scan format. It describes CSV, JSON, JSONL and Excel sources, their effective configuration, datasets, fields, detectors, record facts, limits and diagnostics.
Version 0.2.0, revision 2 (2026-10-10)
Section titled “Version 0.2.0, revision 2 (2026-10-10)”Adds run (start, duration, timings, workers, peak memory), source.shape
(global size of the source), the CSV quote, bom and line_ending, and
diagnostic details; measures_limited is one diagnostic per field.
Also in revision 2:
values.cardinalitynames itsdomain(original), itsaccuracyand itsidentity_accuracy. A limited count addsupper_boundand, for the reasondistinct_limit,threshold_reached_at_recordandoverflow_detected_at_record.scan.limits.max_samplesis the size of the sample the scan keeps per field: 10,000 by default (100 before), at most 100,000 and at mostmax_distinct_per_field. The effective value is inconfig.values.samplesis the sample of the original values (not the analytical ones):selectionisall(every distinct value, first-seen order) orbottom_k(thekvalues of smallest hash priority, withk,hashandthreshold), which replacesuniform_distinctandfirst_seen. Each entry gainsid,count_accuracyandidentity_accuracy. For a sensitive fieldkeptcan be belowof.values.cardinality.estimate: the KMV estimate of the distinct count of a limited field (accuracyestimated), when the sample saw the whole field.scan.random_seedis an integer from 0 to 2^64 - 1.- Values longer than
scan.limits.max_stored_value_lengthhave their own small sample,values.samples.long_texts: a bounded preview, the original lengths,truncatedand a digest of the complete text, withfingerprint_basedidentity (see Long texts). They are no longer excluded from the cardinality estimate, which adds the stored values and the long texts (estimate.parts;kandtheta_hexmove into the parts). New limits:max_long_samples(100),long_preview_chars(200) andmax_sample_memory(512 MiB). - The samples have a memory budget:
values.samplesandlong_textscan carryrequested_kandreasonmemory_budgetwhen a size was cut, and the scan adds the warningsample_budget(see Sample sizes). A CSV batch of rows and a batch of streamed values are also bounded by size. - A
bottom_ksample is completed byvalues.samples.by_length: a small bottom-k of the values of each length, so a rare length stays visible, with the lengths and values beyond the limits summed inother(see Samples by length). New limits:max_samples_per_length(100) andmax_length_strata(64);count_accuracycan now bepartial, for a length tracked after its first occurrence. The warningsample_budgetand the sizes in force cover them too. - Each dataset has a row sample,
records.sample: the first and last rows and a reproducible choice of rows of the interior, with the cells of the same record for every projected field (see Row sample). New settingsrecords.sample.rows(1,000),edge(50),cell_chars(200) andmax_bytes(16 MiB). A Report keeps the block with a subset of its rows. derived.patterns(optional): the structural patterns inferred from the samples of each text field, with their signature, rendering, coverage of the distinct values, overlaps and what was excluded (see Inferred patterns). New settingsscan.pattern_inference.enabled,max_patterns,max_value_lengthandmin_distinct.run.timingsgainsderived. A field on which the inference raises gets afailedentry and apattern_inference_failederror.observations(optional): the findings of the scan about the data, apart from its limits, with a status (informational,candidate_anomaly,confirmed_issue), summaries and the flagshas_observations,has_quality_issuesandhas_limits(see Observations and limits). New settingsscan.observations.enabledandmax_items.run.timings.derivedincludes them. A failure raises anobservations_failederror and leaves the member out.- A normalization stage that changed nothing has
equals, the stage whose count it takes, and keeps that count even when the table of distinct values was released;variant_groupsis complete and empty when no stage changed any string (see Normalization stages). records.previewis removed: the first records are the start of the row sample, with the same cells. The settingscan.records.previewis refused with the place of its replacements (scan.records.sample.edge, andreport.row_samplefor the rows a report shows).run.counters(optional): the work of the samplers, written bytabalyst scan --counters.- The scan document is written without indentation by default; the new
output.json_indentsetting restores indentation (see Writing the document). The content does not change. - The JSON Schema of the document and a validator of its invariants are part of the package (see Scan format).
values.sampleshaskeptandof: the number of values it lists and the number the scan sampled. A scan has both equal; a report keeps fewer.
Version 0.2.0, revision 3
Section titled “Version 0.2.0, revision 3”numeric.coverage(of,share): the analyzable values (values.count) the numeric statistics describe. A column of amounts written with a currency sign covers only its bare numbers.- A field that a code detector (
postal_code,phone,uuid,ip_address) reads for at least half of its eligible values, and for at least as many values as the numeric block holds, hasnumericset tonot_applicablewith the reasoncode_interpretation: the median of postal codes means nothing. - The observation rules are version 2 and gain
minority_length. - Ambiguous numbers such as
1,234are read in the convention that the other numbers of their column prove (at leastdetectors.number.min_evidence, none proving the other) and counted innumeric; with a decimal comma proven,1.250counts as 1250.details.ambiguity.resolutionofnumbernames it withsourceevidence, theevidenceand the numbersresolved. New settingsdetectors.<id>.infer_conventionandmin_evidencefornumber,currency,percentageandquantity. The Scan counts no number of an amount and does not change for the last three: the second pass reads the values with a unit by the evidence of their own detector.
Version 0.2.0, revision 4 (Tabalyst 0.7.0)
Section titled “Version 0.2.0, revision 4 (Tabalyst 0.7.0)”Revision 4 gathers the changes of the hierarchical pattern discovery and of the classification of the fields; it ships with Tabalyst 0.7.0.
-
Every field has a
classification(family,type,basis,state,traits,evidence,alternatives,rule_version): see Classification.engine.classificationrecords the version of the rules that wrote it, and a Scan classified by other rules than the current ones draws a warning that says to scan again. -
The four bounds of the Text types are the new setting
scan.classification(very_short_max_length,short_max_length,medium_max_length,long_max_length, 5 / 20 / 50 / 255 by default). They replace the keys ofreport.string_analysisof the same names, which are now refused with their new place. They are part of the scan configuration, so changing one changesconfig_sha256. -
The classification now reads the detectors: Enum, Boolean, Constant, Currency, Percentage, Quantity, Datetime, Time, Email, Phone, Postal code, URL, IP address and UUID join Integer, Number, Date and Text. Two credible types that cannot be told apart give a fallback with
stateundeterminedand both inalternatives. The traitssingle_pattern,multiple_patternsandconstantare emitted.engine.classificationis{"catalogue": 2}. Still revision 4: the classification is not published yet. -
The classification reads the name of the field with a versioned set of rules in French and English: see Names. A name reinforces a type the values support (evidence with the role
supports), breaks a tie between supported types (the result is then acandidateand the other types areoutranked_by_nameorweakened_by_nameinalternatives), or weakens one (evidence with the rolecontradicts). A type only the name suggests is aname_onlyalternative, never the result.engine.classificationbecomes{"catalogue": 3, "names": 1}andrule_versioncombines both (catalogue * 1000 + names, 3001 here). Still revision 4. -
The classification reads the identifier names:
identifier/idandidentifier/unique_id(see Identifiers). A name rule of the identifier group plus compatible values (a text or whole-number column with at least two present values, none outside its family) gives an ID; it replaces a Text or a plain Integer, and the replaced type is an alternative with the reasonreplaced_by_role. A Unique ID also has no missing position, is not under a JSON array and has an exact analytical distinct count equal to the present values; its evidence carries the measuresvalues.cardinalityandmissing.countwith their scope,partialwhen the scan excluded records. The state of both iscandidate. A UUID keeps its type with the alternativeuuid_name. New traitsobserved_uniqueandfirst_column;high_cardinalitystays unused.engine.classificationbecomes{"catalogue": 4, "names": 2}(rule_version4002). Still revision 4. -
values.samples.caseskeeps the cases of the rules a field is observed to break: the cells that end, start or repeat whitespace (kindleading,trailing,repeated) and, for a field with a minority length, the cells of each analytical length other than the dominant one. Each class has its exact number of cells and a bottom-k of original texts with exact counts, so a rare case no longer depends on the sample of distinct values (see Cases of the observed rules). New settingsscan.limits.max_cases_per_class(5, 0 turns the cases off) andmax_case_lengths(32, 0 turns off those by length); an eighth ofmax_sample_memoryis reserved for them. -
The default of
scan.exposure.sensitive_valuesbecomesshow(it wasmask): sensitive columns are not handled in this beta, so their values are listed and patterns are inferred for them like for any other column.maskandhidekeep their meaning. The structure of the document does not change; thesensitivemember of a field still says whether a sensitive detector matched, andexposurenow readsshowunless the setting asks otherwise. -
Every entry of
derived.patternshas aneligibility(state,score,reasons): whether the field is worth a search for patterns, decided before any pattern is built (see Inferred patterns). Only aneligiblefield is inferred. Thereasonof anot_applicableentry is the first reason that kept the field out;technical_typeis gone. -
A column of whole numbers of 4 digits or more, such as a register number, is now read as text and can have patterns. Shorter whole numbers, decimal numbers, free text and a handful of words are
low_relevance; booleans and objects or arrays only areexcluded. -
string_characteristicsbecomes a character profile and keeps its nine counters. It adds counters by class of character (ASCII lower and upper case letters, other Latin letters, other letters, ASCII and other digits), by start (starts_uppercase,starts_lowercase), by whitespace (with_tabs,with_non_breaking_spaces), by composition (ascii_only,letters_only,digits_only,alphanumeric_only,letters_and_digits) and for problem characters (with_invisible_characters,with_replacement_character), the totalcharacters, the exact countdistinct_charactersand the inventoryspecial_characters, both measure envelopes. New settingsscan.limits.max_special_characters(256, 0 turns the inventory off) andmax_distinct_characters(1,024). -
derived.charactersadds a character map of each field: the most frequent characters, the scripts and categories of the characters, the case of the ends of the values, title case and the emoji, counted from the samples. New settingscan.limits.max_top_characters(20, 0 turns the map off). -
The inferred patterns become a hierarchy of families (grammar 2).
derived.patternshas analgorithm.versionnext to itsgrammar, and acompleteentry haspopulation,families,otherandlimitsin place ofcandidates,other,overlapsandlimits. A family gathers the values of one structure, or of structures merged at the edge of a block (AK-1234withAK-1234X, listed incomposition), and lists itspatterns: one pattern per combination of lengths, each of fixed length with its constants folded into the signature ('INV-'999,'INV-'9999). A pattern holding several merged structures lists anodesentry for each. Every count is given in distinct values (d) and occurrences (o).candidates,generalization,merged_structures,confidence,lengthsandoverlapsare gone: there is no compatibility with the flat list of revision 3. -
The notation of the signatures changes (grammar 2):
Aa capital,aa lowercase letter,@a letter of either case,Xa capital or a digit,xa lowercase letter or a digit,#an ASCII letter or a digit,9a digit. The token kindsupper_digitandlower_digitare new, andalphanumericno longer reads non-ASCII letters. -
A structure with one value is a family of its own, and the generalization of the smallest structures into wider ones is gone. A combination of lengths that
max_patterns_per_familykeeps out of a family is counted in alengthstable of that family. -
scan.pattern_inferencehas new settings:max_values(10,000, at most 20,000),max_families(16),max_patterns_per_family(32),max_depth(3, at most 4);max_patternsnow counts the patterns and nodes of a field (128, at most 1,024). -
The nodes of a pattern are no longer only the structures of a composition. A node has a
facet(composition, orhead:N,tail:Nandtoken:Nfor the first or last characters of a token and a whole token), again(bits_d,bits_o,sigma) for a node of content, and its ownnodes(a longer prefix, or a facet that depends on it). A facet of content that no node reads is a table of the family (tables, with the samefacetnames and thegainof the facet); the cells andothersof a table add up to the family. A pattern folds at most three characters of a token as a constant, unless every value repeats the token. The limitmax_depthnames the setting of the same name when the tree was cut. -
A node has a
value: the text it fixes at its facet (the head, the tail or the whole token, or the structure of a node ofcomposition), andalsowhen a node whose only child covers 95% of it merged into it (what that node fixed). The second pass reads a node by it when it counts every occurrence. -
The
email,phoneandpostal_codedetectors (version 2) addkeysto theirdetails: the keys they declare, counted exactly on the whole file in the first pass and kept by floors (the extension and the domain of an address read from the right, the region and the area code or zone of a number, the first character and the area of a postal code). A level lists itskeys(key,occurrences, and thekeysandothersbelow) and itsothers(keys,occurrences);statusiscomplete,limited(max_trackedornot_tested),not_applicable(no value matched) ordisabled(exposure). The keys are published on a masked field too, with no example or row of the sample. New settingscan.exposure.detector_keys(aggregate, ormaskto bring the masking back).
After Tabalyst 0.7.0 (revision 4)
Section titled “After Tabalyst 0.7.0 (revision 4)”details.ambiguity.same_dayof thedatedetector counts the dates whose two readings are the same day (06-06-2020). They are matched with no format, and no longer ambiguous.