Skip to content
Beta · version 0.8.1 · commands and the JSON format may still change.

Scan format changelog

A scan document written by tabalyst scan identifies its experimental contract with three fields:

{
"format": "tabalyst.scan",
"format_version": "0.2.0",
"format_revision": 4
}

format identifies the document kind. format_version names the compatibility family, and format_revision increases for meaningful structural or semantic changes within it. The JSON profile has its own version and changelog. Beta revisions may be incompatible; automatic migration is not provided. tabalyst report --scan reads the current Scan version only: scan the source again to use an older document.

This is the starting point for the current Scan format. It describes CSV, JSON, JSONL and Excel sources, their effective configuration, datasets, fields, detectors, record facts, limits and diagnostics.

Adds run (start, duration, timings, workers, peak memory), source.shape (global size of the source), the CSV quote, bom and line_ending, and diagnostic details; measures_limited is one diagnostic per field.

Also in revision 2:

  • values.cardinality names its domain (original), its accuracy and its identity_accuracy. A limited count adds upper_bound and, for the reason distinct_limit, threshold_reached_at_record and overflow_detected_at_record.
  • scan.limits.max_samples is the size of the sample the scan keeps per field: 10,000 by default (100 before), at most 100,000 and at most max_distinct_per_field. The effective value is in config.
  • values.samples is the sample of the original values (not the analytical ones): selection is all (every distinct value, first-seen order) or bottom_k (the k values of smallest hash priority, with k, hash and threshold), which replaces uniform_distinct and first_seen. Each entry gains id, count_accuracy and identity_accuracy. For a sensitive field kept can be below of.
  • values.cardinality.estimate: the KMV estimate of the distinct count of a limited field (accuracy estimated), when the sample saw the whole field.
  • scan.random_seed is an integer from 0 to 2^64 - 1.
  • Values longer than scan.limits.max_stored_value_length have their own small sample, values.samples.long_texts: a bounded preview, the original lengths, truncated and a digest of the complete text, with fingerprint_based identity (see Long texts). They are no longer excluded from the cardinality estimate, which adds the stored values and the long texts (estimate.parts; k and theta_hex move into the parts). New limits: max_long_samples (100), long_preview_chars (200) and max_sample_memory (512 MiB).
  • The samples have a memory budget: values.samples and long_texts can carry requested_k and reason memory_budget when a size was cut, and the scan adds the warning sample_budget (see Sample sizes). A CSV batch of rows and a batch of streamed values are also bounded by size.
  • A bottom_k sample is completed by values.samples.by_length: a small bottom-k of the values of each length, so a rare length stays visible, with the lengths and values beyond the limits summed in other (see Samples by length). New limits: max_samples_per_length (100) and max_length_strata (64); count_accuracy can now be partial, for a length tracked after its first occurrence. The warning sample_budget and the sizes in force cover them too.
  • Each dataset has a row sample, records.sample: the first and last rows and a reproducible choice of rows of the interior, with the cells of the same record for every projected field (see Row sample). New settings records.sample.rows (1,000), edge (50), cell_chars (200) and max_bytes (16 MiB). A Report keeps the block with a subset of its rows.
  • derived.patterns (optional): the structural patterns inferred from the samples of each text field, with their signature, rendering, coverage of the distinct values, overlaps and what was excluded (see Inferred patterns). New settings scan.pattern_inference.enabled, max_patterns, max_value_length and min_distinct. run.timings gains derived. A field on which the inference raises gets a failed entry and a pattern_inference_failed error.
  • observations (optional): the findings of the scan about the data, apart from its limits, with a status (informational, candidate_anomaly, confirmed_issue), summaries and the flags has_observations, has_quality_issues and has_limits (see Observations and limits). New settings scan.observations.enabled and max_items. run.timings.derived includes them. A failure raises an observations_failed error and leaves the member out.
  • A normalization stage that changed nothing has equals, the stage whose count it takes, and keeps that count even when the table of distinct values was released; variant_groups is complete and empty when no stage changed any string (see Normalization stages).
  • records.preview is removed: the first records are the start of the row sample, with the same cells. The setting scan.records.preview is refused with the place of its replacements (scan.records.sample.edge, and report.row_sample for the rows a report shows).
  • run.counters (optional): the work of the samplers, written by tabalyst scan --counters.
  • The scan document is written without indentation by default; the new output.json_indent setting restores indentation (see Writing the document). The content does not change.
  • The JSON Schema of the document and a validator of its invariants are part of the package (see Scan format).
  • values.samples has kept and of: the number of values it lists and the number the scan sampled. A scan has both equal; a report keeps fewer.
  • numeric.coverage (of, share): the analyzable values (values.count) the numeric statistics describe. A column of amounts written with a currency sign covers only its bare numbers.
  • A field that a code detector (postal_code, phone, uuid, ip_address) reads for at least half of its eligible values, and for at least as many values as the numeric block holds, has numeric set to not_applicable with the reason code_interpretation: the median of postal codes means nothing.
  • The observation rules are version 2 and gain minority_length.
  • Ambiguous numbers such as 1,234 are read in the convention that the other numbers of their column prove (at least detectors.number.min_evidence, none proving the other) and counted in numeric; with a decimal comma proven, 1.250 counts as 1250. details.ambiguity.resolution of number names it with source evidence, the evidence and the numbers resolved. New settings detectors.<id>.infer_convention and min_evidence for number, currency, percentage and quantity. The Scan counts no number of an amount and does not change for the last three: the second pass reads the values with a unit by the evidence of their own detector.

Version 0.2.0, revision 4 (Tabalyst 0.7.0)

Section titled “Version 0.2.0, revision 4 (Tabalyst 0.7.0)”

Revision 4 gathers the changes of the hierarchical pattern discovery and of the classification of the fields; it ships with Tabalyst 0.7.0.

  • Every field has a classification (family, type, basis, state, traits, evidence, alternatives, rule_version): see Classification. engine.classification records the version of the rules that wrote it, and a Scan classified by other rules than the current ones draws a warning that says to scan again.

  • The four bounds of the Text types are the new setting scan.classification (very_short_max_length, short_max_length, medium_max_length, long_max_length, 5 / 20 / 50 / 255 by default). They replace the keys of report.string_analysis of the same names, which are now refused with their new place. They are part of the scan configuration, so changing one changes config_sha256.

  • The classification now reads the detectors: Enum, Boolean, Constant, Currency, Percentage, Quantity, Datetime, Time, Email, Phone, Postal code, URL, IP address and UUID join Integer, Number, Date and Text. Two credible types that cannot be told apart give a fallback with state undetermined and both in alternatives. The traits single_pattern, multiple_patterns and constant are emitted. engine.classification is {"catalogue": 2}. Still revision 4: the classification is not published yet.

  • The classification reads the name of the field with a versioned set of rules in French and English: see Names. A name reinforces a type the values support (evidence with the role supports), breaks a tie between supported types (the result is then a candidate and the other types are outranked_by_name or weakened_by_name in alternatives), or weakens one (evidence with the role contradicts). A type only the name suggests is a name_only alternative, never the result. engine.classification becomes {"catalogue": 3, "names": 1} and rule_version combines both (catalogue * 1000 + names, 3001 here). Still revision 4.

  • The classification reads the identifier names: identifier/id and identifier/unique_id (see Identifiers). A name rule of the identifier group plus compatible values (a text or whole-number column with at least two present values, none outside its family) gives an ID; it replaces a Text or a plain Integer, and the replaced type is an alternative with the reason replaced_by_role. A Unique ID also has no missing position, is not under a JSON array and has an exact analytical distinct count equal to the present values; its evidence carries the measures values.cardinality and missing.count with their scope, partial when the scan excluded records. The state of both is candidate. A UUID keeps its type with the alternative uuid_name. New traits observed_unique and first_column; high_cardinality stays unused. engine.classification becomes {"catalogue": 4, "names": 2} (rule_version 4002). Still revision 4.

  • values.samples.cases keeps the cases of the rules a field is observed to break: the cells that end, start or repeat whitespace (kind leading, trailing, repeated) and, for a field with a minority length, the cells of each analytical length other than the dominant one. Each class has its exact number of cells and a bottom-k of original texts with exact counts, so a rare case no longer depends on the sample of distinct values (see Cases of the observed rules). New settings scan.limits.max_cases_per_class (5, 0 turns the cases off) and max_case_lengths (32, 0 turns off those by length); an eighth of max_sample_memory is reserved for them.

  • The default of scan.exposure.sensitive_values becomes show (it was mask): sensitive columns are not handled in this beta, so their values are listed and patterns are inferred for them like for any other column. mask and hide keep their meaning. The structure of the document does not change; the sensitive member of a field still says whether a sensitive detector matched, and exposure now reads show unless the setting asks otherwise.

  • Every entry of derived.patterns has an eligibility (state, score, reasons): whether the field is worth a search for patterns, decided before any pattern is built (see Inferred patterns). Only an eligible field is inferred. The reason of a not_applicable entry is the first reason that kept the field out; technical_type is gone.

  • A column of whole numbers of 4 digits or more, such as a register number, is now read as text and can have patterns. Shorter whole numbers, decimal numbers, free text and a handful of words are low_relevance; booleans and objects or arrays only are excluded.

  • string_characteristics becomes a character profile and keeps its nine counters. It adds counters by class of character (ASCII lower and upper case letters, other Latin letters, other letters, ASCII and other digits), by start (starts_uppercase, starts_lowercase), by whitespace (with_tabs, with_non_breaking_spaces), by composition (ascii_only, letters_only, digits_only, alphanumeric_only, letters_and_digits) and for problem characters (with_invisible_characters, with_replacement_character), the total characters, the exact count distinct_characters and the inventory special_characters, both measure envelopes. New settings scan.limits.max_special_characters (256, 0 turns the inventory off) and max_distinct_characters (1,024).

  • derived.characters adds a character map of each field: the most frequent characters, the scripts and categories of the characters, the case of the ends of the values, title case and the emoji, counted from the samples. New setting scan.limits.max_top_characters (20, 0 turns the map off).

  • The inferred patterns become a hierarchy of families (grammar 2). derived.patterns has an algorithm.version next to its grammar, and a complete entry has population, families, other and limits in place of candidates, other, overlaps and limits. A family gathers the values of one structure, or of structures merged at the edge of a block (AK-1234 with AK-1234X, listed in composition), and lists its patterns: one pattern per combination of lengths, each of fixed length with its constants folded into the signature ('INV-'999, 'INV-'9999). A pattern holding several merged structures lists a nodes entry for each. Every count is given in distinct values (d) and occurrences (o). candidates, generalization, merged_structures, confidence, lengths and overlaps are gone: there is no compatibility with the flat list of revision 3.

  • The notation of the signatures changes (grammar 2): A a capital, a a lowercase letter, @ a letter of either case, X a capital or a digit, x a lowercase letter or a digit, # an ASCII letter or a digit, 9 a digit. The token kinds upper_digit and lower_digit are new, and alphanumeric no longer reads non-ASCII letters.

  • A structure with one value is a family of its own, and the generalization of the smallest structures into wider ones is gone. A combination of lengths that max_patterns_per_family keeps out of a family is counted in a lengths table of that family.

  • scan.pattern_inference has new settings: max_values (10,000, at most 20,000), max_families (16), max_patterns_per_family (32), max_depth (3, at most 4); max_patterns now counts the patterns and nodes of a field (128, at most 1,024).

  • The nodes of a pattern are no longer only the structures of a composition. A node has a facet (composition, or head:N, tail:N and token:N for the first or last characters of a token and a whole token), a gain (bits_d, bits_o, sigma) for a node of content, and its own nodes (a longer prefix, or a facet that depends on it). A facet of content that no node reads is a table of the family (tables, with the same facet names and the gain of the facet); the cells and others of a table add up to the family. A pattern folds at most three characters of a token as a constant, unless every value repeats the token. The limit max_depth names the setting of the same name when the tree was cut.

  • A node has a value: the text it fixes at its facet (the head, the tail or the whole token, or the structure of a node of composition), and also when a node whose only child covers 95% of it merged into it (what that node fixed). The second pass reads a node by it when it counts every occurrence.

  • The email, phone and postal_code detectors (version 2) add keys to their details: the keys they declare, counted exactly on the whole file in the first pass and kept by floors (the extension and the domain of an address read from the right, the region and the area code or zone of a number, the first character and the area of a postal code). A level lists its keys (key, occurrences, and the keys and others below) and its others (keys, occurrences); status is complete, limited (max_tracked or not_tested), not_applicable (no value matched) or disabled (exposure). The keys are published on a masked field too, with no example or row of the sample. New setting scan.exposure.detector_keys (aggregate, or mask to bring the masking back).

  • details.ambiguity.same_day of the date detector counts the dates whose two readings are the same day (06-06-2020). They are matched with no format, and no longer ambiguous.