Skip to content
Beta · version 0.8.1 · commands and the JSON format may still change.

JSON profile changelog

The JSON profile written by tabalyst report identifies its experimental contract with two fields:

{
"format": "tabalyst.report",
"format_version": "0.2.0",
"format_revision": 6
}

format identifies the document kind: a report is a scan document with a report block. format_version names the compatibility family. format_revision increases for a meaningful structural or semantic change within that family. Styling, documentation, performance improvements and corrections that preserve the JSON contract do not change it. Beta revisions may be incompatible; automatic migration is not provided.

This is the starting point for the current JSON profile. It describes the existing Report profile, including CSV, JSON, JSONL and Excel sources, effective settings, dataset summaries, column analysis, normalization, detectors, issues, record previews, limits and JSON structure.

The profile becomes a superset of the Scan document: it has format: "tabalyst.report" and holds the scan base at the same paths (engine, status, run, source, config, config_sha256, scope, datasets, diagnostics), plus a report block.

  • The report’s own data moves under report: generated_at, processing_seconds, the presentation settings (report.config, with only string_analysis and value_examples, the report object of the configuration files; the root config is the scan configuration), base_scan (identity of the base) and the projections (report.datasets, with the summary, columns and structure of each dataset). The root datasets are the scan datasets.
  • source is the scan’s: filename becomes name, delimiter is source.csv.delimiter (null csv for other sources).
  • The column copies of scan facts are removed: scan_details, numeric, string_profile and date_profile. They are read from the fields of the base.
  • The samples of the base are reduced and declare kept and of. How many values the report keeps per column is a presentation setting, report.config.value_examples.kept_per_column (1 to 1,000, 100 by default).
  • A report carries the optional derived member of its scan, the inferred patterns, as it is: the reduced samples cannot compute them again.
  • A report carries the optional observations member of its scan, the observations and limits, as it is.
  • The report reads its findings from the base instead of storing its own: report.datasets[].issues, preview and limits, columns[].with_issues and summary.with_issues_column_count are removed. A column has quality issues when an observation about its field is a confirmed issue or a candidate anomaly; a remark such as missing values is not an issue (see Findings).
  • The rows the report shows are the records.sample of the base, reduced: the first, the interior rows of smallest priority and the last rows, as the new presentation setting report.config.row_sample (start, middle, end, 10 each by default) asks. kept and of say what was kept.
  • A report carries an optional analyses member, what its second pass did: who decided it, the outcome, and the second pass document with its plan and its results. The HTML report adds the Second pass section, the validated occurrences of each pattern, the dates per period, and the exact distinct count and median that a scan limit had stopped. The second pass document inside analyses also holds the correlation of pairs of numeric columns (partner on an analysis, the result pearson, relations and exploration in the plan) and the tables of a category against a feature (the result crosstab, relations.tables, the domain mixed). The HTML report shows both in a Relationships section before the Second pass section.
  • The HTML report adds the Observations and Patterns sections, qualifies every count as exact, estimated, sample, at least or not computed, and writes integers with all their digits (see Large integers).

The report follows the scan base at revision 3 (see the Scan format changelog): numeric.coverage, the refusal of numeric for codes, the minority_length observation and the ambiguous numbers read in the convention their column proves, which the numeric statistics of the report now include. The report adds no member of its own. The second pass document inside analyses is at revision 2: it may hold the numbers of amounts, percentages and quantities with a unit (the result measures), which the Second pass section summarizes in one line, the values that patterns and detectors leave out (residual_values) and the repeated values of identifiers and keys (duplicates, duplicates_limited), the line and the rank correlation of a pair (slope, intercept, rank), and the tables of a category against the values of another column with the rules you gave (provided_rules), and the numbers of a column described by category (category_stats: count, extremes, mean, spread and histogram of each category, and the share of the variance the categories explain), the shape of the numbers (numeric_shape: asymmetry, kurtosis, outliers of Tukey, decimals) and the parts of the dates (datetime_parts: year-month, weekday, hour, ambiguous dates under each reading), the drift of a column along the file (position_drift: the records, values and missing values of each tenth of the file) and the columns that miss a value together (missing_matrix), the lengths of the arrays and the sets of keys of a JSON file (array_profile, structure_variants), the words, lines and digits of long texts and their 128-bit fingerprints (text_profile, fingerprint_distinct) and the masks and classes of a masked column (sensitive_forms), which holds no value of it, and the distinct count of a masked or hidden column.

Version 0.2.0, revision 4 (Tabalyst 0.7.0)

Section titled “Version 0.2.0, revision 4 (Tabalyst 0.7.0)”

The four length bounds that named the String analysis classes (very_short_max_length, short_max_length, medium_max_length, long_max_length) are no longer in report.string_analysis: they are the scan settings scan.classification, and the report reads the bounds recorded in the scan it presents. The scan base carries a classification for each field (see the Scan changelog). Each column of the report carries its classification, the one of the scan field refined by the second pass: an identifier/id whose distinct count the scan could not complete becomes identifier/unique_id when exact_distinct proves it, and a second pass that is absent, stale or incomplete changes nothing. It replaces columns[].semantic_type. summary.semantic_type_counts and summary.semantic_type_percents are replaced by summary.type_family_counts, summary.type_family_percents, summary.type_counts and summary.type_percents, keyed by family and family/type. inferred_type and its counts stay. The Columns table and the Column types card of the HTML report show the classification instead of the primary interpretation. length_distribution_max_length cannot exceed scan.classification.medium_max_length.

Each column of the report carries a summary and each dataset a table: the findings (Missing, Errors and Issues with their evaluation states), the facts of the Overview, the Details, the evidence behind the Issues (finding_details: the spelling variants, the formats of the ambiguous dates, the minority patterns, the minority lengths, the whitespace and the mixed types; the examples of the lengths and of the whitespace come from the cases of the scan, with examples_from cases, a complete flag per length and the exact cells per kind of whitespace in whitespace.kinds), the signals and the quality counts, built once with the document and already through the exposure. They hold codes and numbers; the HTML writes the words from them when it renders, and computes none of them. The values of a column are in summary.values (selection, sampled_distinct_count, examples, values and cut), through the exposure: a column exposed as mask or hide has no raw value in its projection. columns[].examples and columns[].value_profile are no longer written. The overview of a column holds its kind, type, facts and notes (partial_scope, ambiguous_left_out), not the sentence.

The plan of analyses.deep lists, for each candidate, only id, analyzer, dataset, field, display, partner, applicability and selection. The method, parameters, outputs, utility, costs, execution, exploration and analyzer_version of a candidate stay in the second pass document written beside the scan, which keeps the whole plan.

The report carries the new result date_order of the second pass under analyses (the records where the first date of a pair of columns is after the second). The report follows the scan base at revision 4 (see the Scan format changelog): the eligibility of each field for the inferred patterns and the character profile of its strings (string_characteristics). A column page says why a column has no pattern with the reason of its eligibility, and lists the characters of the profile. The inferred patterns are the families and patterns of grammar 2 (see the Scan changelog); the page lists the patterns of a column. The patterns carry nodes of content (the first or last characters of a token, a whole token) with their gain, and tables of content, as the Scan changelog says. The pattern_counts result of the second pass that the report carries under analyses follows the hierarchy too (revision 3 of the second pass format): families, patterns and nodes with their occurrences, distinct values and tables. The report adds no member of its own, but it carries the character map (derived.characters) and the character_profile result of the second pass, and the column page draws them. The HTML page of a column presents the hierarchy (families, patterns, indented nodes and tables, readable without JavaScript) and the table of the dataset names the family of each pattern; nothing of the JSON changes. The report carries the keys of the email, phone and postal_code detectors (see the Scan changelog) inside the detector details; the column page does not draw them yet.

The signals and the quality counts of a dataset table gain what the overview of the HTML report reads. A signal carries population: what its sum is out of, the values every column the check covered offered it, and null when a column with the finding has no population (groups of spelling variants). A finding of a quality column (table.quality) carries its cls (error or issue) and its percent (the share of its population, null without one), so that the figures of each class are counted without guessing it from the code. The report shows up to eight signals instead of six.

Each column of an identifier role (identifier/id or identifier/unique_id, not under a JSON array) gets summary.identifier_uniqueness, built with the summary and null for any other column. It separates the repetitions of the ID that exact copies of whole rows explain from those that remain between distinct records. It holds codes and exact integers only, never a sentence or an ID value:

  • rule_version, status (complete, limited, not_assessed, failed), interpretation (raw_unique, unique_after_row_deduplication, unique_among_present_ids, repeated_between_distinct_records, not_assessed) and reason, a stable code that says why a conclusion is missing or refused;
  • scope: the dataset, the records (source_records), the distinct records, the ID values and missing positions, the domain of the values, the row_identity (full_decoded_csv_row_v1 for a CSV, scan_record_v1 for other sources) and whether the scope is complete;
  • raw, after_row_deduplication: unique, id_distinct and id_redundant, with null for what is not known;
  • explanation: duplicate_row_extra_copies, id_repetitions_explained and id_repetitions_residual, and, for a column that lacks a value whose copies were counted, duplicate_row_extra_copies_without_id (the member is absent otherwise);
  • evidence: where the distinct count and the extra copies come from (the Scan, the second pass, whose duplicate_rows_count counts the copies with the engine of Dedup and appears as the source deep.duplicate_rows_count, or a Dedup report, with its file name, revision and origin; the distinct IDs come from the Scan, from the second pass counting the original values (deep.exact_distinct) or from the second pass counting them after nfc, trim and collapse_whitespace (deep.analytical_distinct)), the SHA-256 of the source, the proofs that were read and refused (rejected) and, when two exact sources disagree, the conflict (of the duplicate_rows or of the id_cardinality). A column that lacks a value also has missing_id_copies (source deep.missing_id_copies, accuracy, identity), absent otherwise.

With missing IDs (reason missing_ids) the object stays limited and no unique is ever true: a record without an ID stays among the distinct rows. When the second pass counted the copies of the rows without an ID (missing_id_copies) and the total of the copies is exact, the figures are given: id_repetitions_explained is E - E_missing (the copies of rows that have an ID), id_repetitions_residual is P - (E - E_missing) - D, raw holds D and P - D, and the interpretation is repeated_between_distinct_records (then after_row_deduplication.unique is false) or unique_among_present_ids; with fewer than two distinct rows that have an ID it stays not_assessed. Without that count nothing is subtracted and every figure stays null.

  • near_unique, null unless the interpretation is repeated_between_distinct_records and no ID is missing: a non-binding reading of the residual repetitions, never a guarantee and never a role. It holds model_version, signal (near_unique, not_near_unique, or population_too_small below 100 distinct rows), residual_repetitions (R_residual), distinct_records (U), tolerance and limiting_bound (relative, root or cap, the term of the model that sets the tolerance). The tolerance is min(U // 100, isqrt(U), 1000): repetitions up to it are near_unique. The constants are provisional and a change of one raises model_version.

identifier.repetition is unchanged: it keeps counting the raw repetitions. The extra copies come from the Scan’s count of duplicate records, or from a valid Dedup report given with --dedup-report (the setting report.dedup_report) or found beside a CSV source as <name>.dedup.json, or from the second pass, which counts them by itself for a small CSV table with an identifier column (or when it was forced with --deep) and the Scan could only bound them. A valid Dedup report spares that count. The report never runs Dedup and reads the source again only through that second pass.

The report carries the new result missing_patterns of the second pass under analyses (revision 5 of the second pass format): for a dataset, under the field $, the records by the exact set of columns they lack a value in, with the records that have every value, the combinations listed and the records left out. The second pass settings in analyses.deep.settings.missing gain max_patterns and max_patterns_listed.