JSON profile changelog
The JSON profile written by tabalyst report identifies its experimental
contract with two fields:
{ "format": "tabalyst.report", "format_version": "0.2.0", "format_revision": 6}format identifies the document kind: a report is a
scan document with a report block. format_version names the compatibility family. format_revision increases
for a meaningful structural or semantic change within that family. Styling,
documentation, performance improvements and corrections that preserve the
JSON contract do not change it. Beta revisions may be incompatible; automatic
migration is not provided.
Version 0.2.0, revision 1 — 2026-10-06
Section titled “Version 0.2.0, revision 1 — 2026-10-06”This is the starting point for the current JSON profile. It describes the existing Report profile, including CSV, JSON, JSONL and Excel sources, effective settings, dataset summaries, column analysis, normalization, detectors, issues, record previews, limits and JSON structure.
Version 0.2.0, revision 2 (2026-10-10)
Section titled “Version 0.2.0, revision 2 (2026-10-10)”The profile becomes a superset of the Scan document: it has format: "tabalyst.report" and holds the scan base at the same paths (engine,
status, run, source, config, config_sha256, scope, datasets,
diagnostics), plus a report block.
- The report’s own data moves under
report:generated_at,processing_seconds, the presentation settings (report.config, with onlystring_analysisandvalue_examples, thereportobject of the configuration files; the rootconfigis the scan configuration),base_scan(identity of the base) and the projections (report.datasets, with the summary, columns and structure of each dataset). The rootdatasetsare the scan datasets. sourceis the scan’s:filenamebecomesname,delimiterissource.csv.delimiter(nullcsvfor other sources).- The column copies of scan facts are removed:
scan_details,numeric,string_profileanddate_profile. They are read from the fields of the base. - The samples of the base are reduced and declare
keptandof. How many values the report keeps per column is a presentation setting,report.config.value_examples.kept_per_column(1 to 1,000, 100 by default). - A report carries the optional
derivedmember of its scan, the inferred patterns, as it is: the reduced samples cannot compute them again. - A report carries the optional
observationsmember of its scan, the observations and limits, as it is. - The report reads its findings from the base instead of storing its own:
report.datasets[].issues,previewandlimits,columns[].with_issuesandsummary.with_issues_column_countare removed. A column has quality issues when an observation about its field is a confirmed issue or a candidate anomaly; a remark such as missing values is not an issue (see Findings). - The rows the report shows are the
records.sampleof the base, reduced: the first, the interior rows of smallest priority and the last rows, as the new presentation settingreport.config.row_sample(start,middle,end, 10 each by default) asks.keptandofsay what was kept. - A report carries an optional
analysesmember, what its second pass did: who decided it, the outcome, and the second pass document with its plan and its results. The HTML report adds the Second pass section, the validated occurrences of each pattern, the dates per period, and the exact distinct count and median that a scan limit had stopped. The second pass document insideanalysesalso holds the correlation of pairs of numeric columns (partneron an analysis, the resultpearson,relationsandexplorationin the plan) and the tables of a category against a feature (the resultcrosstab,relations.tables, the domainmixed). The HTML report shows both in a Relationships section before the Second pass section. - The HTML report adds the Observations and Patterns sections, qualifies every count as exact, estimated, sample, at least or not computed, and writes integers with all their digits (see Large integers).
Version 0.2.0, revision 3
Section titled “Version 0.2.0, revision 3”The report follows the scan base at revision 3 (see the
Scan format changelog): numeric.coverage, the
refusal of numeric for codes, the minority_length observation and the
ambiguous numbers read in the convention their column proves, which the
numeric statistics of the report now include. The report adds no member of its
own. The second pass document inside analyses
is at revision 2: it may hold the numbers of
amounts, percentages and quantities with a unit (the result measures), which
the Second pass section summarizes in one line, the values that patterns and
detectors leave out (residual_values) and the repeated values of identifiers
and keys (duplicates, duplicates_limited), the line and the rank correlation of a
pair (slope, intercept, rank), and the tables of a category against the values
of another column with the rules you gave (provided_rules), and the numbers of a column
described by category (category_stats: count, extremes, mean, spread and histogram of each
category, and the share of the variance the categories explain), the shape of the numbers
(numeric_shape: asymmetry, kurtosis, outliers of Tukey, decimals) and the parts of the dates
(datetime_parts: year-month, weekday, hour, ambiguous dates under each reading), the drift of a column along the
file (position_drift: the records, values and missing values of each tenth of the file) and the columns that
miss a value together (missing_matrix), the lengths of the arrays and the sets of keys of a JSON file
(array_profile, structure_variants), the words, lines and digits of long texts and their 128-bit
fingerprints (text_profile, fingerprint_distinct) and the masks and classes of a masked column
(sensitive_forms), which holds no value of it, and the distinct count of a masked or hidden column.
Version 0.2.0, revision 4 (Tabalyst 0.7.0)
Section titled “Version 0.2.0, revision 4 (Tabalyst 0.7.0)”The four length bounds that named the String analysis classes (very_short_max_length,
short_max_length, medium_max_length, long_max_length) are no longer in
report.string_analysis: they are the scan settings scan.classification, and the
report reads the bounds recorded in the scan it presents. The scan base carries a
classification for each field (see the Scan changelog). Each column of the report carries its
classification, the one of the scan field refined by the second pass: an identifier/id whose
distinct count the scan could not complete becomes identifier/unique_id when exact_distinct
proves it, and a second pass that is absent, stale or incomplete changes nothing. It replaces
columns[].semantic_type. summary.semantic_type_counts and summary.semantic_type_percents are
replaced by summary.type_family_counts, summary.type_family_percents, summary.type_counts and
summary.type_percents, keyed by family and family/type. inferred_type and its counts stay.
The Columns table and the Column types card of the HTML report show the classification instead of
the primary interpretation. length_distribution_max_length cannot exceed scan.classification.medium_max_length.
Each column of the report carries a summary and each dataset a table: the findings (Missing,
Errors and Issues with their evaluation states), the facts of the Overview, the Details, the evidence behind the Issues
(finding_details: the spelling variants, the formats of the ambiguous dates, the minority patterns, the minority lengths, the whitespace and the mixed types; the examples of the lengths and of the whitespace come from the cases of the scan, with examples_from cases, a complete flag per length and the exact cells per kind of whitespace in whitespace.kinds), the signals and the quality counts, built once with the document and already through the
exposure. They hold codes and numbers; the HTML writes the words from them when it renders, and
computes none of them. The values of a column are in summary.values (selection,
sampled_distinct_count, examples, values and cut), through the exposure: a column
exposed as mask or hide has no raw value in its projection. columns[].examples and
columns[].value_profile are no longer written. The overview of a column holds its kind, type,
facts and notes (partial_scope, ambiguous_left_out), not the sentence.
The plan of analyses.deep lists, for each candidate, only id, analyzer, dataset, field,
display, partner, applicability and selection. The method, parameters, outputs,
utility, costs, execution, exploration and analyzer_version of a candidate stay in the
second pass document written beside the scan, which keeps the whole plan.
The report carries the new result date_order of the second pass under analyses (the records where the
first date of a pair of columns is after the second).
The report follows the scan base at revision 4 (see the
Scan format changelog): the eligibility of each
field for the inferred patterns and the character profile of its strings
(string_characteristics). A column page says why a column has no pattern with
the reason of its eligibility, and lists the characters of the profile. The
inferred patterns are the families and patterns of grammar 2 (see the Scan
changelog); the page lists the patterns of a column. The patterns carry nodes of content (the
first or last characters of a token, a whole token) with their gain, and tables of
content, as the Scan changelog says. The pattern_counts result of the second pass that the report carries
under analyses follows the hierarchy too (revision 3 of the second pass format):
families, patterns and nodes with their occurrences, distinct values and tables. The report adds no member of its own, but it carries the character map (derived.characters) and the character_profile result of the second pass, and the column page draws them. The HTML page of a column presents the hierarchy (families, patterns, indented nodes and tables, readable without JavaScript) and the table of the dataset names the family of each pattern; nothing of the JSON changes.
The report carries the keys of the email, phone and postal_code detectors (see the Scan changelog) inside the detector details; the column page does not draw them yet.
The signals and the quality counts of a dataset table gain what the overview of the HTML report reads. A signal
carries population: what its sum is out of, the values every column the check covered offered it, and null
when a column with the finding has no population (groups of spelling variants). A finding of a quality column
(table.quality) carries its cls (error or issue) and its percent (the share of its population, null without one), so that the figures of each class are counted
without guessing it from the code. The report shows up to eight signals instead of six.
Version 0.2.0, revision 5
Section titled “Version 0.2.0, revision 5”Each column of an identifier role (identifier/id or identifier/unique_id, not under a JSON array)
gets summary.identifier_uniqueness, built with the summary and null for any other column. It
separates the repetitions of the ID that exact copies of whole rows explain from those that remain
between distinct records. It holds codes and exact integers only, never a sentence or an ID value:
rule_version,status(complete,limited,not_assessed,failed),interpretation(raw_unique,unique_after_row_deduplication,unique_among_present_ids,repeated_between_distinct_records,not_assessed) andreason, a stable code that says why a conclusion is missing or refused;scope: the dataset, the records (source_records), the distinct records, the ID values and missing positions, the domain of the values, therow_identity(full_decoded_csv_row_v1for a CSV,scan_record_v1for other sources) and whether the scope iscomplete;raw,after_row_deduplication:unique,id_distinctandid_redundant, withnullfor what is not known;explanation:duplicate_row_extra_copies,id_repetitions_explainedandid_repetitions_residual, and, for a column that lacks a value whose copies were counted,duplicate_row_extra_copies_without_id(the member is absent otherwise);evidence: where the distinct count and the extra copies come from (the Scan, the second pass, whoseduplicate_rows_countcounts the copies with the engine of Dedup and appears as the sourcedeep.duplicate_rows_count, or a Dedup report, with its file name, revision and origin; the distinct IDs come from the Scan, from the second pass counting the original values (deep.exact_distinct) or from the second pass counting them afternfc,trimandcollapse_whitespace(deep.analytical_distinct)), the SHA-256 of the source, the proofs that were read and refused (rejected) and, when two exact sources disagree, theconflict(of theduplicate_rowsor of theid_cardinality). A column that lacks a value also hasmissing_id_copies(sourcedeep.missing_id_copies,accuracy,identity), absent otherwise.
With missing IDs (reason missing_ids) the object stays limited and no unique is ever true:
a record without an ID stays among the distinct rows. When the second pass counted the copies of the
rows without an ID (missing_id_copies) and the total of the copies is exact, the figures are
given: id_repetitions_explained is E - E_missing (the copies of rows that have an ID),
id_repetitions_residual is P - (E - E_missing) - D, raw holds D and P - D, and the
interpretation is repeated_between_distinct_records (then after_row_deduplication.unique is
false) or unique_among_present_ids; with fewer than two distinct rows that have an ID it stays
not_assessed. Without that count nothing is subtracted and every figure stays null.
near_unique,nullunless the interpretation isrepeated_between_distinct_recordsand no ID is missing: a non-binding reading of the residual repetitions, never a guarantee and never a role. It holdsmodel_version,signal(near_unique,not_near_unique, orpopulation_too_smallbelow 100 distinct rows),residual_repetitions(R_residual),distinct_records(U),toleranceandlimiting_bound(relative,rootorcap, the term of the model that sets the tolerance). The tolerance ismin(U // 100, isqrt(U), 1000): repetitions up to it arenear_unique. The constants are provisional and a change of one raisesmodel_version.
identifier.repetition is unchanged: it keeps counting the raw repetitions. The extra copies come from
the Scan’s count of duplicate records, or from a valid Dedup report given with
--dedup-report (the setting report.dedup_report) or found beside a CSV source as
<name>.dedup.json, or from the second pass, which counts them by itself for a small CSV table with an identifier column (or when it
was forced with --deep) and the Scan could only bound them. A valid Dedup report spares that count.
The report never runs Dedup and reads the source again only through that second pass.
Version 0.2.0, revision 6
Section titled “Version 0.2.0, revision 6”The report carries the new result missing_patterns of the second pass under analyses
(revision 5 of the second pass format): for a dataset, under the field
$, the records by the exact set of columns they lack a value in, with the records that have
every value, the combinations listed and the records left out. The second pass settings in
analyses.deep.settings.missing gain max_patterns and max_patterns_listed.