Configuration

Tabalyst reads strict JSON configuration files. JSON does not allow comments; this page is the descriptive reference for supported settings. Commands load the files passed with --config, and the Python API those passed with config_path; no file is loaded automatically. tabalyst.json is only a conventional name.

One file can configure every command, with one object per tool. The scan object holds the analysis settings, read by tabalyst scan and tabalyst report, described in Scan settings. The report object holds the presentation settings of the report, described in Report settings. The top-level output object sets how the documents are written, described in Output settings. The top-level deep object sets the histograms of the second pass, described in Deep settings. The top-level dedup object sets tabalyst dedup, described in Dedup settings. The top-level csv object holds the CSV settings of tabalyst sample and tabalyst dedup:

{
  "report": {"string_analysis": {"examples_per_length": 5}},
  "csv": {"delimiter": ";"},
  "scan": {
    "csv": {"delimiter": ";"},
    "values": {"null_markers": ["N/A"]},
    "records": {"duplicates": true}
  }
}

Every command validates the whole file, including the sections it does not use, so a misspelled or unknown setting is always an error. Explicit command-line values, such as --delimiter and --encoding, are the final override.

Report settings

The report is built on Tabalyst Scan: it reuses a current stored scan when possible, or reads the CSV with the effective scan settings, then presents the result with the settings of the report object below. They only shape the presentation: changing them never scans the source again.

{
  "report": {
    "string_analysis": {
      "length_distribution_max_length": 50,
      "examples_per_length": 10
    },
    "value_examples": {
      "kept_per_column": 100,
      "full_distribution_max_distinct": 50,
      "short_text_max_length": 20,
      "short_text_percentile": 0.95,
      "short_text_result_size": 20,
      "long_text_result_size": 20,
      "long_text_truncate_at": 30,
      "truncation_suffix": "...",
      "inline_display_size": 3
    },
    "row_sample": {"start": 10, "middle": 10, "end": 10},
    "dedup_report": null
  }
}

report.dedup_report is the file or folder of the Dedup report that proves the copies of rows behind the uniqueness of identifier columns, like the option --dedup-report of tabalyst report, which replaces it. CSV sources only; the path is not stored in the report.

Row sample

The report shows rows of the row sample of the scan: the first report.row_sample.start records, the last end records and middle records drawn from the interior, 10 of each by default, at most 1,000 of each and never more than the scan kept (see Records). The sampled rows are the ones with the smallest priorities, a subset of the scan’s, and are not uniform over the whole file. The setting never limits the analysis of the complete file, and changing it never scans the source again. Values of sensitive columns are masked or hidden as in the rest of the report (see How the report uses the scan settings).

String analysis

Present values in text columns are classified from their maximum length as very_short, short, medium, long or very_long, with the bounds of scan.classification (5, 20, 50 and 255 characters by default). The Text type of a column is decided on the same bounds, but on the median length: a column can be Short as a type and medium in this section when a few values are long. A fixed length is recorded separately when minimum and maximum match. Missing values do not contribute to lengths. Mean and median length are calculated for every text column; the report leaves them blank when the fixed length already conveys the same information.

length_distribution_max_length cannot exceed scan.classification.medium_max_length; the two settings are in different objects, so the report stops with an error when they disagree. When the column maximum is at most length_distribution_max_length, JSON also contains the exact occurrence count of every observed length. Each length lists up to examples_per_length example values, taken from the most frequent values of the column, then from its sampled values. distinct_length_count remains available for every text column, including columns whose distribution is omitted. The report orders length groups by occurrence and shows three examples per length in its String analysis tooltip.

Examples and value profiles

• kept_per_column: how many values of the scan sample the report keeps and draws its examples from, from 1 to 1,000 (default 100). The report keeps the whole scan sample when it has no more values, otherwise this many of its values: the ones of smallest hash priority, which the scan sample would also keep at that size. The report declares how many it kept and how many the scan listed in values.samples (kept and of). A larger scan.limits.max_samples therefore does not change the examples of the report. Choosing diverse examples compares the values with each other, so the time grows faster than the number: on the 34 columns of the insurance example, a report from a scan takes about 1 second at 100 values, 3 seconds at 300 and 30 seconds at 1,000, the maximum.
• full_distribution_max_distinct: through this limit, every present distinct value and its occurrence count are retained in JSON. At 50, the distribution is still complete; at 51, Tabalyst selects values among a reproducible sample of at most kept_per_column distinct values.
• short_text_max_length and short_text_percentile: a column is considered short text when this percentile of its sampled lengths does not exceed the configured length.
• short_text_result_size: number of values retained for the tooltip when a short-text column has more than 50 distinct values. Its default is 20.
• long_text_result_size: equivalent limit for long text. Its default is also 20.
• long_text_truncate_at and truncation_suffix: display limit for retained long-text values in both JSON and the report.
• inline_display_size: number of values shown directly in the Examples cell. They are always the most frequent retained values.

The Examples cell shows +N for a complete distribution. For a sampled profile, it shows ++; the tooltip then shows, for example, 20 / 2,992, meaning retained values / actual distinct values. Tooltip values are sorted by descending occurrence count. For a sensitive column, the cell shows masked or hidden instead, and the tooltip lists masked values.

How the report uses the scan settings

Report behavior	Scan settings
Reading the CSV	scan.csv, --delimiter, --encoding
Missing cells	scan.values: null_markers, null_markers_case_sensitive and missing
Normalization counts, distinct values and variant groups	scan.normalization: nfc, trim, collapse_whitespace, casefold, strip_accents
Inferred types	scan.types.minimum_confidence, with the number and date detectors
Date analysis	scan.detectors.date
Column types	scan.detection.minimum_share, every detector and scan.classification
Masked values	scan.exposure.sensitive_values
Row sample (first, middle and last rows)	scan.records.sample for the rows the scan keeps, report.row_sample for the rows the report shows
Duplicate rows	scan.records.duplicates and scan.limits.max_tracked_records
Limits and diagnostics	scan.limits and scan.errors

• Dates. A column whose present values are dates is date, even with several formats or ambiguous values. Ambiguous values such as 02/03/2025 are never resolved from other values of the column: the report counts them as ambiguous, shows how many unambiguous values use each order and, when only one order appears, suggests setting scan.detectors.date.ambiguous_order. The ambiguous_dates issue counts them.
• Column types. The classification of the scan field: a detector matching at least scan.detection.minimum_share of the present values (such as email, phone or postal_code), the technical type, the length of a text, and the name of the column for an identifier. The second pass can refine an identifier.
• Sensitive values. Sensitive columns are not handled in this beta: by default every value is shown, including in examples, value profiles and the data sample. Set scan.exposure.sensitive_values to mask to replace the values of columns where a sensitive detector matches, such as email addresses and phone numbers (jane@example.com becomes aaaa@aaaaaaa.aaa), or to hide to remove them.
• Limits. When a scan limit stops a measure, such as the distinct values of a column with more than scan.limits.max_distinct_per_field of them, the report shows limited instead of a number and lists the column in the limited_measures issue.
• Duplicate rows. With more distinct rows than scan.limits.max_tracked_records, the duplicate count is a lower bound, shown as ≥ 12. With scan.records.duplicates set to false, duplicates are not checked and the report shows –.

The report reads the scan.csv settings, not the top-level csv settings. When a file sets both with different values, the report stops with an error, since the file would otherwise be read with settings it does not expect; a value given with --delimiter or --encoding always wins. The report also needs every column: a CSV wider than scan.limits.max_fields is an error.

A report built from a scan document with tabalyst report --scan uses the settings recorded in the document. Its configuration files still give the presentation settings; settings given in their scan object must have the values recorded in the document, otherwise the report stops with an error (see Report from a scan).

Settings that moved

These top-level settings are rejected with a message naming their new location. The analysis settings moved to scan, the presentation settings to report:

Former setting	New location
missing_values	scan.values.null_markers for markers such as "N/A"; empty and whitespace-only cells are missing by default (scan.values.missing)
normalization.trim	scan.normalization.trim
normalization.collapse_internal_whitespace	scan.normalization.collapse_whitespace
date_detection	scan.detectors.date
type_inference.minimum_confidence	scan.types.minimum_confidence
enum_detection	scan.detectors.enumeration: maximum_distinct_values is maximum_distinct, and minimum_row_count is minimum_values, which counts present values, not rows
value_examples.candidate_sample_size	scan.limits.max_samples
value_examples.random_seed	scan.random_seed
preview_rows	scan.records.sample.edge and report.row_sample
string_analysis	report.string_analysis
report.string_analysis.very_short_max_length, short_max_length, medium_max_length and long_max_length	scan.classification.<same key>: the Text type they decide is written in the scan
value_examples (other than the two below)	report.value_examples

Output settings

{
  "output": {
    "json_indent": false
  }
}

• output.json_indent: false by default. The scan document and the report JSON are then written without indentation, on one line, which is about 60% smaller. With true they are indented by two spaces and each sample entry stays on one line (about half the size of a fully indented file), which is easier to read when debugging. Only the layout changes: both forms hold the same document. The execution history (executions.json) and the Inspect file are always indented, being small and meant to be read.

Deep settings

The top-level deep object configures the second pass of tabalyst scan --deep and of tabalyst report, in Add histograms and exact medians. It is not part of the scan object: changing it never makes a stored scan stale.

{"deep": {"bins": 50, "bin_strategy": "sample_quantile"}}

Setting	Default	Meaning
deep.bins	100	Number of bins of the histogram of a numeric field that is not counted per integer, and of the numbers of amounts, percentages and quantities (at most, for those in bins whose width is a power of two). From 2 to 1,000
deep.mode	auto	What tabalyst report does about the second pass. auto: the planner adds the analyses that are useful and cheap. off: none, for a script that must read each file once. force: the same as --deep, with its confirmation. tabalyst scan --deep is an explicit request and runs whatever the mode
deep.pearson.max_columns	50	Numeric columns of a dataset explored for pairs, from 2 to 200. The least present columns beyond it are left out, and the plan counts them
deep.pearson.max_work	1000000	Pairs times rows compared by the exploration of the row sample. It lowers the number of columns explored when max_columns is not enough
deep.pearson.max_candidates	20	Pairs the plan lists at most, the best first. A forced second pass takes them all
deep.pearson.automatic_pairs	8	Pairs tabalyst report picks by itself at most; 0 picks none. A pair is picked when the lower end of its 95% interval is 0.45 or more
deep.pearson.explore_min_pairs	30	Rows both columns need in the sample for the exploration to score a pair
deep.pearson.min_pairs	10	Rows both columns need in the file for a coefficient to be defined. Changing it computes the pairs again
deep.pearson.rank_bins	64	Bins per column of the grid the rank (Spearman) correlation is read from; 0 turns the rank correlation off, in the result and in the rating of the pairs. From 4 to 1,024. Changing it computes the pairs again
deep.crosstab.max_categories	30	Categories a table keeps as rows, the most frequent; the others are other. From 2 to 200. Changing it computes the tables again
deep.crosstab.max_classes	12	Classes of a length, a prefix or a value, other included, from 3 to 50. Changing it computes those tables again
deep.crosstab.max_distinct	100	Most distinct values a column may have to be a category, from 2 to 1,000 (the scan lists the frequencies of 100 values by default, scan.limits.max_listed_frequencies)
deep.crosstab.prefix_length	2	Characters of a prefix, from 1 to 8. Changing it computes the prefix tables again
deep.crosstab.examples	3	Rows kept as examples for each cell of a table, from 0 to 10
deep.crosstab.max_exception_cells	20	Cells of exceptions a table lists at most, the largest first, from 1 to 1,000; the result says whether that was all of them. Changing it computes the tables again
deep.crosstab.rule_purity, deep.crosstab.rule_support	90, 30	A category gives a candidate rule when at least rule_support counted rows are in its dominant class and rule_purity percent of them
deep.crosstab.max_columns, deep.crosstab.max_work	50, 1000000	Columns explored for tables, and tables times rows coded by the exploration of the row sample
deep.crosstab.max_candidates, deep.crosstab.automatic_tables	20, 2	Tables the plan lists at most, and tabalyst report picks by itself at most (0 picks none). A table is picked when its corrected Cramér’s V is 0.45 or more
deep.crosstab.explore_min_rows	30	Rows a table needs in the sample for the exploration to score it
deep.stats.max_categories	30	Categories the numbers are described for, the most frequent; the others are other. From 2 to 200. Changing it computes those analyses again
deep.stats.bins	50	Bins every category is counted in, placed as deep.bin_strategy says, from 2 to 1,000. Changing it computes those analyses again
deep.stats.max_distinct	100	Most distinct values a column may have to be a category, from 2 to 1,000
deep.stats.max_columns, deep.stats.max_work	50, 1000000	Columns explored for numbers by category, and pairs times rows compared by the exploration of the row sample
deep.stats.max_candidates, deep.stats.automatic_stats	20, 2	Pairs the plan lists at most, and tabalyst report picks by itself at most (0 picks none). A pair is picked when the square root of its corrected correlation ratio is 0.45 or more
deep.stats.explore_min_rows	30	Rows a pair needs in the sample for the exploration to score it
deep.shape.max_values	100000	Different numbers of a column counted exactly by numeric_shape. Beyond, the numbers go on in bins: counts, extremes and decimals stay exact, the quartiles and the outliers of Tukey are bracketed. From 1 to 10,000,000. Changing it computes the shape again
deep.drift.max_tracked	1000	Most different values a column may have for position_drift to follow the values themselves along the file; the scan must list all of them with their counts, and it lists 100 by default (scan.limits.max_listed_frequencies). Beyond, only the missing values are followed. From 2 to 10,000. Changing it computes the drift again
deep.drift.max_listed	12	Values listed with their counts in each tenth of the file, the most frequent first; the others are summed up. From 1 to 100. Changing it computes the drift of the columns whose values are followed again
deep.missing.max_columns	100	Columns of a dataset that lack a value that missing_matrix accepts; a dataset with more is left out, because the pairs grow with the square of the columns. From 2 to 1,000. Changing it computes the matrix again
deep.missing.max_listed	10	Partners listed for a column, the ones that lack a value most often with it. From 1 to 1,000. Changing it computes the matrix again
deep.date_order.min_share	0.99	Share of the sampled rows with both dates where the first date must be on or before the second for a pair of date columns to be counted by date_order. From 0.5 to 1. Changing it changes the pairs the plan lists, not a result
deep.date_order.max_listed	10	Broken records listed in a date_order result, in file order. From 1 to 1,000. Changing it computes the order again
deep.date_order.max_candidates	20	Pairs of date columns at most that the plan lists. From 0 to 1,000
deep.date_order.automatic_pairs	20	Pairs at most that tabalyst report takes by itself. From 0 to 1,000
deep.date_order.explore_min_pairs	30	Sampled rows with both dates a pair needs to be scored
deep.date_order.max_columns, deep.date_order.max_work	50, 1000000	Date columns of a dataset and pairs times rows the exploration of the sample accepts; the least present columns are left out and counted
deep.arrays.max_lengths	10000	Different lengths array_profile counts one by one; a JSON field whose arrays spread over a wider range (longest minus shortest) is left out. From 2 to 100,000. Changing it computes the lengths again
deep.structure.max_variants	1000	Sets of keys structure_variants tracks, each counted exactly from its first object; objects with another set are summed up and the result says it is not complete. From 2 to 10,000. Changing it computes the sets again
deep.structure.max_listed, deep.structure.max_keys	10, 30	Sets listed, the most frequent first (from 1 to 100), and keys named in a list (from 1 to 200). Changing them computes the sets again
deep.texts.min_length	80	Characters a value must reach for its field to be a field of texts, for text_profile and fingerprint_distinct; a field whose values the scan sampled as long texts is one too. From 1
deep.texts.max_listed	10	Repeated texts fingerprint_distinct lists, by digest and count, the most repeated first. From 1 to 100. Changing it computes the fingerprints again
deep.sensitive.enabled	false	Lets the second pass count the columns whose exposure is mask or hide: the exact distinct count of both, and for mask the masks of the values and their classes (sensitive_forms), never a value. Off, such a column is left out of every analysis. Changing it changes the plan
deep.sensitive.min_count	5	The floor of sensitive_forms: a mask or a class that fewer values take is summed up with the others, and no figure is given about it. From 2 to 1,000. Changing it computes the masks again
deep.sensitive.max_listed, deep.sensitive.max_tracked	20, 10000	Masks listed (from 1 to 1,000) and tracked, each counted exactly from its first occurrence; values with another mask are summed up and the result says it is not complete (from 10 to 100,000). Changing them computes the masks again
deep.datetime.infer_order	true	Read the ambiguous dates (03/04/2020) in the order that the unambiguous dates of the column prove when they all give the same one (day-month or month-day), and count them with the others. false, or both orders in the column, keeps them ambiguous. An order of detectors.date.ambiguous_order always wins. Changing it computes the distribution over time and the parts again
deep.datetime.min_evidence	3	The fewest unambiguous dates that must prove the order (day-month or month-day) for deep.datetime.infer_order to read the ambiguous dates that way; below it they stay ambiguous. From 1. Changing it computes the distribution over time and the parts again
deep.datetime.as_of	none	A calendar date (YYYY-MM-DD): datetime_parts counts the dates after it. Never the date of the day, so that a result does not change with the clock. Changing it computes the parts again
deep.measures.max_groups	20	Currencies or units of a field of amounts, percentages or quantities that get metrics of their own, the most frequent first; the others are counted together in other. From 1 to 1,000. Changing it computes the extraction again
deep.measures.max_values	100000	Different pairs of a currency or unit and a number counted exactly in a field. Beyond, every group goes on in bins: counts, sums and extremes stay exact, the median and quantiles are estimated. From 1 to 10,000,000. Changing it computes the extraction again
deep.patterns.table_bytes	65536	Memory of the table that counts the values of a facet of a pattern family, per facet and field (128 bytes a cell). When it is full the most frequent values are kept and the result says it is limited. From 1,024 bytes to 64 MiB. Changing it computes the analysis again
deep.patterns.distinct_bytes	1048576	Memory of the set that counts the distinct values of the patterns of a field read from the file. Beyond it their distinct values are a lower bound (at_least). From 1,024 bytes to 512 MiB. Changing it computes the analysis again
deep.residuals.max_listed	20	Values listed per source and state in the values that patterns and detectors leave out, the most frequent first. From 1 to 1,000. Changing it computes the analysis again
deep.residuals.max_tracked	10000	Distinct values counted per source and state; the occurrences of the values met after that are only counted. From 1 to 1,000,000
deep.residuals.max_value_length	100	A value longer than this many characters is counted and never listed. From 1 to 10,000
deep.residuals.max_detectors	3	Detectors replayed per field, the one that read the most values first. From 1 to 20
deep.residuals.records	false	Give the first record of each value even when the scan holds every distinct value, which makes the second pass read the source
deep.duplicates.max_listed, deep.duplicates.max_tracked	20, 10000	Repeated values listed, the most repeated first (1 to 1,000), and repeated values whose count is tracked (1 to 1,000,000); the occurrences of the others are only counted
deep.duplicates.records	false	Give the first record of each repeated value even when the scan holds every distinct value, which makes the second pass read the source
deep.copies.memory_mb	256	Accounted size, in MiB, of the table of distinct rows of the count of the extra copies of rows (duplicate_rows_count, and missing_id_copies for the rows that lack an ID) before it moves to a temporary database. tabalyst report counts by itself only a table that is expected to fit the memory of an automatic choice (about 128 MiB). From 1
deep.copies.disk_mb	2048	Estimated size, in MiB, of that temporary database; the count stops, limited, when it passes. 0 forbids the disk, so the count stops at memory_mb. From 0
deep.copies.max_rows	10000000	Records the count reads; a longer file stops it, limited. From 1
deep.bin_strategy	linear	linear: bins of equal width between the exact minimum and maximum. sample_quantile: inner edges at the quantiles of the sample kept by the scan, for a very skewed distribution; it falls back to linear when the sample is too small, and says so

The counts per bin are exact with either strategy.

Smart settings

The top-level smart object configures the plan of a smart sample, in Plan the analyses a scan could add, and the sample that tabalyst sample --smart builds from it. It is not part of the scan object or of deep: changing it never makes a stored scan or a second-pass result stale.

{"smart": {"reserve_share": 0.3, "weights": {"cap": 12}}}

Setting	Default	Meaning
smart.max_enum_values	100	Most distinct values a column may have to get one target per value. From 1 to 10,000
smart.max_exact_values	500	Most distinct values a column may have to get one target per value when the second pass counted every value exactly (above smart.max_enum_values, which limits the values the first pass lists). From 1 to 10,000
smart.near_share	0.08	Share of the rows asked for that the groups of near values may take at most (the typos of a category, see Smart sample), from 0 to 1. The groups are shared between the columns in turn, and a sample too small for its mandatory targets gets none. 0 turns them off
smart.near_groups_per_column	10	Groups of near values at most in one column, from 0 to 1,000. 0 turns them off
smart.near_groups	100	Groups of near values at most in the dataset, from 0 to 100,000. 0 turns them off. The plan counts the groups the three limits leave out among the targets it left out
smart.numeric_bins	10	Bins of equal width between the minimum and the maximum of a numeric column, from 2 to 100. A column with fewer than three possible bins gets none
smart.candidates_per_target	2	Rows the selection holds in memory for each target, from 1 to 3
smart.reserve_share	0.2	Share of the rows asked for that is kept for ordinary rows, from 0 to 0.9. It gives way to mandatory targets
smart.worst_rows	3	Rows with the most weight of anomalies at once that the sample keeps whatever their targets, from 0 (off) to 100. A sample gives them at most a twentieth of its size
smart.max_targets	10000	Targets the plan keeps at most, the heaviest first; the plan counts the others
smart.max_candidate_bytes	67108864	Memory the candidate rows may take, in bytes of source text
smart.weights.gain, smart.weights.floor, smart.weights.cap, smart.weights.unknown_rarity	1, 0.000001, 20, 0.5	A target weighs min(cap, base * (1 + gain * rarity)). The rarity goes from 0 (the target covers every row) to 1 (it covers a share of floor or less); it is unknown_rarity when the frequency is not known
smart.weights.base.*	see below	The base weight of each kind of target

The base weights are confirmed_issue 10, candidate_anomaly 6, value 5, relation_exception 5, pattern 4, extreme 4, signal 3, bin 2 and missing 6 (the empty cells of a column), remark 5 (a remark of the scan that may be valid: cells trimmed or with collapsed whitespace) and character 5 (an unexpected character in a column) exact_value 2 (a value of a column the second pass counted exactly, above smart.max_enum_values) and near_value 4 (a frequent value with typos next to it); the rarity of the target raises them. A weight never falls when the target gets rarer or when its base rises, and never passes cap.

Sample settings

{
  "csv": {
    "encoding": "utf-8-sig",
    "delimiter": ","
  }
}

• csv.encoding: source-file encoding for tabalyst sample and tabalyst dedup. The default accepts UTF-8 with or without a BOM; cp1252 is useful for some Windows-produced files.
• csv.delimiter: single-character separator, a comma by default.

Dedup settings

tabalyst dedup and tabalyst.dedup_csv() read the dedup object, and the csv object above for the delimiter and the encoding of the files they read and write.

{
  "dedup": {
    "max_rows": 10000000,
    "memory_budget_mb": 512,
    "verify_content_on_skip": false
  }
}

• dedup.max_rows: the most data records of a source, 10,000,000 by default and at least 1. A source with more records fails with row_limit_exceeded. The option is --max-rows.
• dedup.memory_budget_mb: the memory in MiB that the rows may take before the run continues on disk, 512 by default and at least 1. It is not a limit of the memory of the process; see Exactness, memory and limits. The option is --memory-budget-mb.
• dedup.verify_content_on_skip: false by default. With true, a source is taken as already processed only when the SHA-256 of the source and of its outputs match the report, as --verify-content does. The command line cannot turn a configured true off, and a forced run (--force) does not use it.

Unknown keys in the dedup object are refused. The command line options override the configuration file, which overrides the defaults.

Scan settings

tabalyst scan, tabalyst report and their Python functions read the scan object of each configuration file. The complete object with its defaults is:

{
  "scan": {
    "csv": {"encoding": "utf-8-sig", "delimiter": ","},
    "json": {
      "collections": null, "discovery_max_depth": 3,
      "flatten": {"enabled": true, "separator": ".", "max_depth": null},
      "arrays": {"mode": "preserve"}
    },
    "excel": {"dataset_path": null, "header_row": null},
    "errors": {"policy": null, "max_locations": 10},
    "values": {
      "null_markers": [],
      "null_markers_case_sensitive": true,
      "missing": ["absent", "null", "empty", "blank", "marker"]
    },
    "normalization": {
      "nfc": true, "trim": true, "collapse_whitespace": true,
      "casefold": true, "strip_accents": true
    },
    "limits": {
      "max_fields": 10000, "max_depth": 64, "max_record_observations": 100000,
      "max_line_bytes": 16777216,
      "max_distinct_per_field": 100000, "max_tracked_values": 2000000,
      "max_stored_value_length": 1000, "max_listed_frequencies": 100,
      "max_samples": 10000, "max_long_samples": 100,
      "long_preview_chars": 200, "max_samples_per_length": 100,
      "max_length_strata": 64, "max_cases_per_class": 5,
      "max_case_lengths": 32, "max_sample_memory": 536870912,
      "max_special_characters": 256, "max_distinct_characters": 1024,
      "max_variant_groups": 100,
      "max_variants_per_group": 20, "max_evidence_examples": 10,
      "max_tracked_records": 2000000, "max_listed_records": 10
    },
    "records": {
      "duplicates": true,
      "sample": {
        "rows": 1000, "edge": 50, "cell_chars": 200,
        "max_bytes": 16777216
      }
    },
    "types": {"minimum_confidence": 0.95},
    "classification": {
      "very_short_max_length": 5, "short_max_length": 20,
      "medium_max_length": 50, "long_max_length": 255
    },
    "detection": {
      "minimum_share": 0.95, "warmup_values": 10000, "probe_interval": 100,
      "rare_share": 0.001
    },
    "detectors": {"number": {"enabled": true}},
    "patterns": [],
    "exposure": {"sensitive_values": "mask"},
    "pattern_inference": {
      "enabled": true, "max_values": 10000,
      "max_families": 16, "max_patterns_per_family": 32,
      "max_patterns": 128, "max_depth": 3,
      "max_value_length": 64, "min_distinct": 2
    },
    "observations": {"enabled": true, "max_items": 1000},
    "random_seed": 42
  }
}

Scans and reports do not read the top-level csv settings: set the delimiter and encoding in scan.csv, or pass --delimiter and --encoding.

Layers and merge rules

From lowest to highest priority: built-in defaults, for a JSON source with no Inspect file the collection that Inspect detects, the scan object of each --config file in the order given, the config of the Inspect file beside the source, then --delimiter, --encoding and --collection. For a workbook, the layers are the same with excel.dataset_path in place of json.collections.

• Objects merge key by key, including detectors.<id>: a file that sets {"detectors": {"number": {"enabled": false}}} keeps the other number settings.
• Lists replace the previous list entirely, such as null_markers, patterns or collections.
• Unknown keys are errors.
• null is accepted only where a setting allows it; there is no syntax to delete a setting.

The scan document embeds the effective configuration and its SHA-256 (config_sha256). A stored scan is reused only when the source content, that configuration, the version of Tabalyst and the version of the pattern discovery that found its derived.patterns are all the same. A second-pass result that validates patterns is reused only for the same discovery. A stored scan whose fingerprint no longer matches its configuration, such as one written before a setting existed, is out of date and scanned again.

Sources and errors

• csv.encoding, csv.delimiter: as for reports. The first record is the header.
• json.collections: for JSON files, a list of absolute collection paths such as "$.customers[]" or "$.customers[].orders[]". Paths must end with [], be unique and not contain one another, and are stored in canonical spelling ($["orders"][] becomes $.orders[]). null means that no collection is named: tabalyst scan and tabalyst report then use the collection that Inspect selects, or stop with exit code 2 when it cannot select one. Only tabalyst.scan() turns null into automatic discovery of every array. For a JSONL file it is null or ["$[]"]. A file that lists json.collections is shared by sources of every format, so it is ignored for JSONL and Excel sources in a mixed batch.
• excel.dataset_path: for Excel files (.xlsx, .xlsm), the table to read: a sheet, "$.Sales", or a named table of a sheet, "$.Sales.Orders". A name that is not a plain identifier is quoted, "$[\"Q1 2026\"]". It is stored in canonical spelling. null means that no table is named: tabalyst scan and tabalyst report then use the table that Inspect selects, or stop with exit code 2 when it cannot select one. tabalyst.scan() has no automatic mode: a workbook without excel.dataset_path is an error.
• excel.header_row: the 1-based row of a sheet that holds the header, instead of the detected one; null (default) detects it in the first 50 rows. A strict integer from 1.
• json.discovery_max_depth: how deep Inspect, and automatic discovery, look for arrays under a root object.
• json.flatten: how nested objects become fields. enabled (default true), separator (default .) joins the keys of a field name such as address.city; it is one character that is not a letter, digit, _, whitespace or one of [ ] " \ $. max_depth (default null, no limit) counts segments from the root of the record, a key or [] each: address is 1, address.city is 2, orders[].amount is 3. A container (object or array) at that depth is kept whole as a complex value and its content is not analyzed: nothing is lost, no warning is raised, and the report shows it as a column of type complex. enabled: false is a depth of 1. The separator changes only the names shown: a key holding it is written ["a/b"], and dataset paths such as $.customers[] always use .. This is not limits.max_depth, which protects the reader and truncates with a warning. Content below the flatten depth is not read for duplicate keys.
• json.arrays (mode, only preserve): arrays never add records. ignore and explode are refused.
• errors.policy: null (default) is the default of the source format, strict for CSV and JSON, tolerant for JSONL. strict stops at the first malformed record, such as a CSV record with the wrong number of fields, a JSON object with a duplicate key or a JSONL line that is not valid JSON. tolerant excludes such records, counts them and marks the scan partial. Invalid syntax in a JSON file, undecodable text and unreadable files stop the scan in both policies.
• errors.max_locations: how many record locations each diagnostic lists.

The json.collections, json.flatten, json.arrays and errors.policy settings are the ones an Inspect file edits, under the names structure.dataset_path, flatten, arrays and errors.policy. For a workbook, they are excel.dataset_path and excel.header_row, under the names structure.dataset_path and structure.header_row.

Values and missing values

• values.null_markers: strings that stand for a missing value, such as "N/A", compared after trimming surrounding whitespace.
• values.null_markers_case_sensitive: set to false to match n/a too.
• values.missing: the categories counted as missing: absent (the key or column does not exist in a record), null, empty (""), blank (whitespace only) and marker. Every component stays listed in the result.

Normalization

Each stage can be turned off. They run in this order: nfc (Unicode composition), trim, collapse_whitespace, casefold and strip_accents. The first three produce the analytical value used by listings and detectors; all stages count the values they change and group variants that become equal, such as Montréal and MONTREAL. Raw values are never modified.

Limits

Limits bound memory and output size. A measure that reaches a limit is marked limited in the result, never estimated. Values above the maximum are rejected before the scan starts.

Setting	Default	Maximum	When reached
max_fields	10,000	1,000,000	New field paths of a dataset are counted but not analyzed.
max_depth	64	1,000	Deeper JSON content is counted but not analyzed.
max_record_observations	100,000	100,000,000	The record is an error, or excluded under tolerant.
max_line_bytes	16,777,216 (16 MiB)	268,435,456 (256 MiB)	JSONL only: a longer line is not parsed. It is an error, or excluded under tolerant. The memory needed to parse a line is about 7 to 13 times its size.
max_distinct_per_field	100,000	50,000,000	Distinct values of the field stop being stored; frequencies and cardinality become limited.
max_tracked_values	2,000,000	500,000,000	Distinct values stored for the whole scan; the largest field is released first.
max_stored_value_length	1,000	1,000,000	Longer values are counted, not stored; a small sample of them keeps a preview and a digest.
max_listed_frequencies	100	100,000	Most frequent values listed per field.
max_samples	10,000	100,000	Size of the sample of distinct values the scan keeps per field, with their exact counts.
max_long_samples	100	10,000	Size of the sample of long texts (above max_stored_value_length) kept per field; 0 keeps none.
long_preview_chars	200	1,000,000	Characters of a long text kept as a preview. It never exceeds max_stored_value_length.
max_samples_per_length	100	10,000	Distinct values kept per tracked length by the samples by length; 0 turns them off.
max_length_strata	64	10,000	Lengths tracked per field by the samples by length; the others are summed in other; 0 turns them off.
max_cases_per_class	5	100	Original texts kept per class of the cases of the observed rules (a kind of whitespace, an analytical length); 0 turns the cases off.
max_case_lengths	32	256	Analytical lengths tracked per field by the cases by length; the others are summed in other; 0 turns them off.
max_sample_memory	536,870,912 (512 MiB)	1,099,511,627,776	Bytes, by a cost model, that the samples of the whole scan may take; a sample is cut to fit and says so (requested_k, reason).
max_top_characters	20	1,000	Characters and emoji sequences the character map lists per field; 0 turns the map off.
max_special_characters	256	100,000	Distinct special characters the character profile lists per field; above it the inventory is limited and lists none; 0 turns it off.
max_distinct_characters	1,024	1,114,112	Distinct characters the character profile counts exactly per field; above it the count is limited.
max_variant_groups	100	100,000	Normalization variant groups listed per field.
max_variants_per_group	20	10,000	Variants listed per group.
max_evidence_examples	10	1,000	Example values per detector state.
max_tracked_records	2,000,000	500,000,000	Distinct records stored for the whole scan to find duplicates, about 80 bytes each; later records are still compared with the stored ones, so duplicate counts become lower bounds.
max_listed_records	10	10,000	Record numbers listed per records block.

max_distinct_per_field cannot exceed max_tracked_values, and max_samples cannot exceed max_distinct_per_field. When max_samples is not set, it is 10,000 or max_distinct_per_field when that is lower; the scan document records the value in force. Counters, statistics and detector coverage stay exact after a limit is reached.

The two settings have different jobs, and they are not the settings of the report:

Setting	Tool	Meaning
scan.limits.max_distinct_per_field	Scan, and Report through the scan it runs	Distinct values counted exactly; beyond it the cardinality is limited
scan.limits.max_samples	Scan	Size of the sample of distinct values kept in the scan document
report.value_examples.kept_per_column	Report	Sample values the report keeps per column, at most 1,000 (default 100): the smallest priorities of the scan sample, so a subset of it
report.value_examples.*	Report	What the report shows and how it picks examples from the values it kept
report.row_sample.start, middle, end	Report	Rows of the row sample the report keeps and shows: the first, the interior rows of smallest priority, the last (0 to 1,000 each, default 10, never more than the scan kept)

Records

• records.preview no longer exists: the first records are the start of the row sample (records.sample.edge). Setting it is an error that names its replacements.
• records.duplicates: set to false to skip duplicate detection, which stores one digest per distinct record, up to limits.max_tracked_records. The duplicate count is then disabled.
• records.sample: the row sample of each dataset. rows is its total size (default 1,000, up to 100,000; 0 turns it off), edge the number of first and of last rows (default 50; 2 * edge cannot exceed rows, so the middle has rows - 2 * edge rows), cell_chars the characters kept per cell (default 200) and max_bytes the budget, by a cost model, of the rows kept by the whole scan (default 16 MiB, from 1 MiB). A larger sample or wider cells need a larger budget; when it is exceeded the middle shrinks and the document says so.

Types and interpretations

• types.minimum_confidence: share of values that must agree for a field to get a technical type such as integer or date, otherwise mixed.
• detection.minimum_share: share of the values that a detector must match to become a candidate interpretation of the field.
• detection.warmup_values: default 10,000, up to 1,000,000. The first distinct values of each field go through every detector. A detector that recognized none of them, not even as invalid or ambiguous, is then skipped for the other values of the field, which count as not_tested in its coverage. number and date are never skipped. A field with fewer distinct values is always analyzed by every detector. Set it to 0 to test every value with every detector, as before this setting existed.
• detection.rare_share: default 0.001, from 0 to 1. A detector that recognized at most this share of the warm-up values, and none of those in the second half of the warm-up, is skipped too: its reactions were rare and have stopped. The default allows 10 values of a 10,000-value warm-up. A detector still recognizing values at the end of the warm-up stays, as in a sorted column where matches become frequent, and so does a sensitive detector that found only invalid values. 0 skips only the detectors that recognized nothing.
• detection.probe_interval: default 100, up to 1,000,000. After the warm-up, about one distinct value in this number, chosen by a hash of the value, still goes through every detector. The scan reports a detector_skipped_reacted warning, meaning that the counts of a skipped detector for the field are incomplete, when it recognizes one of these probes after recognizing nothing in the warm-up, or, for a rare detector, when it recognizes at least 10 probed values, more often than detection.rare_share allows. 0 disables probes.

A detector skipped after the warm-up may miss rare values that appear later in the file, including sensitive ones such as an email address in a comment column: such a value would not be masked. Set detection.warmup_values to 0 when every sensitive value must be found.

Text length bounds

classification holds the four bounds that name the Text types of a column: Very short up to very_short_max_length characters, Short up to short_max_length, Medium up to medium_max_length, Long up to long_max_length and Very long above. The type is decided on the exact median of the lengths of the present values, not on the maximum. The bounds must be whole numbers of at least 1 and strictly increasing.

The report.string_analysis section reads the same bounds on the maximum length (see String analysis). They are scan settings because the Text type is written in the scan document: changing one changes the scan configuration, so a stored scan is stale and the source is scanned again. They were report.string_analysis settings before; the old keys are refused with their new place (see Settings that moved).

Detectors

Every detector has enabled (default true) and the settings below. Other detector ids are errors.

Detector	Settings and defaults
number	conventions: ["dot", "comma"], the decimal separators accepted; ambiguous_convention: null, or dot or comma to read values such as 1,234 one way; infer_convention: true, read them in the convention the other numbers of the column prove; min_evidence: 3, the fewest numbers that must prove it.
date	orders: ["YMD", "MDY", "DMY"]; separators: ["-", "/", "."]; ambiguous_order: null, or MDY or DMY to read values such as 02/03/2025 one way; month_languages: ["en", "fr"].
boolean	pairs: [["true", "false"], ["yes", "no"], ["y", "n"], ["oui", "non"], ["vrai", "faux"]].
enumeration	minimum_values: 500; maximum_distinct: 49; case_sensitive: true.
email	max_tracked_domains: 10,000; max_listed_domains: 20.
url	schemes: ["http", "https", "ftp"]; www: true, accept addresses starting with www.; max_tracked_hosts: 10,000; max_listed_hosts: 20.
phone	regions: ["nanp", "fr"]; nanp covers Canada and the United States.
postal_code	regions: ["ca", "us"], Canadian postal codes and ZIP codes.
currency	conventions, ambiguous_convention, infer_convention and min_evidence, as for number.
percentage	conventions, ambiguous_convention, infer_convention and min_evidence, as for number.
quantity	conventions, ambiguous_convention, infer_convention and min_evidence, as for number; max_tracked_units: 10,000; max_listed_units: 20.
uuid	No other setting.
ip_address	versions: ["ipv4", "ipv6"].

ambiguous_convention and ambiguous_order resolve ambiguous values. Without them, the scan reads ambiguous numbers such as 1,234 in the convention that at least min_evidence other numbers of the same column prove, when none proves the other; a decimal comma proven that way also makes 1.250 read as 1250. The currency, percentage and quantity settings do not change the scan, which counts no number of an amount: the second pass applies them to the numbers of the values with a unit, from the evidence of the same detector (1,250 Kg, $1.250). Ambiguous dates are read by the evidence of their column in the second pass only (deep.datetime.infer_order).

Patterns

patterns adds your own detectors, reported as pattern:<id>:

{
  "scan": {
    "patterns": [
      {
        "id": "customer_number",
        "regex": "C-\\d{4}",
        "description": "Customer number",
        "accepts": ["string"],
        "sensitive": false,
        "max_input_length": 256
      }
    ]
  }
}

• id: 1 to 64 letters, digits, _ or -, unique.
• regex: a Python regular expression of at most 1,000 characters that must match the whole value.
• accepts: native types tested, among string, integer, number and boolean; default ["string"].
• sensitive: true masks the fields where the pattern matches.
• max_input_length: longer values are not tested; default 256, at most 10,000.

At most 200 patterns. Regular expressions run without a timeout: use only patterns you trust, since some expressions can take a very long time on some values.

Pattern inference

After a scan, the structural patterns of each text field are inferred from its samples (see Inferred patterns). They are not the configured patterns above.

Setting	Default	Maximum	Meaning
pattern_inference.enabled	true		false skips the inference and leaves derived out of the document.
pattern_inference.max_values	10,000	20,000	Distinct values of a field that the discovery reads.
pattern_inference.max_families	16	1,024	Most families per field; the others are counted in other.
pattern_inference.max_patterns_per_family	32	1,024	Most patterns per family; the other lengths are counted in a table.
pattern_inference.max_patterns	128	1,024	Most patterns and nodes per field; the nodes of the lowest gain go first.
pattern_inference.max_depth	3	4	Levels of the tree under a family: the pattern, the nodes that read the first or last characters of a token, then the nodes that depend on them.
pattern_inference.max_value_length	64	1,000	Longer values do not take part and are counted as excluded.
pattern_inference.min_distinct	2	1,000	Distinct values a field needs to be assessed for patterns.

Observations

After a scan, the observations, quality issues and limits of its facts are written in observations (see Observations and limits).

Setting	Default	Maximum	Meaning
observations.enabled	true		false leaves observations out of the document.
observations.max_items	1,000	100,000	Most observations, and most limits, that are listed; the counts of the summaries stay complete.

Sensitive values and sampling

• exposure.sensitive_values: how values of sensitive fields (email addresses, phone numbers, IP addresses and sensitive patterns) appear in the result. show (default) keeps them; mask replaces letters with A or a and digits with 9; hide removes the values and keeps the counts. Sample files always keep the values of the source.
• exposure.detector_keys: aggregate (default) publishes the keys that the email, phone and postal_code detectors declare (extension and domain, region and area code, postal area) even on a masked or hidden field, as counts of at least 5 occurrences with no example or row of the sample; mask withholds them on such a field.
• random_seed: seed of the hash that orders the value samples, an integer from 0 to 2^64 - 1, so that two scans of the same file with the same settings are identical.