Skip to content
Beta · version 0.8.1 · commands and the JSON format may still change.

Insurance rare events

The rare_events profile of the insurance example injects defects that are rare on purpose: from one single case to 500, whatever the size of the file. It exists to test whether a method, such as a sampling, a data quality check or a detector, still finds the cases that make up 0.001 to 0.5 percent of the rows. It is a scenario profile: every case belongs to a named scenario and the run writes its ground truth.

Terminal window
tabalyst generate --example insurance --rows 100000 -d out/ --anomaly-profile rare_events
Created: out/insurance-main.csv
Created: out/insurance-ontario.csv
Created: out/insurance-quebec.csv
Created: out/insurance-combined.csv
Created: out/insurance-combined.rare_events.csv
Created: out/insurance-combined.rare_events.changes.csv
Created: out/insurance-combined.rare_events.summary.json
Created: out/insurance-combined.rare_events.truth.csv
Created: out/insurance-combined.rare_events.rules.json
Created: out/insurance-combined.rare_events.metrics.json
Created: out/insurance-combined.rare_events.manifest.json
Done in 35.0s, 1,251 cells changed

The profile keeps the 34 columns of the default file. It needs at least 100,000 rows: with fewer, the command stops before it writes anything, because 500 cases in a short file are no longer rare.

Error: Profile rare_events needs at least 100,000 rows on insurance-combined (below that its rare cases are not rare); 99,999 requested.

The counts are absolute: they do not grow with the file. Each scenario has a rarity level, which is the band its total falls in.

ScenarioCasesLevelThe case
R011 (on 2 rows)singleA contract identifier reused on a second contract
R025extremely_rareA birth date a century later, so in the future (for a reference date before 2040)
R0520very_rareAn annual premium multiplied by 100
R0625very_rareA start date written in the convention of another branch
R0750very_rareA postal code of another province than the customer’s
R0850very_rareA contract that ends before it starts
R10100rareAn annual premium replaced by text (N/A, inconnu, à venir)
R11250uncommonA contract status that is not one of the four statuses
R12500uncommonAn agency code that belongs to another branch
B01100rareA contract that ends the day it starts (valid)
B02100rareA record modified the day it was created (valid)
LevelCases of a scenario
single1
extremely_rare2 to 9
very_rare10 to 74
rare75 to 199
uncommon200 to 999
frequent1,000 or more

The count is a promise: the profile applies exactly that many cases, or the command fails and publishes nothing. If a scenario asks for more cases than the file has rows that can take them, the message names the operation and the rows available.

R01 copies the contract identifier of one of the 1,000 rows before it, as a double entry would. The case is one, on two rows, so the truth tells it with two relational lines that share a group_id, R01-<first row>, one for each row. Each row keeps its own identifier as canonical_value: the identifier the second contract should have. M21 also reports the two rows as rule_violation lines.

kind,scenario_id,rule_id,group_id,event_id,row_number,column,canonical_value
relational,R01,M21,R01-32669,,32669,id_contrat,CTR-1000032668
relational,R01,M21,R01-32669,370,33555,id_contrat,CTR-1000033554

B01 and B02 are not defects. They sit on the boundary that a rule allows, so a check that flags them is wrong. Their truth lines have the kind valid_atypical, the severity info and the expected detection none, and the rules M01 and M02 do not report them. Use them to measure false positives. The clean file holds no such case: they are written by the profile, so they appear in changes.csv too.

metrics.json holds, for each scenario, the exact probability that a uniform sample without replacement holds none of its cases, for samples of 500, 1,000, 5,000, 10,000 and 50,000 rows. It is computed for the rows of your file, not copied from a table:

"R02": {
"cases": 5,
"probability": {
"500": 0.9752482630325081,
"1000": 0.9509890892755107,
"5000": 0.7737768648540725,
"10000": 0.5904834388286798,
"50000": 0.031246875015626563
},
"reason": null
}

Here a sample of 5,000 rows out of 100,000 misses all five future birth dates 77 times in 100. A sample larger than the file has no probability (null). R01 has none either: its case spans two rows, so the single-row formula does not apply, and the entry says "reason": "multi_row_case".

For 275,000 rows and a sample of 5,000, a scenario of 500 cases is missed 0.01 percent of the time, one of 100 cases 16 percent, 50 cases 40 percent, 5 cases 91 percent and a single case 98 percent.