docs: SHAPE historical import documentation
This commit is contained in:
parent
56664068f8
commit
25d1aa493c
1 changed files with 230 additions and 0 deletions
230
ondeck/docs/shape-historical-import.md
Normal file
230
ondeck/docs/shape-historical-import.md
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
# SHAPE Historical Data Import
|
||||
|
||||
> **Script:** `scripts/import-shape-historical.ts`
|
||||
> **Last reviewed:** April 2026
|
||||
|
||||
---
|
||||
|
||||
## For the Team
|
||||
|
||||
### What this script does
|
||||
|
||||
This script imports historical SHAPE task completion data from the team's SharePoint Excel files into OnDeck. It is a **one-time migration tool** — not part of the ongoing sync.
|
||||
|
||||
Each team member has a folder on SharePoint at:
|
||||
```
|
||||
Claims/SHAPE Accounts/{MEMBER_FOLDER}/{Client}.xlsx
|
||||
```
|
||||
|
||||
The script reads every client Excel file for each advocate, matches it to a client in OnDeck, and records whether each SHAPE task was completed, marked N/A, or left blank.
|
||||
|
||||
### Team member folders
|
||||
|
||||
| SharePoint Folder | Advocate |
|
||||
|---|---|
|
||||
| `CHRIS` | Christine Gove |
|
||||
| `DAWN` | Dawn Boland |
|
||||
| `Jeanne` | Jeanne Strong |
|
||||
| `LUKE` | Luke Billman |
|
||||
| `MIMI` | Mimi Rawlings |
|
||||
|
||||
### Running the script
|
||||
|
||||
**Always do a dry run first.** The dry run reads SharePoint and the database but writes nothing.
|
||||
|
||||
```bash
|
||||
# Dry run (safe — no changes made)
|
||||
npx tsx scripts/import-shape-historical.ts
|
||||
|
||||
# Execute (writes to database)
|
||||
npx tsx scripts/import-shape-historical.ts --execute
|
||||
|
||||
# Save report to a custom location
|
||||
npx tsx scripts/import-shape-historical.ts --output=/home/user/import-report.txt
|
||||
```
|
||||
|
||||
The report is always written to `/tmp/shape-import-report.txt` (or your `--output` path). Check it before running `--execute`.
|
||||
|
||||
### What the report tells you
|
||||
|
||||
After running, the report shows:
|
||||
|
||||
| Stat | Meaning |
|
||||
|---|---|
|
||||
| **Files processed** | Excel files successfully downloaded and parsed |
|
||||
| **Clients matched** | Clients in the Excel that were found in OnDeck |
|
||||
| **Clients unmatched** | Clients that could not be matched — need manual review |
|
||||
| **Fuzzy matches** | Clients matched by approximate name — review for correctness |
|
||||
| **Tasks updated** | Existing tasks whose status was set to Completed or N/A |
|
||||
| **Tasks assigned** | Tasks that had the advocate linked to them |
|
||||
| **Tasks created** | Tasks from the Excel that didn't exist in OnDeck yet |
|
||||
| **Duplicates deleted** | Extra copies of the same task removed from the DB |
|
||||
| **Ad-hoc created** | "Additional Services" rows and unmatched tasks saved as free-form tasks |
|
||||
| **Advocates assigned** | Clients that had their Claims Advocate set from this import |
|
||||
|
||||
### What gets skipped
|
||||
|
||||
The script will **not** import a row if:
|
||||
- The client name in the Excel cannot be matched to any OnDeck client (logged as "unmatched")
|
||||
- The file is named `archive*`, contains `template`, `master workbook`, or starts with `all shape` — these are intentionally excluded
|
||||
|
||||
Nothing else is filtered. Every task row in every matched client file is processed.
|
||||
|
||||
### Does deduplication remove tasks from Excel?
|
||||
|
||||
**No.** Deduplication only removes extra copies of the *same task* that already exist in the database (e.g. from a previous partial run of the script). It does not skip or discard any rows from the Excel files.
|
||||
|
||||
Specifically: if the database already has two identical tasks for the same client and template within ±5 days of the same due date, the older one is kept and the newer duplicate is deleted. The Excel row is still processed normally against the surviving task.
|
||||
|
||||
### What to do with unmatched clients
|
||||
|
||||
After running the dry run, check the **"Unmatched Clients"** section of the report. For each one:
|
||||
|
||||
1. Check if the client exists in OnDeck under a different spelling
|
||||
2. If yes — either rename the client in OnDeck to match, or rename the Excel file
|
||||
3. If no — create the client in OnDeck first, then re-run the import
|
||||
|
||||
---
|
||||
|
||||
## Technical Reference
|
||||
|
||||
### Architecture overview
|
||||
|
||||
```
|
||||
SharePoint (Graph API)
|
||||
└── Claims/SHAPE Accounts/
|
||||
├── CHRIS/ → Christine Gove (userId: cmkm36yc90021y7vb9783nitl)
|
||||
├── DAWN/ → Dawn Boland
|
||||
├── Jeanne/ → Jeanne Strong
|
||||
├── LUKE/ → Luke Billman
|
||||
└── MIMI/ → Mimi Rawlings
|
||||
|
||||
Each .xlsx file:
|
||||
├── One or more client blocks per sheet
|
||||
│ ├── Client Name
|
||||
│ ├── Effective Date (Excel serial number)
|
||||
│ └── Task rows: [Task Name | Days After Renewal | Date Completed | Notes]
|
||||
└── "Additional Services Provided" section (ad-hoc tasks)
|
||||
```
|
||||
|
||||
### Processing pipeline
|
||||
|
||||
```
|
||||
1. fixTemplates() Fix known template name/offset discrepancies in DB
|
||||
2. loadDbState() Load SHAPE/Shape2 designations, templates, and clients
|
||||
3. discoverFiles() List Excel files from SharePoint via Graph API
|
||||
4. processFile() × N For each file:
|
||||
└── parseExcelFile() Parse XLSX without external libraries (custom ZIP+XML)
|
||||
└── parseExcelBlocks() Extract client blocks and task rows
|
||||
└── matchClient() Match Excel client name → OnDeck client
|
||||
└── processTaskRow() × N For each task row:
|
||||
└── findMatchingTasks() Look up existing task in DB (±5-day window)
|
||||
└── deduplication Delete extras if >1 match
|
||||
└── update or create Set status, completedAt, assign advocate
|
||||
└── createAdHocTask() × N For "Additional Services" rows
|
||||
5. printReport() Write summary to stdout and output file
|
||||
```
|
||||
|
||||
### XLSX parsing
|
||||
|
||||
The script parses `.xlsx` files without any npm dependencies. `.xlsx` is a ZIP archive containing XML files. The parser:
|
||||
|
||||
1. Finds the End of Central Directory record to locate the ZIP central directory
|
||||
2. Reads each file entry (stored or deflate-compressed via Node's `zlib.inflateRawSync`)
|
||||
3. Parses `xl/sharedstrings.xml` to build a string table
|
||||
4. Identifies the target sheet — prefers `SHAPE2` > `SHAPE` > `CLIENT` tab names, falls back to sheet1
|
||||
5. Parses the sheet XML into a `(row, col) → string` grid
|
||||
6. Scans for `Client Name:` anchors and reads blocks downward
|
||||
|
||||
### Client name matching (3 tiers)
|
||||
|
||||
```
|
||||
1. Exact normalized match
|
||||
- Strip "the", "Inc", "LLC", "Corp", etc., lowercase, collapse whitespace
|
||||
- Direct map lookup
|
||||
|
||||
2. Contains match
|
||||
- One normalized name contains the other as a substring
|
||||
|
||||
3. Jaccard token similarity ≥ 0.65
|
||||
- Tokenize both names, compute intersection/union ratio
|
||||
- Only used if tiers 1 and 2 fail
|
||||
```
|
||||
|
||||
Fuzzy matches (tiers 2–3) are listed in the report for manual review.
|
||||
|
||||
### Template matching (3 tiers)
|
||||
|
||||
```
|
||||
1. Exact normalized name (after alias substitution)
|
||||
2. Contains match on normalized name
|
||||
3. Days-offset fallback: approxOffset = -(365 - daysAfterRenewal), ±15-day window
|
||||
```
|
||||
|
||||
Known aliases handled (`TEMPLATE_NAME_ALIASES`):
|
||||
- `request 125 day loss runs` → `request 120 day loss runs`
|
||||
- `request 89 day loss runs` → `request 90 day loss runs`
|
||||
- `claim review (six month)` / `claims review` → `claim review`
|
||||
|
||||
### Deduplication logic
|
||||
|
||||
`findMatchingTasks(clientId, templateId, dueDate)`:
|
||||
- Queries for tasks matching `clientId + templateId` with `dueDate ±5 days`
|
||||
- Returns the oldest match as the canonical task
|
||||
- Returns all additional matches as `duplicateIds` → these are hard-deleted
|
||||
|
||||
This only cleans up DB duplicates from previous partial runs. It never suppresses Excel rows.
|
||||
|
||||
### Date completed interpretation
|
||||
|
||||
| Excel cell value | Result |
|
||||
|---|---|
|
||||
| Empty / `??` / `?` | `NOT_STARTED` |
|
||||
| `n/a` / `na` | `NA` (with standard naReason) |
|
||||
| Numeric Excel serial (40000–60000) | `COMPLETED`, date converted |
|
||||
| Anything else | `NOT_STARTED` |
|
||||
|
||||
### Task creation rules
|
||||
|
||||
| Situation | Action |
|
||||
|---|---|
|
||||
| Task exists in DB, status = `NOT_STARTED`, Excel says completed/NA | Update status, completedAt, assign advocate |
|
||||
| Task exists in DB, already completed | Assign advocate only (never overwrite completion) |
|
||||
| Task does not exist, Excel says completed/NA | Create task with completed status, assign advocate |
|
||||
| Task does not exist, Excel says `NOT_STARTED` | Create task as `NOT_STARTED`, assign advocate |
|
||||
| Template not matched, Excel says completed/NA | Create ad-hoc task: `[Unmatched Task] {row name}` |
|
||||
| Template not matched, `NOT_STARTED` | Skip (no DB write) |
|
||||
|
||||
### Advocate assignment
|
||||
|
||||
- Advocate is set at the **client level** (`claimsAdvocateId`) only if the client has none yet
|
||||
- Existing advocate assignments are **never overwritten**
|
||||
- Advocate is also linked at the **task level** via `TaskAssignment` upsert
|
||||
|
||||
### Template fixes run on each execution
|
||||
|
||||
Even in dry-run mode, these are applied to the DB on `--execute`:
|
||||
- `Request 125 day loss runs` (offset -125) → renamed to `Request 120 day loss runs`, offset -120
|
||||
- Templates with offset -89 → corrected to -90, names updated accordingly
|
||||
- Missing `Claim Review` at -185 days (Shape designation) → created if absent
|
||||
|
||||
### Environment variables required
|
||||
|
||||
```
|
||||
DATABASE_URL PostgreSQL connection string
|
||||
AZURE_AD_TENANT_ID Microsoft Entra tenant ID
|
||||
AZURE_AD_CLIENT_ID App registration client ID
|
||||
AZURE_AD_CLIENT_SECRET App registration client secret
|
||||
```
|
||||
|
||||
The Azure app registration needs `Files.Read.All` (or `Sites.Read.All`) on Microsoft Graph.
|
||||
|
||||
### Re-running safely
|
||||
|
||||
The script is fully idempotent on `--execute`:
|
||||
- Tasks are updated, not duplicated (±5-day deduplication window)
|
||||
- Advocate assignments use upsert
|
||||
- Ad-hoc tasks check for existing `(clientId, title, isAdHoc=true)` before creating
|
||||
- Template fixes use `updateMany` with specific `where` conditions
|
||||
|
||||
Re-running after adding new clients or fixing name mismatches is safe.
|
||||
Loading…
Add table
Add a link
Reference in a new issue