lorentz
3bfda9e585
Full implementation of the Howl email triage system: - Microsoft Graph API integration with MSAL auth (client-credentials and delegated modes) - Claude LLM classification via tool use for structured output - PostgreSQL database with customers, vendors, whitelist, and email_log tables - Alembic migration for full schema - APScheduler daemon with graceful shutdown - Typer CLI (run, dry-run, status commands) - Business rule classifier with overrides (whitelist protection, low-confidence fallback) - Action executor (move to folders, flag, escalate with webhook) - 35 passing unit tests - README, SETUP, and NEXT_STEPS documentation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
90 lines
4.8 KiB
Markdown
90 lines
4.8 KiB
Markdown
# Next Steps & Roadmap
|
||
|
||
## Immediate (before going live)
|
||
|
||
### 1. Azure app registration
|
||
Follow `SETUP.md` to register an app in Azure AD and grant the required Graph API permissions. This is a prerequisite for any real email processing.
|
||
|
||
### 2. Seed your database
|
||
Populate `customers`, `customer_emails`, `vendors`, `vendor_emails`, and `whitelist` with your real contact data before running the daemon. The more complete this data is, the better the LLM's classifications will be.
|
||
|
||
### 3. Run a dry-run validation
|
||
```bash
|
||
DRY_RUN=true howl dry-run --limit 20
|
||
```
|
||
Review the `email_log` table. Check that:
|
||
- Known customers/vendors are being matched correctly
|
||
- LLM classifications feel accurate for your email types
|
||
- Confidence scores make sense (low scores = good candidates for whitelist/notes improvements)
|
||
|
||
### 4. Tune confidence threshold
|
||
The default `LLM_CONFIDENCE_THRESHOLD=0.60` sends uncertain emails to `Needs Review`. Monitor the `Needs Review` folder for the first week and adjust:
|
||
- Too many emails going there → lower the threshold or add more DB records
|
||
- Wrong actions being taken → raise the threshold
|
||
|
||
---
|
||
|
||
## Short-term improvements
|
||
|
||
### Admin UI for the email log
|
||
A simple read-only web view (Flask or FastAPI) to browse `email_log` entries, see LLM reasoning, and mark records as reviewed. Currently you need `psql` or `howl status` to inspect results.
|
||
|
||
### CSV / bulk import for contacts
|
||
A `howl import-customers customers.csv` and `howl import-vendors vendors.csv` CLI command to populate the database from spreadsheet exports. Would accept columns like `name, company, email, notes`.
|
||
|
||
### Human review workflow
|
||
A way to mark emails in the `Needs Review` folder as approved/rejected after human inspection, writing back to `email_log.action_overridden`. This data could seed a feedback loop for improving prompts.
|
||
|
||
### Multiple mailbox support
|
||
Currently `GRAPH_MAILBOX` is a single address. Supporting a list of mailboxes (e.g., `info@`, `support@`, `billing@`) would let one daemon instance manage several inboxes, each potentially with different folder structures.
|
||
|
||
### Auto-reply drafting
|
||
For emails classified as `customer_inquiry`, have Claude draft a suggested reply and save it as a draft in M365 (Graph supports creating draft messages). A human reviews and sends — no risk of auto-sending.
|
||
|
||
---
|
||
|
||
## Medium-term improvements
|
||
|
||
### Domain-based routing rules
|
||
Beyond exact email address matching, support routing rules like:
|
||
- All `@acme.com` email → Customers folder regardless of LLM classification
|
||
- All `@noreply.*` addresses → skip LLM, move directly to Vendors
|
||
|
||
### Feedback loop / fine-tuning data collection
|
||
Export `email_log` rows where `action_overridden=true` as training/evaluation data. Use these to improve prompts or evaluate whether a switch in model or threshold would reduce overrides.
|
||
|
||
### Attachment handling
|
||
Currently only the email body is analysed. Adding attachment type awareness (e.g., flagging `.pdf` attachments from vendors as likely invoices) would improve invoice detection accuracy.
|
||
|
||
### Metrics and alerting
|
||
Export processing counts, failure rates, and LLM token usage to a metrics system (Prometheus, Datadog, etc.). Alert on elevated failure rates or processing backlogs.
|
||
|
||
### Scheduled summaries
|
||
A daily digest email or Teams/Slack message summarising: emails processed, actions taken, items needing review. Useful for staying aware of daemon activity without querying the database.
|
||
|
||
---
|
||
|
||
## Database considerations
|
||
|
||
### Adding indexes for reporting queries
|
||
As `email_log` grows, queries by date range or `sender_type` will benefit from additional indexes:
|
||
```sql
|
||
CREATE INDEX idx_email_log_type_date ON email_log (sender_type, received_at DESC);
|
||
CREATE INDEX idx_email_log_classification ON email_log (llm_classification);
|
||
```
|
||
|
||
### Archiving old records
|
||
After 90 days, `email_log` rows for completed messages can be moved to a cold archive table or exported to object storage to keep the active table fast.
|
||
|
||
### Contact data enrichment
|
||
Add fields to `customers` and `vendors` for priority tier, account manager, and SLA level. Pass these to the LLM prompt so high-priority customers get escalated rather than just moved to the folder.
|
||
|
||
---
|
||
|
||
## Security considerations
|
||
|
||
- **Rotate the Azure client secret** every 6–12 months and update `.env`
|
||
- **Restrict `.env` and `.msal_cache.bin` permissions**: `chmod 600 .env .msal_cache.bin`
|
||
- **Do not log email bodies** — the current implementation only logs metadata; ensure `LOG_LEVEL=DEBUG` is not used in production as it may expose content
|
||
- **Review whitelist entries regularly** — expired entries are automatically ignored, but stale permanent entries should be audited
|
||
- **Monitor for phishing mis-classifications** — review the `Junk Email` folder periodically to check that legitimate email isn't being over-aggressively filtered
|