91 lines
4.8 KiB
Markdown
91 lines
4.8 KiB
Markdown
|
# Next Steps & Roadmap
|
|||
|
|
|
|||
|
|
## Immediate (before going live)
|
|||
|
|
|
|||
|
|
### 1. Azure app registration
|
|||
|
|
Follow `SETUP.md` to register an app in Azure AD and grant the required Graph API permissions. This is a prerequisite for any real email processing.
|
|||
|
|
|
|||
|
|
### 2. Seed your database
|
|||
|
|
Populate `customers`, `customer_emails`, `vendors`, `vendor_emails`, and `whitelist` with your real contact data before running the daemon. The more complete this data is, the better the LLM's classifications will be.
|
|||
|
|
|
|||
|
|
### 3. Run a dry-run validation
|
|||
|
|
```bash
|
|||
|
|
DRY_RUN=true howl dry-run --limit 20
|
|||
|
|
```
|
|||
|
|
Review the `email_log` table. Check that:
|
|||
|
|
- Known customers/vendors are being matched correctly
|
|||
|
|
- LLM classifications feel accurate for your email types
|
|||
|
|
- Confidence scores make sense (low scores = good candidates for whitelist/notes improvements)
|
|||
|
|
|
|||
|
|
### 4. Tune confidence threshold
|
|||
|
|
The default `LLM_CONFIDENCE_THRESHOLD=0.60` sends uncertain emails to `Needs Review`. Monitor the `Needs Review` folder for the first week and adjust:
|
|||
|
|
- Too many emails going there → lower the threshold or add more DB records
|
|||
|
|
- Wrong actions being taken → raise the threshold
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Short-term improvements
|
|||
|
|
|
|||
|
|
### Admin UI for the email log
|
|||
|
|
A simple read-only web view (Flask or FastAPI) to browse `email_log` entries, see LLM reasoning, and mark records as reviewed. Currently you need `psql` or `howl status` to inspect results.
|
|||
|
|
|
|||
|
|
### CSV / bulk import for contacts
|
|||
|
|
A `howl import-customers customers.csv` and `howl import-vendors vendors.csv` CLI command to populate the database from spreadsheet exports. Would accept columns like `name, company, email, notes`.
|
|||
|
|
|
|||
|
|
### Human review workflow
|
|||
|
|
A way to mark emails in the `Needs Review` folder as approved/rejected after human inspection, writing back to `email_log.action_overridden`. This data could seed a feedback loop for improving prompts.
|
|||
|
|
|
|||
|
|
### Multiple mailbox support
|
|||
|
|
Currently `GRAPH_MAILBOX` is a single address. Supporting a list of mailboxes (e.g., `info@`, `support@`, `billing@`) would let one daemon instance manage several inboxes, each potentially with different folder structures.
|
|||
|
|
|
|||
|
|
### Auto-reply drafting
|
|||
|
|
For emails classified as `customer_inquiry`, have Claude draft a suggested reply and save it as a draft in M365 (Graph supports creating draft messages). A human reviews and sends — no risk of auto-sending.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Medium-term improvements
|
|||
|
|
|
|||
|
|
### Domain-based routing rules
|
|||
|
|
Beyond exact email address matching, support routing rules like:
|
|||
|
|
- All `@acme.com` email → Customers folder regardless of LLM classification
|
|||
|
|
- All `@noreply.*` addresses → skip LLM, move directly to Vendors
|
|||
|
|
|
|||
|
|
### Feedback loop / fine-tuning data collection
|
|||
|
|
Export `email_log` rows where `action_overridden=true` as training/evaluation data. Use these to improve prompts or evaluate whether a switch in model or threshold would reduce overrides.
|
|||
|
|
|
|||
|
|
### Attachment handling
|
|||
|
|
Currently only the email body is analysed. Adding attachment type awareness (e.g., flagging `.pdf` attachments from vendors as likely invoices) would improve invoice detection accuracy.
|
|||
|
|
|
|||
|
|
### Metrics and alerting
|
|||
|
|
Export processing counts, failure rates, and LLM token usage to a metrics system (Prometheus, Datadog, etc.). Alert on elevated failure rates or processing backlogs.
|
|||
|
|
|
|||
|
|
### Scheduled summaries
|
|||
|
|
A daily digest email or Teams/Slack message summarising: emails processed, actions taken, items needing review. Useful for staying aware of daemon activity without querying the database.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Database considerations
|
|||
|
|
|
|||
|
|
### Adding indexes for reporting queries
|
|||
|
|
As `email_log` grows, queries by date range or `sender_type` will benefit from additional indexes:
|
|||
|
|
```sql
|
|||
|
|
CREATE INDEX idx_email_log_type_date ON email_log (sender_type, received_at DESC);
|
|||
|
|
CREATE INDEX idx_email_log_classification ON email_log (llm_classification);
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Archiving old records
|
|||
|
|
After 90 days, `email_log` rows for completed messages can be moved to a cold archive table or exported to object storage to keep the active table fast.
|
|||
|
|
|
|||
|
|
### Contact data enrichment
|
|||
|
|
Add fields to `customers` and `vendors` for priority tier, account manager, and SLA level. Pass these to the LLM prompt so high-priority customers get escalated rather than just moved to the folder.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Security considerations
|
|||
|
|
|
|||
|
|
- **Rotate the Azure client secret** every 6–12 months and update `.env`
|
|||
|
|
- **Restrict `.env` and `.msal_cache.bin` permissions**: `chmod 600 .env .msal_cache.bin`
|
|||
|
|
- **Do not log email bodies** — the current implementation only logs metadata; ensure `LOG_LEVEL=DEBUG` is not used in production as it may expose content
|
|||
|
|
- **Review whitelist entries regularly** — expired entries are automatically ignored, but stale permanent entries should be audited
|
|||
|
|
- **Monitor for phishing mis-classifications** — review the `Junk Email` folder periodically to check that legitimate email isn't being over-aggressively filtered
|