Smart Optimizer: AI-Assisted Transform Configuration in Monad

TL;DR

• Smart Optimizer is an AI assistant built into Monad's transform editor that analyzes your input data and recommends operations like field removal, field addition, and PII masking
• Choose from four optimization strategies: reduce total data size, mask PII, preserve observability data, or preserve security-relevant fields
• Every suggestion includes an explanation and can be accepted, edited, or discarded individually
• Output preview updates in real time with before/after comparison and percentage change in data size
• Designed as a starting point, not an autopilot. You stay in control of what your transforms do

You know the workflow: a new log source comes online, someone has to sit in the transform editor before data reaches the SIEM, scroll through sample records, and decide field by field what stays, what gets dropped, and what needs masking. Okta audit logs alone can carry 40+ fields per event. AWS CloudTrail is worse. Multiply that across a dozen sources and you're spending hours on work that you wished would take minutes.

Most teams handle this one of two ways. Some skip the tuning entirely and send everything downstream. That's the path of least resistance until the SIEM bill arrives and someone asks why you're paying to index transactionInfo.displayMessage on every authentication event.

Others do the manual work, one field at a time, building transforms by hand based on whatever sample they happened to pull. It works, but it's slow, it doesn't transfer well between sources, and the decisions live in one person's head until they leave.  Plus, there’s no guarantee that the pulled sample is actually representative of all fields from that data source.

Both approaches hit the same wall: knowing which fields matter is tribal knowledge. The senior engineer who's tuned Okta transforms for years can spot noise fields on sight, that instinct comes from incident response, detection writing, and hours of staring at raw logs. It doesn't transfer to the next hire or junior engineers. So transform configuration bottlenecks on the few people who have the expertise, and the rest of the team is stuck at a blank screen.

What the Smart Optimizer does

Smart Optimizer is an AI assistant built into Monad's transform editor. It analyzes the sample input data already loaded in your editor and recommends transform operations, things like dropping low-value fields, removing redundant metadata, or masking PII. Each recommendation comes with an explanation of why the field was flagged, so you're reviewing reasoning, not just a list of field names.

The key distinction: Smart Optimizer doesn't apply anything automatically. It suggests. You review each recommendation individually, accept it, edit it, or throw it out. This matters when you're dealing with security logs and sensitive data. One wrong transform can silently drop fields your detections depend on, malform records your SIEM expects in a specific schema, or strip context that an analyst needs during an investigation. Keeping a human in the loop isn't a nice-to-have here, it's a requirement.

The Smart Optimizer can sample data directly from your live pipeline, so you're reviewing recommendations against real traffic, not a static payload you pasted into the editor.

The optimizer can also recommend adding fields when the data warrants it. If your records are missing a normalized timestamp or another field that downstream systems expect, it may suggest adding one. This isn't the primary use case (the system is built around size optimization and PII handling), but the optimizer has context on the full set of operations Monad supports, so it can surface additions when they're relevant.

How it works in practice

Smart Optimizer lives in the transform configuration step. If you're building a new transform and haven't added any operations yet, you'll see a “Suggest operations” button in the optimizer bar at the bottom of the editor.

Click it, and you choose a strategy based on what kind of data you're working with:

• Optimize data size: Audit logs, app logs, system logs, compliance/archival data
• Optimize security data size: Alerts, findings, auth logs, network flow logs, EDR logs, IAM activity logs
• Optimize observability/telemetry data size: Metrics, traces, APM data, infrastructure logs, container logs
• Mask PII: Any data containing personally identifiable information

The strategy selection matters because "low-value" means different things depending on context. A field like network.bytes is filler in an authentication log but the key signal when you're building baselines for data exfiltration detection. Smart Optimizer adjusts its recommendations based on the optimization goal you pick.

After you request suggestions, you get back a list of recommended operations. Each one tells you what it wants to do and why. You work through them one at a time, accepting, editing, or discarding. Accepted suggestions become operations in your transform, same as if you'd added them manually. You can run the optimizer again afterward, combine its suggestions with your own operations, or start over entirely.

What this actually changes

The time savings are the obvious part. Instead of staring at raw JSON trying to estimate which fields are worth the per-GB ingest cost your SIEM charges for them, you get a starting point that's already identified the likely candidates for removal or masking. A transform that took 30 to 45 minutes of manual field inspection can get a solid first pass in under a minute.

It also changes how you build transforms going forward. Smart Optimizer gives you a foundation, not a final answer. Accept the recommendations that make sense, then layer your own manual operations on top — custom field renames, conditional logic, source-specific parsing rules, whatever your pipeline needs. You're not locked into what the AI suggests. You're starting from a reasonable baseline instead of a blank canvas, and building up from there.

But the more interesting effect is on teams that were skipping the tuning step altogether. If your team has been sending unfiltered logs to the SIEM because nobody had time to build proper transforms, Smart Optimizer lowers the effort threshold enough that the work actually gets done. That translates directly into reduced ingest volume and lower SIEM costs, not because the tool is doing something magical, but because it's removing the friction that kept the optimization from happening in the first place.

A few things worth being honest about. Suggestions are based on the sample data loaded in the editor, so the quality of the recommendations depends on how representative that sample is. If your sample doesn't include the edge cases your detections care about, the optimizer won't know to preserve those fields. Each run is also independent: the optimizer doesn't retain memory of previous suggestions you accepted or discarded, so it may re-flag fields you've already reviewed. This is a starting point, not a replacement for understanding your data. Review before you apply.

Try it

Smart Optimizer is available now in the Monad transform editor. If you're already a Monad customer, open any transform configuration and look for the Smart Optimizer bar at the bottom of the editor.

If you're not using Monad yet and you're tired of building RegEx parsers and normalizing at query-time across dozens of sources, schedule a demo or reach out at product@monad.com. We'll walk you through Smart Optimizer on your own data.

‍