Azure Log Analytics Workspace: Design Decisions That Matter at Scale

The first time most teams spin up a Log Analytics Workspace, they do it by accident. The Sentinel wizard creates one for you. The Azure Monitor diagnostic settings blade creates one. A Defender for Cloud recommendation points you at one. Three months later you have four workspaces, fragmented data, duplicate ingestion costs, and KQL queries that have to cross-workspace join to answer a single alert. We have cleaned this up in enough tenants to know the pattern by heart.

Here is what to know before you commit.

What a Log Analytics Workspace Actually Is

The workspace is a blob of managed ColumnStore storage that Azure Monitor, Sentinel, and dozens of other services write into. Queries run against it with KQL. Retention and pricing are workspace-level settings. Every table in the workspace holds a specific data type, and the schema is fixed per table name; you cannot rename columns or restructure a built-in table.

Ingestion flows two ways. The first is the Azure platform itself, anything that sends via Diagnostic Settings or the legacy MMA/OMS agent. The second is the Data Collection Rule (DCR) pipeline, which is the current path for AMA-based agents, custom logs via REST, and Sentinel data connectors. Both land in the same workspace, same KQL surface.

One thing that trips people up early: the workspace is regional. Data written to a West US 2 workspace does not leave West US 2 unless you explicitly cross-query it. For regulated workloads, that is a feature. For global SOC teams who thought they could have one pane of glass across three geographies with one workspace, that is a problem.

Workspace Topology: One or Many

The canonical debate is single workspace vs. per-environment or per-team workspaces. Neither is universally correct, but the default should be: one workspace per tenant for security data, separate workspace for noisy operational telemetry.

The reason is Sentinel. Sentinel licenses at the workspace level, and its analytics rules, incident correlation, and UEBA all operate within a single workspace boundary. Split your security data across two workspaces and you are writing cross-workspace KQL in every detection rule, which is slower to execute, harder to maintain, and occasionally breaks on large result sets.

For operational logs, (VM performance, app traces, container stdout) a second workspace with shorter retention and a lower commitment tier makes sense. These logs are high-volume, short-lived, and rarely needed by your SOC. Blending them into the Sentinel workspace inflates the billable ingestion volume.

A typical topology we deploy:

tenant
├── workspace: sec-prod-eastus (Sentinel-enabled, 90d hot + 2yr archive)
│   ├── SecurityEvent, Syslog, AzureActivity, SignInLogs
│   ├── CommonSecurityLog (CEF via AMA)
│   └── Custom tables: ThreatIntel_CL, EndpointAlert_CL
└── workspace: ops-prod-eastus (no Sentinel, 30d hot)
    ├── Perf, Heartbeat, ContainerLog
    └── AppTraces, AppExceptions

Cross-workspace queries from the security workspace are possible with workspace("ops-prod-eastus").Perf | ... but we keep them rare and only in investigation playbooks, not live detection rules.

Commitment Tiers and the Ingestion Cost Trap

Pay-as-you-go pricing for Log Analytics is $2.30 per GB ingested (varies by region). At any sustained volume above about 100 GB/day, you should be on a commitment tier. The 100 GB/day tier runs around $196/day versus $230 at PAYG: the math is not subtle.

The trap is that teams watch the wrong number. They look at the workspace-level ingestion total and see 80 GB/day and think they are fine. Then they realize that Windows Security Event logs from a single mis-scoped DCR are producing 60 GB of that, mostly audit success noise that no rule ever queries.

Run this KQL weekly in every workspace you manage:

Usage
| where TimeGenerated > ago(7d)
| where IsBillable == true
| summarize
    TotalGB = round(sum(Quantity) / 1024, 2),
    DailyAvgGB = round(sum(Quantity) / 1024 / 7, 2)
    by DataType
| order by TotalGB desc
| take 20

Sort by TotalGB. The top three tables in that list will almost always tell you where your money is going. SecurityEvent with EventID 4624/4625 at high volume is the most common offender. You can scope the DCR to collect only specific EventIDs without losing meaningful coverage.

Data Collection Rules: The Right Abstraction to Learn

If you are still deploying the Microsoft Monitoring Agent (MMA/OMS), you are on borrowed time. MMA reaches end-of-support in November 2024 and is already absent from new marketplace images. The Azure Monitor Agent (AMA) with DCRs is the replacement, and it is strictly better.

A DCR lets you filter and transform at the collection point before data ever hits the workspace. For Windows Security Events, this means you can drop 4688 process creation events that have no command line populated, or filter 4624 logons where the LogonType is 3 and the account ends in $ (machine accounts authenticating to file shares: high volume, almost never an alert). The transform runs on ingestion and the dropped rows are never billed.

The DCR association model is also cleaner operationally. You associate one DCR with thousands of machines and change the filter in one place. With MMA, you had workspace-level collection settings that applied globally with no per-DCR transform capability.

KQL Fundamentals That Pay Off Immediately

KQL is not difficult but it rewards learning the evaluation order. The query engine reads left to right: each operator narrows or transforms the pipe before passing to the next. Filter as early and as specifically as possible.

Two patterns we use constantly:

// Pattern 1: time-bound + indexed filter first, compute last
SecurityEvent
| where TimeGenerated > ago(1h)          // always first
| where EventID == 4625                   // indexed column, push early
| where AccountType == "User"
| summarize FailedAttempts = count() by Account, Computer, bin(TimeGenerated, 5m)
| where FailedAttempts > 10

// Pattern 2: join on pre-filtered subsets, not full tables
let suspicious_ips =
    ThreatIntelligenceIndicator
    | where TimeGenerated > ago(7d)
    | where IndicatorType == "ip"
    | distinct NetworkIP;
SigninLogs
| where TimeGenerated > ago(1h)
| where IPAddress in (suspicious_ips)
| project TimeGenerated, UserPrincipalName, IPAddress, Location, ResultType

The let binding in pattern 2 matters. Without it, the join pulls the full ThreatIntelligenceIndicator table on every run. With it, the materialized lookup is computed once.

Retention, Archive, and the Long-Tail Problem

Workspace retention has two tiers. Hot retention (fully queryable via KQL) runs from 4 to 730 days depending on table-level settings. Archive tier extends up to 12 years at roughly $0.023/GB/month and is queryable only via Search Jobs, which kick off an async restore.

For Sentinel workspaces, the practical configuration is 90 days hot for most tables, then archive. The exception is tables you query in detection rules or workbooks daily: keep those at 90 days and do not shorten them to save cost without checking the detection rules first. We have seen teams drop SecurityEvent hot retention to 30 days and then wonder why their 60-day brute-force trending alert fires on nothing.

Search Jobs let you query archived data without restoring the whole table. The job runs asynchronously, writes results to a new _SRCH table, and you query that. Response time is measured in minutes to hours depending on data volume, so do not plan to use this path in an active incident. Use it for historical investigations and compliance pulls.

Connecting Sentinel Without Doubling Your Ingestion Bill

When you enable Sentinel on a workspace, certain Microsoft-native data connectors are billed under the Sentinel capacity rather than the Log Analytics ingestion rate. These include Defender for Endpoint, Defender for Cloud, Entra ID sign-in and audit logs, and Office 365 activity logs.

The billing distinction matters. Teams sometimes see "free" Sentinel connectors and enable every one available. Then they connect a third-party firewall via CEF, a Linux fleet via Syslog, and a cloud trail via a custom connector, all of which bill at the full Log Analytics rate. The workspace ingestion summary KQL above will tell you which tables are driving cost, but knowing whether a given table is Sentinel-billed or LA-billed requires checking the Microsoft Sentinel pricing docs for your connector.

The connectors that are genuinely free to ingest under most Sentinel plans are narrower than the connector catalog implies. Budget conservatively until you have a week of actual ingestion data in hand.

When to Call Us

If you are standing up a net-new Log Analytics and Sentinel deployment, inheriting a fragmented multi-workspace mess, or trying to get ingestion costs under control before a contract renewal, reach out to the team at KeMeT Tech. We scope these engagements in a day and typically find the savings to pay for the work in the first billing cycle.

For a broader look at how we approach Azure-native monitoring and detection architecture, see our detection engineering practice page.