Design a Log Analytics workspace architecture

Azure Log Analytics is a service offered by Microsoft as part of the Azure Monitor suite of tools. It is a powerful tool for collecting, analyzing, and visualizing data from a variety of sources, including Azure resources, on-premises servers, and other cloud platforms.

One of the key features of Azure Log Analytics is its ability to query and analyze log data in real-time. This allows you to quickly identify trends, patterns, and issues that may be impacting the performance and stability of your systems. You can also use Log Analytics to set up alerts and notifications, so you can be notified as soon as a problem arises and take action to resolve it.

Another important feature of Azure Log Analytics is its integration with other Azure services. For example, you can use Log Analytics to monitor the performance and availability of your Azure virtual machines, storage accounts, and other resources. This allows you to get a comprehensive view of your entire Azure environment, and take proactive measures to ensure that your systems are running smoothly.

In addition to its real-time analysis and integration with other Azure services, Azure Log Analytics also offers a range of visualizations and reporting options. You can use these to create custom dashboards and reports that help you understand your data and identify trends and patterns.

Types of telemetry

1. Metrics
2. Logs

Visibility into 4 key areas

• Activities
• Performance metrics
• Health
• Availability
  

Data structure

Each workspace contains multiple tables that are organized into separate columns with multiple rows of data. Each table is defined by a unique set of columns. Rows of data provided by the data source share those columns. Log queries define columns of data to retrieve and provide output to different features of Azure Monitor and other services that use workspaces.

Design

Your design should always start with a single workspace to reduce the complexity of managing multiple workspaces and in querying data from them. There are no performance limitations from the amount of data in your workspace. Multiple services and data sources can send data to the same workspace. As you identify criteria to create more workspaces, your design should use the fewest number that will match your requirements.
Designing a workspace configuration includes evaluation of multiple criteria. But some of the criteria might be in conflict. For example, you might be able to reduce egress charges by creating a separate workspace in each Azure region. Consolidating into a single workspace might allow you to reduce charges even more with a commitment tier. Evaluate each of the criteria independently. Consider your requirements and priorities to determine which design will be most effective for your environment.

Data retention and archive

At least a year is an absolute requirement to be compliant (ISO, NEST,,,etc)

Total retention time = interactive retention + archive period => Up to 7 years

• Each workspace has a default retention (31/90 days -if AppInsight is enabled-) and up to (720 days interactive retention) policy that's applied to all tables. You can set a different retention policy on individual tables only.
  
• Archived data stays in the same table, alongside the data that's available for interactive queries
  

Permissions/Access Control

Log analytics workspace might contain sensitive data, depending on the type of logs that are being collected and analyzed. some of the information may be considered sensitive. It is important to carefully consider the potential risks and take appropriate measures to protect any sensitive data that may be present in a log analytics workspace. This could include implementing access controls, encryption, and other security measures to prevent unauthorized access to the data

Permission to access data in a Log Analytics workspace is defined by the access control mode, which is a setting on each workspace. You can give users explicit access to the workspace by using a built-in or custom role. Or, you can allow access to data collected for Azure resources to users with access to those resources.

The following diagram shows a cloud security architecture as the flow of information from our environment and how it's secured as is moves to Azure Monitor

See Manage access to log data and workspaces in Azure Monitor for information on the different permission options and how to configure permissions.

The recommendation is to configure custom roles for log analytics, where the admins can have read only access on a specific logs/tables that contains sensitive data

Cost

The cost of using Azure Log Analytics depends on the volume of data you collect and the features you use.

• Free
• PerNode
• Premium
• Standard
• Standalone
• Unlimited
• CapacityReservation
• PerGB2018  (Default one)

In addition to the base cost of using Azure Log Analytics, you may also incur additional charges for other services or resources that you use in conjunction with it, such as Azure Storage or Azure Stream Analytics.

It's important to carefully consider your data collection and analysis needs when choosing a pricing tier for Azure Log Analytics. You can use the Azure Pricing Calculator to estimate the cost of using the service based on your specific requirements.

Overall, Azure Log Analytics is a powerful tool for any organization that relies on Azure for its cloud computing needs. It can help you optimize the performance and stability of your systems, and provide valuable insights into your data. So, it is always a good idea to consider using Azure Log Analytics as part of your overall cloud strategy.