Azure 클라우드 — Application infrastructure & Monitoring
1. Application Infrastructure
First!! App Service Plan
먼저 App Service Plan부터 생성되어야 합니다.
Basic: SSL/custom domain, scale manually
- P1V2 : D series
- A/B testing : staging slots
- Storage: 50% gigabyte
Stamp fee로 요금 부과
Azure App Service is an HTTP-based service for hosting web applications, REST APIs, and mobile back ends. You can develop in your favorite language, be it .NET, .NET Core, Java, Ruby, Node.js, PHP, or Python. Applications run and scale with ease on both Windows and Linux-based environments.
배포 방법 1: visual studio code 사용
배포 방법 2: Slot 사용 — A/B testing
always on feature, configure auto-scaling to increase counts from two to 10 based on HTTP queue length. improve the performance of the application.
→ Store Content close to end users: Azure Content Delivery Network
A content delivery network (CDN) is a distributed network of servers that can efficiently deliver web content to users. CDNs store cached content on edge servers in point-of-presence (POP) locations that are close to end users, to minimize latency.
Azure Content Delivery Network (CDN) offers developers a global solution for rapidly delivering highbandwidth content to users by caching their content at strategically placed physical nodes across the world. Azure CDN can also accelerate dynamic content, which cannot be cached, by leveraging various network optimizations using CDN POPs. For example, route optimization to bypass Border Gateway Protocol (BGP).
→ Store Content close to the application: Azure Redis Cache
Azure Cache for Redis is based on the popular software Redis. It is typically used as a cache to improve the performance and scalability of systems that rely heavily on backend data-stores. Performance is improved by temporarily copying frequently accessed data to fast storage located close to the application. With Azure Cache for Redis, this fast storage is located in-memory with Azure Cache for Redis instead of being loaded from disk by a database.
Logic App(no code application)
WebApp을 사용해도, 프로그래머에게는 여전히 백업이나 Scaling에 관한 책임이 있습니다. 하지만 Logic App으로 프로그램의 Input, Output, Trigger 요소와 같은 논리적인 요소를 정의하면, 코드 작성 없이 자동으로 프로그램이 생성됩니다.
You can schedule a powershell script with Azure Logic Apps.
Azure Logic Apps is a cloud service that helps you automate and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations. Logic Apps simplifies how you design and build scalable solutions for app integration, data integration, system integration, enterprise application integration.
- Azure notification hub: used for notifications
- event grid: Azure Event Grid is a fully-managed event routing service running on top of Azure Service Fabric. Event Grid distributes events from different sources, such as Azure Blob storage accounts or Azure Media Services, to different handlers, such as Azure Functions or Webhooks. Event Grid was created to make it easier to build event-based and serverless applications on Azure.
- event hub: listening to events emitted by Azure resources. → partitions(As Event Hubs receives communications, it divides them into partitions.), capture(Event Hubs can send all your events immediately to Azure Data Lake or Azure Blob storage for inexpensive, permanent persistence.), Authentication
→ Publishers can use either HTTPS or AMQP. AMQP opens a socket and can send multiple messages over that socket → (AMQP’s performance is nice.)
→ Event Hubs default to 4 partitions. Partitions are the buckets within an Event Hub. Each publication will go into only one partition. Each consumer group may read from one or more than one partition.
→ The maximum size for a single publication (individual or batch) that is allowed by Azure Event Hub is 1 MB.
→ To perform real-time reporting by using Microsoft Power BI, you must first select Stream to an event hub.
- The Azure Service Bus cloud service uses the Advanced Message Queueing Protocol (AMQP) 1.0 as its primary means of communication.
You need to recommend a solution to reduce the overhead associated with sending events to the hub.
Configure the application to send events by using the AMQP protocol
- service bus: work with azure queues and topics.
The API can be included in a query string variable named “code”, or it can be included in an x-functions-key HTTP header.
- You can send email by using SendGrid bindings in Azure Functions.
- When you’re using the Consumption plan, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.
Azure function in the consumption plan: able to access the private IP address of a Microsoft SQL server instance that runs on a Azure virtual machine. Cost must be minimized.
→ basic hosting plan: consumption plan, Premium plan, dedicated(app service)plan
Consumption Plan: For the Consumption plan, you don’t have to pay for idle VMs or reserve capacity in advance. Connect to private endpoints with Azure Functions As enterprises continue to adopt serverless (and Platform-as-a-Service, or PaaS) solutions, they often need a way to integrate with existing resources on a virtual network. These existing resources could be databases, file storage, message queues or event streams, or REST APIs.
cf. Azure Container
modernization of application is to use Containers. Azure Container Services.
Containers are becoming the preferred way to package, deploy, and manage cloud applications. Azure Container Instances offers the fastest and simplest way to run a container in Azure, without having to manage any virtual machines and without having to adopt a higher-level service.
Azure Container Instances is a great solution for any scenario that can operate in isolated container.
cf. Azure VM
Azure subscription that contains Azure virtual machines that run Windows Server. You need to centrally monitor all warning events in the System logs of the virtual machines.
→ Resource to create in Azure: A log Analytics workspace
→ Configuration to perform on the virtual machines: Install the Microsoft Monitoring Agent.
Data from a variety of sources, such as the application event log, the operating system (Windows and Linux), Azure resources, and custom data sources.
Azure Monitor collects two types of data: metrics and logs. Metrics are numerical values that describe some aspect of a system at a particular time. Logs contain different kinds of data, such as event information, organized into records.
→ generate a monthly report of all the new Azure Resource Manager resource deployments in your subscription
a metric alert that uses a dynamic threshold
You need to monitor the resource usage of the elastic pool for anomalous database activity based on historic usage patterns. The solution must minimize administrative effort.
ITSMC(IT Service Management Connector)
IT Service Management Connector (ITSMC) allows you to connect Azure to a supported IT Service Management (ITSM) product or service. Azure services like Azure Log Analytics and Azure Monitor provide tools to detect, analyze, and troubleshoot problems with your Azure and non-Azure resources. But the work items related to an issue typically reside in an ITSM product or service. ITSMC provides a bi-directional connection between Azure and ITSM tools to help you resolve issues faster.
On-premise, Cloud모두 같이 중앙 집중관리할 수 있는 서비스입니다.
Azure Log Analytics
Give the ability to correlate Azure resource usage and the performance data with the actual application configuration and performance data.
Use Log Analytics to write your queries. The agent gathers security-related information from resources into a workspace.
→ You use cross-resource querying to analyze the log data collected from separate workspaces.
→ The agent gathers security-related information from resources into a workspace.
Alerts in Azure Monitor can identify important information in your Log Analytics repository. They are created by alert rules that automatically run log searches at regular intervals, and if results of the log search match particular criteria, then an alert record is created and it can be configured to perform an automated response. (including specific user sign-in events)
The Activity log is a platform log in Azure that provides insight into subscription-level events. This includes such information as when a resource is modified or when a virtual machine is started.
Activity logs are kept for 90 days. You can query for any range of dates, as long as the starting date isn’t more than 90 days in the past.
Through activity logs, you can determine:
what operations were taken on the resources in your subscription, who started the operation, when the operation occurred, the status of the operation
Log Analytics Performance
서비스마다 App Performance 특징이 다르다.
- Log stream => give real time application logging
- Web server logging => file system
- Event : event
- SysLog: system log
- Diagnostics setting => workspace : screen message
Log Analytics Workspace
On-Premise를 포함해서, 로그 데이터를 한곳에 모아 모니터링 할 수 있습니다.
event logs → event
system logs → syslog
The workspace is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning.
Azure Application insights
visualize the relationships between application components.
track requests and exceptions to a specific line of code within the application.
Analyze how many users return to the application and how often they select a particular dropdown value.
Give the ability to visualize the relationships between application components.
Give the ability to track requests and exceptions to specific lines of code from within the application.
Give the ability to actually analyze how uses return to an application and see how often they only select a particular drop-dow value.
monitor release pipelines: Use a continuous monitoring gate to monitor release pipelines. Use the gate to stop deployment when an issue has been identified. Deployment will continue automatically when the issue is resolved.
different locations: Availability tests let you monitor your application from multiple locations in the world.
Application Insights can be used for smart alerting to detect anomalies, whilst Azure Monitor Logs can be used for the live monitoring and use of adhoc queries on stored JSON data
third-party hosting providers → collect log and diagnostics data from all the subscriptions into a centralized repository. detect threats. automatic responses to known events.
correlate Azure resource usage and performance data with application configuration and performance data.
The performance diagnostics tool helps you troubleshoot performance issues that can affect a Windows or Linux virtual machine (VM). Supported troubleshooting scenarios include quick checks on known issues and best practices, and complex problems that involve slow VM performance or high usage of CPU, disk space, or memory.
You can run performance diagnostics directly from the Azure portal, where you can also review insights and a report on various logs, rich configuration, and diagnostics data. We recommend that you run performance diagnostics and review the insights and diagnostics data before you contact Microsoft Support.
settings for an Azure SQL Databases.
Diagnostics data can be reviewed in Azure SQL Analytics.
Azure VM Diagnostics Extension
use a third-party solution to parse event logs from the virtual machines stored in an Azure storage account.
save the event logs from the virtual machines to the Azure Storage account.
minimize costs and complexity
Kusto Log Queries
어떤 종류의 data 로그를 원하는지 쿼리할 수 있습니다.
Monitor (AZURE SQL DATABSE)
Azure sql diangnostics
add a new diagnostic setting that archives SQL insights logs to storage2.
add a new diagnostic setting that sends SQL insights logs to Workspace2.
add a new diagnostic setting that sends SQL insights logs to Hub1.
Log diagnostic telemetry: Most important among the diagnostic telemetry that you can export is the Intelligent Insights (SQLInsights) log. This diagnostic telemetry can be streamed to one of the following Azure resources for analysis. Log Analytics workspace, Azure Event Hubs, Azure Storage …
→ you can have multiple settings for diagnostics and stream them onto different locations.
BACPAC file (AZURE SQL DATABSE)
A BACPAC is a Windows file with a .bacpac extension that encapsulates a database’s schema and data. The primary use case for a BACPAC is to move a database from one server to another — or to migrate a database from a local server to the cloud — and archiving an existing database in an open format.
You can import a SQL Server database into Azure SQL Database or SQL Managed Instance using a BACPAC file. You can import the data from a BACPAC file stored in Azure Blob storage (standard storage only) or from local storage in an on-premises location. To maximize import speed by providing more and faster resources, scale your database to a higher service tier and compute size during the import process. You can then scale down after the import is successful.
The amount of time that SQL Insights data will be stored in blob storage is indefinite.
The maximum amount of time that SQL insights data can be stored in Azure Log Analytics is 730 days.
Query Performance Insight
Query Performance Insight provides intelligent query analysis for single and pooled databases. It helps identify the top resource consuming and long-running queries in your workload. This helps you find the queries to optimize to improve overall workload performance and efficiently use the resource that you are paying for.
Azure Service Health
It provides personalized alerts and guidance where Azure service issues affect you. It can notify you, help you understand the impact of issues, and keep you updated as the issue resolves. It can also help you prepare for planned maintenance and changes that could affect the availability of your resources.
You receive traffic data from thousands of sensors and analyze it to make your recommendations. The amount of incoming data varies throughout the day;
The queue will handle spikes in traffic and ensure no data is lost. If the VM cannot keep up with the flow of incoming messages, it will process the message backlog during low-traffic times.
Storage account names must be globally unique. Queue names must be unique within their containing storage account. This means the combination of storage account name and queue name uniquely identifies a queue.
The operations ‘get message’ and ‘delete message’ are separate. → By design, messages are not automatically deleted from a queue after they are retrieved for processing. This helps ensure that every message is processed to completion. If a consumer application crashes during processing, the message is still available to be processed by a different instance of the consumer app.
Azure Service Bus Queues
A service bus queue can be up to 256 KB for standard tier, and 1MB for the premium tier.
A topic allows multiple destination components to subscribe. This means that each message can be delivered to multiple receivers.
Service Bus is a transactional message broker and ensures transactional integrity for all internal operations against its message stores. All transfers of messages inside of Service Bus, such as moving messages to a dead-letter queue or automatic forwarding of messages between entities, are transactional.
→ enable the cloud services to asynchronously communicate transaction information by using REST messages.
Azure Storage Queues
Even though a queue is a first-in-first-out data structure, Azure Storage queues do not guarantee it.
An Azure Storage queue message must be smaller than 64 KB.
Relay : Expose hybrid services securely with Azure Relay
A relay is used for two-way communication and it provides bidirectional connections across network boundaries.
Resource, condition, actions, alert details
An alert rule can have multiple VMs as target.
The following are key attributes of an alert rule: Target Resource — Defines the scope and signals available for alerting. A target can be any Azure resource. Example targets: Virtual machines. Storage accounts. Log Analytics workspace. Application Insights. For certain resources (like virtual machines), you can specify multiple resources as the target of the alert rule. Various alerts may use the same action group or different action groups depending on the user’s requirements.
Automatically, using machine learning algorithms.
→ New, Acknowledged, Closed
Traffic Analytics is a cloud-based solution that provides visibility into user and application activity in cloud networks. Traffic analytics analyzes Network Watcher network security group (NSG) flow logs to provide insights into traffic flow in your Azure cloud. With traffic analytics, you can:
Identify security threats to, and secure your network, with information such as open-ports, applications attempting internet access, and virtual machines (VM) connecting to rogue networks. Visualize network activity across your Azure subscriptions and identify hot spots.
Understand traffic flow patterns across Azure regions and the internet to optimize your network deployment for performance and capacity.
Pinpoint network misconfigurations leading to failed connections in your network.
Azure Service Map
Service Map automatically discovers application components on Windows and Linux systems and maps the communication between services. With Service Map, you can view your servers in the way that you think of them: as interconnected systems that deliver critical services. Service Map shows connections between servers, processes, inbound and outbound connection latency, and ports across any TCP-connected architecture, with no configuration required other than the installation of an agent.
Azure Active Directory Audit Log
→ Azure Event Hubs → Azure Functions → Azure CosmosDB
3. Azure Migrate & Azure Site-recovery
Azure Migrate: migrate the virtual machine to Azure: ensure that the virtual machines remain available when the migration of the disks is in progress.
Azure Site-recovery: that would be used to replicate the disks of the virtual machines to Azure. The solution needs to ensure that the virtual machines remain available when the migration of the disks is in progress.
→ recommend how many and what size Azure virtual machines will be required to move the current workloads to Azure. The solution must minimize administrative effort.
- The application consumes data from multiple databases. Application code references database tables using a combination of the server, database, and table name. You need to migrate the application data to Azure.
→ SQL Server Stretch Database
→ SQL Managed instance
- on-premises file server to Blob Storage: an Azure Import/Export job, Azure Data factory
- The log files are generated by user activity to Apache web servers. The log files are in a consistent format. Approximately 1 GB of logs are generated per day. Microsoft Power Bl is used to display weekly reports of the user activity. → Replace Azure Data Factory with CRON jobs that use AzCopy.
- AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account.
- Cron is one of the most useful utility that you can find in any Unix-like operating system. It is used to schedule commands at a specific time.
Data Migration Assistant
- AzCopy: work with data in Azure storage account
- Azure CosmosDB Data Migration tool: migration of data to CosmosDB
- Data Management Gateway: building a gateway with the on-premise infrastructure
- Data Migration Assistant: migrate the data. it has support for various versions of Microsoft SQL Server as shown below.
API Management policies
this section provides a reference for the following API Management policies. Policies are a powerful capability of the system that allow the publisher to change the behavior of the API through configuration. Policies are a powerful capability of the system that are executed sequentially on the request or response of an API, Popular Statements include format conversion from XML to JSON and call rate limiting to restrict the amount of incoming calls from a developer. Many more policies are available out of the box.
- Convert XML to JSON
- Set HTTP Header: set the header of the response.
ex) You plan to migrate DB1 and DB2 to Azure. Support server-side transactions across DB1 and DB2 and Minimize administrative effort to update the solution. → two SQL Server databases on an Azure virtual machine (When both the database management system and client are under the same ownership (e.g. when SQL Server is deployed to a virtual machine), transactions are available and the lock duration can be controlled.)
a Recovery Services vault and then using Azure Site Recovery
→ ensure that the virtual machines remain available during the migration of the disks.
Site Recovery helps ensure business continuity by keeping business apps and workloads running during outages. Site Recovery replicates workloads running on physical and virtual machines (VMs) from a primary site to a secondary location. When an outage occurs at your primary site, you fail over to secondary location, and access apps from there. After the primary location is running again, you can fail back to it.
SQL server: data migration assistant
Table, NoSQL: Azure Cosmos DB Data Migration Tool