Azure Data Factory Test

The value of this measure should be low.

Large datasets can significantly increase processing time. The more data being moved or transformed, the longer the pipeline will take.

Failed pipeline runs in Azure Data Factory (ADF) can disrupt data workflows and impact downstream processes.

An alert will be triggered whenever the total Failed pipeline runs metrics is greater than 0.

Ideally, the value of this measure should be singnificantly high.

The value of these measures should be low.

The pipeline and activity runs will be cancelled when a user executes a manual cancellation through the Azure portal or API. Also, if a parent pipeline is cancelled, any child pipelines or activities may also get cancelled.

Failed activity runs in Azure Data Factory (ADF) can disrupt data workflows and complicate data processing.

Ideally, the value of this measure should be singnificantly high.

Failed trigger runs in Azure Data Factory (ADF) can lead to disruptions in scheduled data workflows.

Ideally, the value of this measure should be singnificantly high.

The value of this measure should be low.

If a parent pipeline is cancelled, any active trigger runs associated with that pipeline may also be cancelled.

Failed starts in Azure Data Factory (ADF) refer to scenarios where triggers or pipelines fail to initiate successfully. This can disrupt data workflows and impact scheduled tasks.

This measure is used to identify resource utilization and failed executions when SSIS is integrated with Azure Data Factory.

Ideally, the value of this measure should be singnificantly high.

The value of this measure should be low.

Cancelled starts in Azure Data Factory (ADF) refer to instances where triggers or pipelines are not initiated due to manual or automated cancellations.

In Azure Data Factory, insufficient resources or high load on the integration runtime may prevent a stop command from processing in a timely manner.

Ideally, the value of this measure should be singnificantly high.

Ideally, the value of this measure should be singnificantly high.

Failed execution in Azure Data Factory (ADF) refers to situations where a pipeline or activity does not complete successfully. The failed execution could disrupt data workflows and affect data processing.

The value of this measure should be low.

Limited resources in the integration runtime can lead to the cancellation of additional executions, especially if the system is under heavy load.

High CPU utilization may indicate that the integration runtime is under heavy load, which can slow down pipeline execution.

Low available memory can lead to slower processing times and may cause activities to fail if they exceed memory limits.

Long queue durations can indicate resource bottlenecks or inefficiencies in pipeline design, affecting overall performance.

A long queue length can signal resource bottlenecks, indicating that the system is struggling to keep up with demand.

A low count of available nodes can indicate that the system is overloaded, potentially leading to longer queue durations and slower execution times.

Efficient use of copy capacity can lead to lower costs by preventing the need for unnecessary resource scaling.

High available capacity percentage may indicate that resources are underutilized, while low values can signal bottlenecks that need addressing.

A long waiting queue indicates potential bottlenecks in resource allocation or pipeline design, which can slow down data processing.

High capacity utilization can indicate that resources are effectively used, while low utilization may suggest inefficiencies or underuse of allocated resources.

A high available capacity percentage may suggest that the pipeline has capacity to handle additional workloads, while a low percentage could indicate that resources are strained.

A long waiting queue can indicate that the system is unable to process incoming requests quickly enough, which may affect overall data processing efficiency.

High utilization can indicate that external resources are effectively leveraged, while low utilization may suggest inefficiencies or underuse.

A high percentage indicates that there is significant unused capacity, suggesting potential for better resource allocation or workload distribution.

A long waiting queue can indicate that external resources are overloaded or that there are constraints preventing activities from executing efficiently.

Managing the count of entities can help maintain performance and prevent slowdowns caused by excessive configurations.

Managing factory size effectively ensures that performance remains optimal, as exceeding limits can lead to throttling and slower execution times.

A high number of entities can impact performance, so monitoring helps identify potential bottlenecks and areas for optimization.

A larger factory size can lead to performance bottlenecks if not managed correctly.

Analyzing job starts can help identify trends in usage, peak times for data processing, and potential bottlenecks in data workflows.

This mesaure is used to identify failures when MVNet/Airflow is integrated with Azure Data Factory.

Tracking job ends helps administrators analyze how long jobs take to complete, which can indicate efficiency and potential bottlenecks in data workflows.

Failed heartbeats refer to the instances where the monitoring system fails to receive a regular signal (or heartbeat) from an integration runtime or a linked service. These heartbeats are crucial for ensuring that components are functioning correctly and are responsive. Monitoring failed heartbeats can provide insights into potential issues within data integration processes.

In Azure Data Factory (ADF), operators are essential components used in data integration and transformation processes. They help define how data is processed, transformed, or transferred between various data sources and destinations.

The overall pipeline activity will typically be marked as Failed if one or more operators fail.

Ideally, the value of this measure should be singnificantly high.

In Azure Data Factory (ADF), an instance task typically refers to the execution of a specific activity or job within a pipeline. Each time a pipeline is triggered, an instance of that pipeline is created, and each activity within the pipeline runs as part of that instance.

The overall status of the pipeline instance will typically be marked as Failed if one or more activities fail.

Ideally, the value of this measure should be singnificantly high.

More number of previously succeeded instance tasks may indicate a well-functioning pipeline, but it can also complicate data state management. The administrators may need to ensure that the outputs from these tasks are still valid and relevant for future executions.

If multiple zombie tasks are killed, the pipeline instance may be marked as Failed, especially if the killed tasks are critical to the workflow. This can disrupt the entire data processing sequence.

In Azure Data Factory (ADF), scheduler heartbeats refer to a mechanism that ensures the orchestration and scheduling components are functioning correctly and actively monitoring the status of various tasks and workflows.

More frequent heartbeats can increase the load on the system, consuming more network bandwidth and processing resources. This could lead to higher operational costs or impact the performance of other tasks and services running in ADF.

Directed Acyclic Graph (DAG) is a graphical representation of tasks and their dependencies, often used in workflow management systems of Azure Data Factory (ADF).

More DAGs can lead to increased complexity in managing data workflows. Identifying issues and debugging failures may take longer, as more processes mean more potential points of failure.

The value of this measure should be low.

DAG Processing Manager Stalls refer to a situation where the component responsible for managing and executing Directed Acyclic Graphs (DAGs) becomes unresponsive or significantly slows down.

The value of this measure should be low.

A DAG file refresh error typically occurs in workflow orchestration systems of Azure Data Factory where the system fails to update or reload a DAG (Directed Acyclic Graph) file. This can lead to issues in task scheduling and execution.

The value of this measure should be low.

Scheduler tasks refer to the individual units of work managed and executed by a scheduling system in Azure Data Factory. These tasks are defined within a Directed Acyclic Graph (DAG) and are responsible for performing specific operations in a data pipeline or workflow.

If a task is killed, any downstream tasks that depend on it may not execute, leading to incomplete data processing or failed workflows.

The value of this measure should be high.

A scheduler orphan task refers to a task in Azure Data Factory that remains in the task queue or task state but has lost its parent dependencies.

Clearing orphan tasks releases resources (CPU, memory, I/O) that were previously tied up by those tasks, allowing the system to allocate them to active workflows and tasks.

When scheduler orphan tasks are adopted in Azure Data Factory, it typically means that those tasks are reassigned or re-integrated into the workflow, often after having been previously orphaned.

A higher count of adopted orphan tasks can complicate workflow management, making it harder to track dependencies and the overall state of tasks.

When the scheduler's critical section is busy in Azure Data Factory, it means that the part of the system is responsible for managing task execution, dependencies, and resource allocation is currently occupied.

A high value of this measure represents that more tasks are waiting for access to the critical section, resulting in longer queues and it would take more time for executing the tasks.

An increase in failed email attempts of scheduler SLA may indicate systemic issues within ADF pipelines, prompting a need to review pipeline designs, triggers, or dependencies.

A surge in task instances can cause delays in execution if the Integration Runtime is unable to handle the workload efficiently, resulting in longer processing times.

DAG is a collection of tasks with defined dependencies. When using callbacks in DAGs, you may encounter exceptions that can disrupt data workflow execution.

Frequent callback exceptions may lead to repeated task failures, causing pipelines to halt or enter an inconsistent state.

A task timeout error in Celery typically indicates that a task is taking longer to execute than the specified timeout limit.

Frequent timeouts can put additional strain on work processes, consuming more CPU and memory as they attempt to handle retries or process long-running tasks.

When tasks are removed from a DAG , it could lead to failures in downstream tasks that expect outputs from the removed tasks.

If tasks are restored to DAG, they will have access to historical run data from previous executions, which may affect their behavior.

As more task instances are created, overall execution times may increase due to resource contention. High contention for resources can result in slower execution of individual tasks.

When more triggers block the main thread, the ADF scheduler becomes slow or unresponsive, leading to delays in scheduling new task instances. This can cause subsequent tasks to wait longer than expected to start.

The failure of triggers can cause delays in the execution of the entire workflow. Tasks that depend on the successful execution of the trigger will remain in a queued or waiting state.

Ideally, the value of this measure should be singnificantly high.

DAG bag size typically refers to the amount of memory and resources used by the DAG bag, which is the collection of all the DAGs defined in ADF environment.

If the DAG bag is excessively large, it may lead to out-of-memory errors, especially in environments with limited resources.

DAG processing import errors in ADF occur when the scheduler encounters issues while trying to import or parse the Python files that define the DAGs. These errors can prevent DAGs from being recognized and executed properly.

Continuous import errors can put a strain on the scheduler, which may consume additional resources trying to process the faulty DAGs, potentially affecting the performance of other DAGs.

As the parse time is more, the scheduler takes more time to load and process DAGs, delaying the scheduling of tasks. This can cause cascading delays in dependent workflows.

If a DAG takes longer to complete its last run, subsequent tasks may be delayed, leading to a backlog in the execution of dependent workflows.

A processor timeout typically occurs when a task in a DAG does not complete within a specified time limit, causing the scheduler to terminate the task and report a failure.

Frequent timeouts can cause tasks to fail, leading to incomplete data workflows. This can disrupt data pipelines and business processes that depend on the DAGs.

In Azure Data Factory (ADF), a scheduler task typically refers to the scheduling of activities within a pipeline to run at specific times or intervals

An increase in running scheduler tasks can lead to resource contention, potentially affecting the performance of data processing activities.

In Azure Data Factory, scheduler task starvation occurs when there are more scheduled pipeline runs than the system can handle concurrently, leading to some tasks being delayed or queued.

A growing number of starved tasks can create backlogs, causing a domino effect where subsequent tasks are also delayed. This can complicate workflows that depend on timely data availability.

By analyzing this count, you can assess whether the pipelines are running as expected. A sudden increase or decrease in execution counts can indicate issues or changes in data processing needs.

A consistently low count may indicate that the pipelines are hitting concurrency limits or that resources are being underutilized.

A very high value of this measure often indicates that the system is experiencing bottlenecks, either due to resource limits (like maximum concurrent activities) or contention for shared resources (like databases or storage).

A consistent high value of this measure may suggest a healthy throughput, while a sudden drop could indicate issues such as task failures or bottlenecks.

If the count of this measure is consistently low, it may indicate that the integration runtime is nearing its capacity or that tasks are taking longer to complete.

A high value of this measure can signal bottlenecks in the data processing pipeline, prompting a review of pipeline design or resource allocation.

A high value of this measure can indicate that the system is effectively utilizing its available resources.

A high value of this measure indicates that the system is unable to process tasks efficiently. This can lead to increased latency, as tasks remain queued longer than expected.

A consistently high value of this measure indicates that multiple instances of a pipeline are running simultaneously, which can impact resource usage and performance.

A longer dependency check time delays the start of downstream tasks, leading to increased overall pipeline execution time. This can affect time-sensitive processes and data freshness.

Prolonged task durations can result in resource contention, where available resources are tied up for longer than necessary. This may lead to queuing of other tasks that depend on the completion of the long-running tasks.

A high value of this measure indicate bottlenecks within specific tasks or dependencies, which can hinder the overall workflow. This could lead to queuing of other tasks waiting for resources to become available.

A high value of this measure indicate that a significant amount of time is spent executing tasks that ultimately do not succeed. This can disrupt the flow of subsequent tasks that depend on the successful completion of the DAG, causing delays throughout the workflow.

A high value of this measure can disrupt downstream tasks that rely on the output of the delayed run. This can create a cascading effect, causing subsequent processes to be delayed as well.

A high value of this measure can indicate inefficiencies in task execution. This may lead to prolonged resource usage, which can impede the performance of other tasks awaiting execution.

A high scheduling delay for the first task indicates that the entire DAG takes longer to execute. This can lead to delays in data processing and reporting, affecting business decisions based on timely data.

In Azure Data Factory (ADF), collected DB bags refer to a specific mechanism related to the management of data and metadata within ADF's execution environment.

As the number of collected DB bags increases, it may slow down data processing and retrieval times. The overhead of managing a larger number of bags can impact the speed of data operations.

High memory usage can slow down processing speeds, leading to longer execution times for data pipelines and tasks. This may affect the overall throughput of data processing.

High CPU usage can result in slower processing speeds for tasks, leading to longer execution times for data pipelines. This can affect overall workflow efficiency and throughput.

Each node represents an activity that requires resources. If there are too many nodes running concurrently, it may lead to resource contention, resulting in slower performance and longer execution times for the pipeline.

The resources in Azure Data Factory represents the pipeline, activity and triggers.

The values reported by this measure and its numeric equivalents are mentioned in the table below:

Note:

By default, this measure reports the Measure Values listed in the table above to indicate the status of the resources in Azure Data Factory.

Use the detailed diagnosis of this measure to know Factory name, Location, Provisioning state and Pipeline enable details.

A high value of this measure can lead to overlapping executions, where multiple instances of the same pipeline are running at the same time. This can complicate data management and may lead to data integrity issues if not handled correctly.