Java transaction monitoring for code-level visibility

Distributed Application Transaction Tracing is a technique to monitor the flow of application requests as they move through the application delivery chain. It involves monitoring the entire journey of the request across your IT infrastructure, from front-end to middleware, message queues, backend services, external dependencies, and databases.

To track application transactions, Application Performance Monitoring (APM) and observability tools typically use bytecode instrumentation and a "tag and follow" approach, where requests are uniquely identified, and processing times are noted at each tier of the delivery chain. This information is aggregated to present simple topology or application dependency map views that clearly highlight application bottlenecks.

Site reliability engineers (SREs), IT Operations and Application developers (AppDev) can use distributed transaction tracing to identify and troubleshoot requests that experience high traffic surges, response time slowness, service latencies or errors. APM tools that leverage distributed tracing can reveal slow HTTP calls, sub-optimal database queries, slow method or function calls, exceptions or errors, and other issues.

Distributed tracing offers multiple and varied stakeholders such as SREs, DevOps, AppDev, helpdesk or support engineers, business owners and IT operations several benefits:

• Easily comprehend the complexities of your application stack: As requests traverse the application, APM automatically builds an interactive topology map that visualizes these relationships without the need for manual configuration or additional instrumentation.

  This provides end-to-end observability into all application components and dependencies, helping you quickly identify and troubleshoot issues. With intuitive topology and dashboards, you can easily comprehend the complexities of your application stack and delivery chain and ensure that there are no gaps or blind spots in your monitoring.

• Understand service dependencies that may cause cascading issues: The end-to-end topology view or dependency map provided by distributed tracing allows SREs to have a comprehensive understanding of all service dependencies involved in each transaction. Such dependencies could be horizontal (between components of the same type) or vertical (between components of different types).

  This big picture view can help SREs identify any performance bottlenecks that may arise due to dependencies between services. For example, sub-optimal SQL can have a cascading effect on several upstream services. By having this bird's-eye view, SREs can make informed decisions about where to focus their attention and optimize the system for maximum performance.

• Get early warnings when microservices are unhealthy: Distributed tracing can provide real-time insights into the performance and behavior of your microservices. By continuously monitoring your system's traces, you can detect anomalies and unusual patterns that could indicate the presence of a problem or failure before it affects end-users. This can help you take proactive measures to address the issue and minimize its impact on your system's overall performance.

• Reduce mean-time-to-resolve (MTTR) and empower even non-tech stakeholders like helpdesk: The individual timing information is aggregated to provide a comprehensive, end-to-end analysis of the overall time it takes to process the transaction. The APM and observability tool also correlates the captured transaction ID with other relevant metadata and service identification information.

  This makes it easier for even non-technical helpdesk or support engineers to quickly triage all issues on the same APM and observability platform and avoid costly and time consuming “war rooms”. Instead, the helpdesk can forward the trouble ticket to the right stakeholder who can make any necessary optimization to resolve the issue.

• Detect performance regressions caused by changes: Distributed tracing can help provide visibility into changes made in different versions of the software, making it easier to track down any issues or performance regressions that may occur because of those changes. This can ultimately reduce the overhead required for deployments and rollbacks, as teams are able to quickly identify and address any issues that arise.
• Track specific user behavior: Distributed tracing enables the measurement of the time it takes to complete important user actions, such as adding to cart or making a payment. Traces provide insights into backend bottlenecks and errors that may be negatively impacting the user experience.
• Polyglot development support: Distributed tracing is programming language agnostic and can work with any monitoring vendor or runtime environment. It allows tracing of requests across different systems and services, providing freedom and flexibility to developers.

• Manual instrumentation can be challenging: This involves modifying your code to enable tracing. This process can be time-consuming and error-prone, leading to bugs in your application. Additionally, the need for manual instrumentation may vary depending on the language or framework used in your application, resulting in missing traces.
• Incomplete or incorrect traces: It can be challenging to do a complete full-stack distributed tracing by yourself. It can be difficult to know which parts of your code to instrument and how to do it correctly. This can result in incomplete or incorrect traces, which can make it harder to diagnose and fix performance issues in your application. Fortunately, automatic APM solutions solve this problem by integrating the frontend and the server-side traces and showing them in one cogent request topology.
• DIY tracing can consume valuable engineering time: Manual instrumentation for distributed tracing can be a time-consuming task that consumes valuable engineering time, especially as your application becomes more complex. Additionally, it can introduce unintended bugs into your application if developers are not familiar with bytecode instrumentation or context propagation techniques for complex microservices.
• Correlating frontend to server-side: There are challenges with many distributed tracing platforms that primarily focus on backend monitoring. As a result, visibility is often limited to when a request reaches the first backend service, making it difficult to identify the root cause of performance issues especially when the culprit is the frontend. SREs may struggle to determine whether the frontend or backend team should address them. To achieve full stack correlation, it's essential to link instances of page or ajax requests (frontend browser telemetry) with instances of business transactions (backend service telemetry).
• Some distributed tracing solutions may have limitations when it comes to monitoring asynchronous processes and collecting payload data – for example, HTTP request body, JMS or Kafka messages. This can make it challenging to identify and debug issues in pre-production environments where full payload data is often required. Therefore, it's important to consider monitoring tools that can drill through asynchronous processes and collect full payload data. This will provide developers with a more complete picture and gain insights into the end-to-end flow of data across services and identify performance bottlenecks or errors that might be causing issues.
• Big data problem: A high-throughput system can generate millions of traces per minute, making it difficult to identify and monitor the most relevant performance data. However, there are tools available to help you prioritize and surface the most useful performance data for your applications.

A full-stack APM and observability tool can preempt the need for manual instrumentation by automatically discovering all the components in your IT infrastructure and enabling end-to-end tracing without any manual intervention. This saves valuable engineering time and reduces the chances of introducing bugs in your application.

Yes and no.

For transaction tracing to be enabled, there are two choices:

• Choice #1: Application code changes - To capture and propagate trace information, you can instrument your code by adding snippets or using tracing libraries provided by the distributed tracing tool. This involves generating unique trace and span IDs for each request, propagating them across service boundaries, and annotating key operations or methods with trace spans. While this choice offers greater control over tracing, it also introduces additional coding and testing overhead and requires manual modifications to your code.
• Choice #2: Automatic APM without code changes - With automatic APM tools, you can enable transaction tracing without modifying your application code. These tools utilize a single agent to instrument each service involved in a request, capturing timings and troubleshooting information within each span. This eliminates the need for extensive manual code modifications and allows for a quick and efficient setup.

Both options enable the tracking of request flows and provide valuable insights for troubleshooting and performance analysis. Depending on your use case and available resources, you may consider alternative approaches, such as automatic APM solutions, which offer simplified instrumentation without the need for extensive code changes.

Distributed transaction tracing can introduce several overheads:

• Latency: Tracing SDKs may introduce additional latency and consume system resources. Be sure to assess the impact and choose an efficient tracing implementation to minimize the performance overhead.
• Network overhead: Distributed tracing generates and transmits trace data across the network. This can increase network traffic and consume bandwidth, especially in highly distributed environments with numerous services generating trace data.
• Storage overhead: Distributed tracing generates a significant amount of data, including span information, metadata, and context. Storing and retaining this data can require substantial storage resources, especially in systems with high transaction volumes or long trace retention periods.

Transaction tracing can help detect a variety of problems in an application. Some common problems that can be detected through transaction tracing include:

• Performance bottlenecks: Transaction tracing can identify slow and inefficient operations, such as slow database queries, network latency, or resource-intensive processes, that contribute to overall performance bottlenecks.
• Response time issues: By tracing the complete journey of a request, transaction tracing can pinpoint the specific components or services that contribute to slow response times, helping to identify and resolve latency issues.
• Error and exception tracking: Transaction tracing can capture errors and exceptions that occur during the execution of a request, providing valuable insights into the root causes of failures and helping in debugging and troubleshooting efforts.
• Dependencies and inter-service issues: Transaction tracing allows for the visualization and analysis of dependencies between different services and components within a distributed system. It can help identify issues related to service communication, such as failures, timeouts, or misconfigurations.
• Resource utilization: By tracing transactions, it is possible to analyze resource consumption patterns and identify areas where resources are underutilized or overutilized, enabling optimization and efficient resource allocation.
• Complex workflow analysis: Transaction tracing enables the understanding of complex workflows and the interactions between different components, making it easier to identify potential bottlenecks, inefficiencies, or areas for optimization.
• Scalability and load balancing: Transaction tracing can highlight load imbalances or uneven distribution of requests across services, aiding in the identification of scalability issues and enabling better load balancing strategies.

These are some common prerequisites to consider for Java transaction tracing:

• Instrumentation library: You will need to have an instrumentation library or agent installed in your Java application. This library is responsible for capturing and collecting the necessary tracing data.
• Runtime environment: Verify that your application environment is supported by the Java Agent provided by your chosen monitoring tool. Some libraries may require specific versions of Java or have compatibility limitations with certain application servers or containers.
• Compatibility with frameworks: Ensure that the instrumentation library you choose is compatible with the Java frameworks and libraries used in your application. Different libraries may have specific support for popular frameworks like Spring, Java Servlets, or JAX-RS.
• Configuration and setup: You will need to configure the instrumentation library to specify the desired level of tracing, enabling or disabling specific features, and defining any custom settings or sampling rates.
• Distributed tracing backend: Set up a compatible distributed tracing backend or platform to receive and process the captured tracing data. This backend could be hosted locally or in the cloud and should be capable of storing and visualizing the traced transactions.

With an APM tool, all this is automatically made available to you out of the box.

Yes, transaction tracing can work even if multiple technologies are involved in an application, such as Java and .NET.

• Instrumentation libraries or agents: Each technology stack (Java, .NET, etc.) may require its own instrumentation library or agent to capture and propagate tracing data. These libraries or agents are responsible for instrumenting the code and capturing the necessary information for tracing.
• Cross-platform communication: For transaction tracing across multiple technologies, it's important to establish a mechanism for cross-platform communication and propagation of trace context. This allows the tracing information to be passed between different components or services implemented in different technologies. Common approaches include using standardized trace context headers or formats, such as OpenTelemetry or W3C Trace Context.
• Tracing backend and visualization: Set up a tracing backend or observability platform that can receive and process the traced data from multiple technologies. This backend should be capable of aggregating and correlating the tracing information to provide a unified view of the distributed transactions involving different technologies.

Automatic APMs offer comprehensive visibility by capturing trace information across different technologies, enabling a holistic view of request flow and transaction behavior. They enhance troubleshooting, facilitate efficient root cause analysis, ensure cross-platform compatibility, drive proactive performance optimization, and streamline collaboration among teams for improved system performance and reliability.

Agentless transaction tracing refers to gathering information about the flow of requests and responses across components in a distributed system without installing any software agent.

Several options exist:

• Network traffic analysis: By capturing and inspecting network packets, it's possible to extract information about the interactions between components and identify the sequence of requests and responses.
• Log parsing: Log files generated by different components can be parsed and analyzed to reconstruct the flow of transactions.

In general, while agentless transaction tracing can provide visibility into the flow of requests and responses, it may have limitations compared to agent-based distributed tracing approaches.