Dynatrace
Observability
Cybersecurity
Articles
News
News
Observability
Cybersecurity

AI Observability, LLMs and Autonomous IT: Key Takeaways from Observability Poland 2026

Artificial intelligence is transforming how organizations build, monitor and manage IT environments. As Large Language Models (LLMs) and autonomous AI agents become part of business-critical applications, traditional monitoring is no longer enough. Organizations need a new approach: AI observability. This was one of the central themes of Observability Poland 2026, which brought together technology experts, IT architects and business leaders in Warsaw on October 22, 2026.

Under the theme See more. Secure more. Automate more., the conference explored the future of observability, AI-powered IT operations, cybersecurity and automation. Key topics included LLM observability, AI agent monitoring, OpenTelemetry, Business-Centric Observability and autonomous IT environments.

Experts from Omnilogy also shared their knowledge, highlighting why modern observability must connect technology with real business needs.

AI Observability: Why Traditional IT Monitoring Is No Longer Enough

Traditional observability focuses on monitoring infrastructure, applications and services through three core types of telemetry: logs, metrics and traces. These signals help teams detect failures, analyze performance and troubleshoot issues across distributed IT environments.

However, applications powered by artificial intelligence, LLMs and AI agents introduce a new level of complexity.

An AI-powered system may operate correctly from a technical perspective, return successful HTTP responses and maintain acceptable response times while still generating inaccurate, misleading or potentially harmful content.

This is where AI observability comes in. It extends traditional observability by monitoring AI model behavior, response quality, token consumption, costs, latency and interactions between AI agents and external tools.

In practice, AI observability helps organizations answer critical questions:

• Are AI models generating accurate and reliable responses?

• What causes hallucinations and errors in LLM outputs?

• How much does each AI request or workflow cost?

• What is the latency at each stage of AI processing?

• Are AI agents using tools and external data sources correctly?

• How do changes in models, prompts or input data affect application performance?

The shift from simply monitoring system availability to understanding actual AI behavior was one of the important themes of Observability Poland 2026.

LLM Observability: Monitoring Large Language Models in Production

Large Language Models are increasingly becoming part of digital products, customer services and internal business processes. Deploying an LLM is only the first step. Organizations also need to understand how these models behave in real production environments.

LLM observability focuses on monitoring the entire lifecycle of an interaction with a language model - from the initial prompt and model invocation to the generated response.

Key areas include:

• Response quality - evaluating the accuracy, relevance and usefulness of generated content.

• Hallucination detection - identifying false or unsupported information.

• Token usage - tracking input and output tokens and their associated costs.

• Latency monitoring - measuring model response times and delays across processing stages.

• Prompt monitoring - understanding how prompt changes affect model behavior and output quality.

• Model performance - comparing different models and identifying performance degradation.

One of the main challenges is that generative AI models do not always produce identical responses to the same prompts. This means that traditional testing and monitoring approaches based solely on predictable outcomes are no longer sufficient.

During the conference, Emilia Rucińska from Soflab addressed this challenge in the context of testing AI-powered bots, highlighting the complexity of evaluating systems that can produce different responses to similar inputs.

This is an important part of the broader discussion around the quality, reliability and control of generative AI systems.

AI Agent Observability: A New Level of Visibility into Autonomous Systems

Another important topic at Observability Poland 2026 was the observability of AI agents.

Unlike traditional chatbots, AI agents can independently execute multi-step tasks, interact with external tools, communicate with other systems and make decisions within defined workflows.

This autonomy creates new challenges for IT teams.

When an agent fails to complete a task correctly, examining its final response may not be enough to identify the root cause. Teams need visibility into the entire execution process, including model calls, decisions, tool interactions and data flows.

AI agent observability provides this visibility by enabling teams to analyze agent behavior at every stage of task execution.

Key capabilities include:

• Tracking multi-step workflows and agent interactions through agent tracing.

• Monitoring LLM calls and external API requests.

• Analyzing how AI agents use tools and external services.

• Identifying incorrect decisions and failed operations.

• Monitoring execution time and end-to-end costs.

• Controlling security, permissions and the actions performed by autonomous agents.

This topic was directly reflected in Patryk Gudalewicz's presentation from Soflab, which explored whether a new LLM model is actually better for business and how to approach AI observability in an agent-driven world.

As agentic AI evolves, human oversight remains essential. The greater the autonomy granted to AI systems, the more important transparency, auditability and appropriate governance become.

OpenTelemetry and GenAI: Building a Common Standard for AI Observability

The development of AI observability also increases the need for standardized approaches to collecting and analyzing telemetry data.

One of the key technologies in this area is OpenTelemetry, an open-source project that enables organizations to collect and export telemetry data independently of a specific observability vendor.

In the context of generative AI, GenAI semantic conventions extend traditional telemetry with information specific to AI-powered applications.

This can include monitoring:

• LLM model invocations and parameters.

• Input and output token consumption.

• LLM operation duration and latency.

• Interactions with tools and external services.

• Data flows across applications using AI agents.

The topic of telemetry standardization was explored during Adriana Villela's presentation from Dynatrace, "OpenTelemetry – Playtime is Over."

One of the key takeaways was that collecting more telemetry does not automatically result in better observability. Organizations also need to focus on data quality, consistency, pipeline optimization and cost management.

OpenTelemetry and the development of GenAI conventions can play an important role in creating a consistent approach to monitoring both traditional applications and AI-powered systems.

Business-Centric Observability: Connecting Technology with Business Outcomes

AI observability and autonomous systems are part of a much broader transformation. Modern observability should provide insights that help organizations make better business decisions.

This perspective was highlighted in "Observability with Business Focus," presented by Tomasz Płoński, CTO at Omnilogy, together with Michał Olszewski.

Business-Centric Observability connects technical telemetry with business processes and their actual impact on an organization.

For example, an application performance issue is not necessarily just a technical incident. It may result in delayed order processing, unavailable services or a poor customer experience.

By correlating data across different layers of the IT environment, organizations can better understand the relationships between infrastructure, applications and business processes.

This approach helps teams identify issues faster and prioritize incidents based on their business impact rather than technical severity alone.

Digital Experience Observability: Understanding Customer Behavior

Another important area is the analysis of digital experiences and user behavior.

During the conference, Sławomir Michalik from Omnilogy presented "Automated Insights into Digital Customer Behavior," focusing on the value of automated analysis of customer interactions with digital services.

Data about how users interact with applications and websites helps organizations understand how their services are actually being used.

Combined with AI-powered analysis, these insights can help identify friction points in customer journeys, detect unexpected behavior and highlight opportunities to improve digital experiences.

This demonstrates how modern observability is moving beyond IT infrastructure to become a valuable tool for improving service quality and customer experience.

From AI Observability to Autonomous IT: The Next Stage of IT Operations

Automation and the evolution of autonomous IT environments were also key themes of Observability Poland 2026.

Experts from Dynatrace and ServiceNow explored how AI can support IT operations, incident orchestration and automated remediation.

In traditional IT operations, systems detect problems and generate alerts. IT teams then investigate the issue, identify its root cause and implement corrective actions.

In more advanced environments, telemetry data can trigger automated workflows involving root cause analysis, impact assessment, remediation decisions and corrective actions.

This is the direction towards Autonomous IT.

However, autonomy does not mean eliminating human oversight. Governance, security controls, access management and the ability to verify AI-driven actions remain essential.

The combination of observability, artificial intelligence and automation has the potential to fundamentally transform how IT Operations, SRE and DevOps teams work.

Observability Poland 2026: A Shared Perspective on the Future of IT

This year's conference highlighted several interconnected areas shaping the future of observability:

1. AI Observability - monitoring the quality, performance, security and cost of AI-powered systems.

2. LLM and AI Agent Observability - understanding the behavior of language models and multi-step autonomous workflows.

3. OpenTelemetry and GenAI - standardizing telemetry and building consistent visibility across IT environments.

4. Business-Centric Observability - connecting technical data with business processes and outcomes.

5. Autonomous IT - using AI to automate incident analysis, orchestration and remediation.

Presentations, discussions and technology demonstrations from experts representing Dynatrace, ServiceNow, Anomali, Soflab and Omnilogy showed that modern observability is no longer limited to infrastructure teams.

It is becoming a strategic component of technology management, cybersecurity and customer experience.

See more. Secure more. Automate more. This is not just the theme of Observability Poland 2026. It also reflects the direction in which modern IT environments are evolving.


We would like to thank all speakers, technology partners, participants and organizers for contributing to the event and sharing their knowledge and experience.

See you at the next edition of Observability Poland!

Want to explore the latest insights on observability, AI and IT automation?

Watch the Observability Poland 2026 session recordings on our YouTube channel and revisit the key discussions and presentations from the conference.

Watch Observability Poland 2026 session recordings



In-house materials


Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//
Observability and security for business resilience
//

Stay informed

Subscribe to our newsletter and get the latest articles, insights, and updates delivered straight to your inbox.