PARTNER FEATURE: As mobile operators expand their 5G footprints and progressively upgrade to 3GPP Release 18 and 5G Advanced, there is a growing imperative to move beyond traditional network operations practices to data- and AI-driven operations (AIOps).

Interest in AIOps extends well beyond the mobile sector: a Morder Intelligence forecast from October 2024, suggested the AIOps market across IT and communications could grow from $27.24 billion in 2024 to $79.91 billion by 2029.

Today’s 5G networks generate vast quantities of data. A network of ten million subscribers could easily generate up to nine petabytes of data per day. Observability solutions can present operators with potentially overwhelming volumes of data. Historically, much of this data has not typically been collected in real-time and it is often not provided in easily consumable standard or open formats. After being deposited into large data lakes, considerable time and effort is often required to assemble relevant data and undertake meaningful analysis.  

In its raw form, packet data and even CDRs can be too granular, unstructured and noisy for immediate AI processing. While AI can be very good at analysing large quantities of data, the value of the analysis is entirely dependent on the quality of the input data selected. Using AI to prepare vast quantities of data becomes prohibitively expensive.

Transformation at source
If they are to truly revolutionise their operations, operators need to deploy smart monitoring, data curation and pipelining to enable cost-effective, real-time, scalable and intelligent AI automation across RAN, Core, Transport, and MEC layers.

Transformation pipelines need to be used to normalize, enrich, and label data at its source reducing the noise and enhancing the quality of inputs to the AIOPs environment. Pipelines can be configured to ensure compliance by anonymizing sensitive fields.

Telecom domain expertise is essential if data from different sources is to be contextualised into meaningful, actionable intelligence. Relevant data needs to be identified and then collected, cleaned, validated and labelled for model training. Human experts must also design and train the AI models and provide high-quality feedback to correct mistakes.

Tokenization: From Raw to Curated
A single raw 5G event record can contain as many as 180 tokens.Through curation, this can be reduced to around 25 tokens — a reduction of around 85 percent —  dramatically lowering GPU utilization and processing costs, especially in public cloud environments such as AWS Bedrock. Not only does curation reduce compute usage, it also provide more precise results and a reduced requirement for data lake storage for redundant data.

Once pipelined, the curated data can be combined with data from other sources such as subscriber demographics, cloud infrastructure metrics, geospatial / environmental data and even social media analytics from subscriber base. . With requisite domain intelligence, algorithms can provide a comprehensive overview and valuable insights which can be fed into AIOps to drive enhanced network performance and user experience.

Packet-Level Precision
Curated data should be extracted for specific use cases. Deep Packet inspection (DPI), where sensors inspect the actual data payloads in network traffic, can show precisely what has been sent, when it happened and how systems across the entire stack responded. When enriched with control plane metadata and identifiers such as IMSI/SUPI, packet data can provide metrics at per cell, slice, handset or subscriber levels. This allows operators to train AI systems with a precise understanding of network behaviour as it relates to each subscriber.

Curated data feeds high-value, low-volume data and enhanced subscriber insights into AIOps pipelines which can reveal monetization opportunities by driving NPS improvement or enhancing SLA management. A single curated feed can be one hundredth the size of the underlying raw data, yet retain maximum analytical value.

Driving Down AI Costs
NETSCOUT’s Omnis AI Streamer has been developed to deliver curated high-fidelity metadata derived from packet flows. This can be used to identify and correlate observability trends, streamline and automate data analysis, uncover historical operational patterns and detect unforeseen issues and security risks that could lead to future service outages or data breaches. Data streams can be tailored for many different use cases.

In operation, the AI Streamer has been shown to deliver significant – up to 93 percent – reductions in data volume, lower GPU memory and processing time, and increased throughput from fewer GPU instances. It can support numerous use cases including network optimization; predictive maintenance; real-time slicing analytics and digital twin applications.

Flexible feed configuration offered by tools such as Omnis AI Streamer enable operators to define a ‘playbook’ of feeds, scheduling intervals, critical metrics and dimensions and filters to ensureonly the necessary curated data is sent to AIOps engines.

For example, QUIC transport protocol latency metrics can be aggregated specifically to monitor and manage the performance of a dedicated 5G slice for premium YouTube users. Any issues can be traced to specific network parameters such as cell or node to create a focused dataset for precise troubleshooting.

User Plane Data is another source of valuable input data for AIOps. Key data elements such TEID, QoS Flow ID, IP addresses, latency, and application signatures can be applied to traffic pattern analysis, SLA breach detection, QoE estimation and App-level performance monitoring

Conclusion
High-quality, high-value and low volume data curated data is essential for AIOps success. By conducting high-quality analytics and filtering at source, it becomes possible to provide “gold-standard” curated data that is AIOps-ready. A curated data approach is also consistent with the TM Forum’s autonomous network initiative and the new use cases and revenue streams that it will unlock for operators. Through improved service quality, predictive maintenance and security enhancements, curated data opens the door to true monetisation.