![Raft [R]DP Blog Post - Thumbnail Image](https://teamraft.com/wp-content/uploads/Raft-RDP-Blog-Post-Thumbnail-@2x.jpg)
Raft is a DefenseTech Product company on a mission to connect humans with data at the edge. One of our products is the Raft Data Platform -[R]DP. We’ve built the [R]DP and operationalized it in several mission environments to enable actionable insights. [R]DP provides data centricity across Real-Time and Historical data in a unified layer. In this post, we’ll showcase a use case for [R]DP for analyzing large-scale network traffic. We’ll also demonstrate how [R]DP supports cell-level access control and ultra-low latency queries for a 100TB+ data set that contains network logs.
[R]DP Architecture
[R]DP consists of custom implementations and open-source technologies that have been brought together in a single data platform with hardened images and rigorously tested security.
[R]DP’s Cybersecurity Use Case
Network Operations Center (NOC) and Security Operations Center (SOC) analysts face significant challenges when monitoring network traffic at scale. A typical enterprise generates terabytes of network logs daily, making it difficult for analysts to detect and respond to threats in real-time. The problem compounds when working with classified data across multiple security domains, where analysts need different levels of access based on their clearance levels.
Consider a SOC analyst investigating a potential distributed denial-of-service (DDoS) attack. They need to quickly analyze current traffic patterns and compare them against historical baselines. However, traditional solutions often force a trade-off: either maintain fast access to recent data while archiving historical data (making historical analysis slow), or keep all data readily available but sacrifice query performance and increase infrastructure costs.
Additionally, when investigating security incidents, analysts often need to collaborate across teams with different security clearances. Some team members may only be authorized to view unclassified data, while others need access to classified information. Managing these access controls at a granular level while maintaining system performance becomes increasingly complex as data volumes grow.
These operational challenges are typically addressed using Security Information and Event Management (SIEM) tools, but many existing solutions struggle with:
- Real-time analysis of high-volume network traffic
- Fast queries across both recent and historical data
- Granular access controls at the cell level without compromising performance
- Cost-effective storage and retrieval of large-scale network logs
- Seamless integration with existing security tools and workflows
[R]DP in Action
Zeek is an open-source network security monitoring tool popular among cybersecurity communities. In a production environment, Zeek network logs would be sent over Kafka for live streaming ingestion into Pinot, a real-time analytics system. To generate the target data size of 100TB for this demonstration, we deployed many ShadowTraffic pods, which published 50GB per minute to [R]DP.
[R]DP Provides a visualization layer (Superset) to build dashboards and query the network logs. [R]DP’s Query Engine processes all queries submitted via Superset to prevent unauthorized access to sensitive data. The query engine retrieves the appropriate security filters from an Identity and Access Management tool like Keycloak. The security filters may restrict access to specific rows and columns of the data based on the logged in users’ roles and attributes (RBAC and ABAC).
The result set in the screenshot above demonstrates that a non-admin user only has access to unclassified (__security__classification = ‘U’) records. It also shows that the user doesn’t have access to the community_id column and has limited access to the uid column as it’s mostly redacted.
As shown above, even after ingesting over 100TB of streaming network data in real-time, the query latency to retrieve the Top Destinations returned in under 1 second. The low latency achieved here is primarily attributed to the pre-computation performed by Pinot’s Star-tree Index (STI). Pinot immediately builds the STI at ingestion time. Therefore, the pre-computed aggregations need only be retrieved from the STI during query time.
The STI configured above lists the dimensions needed for grouping and filtering in order of descending cardinality. In other queries we tested with low selectivity, we observed that using the STI provided about a 1000x improvement in query latency compared to using Pinot’s inverted index.
The dashboard above is designed to provide visibility in a targeted DDOS attack. Top Sources reports the IPs from the botnet targeting the #1 ranked IP in Top Destinations. Since the dashboard is updated in near real-time it allows analysts to proactively address active threats.
Conclusion
In this demonstration, we showed how [R]DP addresses critical operational challenges faced by SOC/NOC teams. By leveraging technologies like Pinot’s Star-tree Index, [R]DP delivers sub-second query performance across 100TB+ of network logs while maintaining cell-level security controls. This enables analysts to perform real-time threat detection and historical analysis without compromising on speed or security. The platform’s ability to handle 50GB per minute of streaming data while providing granular access controls makes it particularly suitable for secured environments where different teams need varying levels of data access. We’d like to thank our fellow Rafters Matthew DeGarmo and Patrick Stevens for their support.
