Accelerating High-Performance Object Storage with Weka.io and Oracle Cloud Infrastructure

Accelerating High-Performance Object Storage with Weka.io and Oracle Cloud Infrastructure

As data volumes keep exploding, many of us in tech face the same challenge: how to store and access massive datasets with both speed and cost-efficiency. Recently, I explored how Weka.io integrates with Oracle Cloud Infrastructure (OCI) to tackle this problem — combining high-performance local storage with cost-effective object storage at scale.

In this post, I’ll share the architecture I tested, some benchmarking results, and why this solution could be interesting for workloads like AI/ML, media pipelines, HPC, and big data analytics.

Architecture Overview

At the heart of this setup is Weka.io’s distributed file system, running on OCI’s Dense.IO compute instances. This provides blazing-fast data ingestion and retrieval for hot data, while seamlessly tiering colder data into OCI Object Storage.

Weka.io File System

Weka.io acts as a high-performance layer capable of handling millions of PUT and GET operations per second. This makes it a good fit for workloads like:

• AI/ML training and inference
• High-performance computing (HPC)
• Media and entertainment pipelines
• Big data analytics

OCI Dense.IO Compute

For the testing, I used OCI Dense.IO E5 instances, which provide:

• 128 OCPUs
• 1.5 TB RAM
• 81.6 TB local NVMe storage
• 100 Gbps networking per node

These Dense.IO instances give ultra-low latency and high IOPS for “hot” data.

OCI Object Storage Tiering

To manage storage costs, less-frequently accessed data gets automatically tiered from local NVMe storage to OCI Object Storage. The cool part is that Weka.io’s tiering is:

• Transparent: Data stays in the same namespace, no manual migration needed
• Scalable: Easily supports petabyte-scale datasets
• Fast: Data pulled back as needed, without massive latency penalties

The Weka.io file system chunks and migrates data into OCI Object Storage, freeing up expensive NVMe capacity while keeping everything accessible.

Deployment on OCI

Here’s how the solution was set up:

1. Deploy a Weka.io Cluster:
   Launched across OCI Dense.IO instances with NVMe SSD storage.
2. Configure Tiering:
   Weka.io policies were set to offload colder data into OCI Object Storage automatically.
3. Benchmark Performance:
   Used the WARP benchmarking tool to test real-world performance metrics.

Benchmarking Results with WARP

I ran several benchmarks across a 9-node Weka.io cluster on Dense.IO E5 instances. Here’s a quick summary of what I saw (Per Node) when testing file sizes commonly found in Smart Camera applications:

PUT Operations

• 420KB Video Files:
  • Average Throughput: ~18.39 GB/s
  • Objects/sec: ~43,782
• 20KB Audio Files:
  • Average Throughput: ~1.97 GB/s
  • Objects/sec: ~98,418

GET Operations

• Average Throughput: ~34.53 GB/s
• Objects/sec: ~82,261

DELETE Operations

• Average Rate: ~56,225 objects/sec
• Peak Rate: ~265,926 objects/sec

Pre-Signed URL Generation

• Average Rate: ~161,827 URLs/sec across 4 nodes

These numbers show the solution's ability to sustain high throughput and low latency — critical for workloads where time-to-data matters.

Why This Matters

For anyone dealing with large-scale data workloads, this architecture offers some clear benefits:

High Performance:
Dense.IO NVMe gives exceptional speed for hot data, perfect for AI training, real-time analytics, or media pipelines.

Cost Efficiency:
Cold data moves to OCI Object Storage automatically, reducing costs without losing accessibility.

Simplified Management:
Integrated tiering and monitoring reduce operational overhead.

Scalability:
Supports millions of requests per second and can expand horizontally as workloads grow.

Enterprise-Ready:
Built on OCI’s infrastructure for security, reliability, and global availability.

Potential Use Cases

Based on my testing and research, here are some great fits for this architecture:

• Media & Entertainment:
  Manage high-resolution video, audio, and assets with fast retrieval and cost-effective storage for archiving.
• AI/ML Pipelines:
  Train models on large datasets stored locally, while tiering older training data to object storage.
• HPC Workloads:
  Run simulations or scientific analyses with high IOPS, then archive results after for a cost-effective approach.
• Big Data Analytics:
  Keep recent data local for fast analysis, push older data to object storage seamlessly.
• Enterprise Backup & Recovery:
  Enable fast restores while leveraging cost-effective backend storage.
• Life Sciences & Genomic Processing:
  Store massive genomic datasets and research files efficiently.

Final Thoughts

Testing Weka.io on Oracle Cloud Infrastructure confirmed for me that it’s a solid way to handle high-performance storage without blowing up costs. You get the speed of local NVMe for active data and the economics of object storage for everything else — all wrapped in a single, manageable namespace capable of handling upwards of a million IOPS.

It’s a promising solution for modern data-driven workloads that demand both performance and scale.

Interested in seeing a scaled up architecture & test? You can see Weka.io pushing 2TB/s at 17.45 Million IOPS here.

Have you tried combining high-performance file systems with cloud object storage? Let me know how you’re tackling storage challenges in your projects!