Solutions

Parallel Storage

Spectrum Scale · Lustre · BeeGFS · VAST

Multi-petabyte parallel filesystems — metadata/data separation, HSM tiering, disaster recovery, and AI training workload performance tuning on IBM Spectrum Scale, Lustre, BeeGFS, VAST Data, and DAOS.

Architecture consultation Get the RFP template

The challenges clients bring to us

IOPS degradation from metadata hot-spots
Poor Spectrum Scale failure-group design → a single disk failure affects everything
Difficulty judging AFM cache fit for a workload
Small-file performance degradation in AI training workloads

Reference Architecture

Parallel Storage — Spectrum Scale · Lustre · BeeGFS · VAST · DAOS with separate metadata / data NSDs, AFM DR cache

02DESIGN DECISIONS

Key design decisions

Why we chose each component, and the assumptions and thresholds behind it — we publish our design decisions as they are.

Q.01

Why a distributed 8-NSD-node (or more) configuration?

In a parallel filesystem, mixing metadata I/O (open/close/stat) and data I/O (read/write) in the same disk pool creates metadata hot-spots that drag down overall IOPS. Separating NSD (Network Shared Disk) nodes into 4 metadata + 4 data (minimum) and isolating the metadata pool onto NVMe-only improves metadata throughput 6–10× for many-small-file workloads (AI training dataset loading).

Q.02

What ratio of NVMe metadata pool to HDD data pool?

Empirically, allocating 5–10% of total used capacity to the NVMe metadata pool keeps metadata I/O from becoming a bottleneck for typical HPC workloads. For AI training (hundreds of millions of small files), raising it to 15–20% is the safe choice. Since performance falls off a cliff edge once the metadata pool fills, always set a 70%-usage threshold alert.

Q.03

AFM Cache vs. direct mount

AFM (Active File Management) lets you use a remote site’s data as if it were a local cache. It is effective for high-WAN-latency DR sites, multiple data centers, and external compute farms; only hot data is cached, so stale-data synchronization overhead is low. However, write-back mode has a consistency model more complex than NFS, so we recommend adopting it only after the operations team is comfortable with Spectrum Scale per-fileset policies.

03PERFORMANCE

Validated performance numbers

Estimated ranges from internal PoCs and customer acceptance tests (HPL · STREAM · IOR · MDTest · osu_*). Real-site results may vary by around ±10% depending on workload and topology.

Metric	Value	Condition
IOR write (sequential, 1 MB)	240 GB/s	12 NSD nodes, 64 client write
IOR read (sequential, 1 MB)	320 GB/s	12 NSD nodes, 64 client read
IOR random (4 KB)	48 GB/s	12 NSD nodes, 64 client, queue depth 64
MDTest file create (per dir)	180 k ops/s	4 metadata NSD (NVMe), single client tree
MDTest file stat	480 k ops/s	4 metadata NSD (NVMe), warm cache
MDTest file remove	120 k ops/s	4 metadata NSD (NVMe), parallel
Sequential read (single client)	24 GB/s	NDR IB, 100GbE bypass disabled
Metadata ops (aggregate)	1.2 M ops/s	AI training dataset load, 32 client

04VENDOR MATRIX

Vendor matrix

Vendors validated at real sites — recommended to match workload characteristics and your operations team's expertise.

IBM Spectrum Scale
Lustre (HPE ClusterStor · DDN ExaScaler)
BeeGFS (ThinkParQ)
VAST Data
DAOS (Intel)

Operations automation — Luxe Series

Our own products that automate operations in this domain

Luxe Ray

Unified storage-array hardware event monitoring

FAQ

Q. How do you decide between Lustre and Spectrum Scale?
The decision depends on the workload I/O pattern (large-sequential vs. metadata-heavy) and the operations team’s staffing. We recommend a PoC.
Q. When is an NFS-over-RDMA solution like VAST Data a good fit?
When NFS compatibility matters and the operations team is not comfortable with metadata/data separation. Note, however, that it is a dedicated appliance with strong vendor lock-in. It is more than good enough for AI inference and notebook environments.

Have a project in this domain?

Talk to an engineer