Distributed Filesystems in Agentic Work
and the Signs You Need One
This is the first post in a short series on Distributed File Systems (DFS). Before diving into which vendors build these systems and why their choices matter, it helps to start with the basics of how the technology should work. When it’s set up correctly, most people don’t notice it. But if it’s not done right, things break, and problems show up in confusing ways. My goal here is to explain what a DFS does for AI and agentic work, why these areas need one, and how to spot when you have this problem, either because you’re missing the capability or because your current setup isn’t right for the job.
What is a DFS
A distributed filesystem creates a single, unified space for data across many machines, and now often across different sites, clouds, and storage types. This lets people share and access data without everyone needing their own copy. This idea isn’t new—it’s been around long before anyone started training models on the data.
Modern DFS matters for AI because of the extra features they offer. They can automatically decide where data should live based on your goals, instead of just keeping it where it was first stored. They let many compute nodes access the same data at once, so the whole cluster can read together without overloading the storage. They also provide copy-on-write clones, which give each user a separate, writable view that shares unchanged data with the original, avoiding unnecessary duplication. Finally, they can move data between storage types or locations while it’s still being used, instead of waiting for downtime. These four features are what make shared storage important for AI and agentic workloads.
At the core, both AI and agentic work need data to be available in more than one place, close to the compute, and accessible by many workers at the same time. When sharing data directly is hard, people tend to make copies. But copies use up storage, can get out of sync with the original, and force compute jobs to move to wherever the data is. What starts as a quick fix can become a long-term problem. The main idea behind a DFS is that it’s easier and cheaper over time to manage one shared space than to deal with lots of separate copies.
Why AI workflows reach for one
An AI pipeline isn’t just one type of work—it’s a series of very different tasks. Ingestion writes large amounts of data in order, training needs steady, high-speed reads for the GPUs, and checkpointing interrupts with bursts of writing that make the GPUs wait. Inference is different again: it’s sensitive to delays, often runs in parallel, and can’t tolerate slowdowns. If you use the same storage setup for all these phases, you’ll end up paying too much in one stage and not giving the GPUs enough in another.
Inference is the stage where the need for a shared namespace becomes most obvious. When a model is handling requests or an agent is making decisions and calling tools, the time it takes to fetch data from object storage becomes the main bottleneck. Object storage is great for storing data that isn’t being used, but it’s not good when users care about how fast they get the first byte or about worst-case delays. The data needs to be physically close to the compute, not just in the same region, and often several GPU nodes need the same data at once. This is a problem of both proximity and concurrency, which is exactly what a global namespace with automated placement and parallel access is designed to solve.
Why agentic development teams reach for one
The second area is newer and comes from changes in how engineers work, not just from infrastructure planning. Once coding agents become reliable enough for teams to use, the work shifts from individual laptops to shared computing resources. Now, a group of agent sandboxes runs in parallel, close to the build and test systems, and several agents can work on the same large code repository at the same time, along with human developers.
A coding agent’s workload is almost the opposite of training. Instead of reading large files in order, it deals with lots of small, random file operations—like checking thousands of files, searching through directories, opening many small files, indexing the project, and building, which touches many small pieces. The important data that needs to be fast is the code repository, its dependencies, build caches, and code index, and this speed is needed for many agents at once. If you give each agent a full copy and a cold cache, you pay the setup cost every time, multiplied by the number of agents. But with copy-on-write clones and a shared warm cache, setup takes seconds instead of minutes, and a single, consistent namespace keeps everyone—humans and agents—on the same page instead of dealing with out-of-sync copies.
The signs you need one — and don’t have it.
When this capability is missing and teams try to work around it, the signs are usually the same.
The most obvious sign is when ‘latency’ becomes a catch-all problem that doesn’t get better with the usual fixes. You add more GPUs or bandwidth, but performance barely improves because the real issue is where the data is, not how much power you have. Another clue is unexplained GPU idle time, even though the dashboard shows good utilization. Throughput is also lower than expected because GPUs are waiting for data between steps. You might also notice storage usage growing faster than your actual data, which points to too many copies. If you can’t easily say how many copies of your biggest dataset or repository exist, you’re probably storing several, along with the extra pipelines to keep them in sync and the occasional bug when two users read different versions.
For agentic development teams, missing this capability shows up in a few ways: setting up workspaces takes minutes, every agent starts with a cold cache, your Git host slows down because of repeated large repo clones, disk snapshots pile up, and you have to spend time fixing things when two agents’ code trees get out of sync. The key question is: where is the data when the computer needs it, whether that’s during inference or building? If your best answer is ‘somewhere in object storage, I think,’ then the problem is already affecting your results, even if you haven’t seen it in your metrics yet.
The signs you have one — and it isn’t working.
This kind of problem is harder to notice because a DFS might look fine on its own dashboard but still not work well for your needs.
The most common issue is when a system does well in benchmarks but performs poorly in real use. This happens because the benchmark tested large, sequential reads, but your actual workload involves lots of small files and metadata operations. So, the storage is fast for what was measured, but slow for what you actually do. This shows up when the metadata path becomes the bottleneck, with tasks that involve lots of file checks and opens slowing down, even if the overall throughput looks good. Performance might be fine for one user but drops when many try to read the same data at once, which is typical for both GPU and agent fleets. To spot this, look at the slowest cases, not the average—p99 and time-to-first-byte are where these problems first appear, and they matter most for multi-step agents or parallel GPU reads.
Some problems are more subtle and have to do with where data is placed and how it’s accessed. You might have a unified namespace, but if the data isn’t actually close to the compute, you lose the main benefit. This often happens when placement policies aren’t set or followed. Sometimes, a setup meant for fast, parallel access quietly switches to a slower, single-stream mode, and you only notice when latency gets worse. Issues with data consistency can also appear, like stale reads or two nodes seeing different versions, usually because the system is tuned for a different workload. Finally, operational problems can show up: a system that looks simple when everything works can become confusing when something fails, and that lack of clarity is its own kind of problem.
Where DFS run changes the calculus
When you run systems on-premises, you control the network and hardware, which is good for this kind of work. You can place fast storage like NVMe drives close to your agents or GPUs, use a parallel filesystem over a low-latency network, and treat local drives as a shared, high-performance tier. Proximity is easy when you control everything. The trade-off is that you have to manage more operations and deal with data being tied to specific hardware. As your needs grow, you have to buy more equipment.
In the cloud, things work differently. Compute resources are flexible, but durable object storage isn’t a good fit for fast access, and local storage on instances doesn’t last. So, teams often end up making separate copies for each user or taking disk snapshots for each instance. This leads to problems like overloaded Git hosts, too many snapshots, slow disk attachment, and being stuck in one availability zone because that’s where the data is. In the cloud, data gravity shows up as extra costs for moving data and being tied to certain zones, not just hardware. In hybrid setups, where your main data is on-premises but you use the cloud for extra compute, you really need a namespace that covers both locations and can move data while it’s being used. Otherwise, every time you burst to the cloud, you have to copy the dataset again.
What this sets up
If you can spot these signs, the rest of the series will make more sense. Each vendor focuses on the same set of capabilities, but in different ways. As you read on, keep these questions in mind: Does the global namespace cover what you need without forcing a migration? Do proximity and parallel access hold up when many users are active? Do copy-on-write clones give you separate views without making extra copies? Is data placement automated based on your goals? Can data move while in use, without downtime? And how much does the system handle on its own? Vendors differ in which of these they prioritize and how well they deliver, and the upcoming posts will look at those differences.
That sets the stage. The next posts will dive into specific vendors and their architectures.