TLDR: I am running some Docker containers on a homelab server, and the containers’ volumes are mapped to NFS shares on my NAS. Is that bad performance?
- I have a Linux PC that acts as my homelab server, and a Synology NAS.
- The server is fast but has 100GB SSD.
- The NAS is slow(er) but has oodles of storage.
- Both devices are wired to their own little gigabit switch, using priority ports.
Of course it’s slower to run off HDD drives compared to SSD, but I do not have a large SSD. The question is: (why) would it be “bad practice” to separate CPU and storage this way? Isn’t that pretty much what a data center also does?
Thee main issues are latency and bandwidth, especially on a server with limited RAM. If you’re careful to manage just the data over NAS, it’s probably fine, especially if the application’s primary job is server data over the same network to clients. That will reduce your effective bandwidth, as data has to go NAS->server->client over the same wires/wifi. If the application does a lot of processing, like a database, you’ll start to compromise that function more noticeably.
On applications with low user count, on a home network, it’s probably fine, but if you’re a hosting company trying to maximize the the data served per CPU cycle, then having the CPU wait on the network is lost money. Those orgs will often have a second, super-fast network to connect processing nodes to storage nodes, and that architecture is not too hard to implement at home. Get a second network card for your server, and plug the NAS into it to avoid dual-transmission over the same wires. [ed: forgot you said you have that second network]
The real issue is having application code itself on NAS. Anytime the server has to page apps in or out of memory, you impose millisecond-scale network latency on top of microsecond-scale SSD latency, and you put 1 Gb/s network cap on top of 3-6 Gb/s SSD bandwidth. If you’re running a lot of containers in small RAM, there can be a lot of memory paging, and introducing millisecond delays every time the CPU switches context will be really noticeable.
I wouldn’t recommend running container volumes over network shares mainly because network instability between NAS and server can cause some really weird issues. Imagine an application having its files ripped from underneath them while they’re running.
I would suggest containers + volumes together on the server, and stuff that’s just pure data on the NAS. So for example, if you were to run a Jellyfin media server, the docker container and its volumes will be on the server, but the video and audio files will be stored on the NAS and accessed via a network share mount.
I think you are saying what I am also saying, but my post was not clear on this:
The container files live on the server, and I use the
volumesection in mydocker-compose.ymlfiles to map data to the NFS share:volumes: - '/mnt/nasvolume/docker/picoshare/data:/data'Would you say this is an okay approach?
Well it’s not “bad practice” per se but it ultimately depends on what you are trying to accomplish and what the underlying architecture is capable of supporting.
Not docker specific but we run enterprise level applications with VMWare esxi hosts accessing vm’s over an iSCSI network share and plan on trying this using NFS datastores over 100gbe with SCM and E1.L SSDs. So this should work fine but this is super fast super expensive hardware and is a lot different from homelab type architecture. Now I have done similar things at home with two esxi hosts using a Drobo for NFS datastores so I could vmotion. Was it super high performance? No. Did it work? Yes.
For a small container based environment I’d probably keep all the containers and storage for the containers local, probably on a fast SSD or NVME drive. Those are usually small. But any large media I’d NFS mount and keep that on the NAS. You can get 1TB NVME drive and pcie adapter for less than $100 on Amazon that would be way fast enough for anything in a home lab.
My 2¢… 😀
