The media server that taught me infrastructure

The gym WiFi was too slow.

That's the whole origin story. I'd be on a treadmill, hit play on something, and watch it buffer through the warm-up. The access point covering that part of the property was doing its best across a long stretch of building, and whatever was left by the time it reached me wasn't enough to stream video.

The obvious fix was to complain about the WiFi. The more interesting one was to stop needing the internet at all: put the media on the local network, where there was gigabit of headroom sitting unused.

So I built a media server. Then I tore it down and rebuilt it twice in about a month, first off the Synology onto Ubuntu on bare metal, then onto Ubuntu running on Proxmox. Somewhere in the middle it stopped being about media entirely.

The real reason I kept going#

Buffering was the trigger. It wasn't the reason I tore a working system down twice to build it again differently.

I wanted to learn Linux properly, and I'd worked out that I'm bad at learning from tutorials. I can follow a lab, complete it, and retain almost nothing, because nothing is at stake. Nobody notices if a lab VM breaks.

But if the media server goes down, people notice. Having something that had to actually work, that people actually used, turned out to be worth more than any structured course. Every outage was a problem I had to genuinely understand rather than a checkbox I could tick.

The media was the excuse. The infrastructure was the point.

Phase 1: a NAS and a media server#

The starting point was a Synology DS923+ that was already in service, plus Jellyfin installed directly onto it through the Synology package system.

flowchart LR
  A[Synology DS923+] --> B[Jellyfin]
  B --> C[Watch things]

That was the entire architecture. No virtualization, no containers I understood, no networking beyond "it has an IP address."

What it taught me was unglamorous and genuinely necessary: how media libraries are organised, why naming conventions matter enormously to a scraper, how file permissions work, and the first real encounter with the idea that a service runs as a user and that user needs access to things.

I spent an embarrassing number of evenings fixing wrong artwork and mis-identified episodes. It felt like wasted time. It wasn't. It was the first time I'd had to think carefully about how a system reads a filesystem.

Phase 2: running out of space, and building storage instead of buying it#

Media collections grow. NAS drives are expensive. There was no budget for this project because this project was not, officially, a project.

So I went looking through the server room properly, and found a pile of old 500GB drives that had come out of machines long since retired. Individually useless, but together a few terabytes.

A five-bay UGREEN DAS enclosure turned that pile into an array. Old disks, a second life, and it cost nothing.

This was the first lesson that actually changed how I think:

You can usually build capacity out of what's already there, if you're willing to understand it well enough. Buying a solution is faster, but building one is where the learning is.

It also introduced the entire vocabulary of storage: RAID levels, mount points, filesystems, what happens when a disk starts throwing errors, and the uncomfortable realisation that a pile of decade-old consumer drives has a failure probability you should take seriously.

Phase 3: leaving Synology, because it was too good#

The Synology worked. That was the problem.

DSM is an excellent product precisely because it hides everything. You click a button and a service appears. I could run a media server with it indefinitely and learn nothing about what was happening underneath, which was the opposite of what I wanted.

So I split the roles. Jellyfin moved onto a Dell OptiPlex Micro, another inventory find, a desktop that had been replaced and set aside. The Synology stayed as network storage. The OptiPlex became the application server.

flowchart LR
  A[Synology DS923+<br/>storage over NFS] --> B[Dell OptiPlex Micro<br/>Ubuntu Server]
  C[UGREEN DAS<br/>reclaimed 500GB disks] --> B
  B --> D[Jellyfin]

Overnight the project went from a web interface to a terminal. Everything I'd been doing by clicking now had to be done with commands I didn't know yet: systemctl, journalctl, apt, SSH keys, users and groups, ownership, fstab, NFS exports, Samba shares.

This is the phase where it stopped being a media server and became a Linux server that happened to serve media. The distinction matters. From here on, every problem was a Linux problem.

Phase 4: Docker, and the permissions wall#

I resisted Docker for a while. Installing a package worked fine. Containers looked like unnecessary abstraction on top of something already solved.

I was wrong for the ordinary reason: I hadn't yet had the problem containers solve. That arrived the first time I needed two services with conflicting dependencies, and again the first time I wanted to move a service to different hardware without rebuilding it from scratch.

Once Jellyfin was containerised, the concepts landed quickly: images, volumes, bind mounts, networks, Compose files, the container lifecycle.

Then I hit the wall that everyone hits.

Almost every early Docker problem I had was a permissions problem. A container runs as some user ID. The files on the host are owned by some other user ID. The container doesn't care what the name is, only what the number is. Getting PUID and PGID right, and understanding that a bind mount is just the host filesystem showing through with host ownership intact, resolved a whole category of failures I'd previously been solving by randomly running chmod 777 and hoping.

That was a lesson in itself: chmod 777 is what you do when you don't yet understand the problem.

Phase 5: hardware transcoding#

Jellyfin was working but struggling. Any client that couldn't direct-play a file forced a CPU transcode, and a small office desktop CPU doesn't have much to give.

The OptiPlex had an Intel iGPU sitting completely idle, which meant Quick Sync was available if I could reach it. That required passing /dev/dri/renderD128 into the container and making sure the container's user was in a group with the right GID to open it. The same UID/GID lesson from the previous phase, wearing a different hat.

The result was dramatic. CPU load on a transcode dropped from pegging most of the cores to something barely noticeable, and multiple simultaneous streams became possible on hardware that had been considered too old to keep.

First time I'd worked with hardware acceleration directly, and the first time I properly understood that "device" in Linux means a file you need permission to open.

Phase 6: the enterprise storage mistake#

This is the phase I got wrong, and it's the most useful one.

The inventory also turned up a Dell EqualLogic SAN. Enterprise storage. Redundant controllers, proper disk shelf, iSCSI. Compared to a plastic USB enclosure full of salvaged laptop drives, it looked like a clear upgrade.

I put real effort into it. Configured iSCSI, presented volumes, mounted them, migrated data onto them.

It never worked properly. I/O errors. Read failures. Volumes that would behave for hours and then stall. I chased it for days: replaced disks, checked cabling, swapped ports, went through controller firmware, read a lot of documentation for a product line that was already end-of-life.

Eventually I accepted the conclusion I'd been avoiding: the array was old, the disks in it were older, and it had been decommissioned for a reason. I was fighting hardware that had already lost.

I migrated everything off it and shut it down.

Those days weren't wasted. I learned iSCSI properly, and I learned how to read storage errors instead of guessing. But the real lesson was about judgement: knowing when to stop investing in something because it's impressive on paper.

Phase 7: storage done properly, with ZFS#

After the SAN, I rebuilt storage on local disks and moved to ZFS in RAIDZ1.

ZFS was chosen for reasons the earlier phases had taught me to care about:

- Checksums on everything. After days of silent read failures on the SAN, a filesystem that can tell me a block is wrong rather than quietly handing back bad data was worth a lot. - Snapshots, which made every subsequent migration far less frightening. - Straightforward management. Pool status in one command, clear resilver behaviour, and no vagueness about array state.

The layout settled into OS on its own SSD, media on the RAIDZ1 array, and the Synology still serving long-term storage over NFS.

I'll be honest about the compromise: RAIDZ1 gives one disk of redundancy, and these are old disks. If a second one fails during a resilver, the pool is gone. That's a risk I accepted deliberately for a system holding replaceable media, and it's why I keep reminding myself that snapshots are not backups. A snapshot doesn't help if the pool itself dies.

Phase 8: Proxmox, and the end of "everything on one box"#

Services kept getting added, and they were all sharing one Ubuntu install. Every upgrade was a risk to everything. There was no isolation, no easy rollback, and no way to test a change without testing it in production.

So I rebuilt the whole thing again, this time on Proxmox VE, with each service in its own LXC container.

flowchart TD
  P[Proxmox VE host] --> J[LXC: Jellyfin]
  P --> T[LXC: TVHeadend]
  P --> F[LXC: future services]
  P --> Z[(ZFS RAIDZ1)]
  N[(Synology NFS)] -.-> J
  Z --> J

This was the biggest single turning point. Snapshot before a change, roll back in seconds if it breaks, and suddenly experimentation stopped being dangerous. Backups became a scheduled job instead of an intention.

It also introduced the single most confusing thing in the project: unprivileged LXC containers shift user IDs. Root inside the container is not root on the host. It's mapped to a high-numbered host user. Which means every bind mount from host storage into a container reopens the exact permissions problem from Phase 4, now with an offset applied.

I lost real time to that before it clicked. When it did click, it was the moment Linux permissions stopped being a thing I fought and became a thing I understood.

Phase 9: networking, and actual segmentation#

Everything had been on one flat LAN. As the number of services grew, that stopped being acceptable. A media server, management interfaces, storage traffic and TV distribution have no business sharing a broadcast domain.

I introduced VLANs, separating administration, storage, services and IPTV.

That meant learning Linux bridges and bridge interfaces, VLAN tagging, static routes, inter-VLAN routing, and firewall rules that were now doing real work because there was actually something to segment.

This is the phase that paid off directly at work. When I later took on a production network migration on the property, VLAN design and inter-VLAN routing weren't theory I'd read about. They were things I'd already broken and fixed on my own time, on equipment nobody would miss.

Phase 10: Live TV, and learning multicast the hard way#

The most ambitious addition, and the one that taught me the most.

The property distributes television over IP using multicast. I could have ignored that entirely. Instead I decided that a live TV stream sitting on a network I had access to was the best networking lesson available to me.

The plan: TVHeadend in its own LXC container, with an interface on the IPTV VLAN, receiving multicast streams and presenting them to Jellyfin as live TV alongside the movie and show libraries.

flowchart LR
  I[IPTV source<br/>multicast streams] --> V[IPTV VLAN]
  V --> T[TVHeadend<br/>LXC container]
  T --> J[Jellyfin]
  J --> C[Clients]

Getting there involved a genuinely new category of problem. Unicast networking mostly works or doesn't. Multicast has a middle state where everything looks correct and no packets arrive.

The tools that made it tractable:

- tcpdump to confirm whether packets were reaching the interface at all - ip maddr show to check which multicast groups an interface had joined - bridge mdb show to see what the Linux bridge believed about group membership - ffprobe and ffplay to test a stream directly, cutting TVHeadend out of the equation entirely

That last one was a technique more than a tool: when a chain of components is failing, test each link in isolation before theorising about the whole. If ffplay can render the stream, the network is fine and the problem is downstream.

Phase 11: the bug that taught me the most#

One symptom, and it bothered me more than anything else in the build.

Live TV would work perfectly, then stop. Restarting TVHeadend fixed it instantly, every time. The obvious move was to accept the restart, schedule it and move on.

I didn't want a cron job papering over something I didn't understand, so I dug in. Restarting fixed it, which meant whatever broke was re-established at startup. What does a multicast receiver do at startup? It sends an IGMP join for the groups it wants.

Which pointed at IGMP membership expiring and never being renewed.

Multicast delivery on a switched network depends on IGMP snooping: the switch listens for join messages and only forwards a group's traffic to ports that asked for it. But membership is soft state, and it times out. Something has to periodically send IGMP queries to prompt receivers to re-announce themselves, and if there's no querier on that segment, memberships silently lapse. The switch stops forwarding and the stream dies. Restarting TVHeadend issues a fresh join and it works again, until the next expiry.

The stream wasn't breaking. The network was correctly forwarding traffic to nobody, because as far as it could tell, nobody had asked recently.

bridge mdb show was what made it visible. I could watch group membership present, then simply absent, with nothing in any application log to explain it.

I remember that one clearly because it's where troubleshooting changed shape for me. The fix mattered less than the method: observe the symptom precisely, work out what a restart actually does, and follow that backwards until you reach a mechanism. Nothing in any tutorial would have taught me that, because the whole point was that the tutorial's happy path wasn't happening.

Where it is today#

flowchart TD
  subgraph Host["Proxmox VE on a Dell OptiPlex Micro"]
    J[LXC: Jellyfin<br/>Quick Sync transcoding]
    T[LXC: TVHeadend]
  end

  Z[(Local ZFS RAIDZ1<br/>reclaimed disks)] --> J
  S[(Synology DS923+<br/>NFS, long-term)] --> J
  IP[IPTV multicast<br/>dedicated VLAN] --> T
  T --> J
  J --> CL[Clients: movies, shows,<br/>music, live TV]

Everything runs in isolated containers on a virtualization host, with media on a checksummed redundant pool, long-term storage over NFS, hardware transcoding on an iGPU that was going to be thrown away, live television pulled off a multicast VLAN, and network segmentation that reflects what the traffic actually is.

For a problem that started as "the video won't load at the gym," that's a considerable amount of yak shaving.

What it actually taught me#

The technology list is long: Linux administration, Docker, ZFS, NFS, Proxmox and LXC, VLANs and inter-VLAN routing, hardware transcoding, iSCSI, IGMP and multicast, packet capture. All of it is on my CV now, and all of it is there because I broke it and fixed it rather than because I watched a course on it.

But the list isn't the valuable part. Three things are:

Constraints are a feature. No budget meant every capability had to come from understanding something well enough to repurpose it. If I'd been able to buy a solution at each step, I'd have learned a fraction as much.

Knowing when to quit is a skill. The SAN taught me more by failing than it ever would have by working. Recognising a losing battle and cutting it loose is an engineering judgement, not an admission of defeat.

Understanding beats memorising. The IGMP bug had no tutorial. Nothing I could have memorised would have solved it. What solved it was knowing enough about how the pieces worked to reason from a symptom to a mechanism, and that only comes from having broken the pieces yourself first.

The system serves movies and live TV to people who have no idea any of this is underneath it, which is exactly as it should be. But it's the most valuable thing I've built, and none of it was on a syllabus.

It started because the WiFi at the gym was too slow. I still think it is the best thing I have done for my own career.