Phase 7: Storage & File Systems
Explore the concepts of Storage & File Systems.
Blob Storage
Object storage (also marketed as blob storage) keeps large, unstructured files - images, video, backups, logs - as self-contained objects in a flat namespace accessed over HTTP, rather than as rows inside a database. It is the default home for big binary data that does not need transactional queries. 🧠 Mental model: Object storage is like a warehouse with numbered shelves. You give it a box (file), it gives you a shelf number (URL). You can't search inside the boxes - you just fetch them by number. It's cheap, huge, and simple.
Distributed File Systems
A distributed file system (DFS) allows multiple clients across a network to read, write, and manage files as if they were on a single local hard drive, even though the data is physically spread across multiple servers. It provides a hierarchical namespace (directories and files) and often supports POSIX-like semantics. 🧠 Mental model: Think of a shared network drive in an office (like a Z: drive). Everyone sees the same folders and files, but the actual hard drives are in a server room down the hall. A true DFS just scales that concept to thousands of machines.
Object vs Block vs File Storage
These are the three fundamental ways to store data in the cloud. Choosing the right one determines your system's performance, scalability, and cost. 🧠 Mental model: Block Storage: A blank notebook. You can write on any page, erase any sentence, and it's fast. File Storage: A filing cabinet. Folders inside folders, organized hierarchically. Object Storage: A valet parking lot. You hand them a car (file), they give you a ticket (URL). You can't change the car's radio station while it's parked; you have to retrieve the whole car, change it, and park it again.
Data Lakes vs Data Warehouses
Data Lakes and Data Warehouses are central repositories for an organization's analytical data, but they differ fundamentally in how data is structured, stored, and queried. A warehouse is highly structured and optimized for business reporting; a lake is unstructured and optimized for raw scale and data science exploration. 🧠 Mental model: A Data Warehouse is a library where every book is cataloged, shelved by genre, and easy to find, but hard to add new unformatted material to. A Data Lake is a massive warehouse where you dump boxes of books, loose papers, and hard drives in a pile. It holds everything cheaply, but you need a data scientist to sort through it.