Colossus: Google’s File System
A file system is the system used by an operating system to organize and manage files on a storage device, such as a hard drive, solid-state drive, or USB flash drive. Imagine it as a librarian for your digital files, keeping them neatly categorized and readily available when you need them.
A file system majorly carries out the following functionalities:
- Organization: Files are grouped into folders, creating a hierarchical structure that makes it easy to find what you’re looking for. Each file has a unique name that helps you identify it. The file system keeps track of important information, that is the metadata about each file, such as its size, creation date, and location on the storage device. And finally, when you open a file, the file system locates it on the storage device and retrieves its contents for you.
- Efficiency: The file system divides the storage device into smaller units called sectors and allocates these sectors to files as needed. This optimizes storage space and ensures files are stored efficiently.
- Security: The file system provides features like permissions and encryption to secure your data. It acts as digital gatekeeper, specifying who can read, write, or execute a file. This prevents unauthorized access and modifications. Moreover, it also encrypts and effectively encodes multiple file formats by scrambling the contents of a file, rendering it unreadable without the correct decryption key. This protects sensitive data even if the file system is compromised.
- Reliability: File systems often integrate with backup tools to create copies of files for disaster recovery by using snapshots and versioning. This allows for restoration of data if it’s lost or damaged.
Colossus is Google’s proprietary distributed file system designed to handle massive amounts of data reliably and efficiently. It powers many of Google’s core services, including Search, Gmail, and YouTube.
Though Google has not released an in-depth explanation of the internal architecture of Colossus, in this blog, I have tried to cover the key components and features that enable Colossus to handle exabytes level of data at Google.
What limitations of existing file systems prompted the development of a new one?
The need for Colossus arose from Google’s rapid growth and data demands. Traditional file systems utilized by Google before (GFS: Google File System) were becoming inadequate in handling the ever-increasing volume of data generated by Google’s core services.
Colossus offered a solution with its scalability, reliability, and efficiency. Its unique features allowed Google to scale data storage and access seamlessly, ensuring smooth operation and fast performance for its services. This flexibility and power have been crucial in enabling Google to manage its massive datasets and cater to billions of users worldwide.
Here are the major limitations of the existing Google File System:
- Single master bottleneck: GFS relied on a single master node for metadata management, creating a scalability bottleneck as data volume and access requests grew. Imagine a single librarian managing a massive library — eventually, things become overwhelming.
- Limited metadata scalability: The centralized metadata storage on the master node couldn’t scale efficiently, hindering performance and hindering the management of petabytes and exabytes of data.
- High latency for real-time applications: GFS was optimized for batch processing, leading to higher latency for real-time applications like Search and Gmail. Think of searching for a specific book in a large library with one librarian — it can take time.
- Static data distribution: GFS had a predefined data chunk size (64 MB) and replication strategy, lacking flexibility for diverse workloads and storage options like flash memory. Imagine using only one type of box for all your belongings — some things might not fit well.
High-Level Architecture of Colossus File System
Here’s a simplified breakdown of its key components:
Client Library
The Client Library provides the interface for applications to interact with Colossus. Think of it as the entry point for applications to access and manage their data in Colossus.
Colossus Control Plane
The Colossus Control Plane comprises of two components, the curators and the custodians.
The curators manage metadata, storing it in BigTable. They act like librarians, keeping track of where all the files are located and their relevant information. Clients talk directly to curators for control operations, such as file creation, and can scale horizontally.
The custodians, on the other hand manage the disk space and perform garbage collection. They act like janitors, ensuring efficient use of storage space and removing any unnecessary data. They play a key role in maintaining the durability and availability of data as well as overall efficiency.
The Metadata
BigTable database stores metadata about files, directories, and blocks. Imagine this as a giant catalog card system for all the files in Colossus, efficiently tracking their location and details.
The original motivation for building Colossus was to solve scaling limits with Google File System (GFS) when trying to accommodate metadata related to Search. Storing file metadata in BigTable allowed Colossus to scale up by over 100x over the largest GFS clusters.
“D” File Servers
These file servers store actual file data. Think of them as the bookshelves where the actual content of the files resides.
Key Features of Colossus File System
Colossus achieves its remarkable scalability through a combination of innovative design and leverage of other Google Cloud services.
Dis-aggregation of Resources
Colossus provides the flexibility to provision for peak demands of real-time workloads such as streaming YouTube video with low latency, at the same time it runs batch analytic workloads cheaply by filling the gaps of idle time.
Abstraction of Hardware Complexity
Colossus abstracts away a lot of physical hardware complexity that would otherwise have been difficult to configure. To ensure every application has the storage it requires, Colossus provides a range of service tiers based on three requirements:
- I/O requirements
- Availability requirements
- Durability requirements
To handle hardware failures, Colossus performs quick background recovery to provide highly durable and available storage.
Maximize Storage Efficiency
To find the sweet spot where applications have enough storage without over-provisioning it, Colossus utilizes the concept of hot data (frequently accessed). Hot data is put on flash storage to achieve lower latency. And eventually, it is evenly distributed across all the drives in a cluster.
Conclusion
Colossus scales seamlessly to handle massive data volumes while maintaining efficient access and high availability. It exemplifies Google’s innovative approach to data storage and management, enabling its services to cater to billions of users across the globe.
By employing integration with Google Cloud Services such as Compute Engine, BigTable and Borg, Colossus provided the flexibility and power that have been crucial in enabling Google to manage its massive datasets and cater to billions of users worldwide.
