ScoutFS
SCALABLE METADATA
MANAGEMENT
ScoutFS is an open-source Linux filesystem specifically created to support mass storage and archiving applications. Metadata is processed on all nodes or a subset of nodes in a ScoutAM cluster.
In a ScoutFS cluster there is no central metadata controller nor any single point of failure. ScoutFS introduces many outstanding new capabilities by radically increasing the number of POSIX files and objects that can reliably be maintained and searched within a global namespace. ScoutFS was designed to manage hundreds of billions of metadata records.
SCOUTFS FEATURES
Open-Source
ScoutFS is an open-source GPLv2, in -kernel Linux filesystem. Open-source metadata combined with open-source data formats gives customers complete control over data collections and aligns with long-term data preservation and autonomy goals.
Exabyte-Scale Capacity
The ScoutFS global namespace is designed to accommodate hundreds of billions of files and objects while delivering responsive search and query results. Mass storage capacity is > 100 Exabytes per system.
Scale-Out
ScoutFS scales horizontally by adding server nodes. Each nodes in a ScoutFS cluster participates in metadata and data management, enabling the system to achieve higher levels of performance with additional compute resources.
Metadata Separation
ScoutFS stores metadata on a separate device from data. Metadata separation allows ScoutFS to direct extremely high metadata IOPS workloads to smaller higher performance NVMe or SSD LUN’s, while allocating data reads and writes to different devices that are optimized for streaming throughput. In extreme-scale environments, the metadata storage may be directed to a dedicated controller to ensure maximum performance.
Highly Availability
The ScoutFS scale-out architecture allows nodes to enter or leave the cluster without disruption. Due to the journaled and atomic design, work is never lost during a node failure. Failover within a cluster is fully managed without the need for external HA frameworks.
Advanced Indexing
Atomic metadata and data sequence numbers are indexed so both inode attribute changes and file content changes may be quickly discovered within the system. Indexed change sequence information eliminates the need for routine file system scans.
Metadata Tagging
Highly scalable user defined and API driven custom metadata tagging is supported to enable the creation and management of enhanced metadata. Enhanced metadata can be searched with responsive performance even among very large file and object populations.
Fast Dump & Restore
ScoutFS supports fast metadata dumps for greater convenience and enhanced system protection. Metadata restores benefit from extremely high file creation rates in the ScoutFS multi-node system. Incremental file system dumps avoid wasted processing time by skipping items that have not changed.
ScoutFS Architecture
The ScoutFS cluster architecture takes advantage of technologies that have emerged in recent years including the commercialization of NVMe devices. ScoutFS stores metadata and data on separate devices. This separation allows low capacity but high performance metadata workloads to be concentrated on relatively small quantities of NVMe or SSD devices, while data is stored on either SSDs or conventional disk arrays optimized for streaming read/write performance. The ScoutFS filesystem was developed specifically to address the unique and challenging requirements of mass storage and archiving workloads.
“Versity is providing us with the best support we have ever had. Even before our subscription started, we felt like Versity was delivering a professional services engagement. The solution has exceeded our expectations.”
— Vice President of IT, International Non-Profit Organization