Jump to page content

Multidimensional Filing System

Caveats

Most if not all pages are just rough notes, and these pages as a whole are far from complete. More notes will be added in time, eventually, maybe.

If, from reading these notes, you conclude that I am off my rocker, you won’t be the first, and you may even be right. These pages may simply position me as an acolyte to the late Gene Ray.

No doubt there are a dozen and one reasons why none of this would ever work, but perhaps somewhere deep down there is a tiny fragment that could be used for something.

Contents

Overview

The concept of a “multi-dimensional file system” was one of Jenni Elaan’s notions, and while it carried an interesting sense of promise, neither of us ever formed a clear idea of what it represented. (The change in name here to “filing system” is a nod to Acorn systems.)

Broadly, the problem at hand is the matter of classifying files and directories. One could argue that there are situations where the same file or directory should be filed under multiple classifications. There are reasons for users to do this, and reasons why the system may also need to do this. Work material or entertainment media can be classified in multiple simultaneous ways and this concept also allows items from self-contained directories to be projected out into other locations.

Owing to changes in how this concept may be implemented, this page is not all consistent or up-to-date.

Dimensions

One question never decided is exactly what form these dimensions take. There are broadly two approaches considered so far: overlaid hierarchies and tags. Both are considered here and potentially both may be used.

Overlaid hierarchy

The overlaid hierarchy idea allows for items on disk to exist in multiple simultaneous locations. Take the overview illustration below:

The item highlighted, D1, could be assigned paths A/D/D1 and W/X/D1. The item B contains its own “B” items (B1, B2 and B3), while the item Y contains the “2” items (A2, B2 and C2). This forms a basis for the application package management system in that subdirectories such as Exec, Lib and Doc can be simultaneously local to the application (as they are in Windows) and projected out into type-specific subdirectories (much more like the POSIX arrangement). This is part of the robustness principle: the only physical action needed to remove package B is to delete the directory B. This will simultaneously remove B/B1 (AKA X/B1), B/B2 (Y/B2) and B/B3 (Z/B3). This is more than simply directory hard-links: Z/B3 is truly the same item as B/B3 and removing either identity will remove the item from both locations rather than leaving a dangling symbolic link or keeping the item in existence via a hard link located elsewhere.

Physically removing item B unexpectedly may still cause issues (and as such, would be highly discouraged) but the OS could self-heal by rebuilding its package cache.

Dimension mapping can be done to both directories and individual files. For example, A/B/B1 (X/B1) may be a directory, while A/B/B2 (Y/B2) may be a file.

Tags

Tags allow individual items to be grouped together with related items. In the broadest sense they are the same principle as tags used on social media sites. From a user perspective, tags are a better-known way to classify data, but it’s not normal practice to form whole folder hierarchies from it. Tags are orthogonal: any combination of tags is valid at a data storage level even if semantically invalid. For example, it would be possible to tag an item simultaneously as “Exec” and “Lib” or “User” and “System”. Directories are protected by permissions to control who can view, read, modify and execute their contents. Tags as commonly encountered provide an asemantic set of terms with no privileges. For the purposes of classifying files on a computer system, they would need to be extended to a much more stringent form. There would need to be privileges associated with who can set what tag on what items.

Tags also introduce other complexities. Where tags are applied to files, each “dimension” would appear as the union of all items tagged as being in that dimension. This would allow any file or directory to appear in multiple places as required; a directory bearing the tag would appear as a child item and contain all of subdirectories and files regardless of how they are tagged. However, it would also allow for multiple items with the same name to exist in the same location. While it’s already likely that files would bear internal IDs (separate from filenames) this would not help users to understand which item is which.

As a note, the idea of files having internal numeric IDs independent of the display name already exists in the Macintosh HFS and HFS+ file systems, and is not unlike the POSIX notion of separating directory entries from inodes. POSIX still requires that files are accessed by name, however, and Mac OS file record tuples still used the name of a file rather than its ID. NTFS seems to have a similar internal arrangement in that regard to HFS.

Applying tags solely at the directory level would be safer but would require workloads such as photo classification to exist as as custom implementation outside of MDFS instead of simply manipulating it directly.

Within the 2019 all-tags design, tags were broadly divided into three classifications: system, state and user. System tags may only be set by the operating system, or at the operating system’s discretion. State tags provide additional state of a file, but do not necessarily affect its filtering. The details below refer to the 2019 all-tags design, but some principles equally apply to dimensions. User tags are purely for users to categorise their work.

Each tag associated with a file or directory also bears the timestamp when the tag was applied.

Each tag has a security descriptor which controls who can see the contents of the tag, and who can apply and who can remove it from files. This would also apply to the 2022 dimension system, for example preventing the Application dimensions being arbitrarily applied in an insecure manner.

State tags

State tags provide non-grouping characteristics, allowing files to be marked without placing them into a separate sub-group. Files can be hidden, or shown differently, according to these tags, but are not treated differently from their peers in terms of grouping.

State tags include:

Deleted
This forms the MDFS equivalent to the Recycle Bin or Trash. “Deleted” files are still exactly where they were before, but are simply hidden from view. The tag-applied timestamp provides deletion order, to allow users to find files that were recently deleted. The “Deleted” dimension must be counter-classified by user permission, to ensure that a user may only see what they deleted. An administrator could however flush all files so tagged.
Favourite
These are files that the user wishes to have ready access to. Tagging a file as a favourite makes this process generic instead of application specific.
Recent
This allows the OS to locate recently-accessed files. This also relies on the tag-applied timestamp to function.

It may also be possible to colour-code files as on the Mac using a set of state tags. Morever, classic Mac–style labels may be possible, using a set of state tags that each have a colour associated. Being state tags, they do not affect file classification and grouping.

Hidden files will not be implemented as the idea is too open to abuse. File visibility will be semantically controlled according to the purpose of each file. Software vendors will not be able to hide files on the computer without also affecting the status of the file in detrimental ways.

Standard dimensions

Some standard dimensions include:

Packages
All packages present on the system.
Applications
All applications present on the system. Multiple versions can exist at once but the user will only see the most recent version or the version that they are holding open. As such, this is a dynamic view.
Temporary
Items thus marked will be deleted immediately upon file handle closure and (following an unexpected shutdown) on boot. Such files are hidden from view. Immediate deletion prevents a build-up of data over time.
Disposable
Akin to Temporary but purged upon sign-out and (following an unexpected shutdown) on boot. Closing the file handle will not remove the file immediately, allowing the owning application to re-open the file. There is a subtle difference between disposable files and a (standard) on-disk cache: inactive disposable files must not be deleted while the owning process is running as they represent data that cannot be reacquired if purged. Disposable files are subject to confirming valid use-cases for the idea, according to how applications treat temporary files.
Cached
Managed exclusively by the Cache Manager, hidden from view and subject to spontaneous deletion to ease system-wide cache pressure. Preserved upon sign-out and boot.
Deleted
Items that have been deleted.
Favourite
These are files that the user wishes to have ready access to.
Recent
This allows the OS to locate recently-accessed files.

It should be clear that many of these dimensions require user-scoping, such that users and indeed system itself only see their own files. As such, a dimension may be a global concept, but it will also be user scoped and dynamically mapped according to need.

True locations and dynamic mapping

The user is rarely going to see the true layout of the disk, due to the various projections that simulate a clean and self-consistent system. For example, if an application is updated on behalf of one user while another user is signed in and using the previous version, the two versions and all the dependencies that each need will sit on disk side-by-side. This is similar to the WindowsApps system in more recent versions of Windows. However, the user never sees these separate versions: the Applications dimension only shows the best available version at that time. This will be the version that the user is using, or the latest version installed. This is covered in more detail under packages.

The true location of an application will be a disambiguated package, which appears in a canonical form under the Applications dimension as seen by the user.

It may be necessary to have a path prefix to differentiate true paths and virtual paths.

Applications

POSIX systems store the various files from packages in different locations, with no visible indication of where each file comes from and how they relate. Windows applications are more self-contained, while macOS and RISC OS applications are typically fully self-contained. The Multidimensional Filing System could provide a means to achieve both approaches simultaneously. For an example application Foo Writer, the application dimension “Foo Writer” would be a chroot-like space laid out as follows:

The application’s primary executable is tagged at the file level so that the Applications::Exec dimension is a simple list of executables akin to /usr/bin or the Macintosh’s Applications directory. Supporting applications not intended to be run separately would be stored inside the application’s directory at the discretion of the developer, not tagged as any system dimension.

This application would be added during installation to the system’s Applications dimension. This is similar to a union system except that no folder contents are merged. The Applications dimensions would then look something like the following:

That is to say, each subfolder of the Foo Writer directory would automatically appear in the corresponding Applications dimension directory as a distinct entry.

Considering the notes below on the similarity to tags, it may be that each subdirectory of Applications above would be its own dimension, with each subdirectory of the application being mapped into the corresponding dimension.

Where an application provides both a user-facing application and a daemon, the daemon executable will be placed into Daemons::Exec and its corresponding manifest in Daemons::Manifests.

The directory contents above is all that is needed to be a complete package, thus the Packages dimension is also applied. Directories in the Applications dimension are user-facing applications, while those in the Packages dimension are packages. For example, a driver package would be in the dimensions Packages and Drivers, while a daemon would be in the dimensions Packages and Daemons. Some additional thought is needed to refine the tagging process to allow all the proper identification to take place, e.g. a single package that contains a daemon and its management applications. The dimension assignment allows the installed programs UI to separate out application packages from driver packages: the user is able to identify what each package is installed for. Packages that only provide library functionality will be added to Packages and Libraries. Dependency information in the package manifests will ensure that the user cannot mistakenly remove functionality, and will allow dependencies to be cleaned up when they become redundant.

File IDs

Every file is identified to the operating system by a unique numeric ID. Ideally these would be GUIDs, so that they can be tracked across volumes: a file could be successfully located no matter which which volume it now resides on (so long as the catalogue of that volume is available for querying). GUIDs require 16 bytes of storage, and if this is considered too great, a bare minimum they will be a sequential ID on their enclosing volume, in which case they would be paired with a volume ID for successful identification. A 32-bit (“mere” 4 bytes) sequential ID would allow an implausible 4 billion files per volume, but a 64-bit integer, while being more space hungry, would be quicker to read and write.

File IDs will be accessible to users, but users will generally have no reason to see or be shown IDs; IDs exist to ensure that the OS can unambiguously locate and reference files no matter what changes occur to their names or tags. This permits open files to be renamed and re-tagged at will, just as renaming open files was permitted on the classic Macintosh. Native Event Model applications will be notified if any open files are renamed or re-tagged so that they can update their user interface. Legacy Model (I/O stream–based) applications such as classic command-line utilities will not receive such notifications, and may fail under these conditions as they are likely to be using file paths, which will become invalidated under such conditions.

Application tracing

The system needs to be able to trace all application-level material back to the application that created it. For material such as scheduled tasks that could be database-backed (e.g. configuration living inside Layer Config) rather than file-backed, it may be necessary to assign every application an ID (which could be GUIDs, or simple 32-bit integers in a look-up table) for use with material that lives outside of MDFS. This is to ensure that, upon application uninstallation, proper clean-up can be applied to avoid cruft accumulating in the system.

Paths

One problem that remains is determining how the system would build a canonical representation of the disk structure. File system exploration and any kind of recursive processing (e.g. space totalling, permissions application) must not be trapped into processing any file more than once. Microsoft’s idea of creating “Documents and Settings” and then renaming it “Users” ultimately led them to make the appallingly grievous error of a cyclic graph file system with redundant subtrees and this must be prevented. User project spaces will be the hardest to produce canonical trees for, unlike system dimensions that are well-defined.

In the 2019 version of this page, dimensions were tags that were going to completely replace paths. Each file would be identified via a canonical tag list, with paths taking the following form:

[volume](tags)filename

The volume name is enclosed inside […] and the tags list inside (…), with either one being otherwise implicit.

In the 2022 version, dimensions are suggested to sit alongside directory hierarchies and allow files and folders to appear in multiple locations according to need. Paths are likely to resemble those of UNIX systems, although paths are an inherently wrong notion for graphical systems as they prevent one or more very useful and commonplace characters being used in filenames. Although there is nothing to stop any character being used in a filename (applications would have be discouraged from blind concatenation and think of paths in a similar vein to SQL prepared statements) presenting a path to a user with either escape characters or ambiguous characters would be unwise.

Hierarchical paths can essentially take two forms: UNIX and non-UNIX. UNIX paths place a magic location outside of all volumes (removable volumes are mounted inside the namespace owned by another volume), while other systems treat every volume separately, with fixed and removable volumes treated equally.

The multidimensional principle means that the root could comprise any number of directories, made from a list of dimensions tagged as root, e.g. Applications, Volumes, Users. Under this principle, nothing would be allowed to exist directly within the root, and creating new directories within the root would be illegal.

Examples might include:

As noted, the root of each volume would itself be tagged into the Volumes dimension causing it to appear under /Volumes.

Volumes

Volume identification is as-of-yet undecided. There are pros and cons for each option:

Filenames

Each file will have a name. Under the canonical-tags-as-paths design, names were required to be unique within each tag combination, specified to ensure that paths are unique.

Names are UTF-8 and may contain any printable character including space. Attempts to use control characters (ASCII 0x00 to 0x1F) will be rejected to avoid programming error arising from failure to handle confusing names, but files with corrupt filenames will still be recoverable via their IDs.

The the canonical-tags-as-paths design, filenames could not begin with an open parenthesis “(”, as this is the start of the tag sequence, or with an open bracket “[”, as this is the start of the volume identifier. Under the conventional path basis, choosing a suitable delimiter is trickier. The UNIX and Mac choices of “/” and “:” are common human syntatic characters that should be allowed in filenames. There are a huge amount of Unicode characters that could be used instead, such as bullet “•” or arrow “→” that are impossible to type on basic keyboard and unlikely to be put into filenames even by users with access to advanced character entry. On a universally Unicode operating system, such characters could easily be added to keyboard input, causing a conflict with paths.

The use of a slash is the most appealing choice to more technical folks, but a slash is also used to write dates in many countries. While it’s also true that such dates will not currently sort correctly in filenames (only Asian big-endian slash-delimited dates will work this way) Windows has since XP sorted numbers in filenames correctly and there is no reason why the OS could not also recognise dates and sort those properly also (see also the notes on sorting, below).

For now, slash is being used here.

Tags

In the 2019 tag-based design, files are grouped and located using tags. Tags are divided up into various classifications, and within a path, tags are presented in the canonical order of dimension, username, application, user tags in alphabetical order, then finally state tags. State tags are not part of the standard path and are normally omitted. State tags are separated from the other tags with a slash.

Permissions

Background

Anyone who has spent time around NTFS knows that NTFS permissions are fundamentally broken by design. Permissions are cached on each file and directory, and these permissions are capable of (and do) get out of sync with the parent permissions. A file or directory will show that it is inheriting permissions from its parent, even though the permissions displayed are manifestly different to the parent. This is a perfect example of data normalisation failure. In addition, changing the permissions on a directory involves rewriting the permissions on every descendent, which can be very time consuming (far more so than the comparable operation on Linux).

NTFS also resets explicit (non-inherited) permissions when moving a directory: the explicit permissions are replaced by those of the new container. For example, moving a network share in Windows from one volume on a server to another (to manage storage capacity) not only deletes the share (instead of relocating it) but it also erases all the custom permissions set on that directory.

Traditional (octal) UNIX permissions are based on the user and not the containing directory, which can be frustrating and awkward as setting directory permissions does not enforce the same on newly-created files. The plus side of UNIX octal permissions is that they are simple and self-evident, and able to be shown in full detail in a simple command-line listing, while Windows permissions are unlimited in length and cannot be concisely presented.

Implementation

MDFS permissions are not yet defined. However, they will be fully normalised: inherited permissions will be read from the container dynamically. With modern RAM capacity and solid-state storage, this is an acceptable trade-off against dealing with denormalised data.

Whether file and directory permissions should be carried over with a move operation is unfortunately situation-dependent. Possibly, the ability to carry over directory permissions during a move would be an optional part of a move operation. File permissions are seldom set specifically so if this has been done, this may be worth issuing a warning on any copy or move request.

Sorting

Consideration needs to be made as to sort order. Windows since XP has sorted numbers in filenames by human standards (i.e. 1 < 2 < 12 < 20) but likely only within the shell, and direct access to directory listings seems to bypass this enhancement. (The kernel/shell split in Windows is a monumental error in general: the whole system should behave as one. Windows has a ridiculous number of separate APIs, and no good ones.) Sorting will always function identically for any user. The system may choose to operate in a neutral locale or split-neutral locale where human sorting is turned off, or directory enumeration could employ a flag to disable human sorting or disable all sorting where speed is critical (e.g. backup software that has no interest in file ordering). Disabling of all sorting is necessary due to the complexity of sorting in a Unicode world.

If slashes are to be allowed in names, then strings like “5/3/22” could be interpreted as dates and sorted accordingly, according to the user’s locale rules (as this is either the 5th of March or 3rd of May in 2022). There is no means to universally sort names unless the originating locale is stored with each file!

Storage

It is clear from the details above that MDFS volumes will require an indexed database to hold the volume contents. It is not going to be practical to walk the directory tree looking for items matching a particular tag or dimension. The HFS and HFS+ file systems for Mac OS used simple balanced tree structures to hold details on all files on a volume, paired with a separate “desktop database” file to contain file comments and file type details. This approach was however criticised for its lack of concurrency although details were not made clear as to why, and whether this limitation was more of a byproduct of the lack of threading and pre-emptive multitasking on the Mac when HFS was conceived, rather than a limitation of database engines in general.

Relational databases implement various levels of locking, and if the volume data were to be held in a database, appropriate lock granularity would be required to limit the extent to which a program could be starved of disk access.

Metadata

Dates

Copying files between volumes carries a curious pitfall: the copied files may not have the expected dates. For example, copying files from one computer to another via File Explorer in Windows will lose all the files’ creation dates. There is no distinction between the creation-of-work date and the appearance-of-file date. One might argue that files should have the following dates:

Thus, it would be possible to take a copy of a file without losing the record of when the file was originally created. This does assume of course that the application performing the copy correctly identifies the nature of the operation. Experience with Windows itself shows that handling of dates is poor. File History does not preserve creation dates (they appear to be set to the date and time that a file was backed up) and it does not restore the dates it did use. Zip files can preserve both creation and modification date, but only if a program like 7-Zip is used to create the archive.

The general essence is that, without good reason, no data should be lost in the course of activity. Creation and modification timestamps both represent user-facing data that should be preserved where possible.

Whether last-access timestamps are still considered relevant remains to be determined, as they incur a considerable overhead.

There is an argument to be made—for reasons of auditing—in favour of protected timestamps, i.e. applications should not be able to directly alter the timestamp on a file, for example to roll back a file’s date to pretend that the file was not changed. This requires the file system to have native support for situations that might otherwise lead to a need to change the date. Some of this is discussed elsewhere on the page and on the saving changes page. Situations where files need the date programmatically setting include restore from backup and when transferring files with date preservation outside of MDFS, such as via SSH and FTPS.

Attributes

Template/stationery pad

Just as with the Macintosh, templates will be a system concept rather than an application one. This raises a few questions.

Firstly, how are icons to be specified? The Macintosh principle is that stationery-aware programs supply additional icons where the first character of the four-character file type is replaced with “s”, e.g. a file type of “TEXT” is changed to “sEXT” when resolving the file icon (per Resources at the White Files). Since filename extensions are now universal, either marking a file as a template requires a filename extension change that follows a particular pattern, or the filename extension is to remain unaffected as the file format itself has not changed. Some means of generating automatic template icons would be useful for where program authors did not anticipate users creating templates from their documents.

Secondly, should this characteristic be defined using a filesystem attribute (as on the Mac), a dimension or tag, or a change in filename extension? The use of a dimension allows templates to be automatically added to an New menu, but users will want templates that are not added to that menu. The contents of any New menu would have to originate from a directory or dimension where items could exist without the relevant attribute set.

Finally, some means is needed to instruct the application to handle the template. The Macintosh approach of stationery-aware applications may suffice, unless there is a way to ensure that all applications incorporate the relevant logic. Office applications add an Edit context menu command to templates (so long as shift is held), and this behaviour needs to be made system-wide, to allow users to edit templates without needing to fiddle with the file status. This requires applications to be made aware of whether the file is to be cloned as a new untitled document, or to be edited directly.

User metadata

Some degree of user metadata is always healthy. Examples include:

User metadata should also apply equally to directories. Ideally the implementation would be identical; see below under hidden files for further comments. Each metadata item would have a type key that references a binary stream, akin to file system forks or NTFS alternate data streams. There is a caveat that this data would not be transferrable to all other file systems, and either these features would simply not function on volumes without adequate metadata support, or some kind of hidden files would be needed to store it (as seen on macOS).

There is a possibility for applications to be allowed to store additional metadata for purposes not defined by the operating system. This would be an accountability violation however as it would allow applications to have secretive implementations that users would have no way to understand and troubleshoot. System-defined metadata types would be guaranteed a means to observe their existence and alter or remove the information, while application-defined metadata would have no meaning to the operating system, which in turn would have no means to display the data.

Dynamic maps

Dynamic maps are process-specific aliases between canonical paths and individual packages. When an application is launched, each reference to a package generates a dynamic map to the selected version. For library packages, the path in /Library to the package is aliased to the most recent version acceptable to the application.

When an application is updated by one user while it’s still open by another (e.g. on a terminal server), the application process’s dynamic maps to the application package and all library packages are repointed to the now deprecated packages in use within that process. Once the application is closed and the dynamic maps are deleted, further requests to map the application package will access the most recent package version.

Hidden files

Hidden files and directories are a common choice on various operating systems. Examples include:

Hidden files and directories are unfriendly, and allow the accountability principle to be violated by creating invisible implementation details. Ideally, hidden files would not exist at all.

Configuration data on UNIX and Linux systems uses hidden files and directories due to a lack of proper system organisation. There will not be configuration files of this nature under Layer Config as per-user configuration will be held in its own dimension out of sight. Application runtime files and long-term storage will have a dimension equivalent to AppData in Windows or ~/Library in macOS.

Folder configuration will not need a hidden file either. Folder characteristics will be set by dimensions or file system metadata.

Windows has deprecated thumbnail databases, although even the most recent version of Windows 10 (22H2 at the time of writing) retains the bug where File Explorer cannot delete a folder with a Thumbs.db file in due to being too stupid to realise that File Explorer itself is holding the file open. There is no current proposal for how thumbnail caching is best handled. Windows historically used NTFS alternate data streams where possible, but this is impossible with FAT-based media. Whether thumbnails should be a property of the images (stored within or alongside the images themselves) or a property of the user (stored within the user’s local profile and duplicated for each user viewing shared material) needs to be assessed.

Custom icons would hopefully be achieved using rich metadata associated with the directories themselves, rather than stored in a special hidden file. The custom icon system in classic Mac OS is a great idea, but it relies on resource forks and has the limitation that setting a custom icon on a file that already had its icon set that way (as was required to get 32-bit icons on applications) would permanently erase the initial custom icon. To get a custom icon on a directory, a hidden file is created so as to associate a resource fork with the directory. Resource forks are unlikely to be introduced into MDFS, but rich metadata—somewhat akin to unlimited forks or NTFS alternate data streams—will ideally apply equally to directories as it does files, and thus any use of this for custom icons will not require special files be created.

Another use for the Hidden flag in Windows is to bury irritations. For example, some applications continually recreate their desktop shortcuts with every update, even if the user originally stated that no such icons should be created. The only way to fight this disease of stupidity in Windows is to hide the icons, as many installers will fortunately neither recreate the shortcut nor remove the Hidden flag. This problem is solved by declarative software packages where the installation process is controlled by the operating system. Another such irritation is folders created for no good purpose or created in the wrong place, such as software (including Microsoft’s own Outlook and Feedback Hub of all things) that puts application data under Documents instead of AppData, or creates worthless folders in Documents or the profile root such as “3D Objects”. Since these folders are either mandatory or forcibly recreated, the only way to fight this ineptitude is to hide the folders. There is no way to enforce that software vendors uphold correct practice, so some consideration is needed to ascertain how to encourage tidiness to the extent that the user does not need to be at odds with the computer.

Hopefully there will be so little call for hidden files and directories that they can be omitted entirely, at least on MDFS volumes. The VFS will still need to understand the concept in order to correctly handle other file systems.

Suspended metadata

Microsoft SQL Server manages to write to database files without triggering file timestamp updates. An SQL Server database can go for months without showing a file change, making the database look dormant. This is likely a performance concession to avoid the overhead of continual timestamp changes, although this raises the question of whether a file’s modified time should be updated only when the file is closed, or during some or all writes while the file is held open (as it would be for a database).

More confusingly, programs under Microsoft Windows can create files on disk without actually causing NTFS to see the space allocation, e.g. Sage maintenance. The disk can run out of space, but a program such as TreeSize Free will fail to show the cause because the offending files are showing the wrong size. This seems to persist even when the files are closed and if anything seems to be another bug in NTFS.

There needs to be a formal specification on deferred/suspended metadata changes: the reasons for doing this, and the means of mitigation, e.g. a command to flush pending metadata changes.

Versioning

There are different approaches to file versioning. Apple and Microsoft require the use of backup storage to hold backups of local files, and Microsoft OneDrive stores the previous versions in the cloud.

For smaller files, simple copy-on-write within the same disk would offer a cloud-free, backup-free offering (Windows File History does not maintain every individual version). Potentially, the previous versions could be cleared from the disk after a certain time limit (to release disk space) following confirmed backups of the files. The danger of copy-on-write versioning onto the same disk is that disk space will deplete without any apparent cause; pretending that no space is occupied by copy-on-write versioning will cause the loss of previous versions if the user tries to use the “unused” disk space for something else and the previous versions are removed to honour the free space claimed by the OS.

What should happen when a user requests that a file become an older version? There are (at least) three possibilities:

Note that one must be very careful when it comes to files that are repeatedly written to, such as log files. (Notwithstanding the discussion under logging and the question of whether any programs would ever have traditional text log files.) If a program opens a log file temporarily to write each line, there will be a separate version for each change, which would not be a good idea.

File open modes

The different file open modes that could conceivably exist include:

Observable write allows a file to be held open for write with other processes able to read the contents (with the caveat that they made read the file in an inconsistent state). This allows for a log file to be held open continually, while still allowing it to be tailed. Observable append would be more sensible.

Cloud storage

Programs like Dropbox and OneDrive are pretty gnarly when it comes to trying to “merge” local and cloud storage as if it were the same thing. OneDrive for example is incapable of underlying the cloud below an existing directory and forces all your directory paths to change.

OneDrive chokes to death on larger directory trees (especially with SharePoint), as it tries to juggle all the files in its head and falls over when this becomes too painful. In general, OneDrive is very fragile and difficult to maintain.

Cloud storage that underlies local storage needs to be a native feature of MDFS that is independent of the service provider, and implemented with a proper listener model that waits for both local and remote change notifications and doesn’t track any files that are not being changed. There should be no need for a database of individual files as the only direct mapping would be between local and cloud directories, with all their contents implicitly mapped between the cloud and local storage automatically.

Cloud synchronisation status (on a file level) and directory mapping would be native to MDFS and able to be implemented properly from the outset instead of bodged into an existing file system in a way that—as one sees with Dropbox and OneDrive—is prone to failure.

Unsubscribing from a cloud directory would immediately remove all the local material instead of (as with OneDrive) inexplicably leaving it all behind.

The system would be self-healing by way of a simple synchronisation pass akin to a Robocopy or rsync pass, with the exception that files only found in the cloud would only be downloaded in full if the service is set up for automatic full offline copies. Once this pass completes, you would have the same files in both locations.

Some means would be needed to accommodate version conflicts where the new file has been changed locally while working offline, and separately changed within the cloud. This could be done by simply forking the file, but this interferes with the file’s name and history.

Exactly what happens of cloud storage is set to underlie a versioning file system is unclear, as the cloud provider can be responsible for holding the versions directly (as with OneDrive). OneDrive does not appear to support copy-on-write, meaning that putting an actively-used Outlook PST file into a folder backed by OneDrive will cause it to keep being re-uploaded in full. The cloud storage system could instead behave identically to a local versioning system with copy-on-write, where no matter how large the file is, so long as only a portion of it is changed, there is no large re-upload.