File system – Part 1: Platform-specific API design

Typically, a file system in a game engine comes with all the bells and whistles like multiple file devices, support for zipped files, encryption, device aliases, asynchronous I/O, and more. But buried underneath every file system lives a low-level implementation (directly using OS/SDK functions), which in turn is used in the high-level file system.

This low-level implementation should be the only platform-specific file implementation in the whole file system, and is the topic of today’s post.

In the Molecule Engine, the low-level file implementation is called OsFile, and handles both synchronous and asynchronous requests.

The synchronous part of the API is pretty straightforward:

class OsFile
  OsFile(const char* path, FileSystem::Mode mode, bool async);

  // Synchronous API

  /// Synchronously reads from the file into a buffer. Returns the number of bytes read, or 0 if the operation failed.
  size_t Read(void* buffer, size_t length);

  /// Synchronously writes from a buffer into the file. Returns the number of bytes written, or 0 if the operation failed.
  size_t Write(const void* buffer, size_t length);

  /// Seeks to the desired position
  void Seek(size_t position);

  /// Seeks to the end of the file
  void SeekToEnd(void);

  /// Skips a certain amount of bytes
  void Skip(size_t bytes);

  /// Returns the current position in the file, or INVALID_SET_FILE_POINTER (0xFFFFFFFF) if the operation failed.
  size_t Tell(void) const;

  // Asynchronous API


I choose not to use or expose FILE_BEGIN, FILE_CURRENT and FILE_END (used for seeking) in the API itself – it’s used internally, but the user shouldn’t have to bother with it. It’s just a relict of the olden days of real stream-based devices anyway, and that’s the reason why Seek(), SeekToEnd() and Skip() exist. Other than that, the synchronous part of the API just does what it says on the tin.

What about the asynchronous API? One solution I have seen in the past was something like the following:

class OsFile

  // Asynchronous API

  /// Asynchronously reads from the file into a buffer
  void ReadAsync(void* buffer, size_t length);

  /// Asynchronously writes from a buffer into the file
  void WriteAsync(const void* buffer, size_t length);

  /// Waits until the asynchronous operation has finished
  void WaitForAsync(void);

  /// Returns the number of bytes transferred in the asynchronous operation
  size_t GetAsyncBytesTransferred(void) const;

Unfortunately, this solution has a serious flaw: You can never initiate more than one asynchronous operation at the same time, because otherwise the calls to WaitForAsync() and GetAsyncBytesTransferred() would be ambiguous. With asynchronous I/O you can never be sure which I/O finished first, hence something like the following won’t work:

OsFile* file = [...];
file->ReadAsync(header, 256);
file->ReadAsync(data, 65536);
file->WaitForAsync(); // which one finished?

Part of the reason for using asynchronous I/O is that you can overlap read and write operations, and do several asynchronous operations at once. I’ve worked with sound middleware which explicitly used those features, and integrating it into the above system was not nice. Hence, let’s strive for something different.

An alternative, better solution would be something like the following:

class OsFile

  AsyncOperation* ReadAsync(void* buffer, size_t length);

  AsyncOperation* WriteAsync(const void* buffer, size_t length);

  void WaitForAsync(AsyncOperation* operation);

  size_t GetAsyncBytesTransferred(AsyncOperation* operation) const;

By returning a unique AsyncOperation instance, the API can use that as an identifier/token for the corresponding, internal asynchronous operation, so multiple reads/writes now work:

OsFile* file = [...];
AsyncOperation* op1 = file->ReadAsync(header, 256);
AsyncOperation* op2 = file->ReadAsync(data, 65536);
// header has been read at this point, data reading still carries on

But still, there is a minor quirk in this design:

  • Ownership of the AsyncOperation instances is transferred to the user, meaning that each AsyncOperation has to be delete‘d manually at the calling site, which is error-prone.

Keeping that in mind, the design I use in the Molecule Engine is the following:

class OsFile

  /// Asynchronously reads from the file into a buffer
  OsAsyncFileOperation ReadAsync(void* buffer, size_t length, size_t position);

  /// Asynchronously writes from a buffer into the file
  OsAsyncFileOperation WriteAsync(const void* buffer, size_t length, size_t position);

As you can see, waiting for an operation and getting the amount of transferred bytes is no longer part of the OsFile API. Instead, OsAsyncFileOperation is a self-contained object which keeps track of the asynchronous operation internally, and offers the following API:

class OsAsyncFileOperation
  OsAsyncFileOperation(HANDLE file, size_t position);
  OsAsyncFileOperation(const OsAsyncFileOperation& other);
  OsAsyncFileOperation& operator=(const OsAsyncFileOperation& other);

  /// Returns whether or not the asynchronous operation has finished
  bool HasFinished(void) const;

  /// Waits until the asynchronous operation has finished. Returns the number of transferred bytes.
  size_t WaitUntilFinished(void) const;

  /// Cancels the asynchronous operation
  void Cancel(void);

  HANDLE m_file;
  ReferenceCountedItem<OVERLAPPED>* m_overlapped;

Because OsAsyncFileOperation stores a reference-counted object internally, it can be copied, assigned, and stored somewhere. The copy-constructor and assignment operator increase m_overlapped‘s referece count, and the destructor delete‘s m_overlapped if its reference count reaches zero:

OsAsyncFileOperation::OsAsyncFileOperation(const OsAsyncFileOperation& other)
  : m_file(other.m_file)
  , m_overlapped(other.m_overlapped)

OsAsyncFileOperation& OsAsyncFileOperation::operator=(const OsAsyncFileOperation& other)
  if (this != &other)
    m_file = other.m_file;
    m_overlapped = other.m_overlapped;

  return *this;

  if (m_overlapped->RemoveReference() == 0)
    delete m_overlapped;

This API allows for simultaneous asynchronous reads/writes, and is completely non-ambiguous to use:

void Store(const OsAsyncFileOperation& op)
  // hold on to operation somehow

OsFile* file = [...];
OsAsyncFileOperation op1 = file->ReadAsync(header, 256);
OsAsyncFileOperation op2 = file->ReadAsync(data, 65536);
const size_t headerRead = op1.WaitUntilFinished();
// whenever op1 and op2 go out of scope, the internal object will be automatically deleted
// and still, the following would work too:

In the next installment of the series, we will see how the high-level file system makes use of the low-level OsFile.


6 thoughts on “File system – Part 1: Platform-specific API design

  1. Pingback: File system – Part 2: High-level API design | Molecular Musings

    • Because real asynchronous I/O has nothing to do with calling synchronous APIs from a different thread.
      Real asynchronous I/O enables reading directly from the hardware, bypassing any internal OS and/or driver buffers. On certain consoles, you almost have direct access to the optical drive. This often comes with certain limitations:

      • Destination buffers must be aligned (e.g. to a 4096-byte boundary).
      • Only multiples of the sector size can be read (e.g. only multiples of 4096 bytes).
      • Reads are only allowed to start at multiples of the sector size (e.g. it is invalid to seek 3000 bytes, and read from there).

      This makes reading of arbitrary-length data at arbitrary positions a bit more complicated, but reads directly from the hardware into memory. Synchronous APIs often do nothing more than reading data asynchronously into OS buffers, waiting until the transfer has finished, and copying the data into the user-supplied buffer. If you want to have fast loading times, you need to go asynchronous. Furthermore, synchronous APIs make it hard (or even impossible) to implement technical requirements on certain consoles.

  2. Hey Stefan !

    I tried implementing these classes but ran into a problem.
    When using the asynchronous API, the calls to functions like ReadFile/WriteFile are blocking even though I use an OVERLAPPED structure. The functions return FALSE and GetLastError() returns ERROR_IO_PENDING so for the OS, the operations seem asynchronous but my thread is still blocked for a certain time. Then, the call to GetOverlappedResult() returns immediately meaning that the operation is already finished.
    I saw some people having the same problem on several forums but found no answer.
    I know there are several cases where asynchronous I/O can become synchronous (Compressed/Encrypted files, etc.) but in my case, I’m just reading a large non-compressed, non-encrypted file, that’s why I would like the functions not to block !
    Did you have the same problem? Do you think I should spawn a thread to handle these calls?
    I’m on Windows 10.

    Thanks in advance.

    • I think I saw similar behaviour when reading small files because Windows would decide to just read it synchronously anyway. Other than that, asynchronous reads from large files seem to work fine. Writes are an entirely different beast altogether though, I think I never used them in practice. I remember reading something about asynchronous writes on Windows on Charles Bloom’s blog.

      For reading engine resources I would recommend doing that from a separate thread.

      • OK, thanks for your answer.
        In my case, I was reading a 200 MB file and ReadFile() took around 60 ms to complete according to Visul Studio, so no async for me, I’ll go with the separate thread for now. My Hard Disk is quite old so this may be the reason…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.