The program I'm working on is going to end up with a set of read-only files containing collections of structure data. My original thought was to have the data split up into reasonably sized sets across a number of files, and load an entire file into memory at once, then switch files when a structure in a different file was needed (originally the need to access a different file would come maybe once every few minutes, nothing insane).

Due to changes, I'm going to have to figure out a way to allow either multiple processes or multiple threads to read the files to serve a higher rate of requests for the structures. Creating an index file to point which structures are at what offset in what file won't be too difficult, however there's a couple things I'm not sure of in fs:

  1. fs::File::open() "Opens" the file, but if I understand, doesn't load any of the file data to memory until a Reader method/function actually pulls it. Does this mean that using fs::File -> open() -> seek() -> read_exact() approach using the index information would minimize the memory usage to what is necessary to get the structure data from the file?

  2. Documentation indicates that "Files are automatically closed when they go out of scope." I'm guessing this means that using fs::File::open() indicates to the file system that the process is using that file, and access blocking for "opened" files is handled OS side and not program side?

  3. The fs::read() function indicates it skips the intermediate variable, would this sidestep the potential issue of a process or thread "holding open" the file and blocking others?

It's entirely possible no approach is non-blocking because the OS has the final say on who can access what file when, and I need to do something like async queuing for threads needing to read. Regardless, I'm going to continue beating my head into this wall, appreciate any insights.

2 posts - 2 participants

Read full topic