Arcanum 'Dat' Files

I talk about how Arcanum stored its game data files in big monolithic files...and why we did that.

Video I reference:

[TheGameStarter]

I love it when you do a bit more technical video like this one. Lots of great insights. Thank you, Tim!

[talideon]

Another reason, especially useful for spinning rust, is that it helps avoid disc seeks by grouping together data that's likely to be needed together.

[Anubis]

Man, it's surreal to hear you talk about this stuff. I remember back to my modding days, unpacking the DAT files, playing with custom backgrounds and item data.
Thanks for going over this stuff!

[imALazyPanda]

I just want to thank you for designing with modders in mind. Ive created a few in my time and am working on a big one with a friend for a game we love but is not mod friendly, they allow them but dont have a framework really for them. Devs who publish a modding tool are greatly appreciated.

[motyd]

Entries is later files overriding the same entry from an earlier file is also the method used by the classic Doom games when loading data from the .wad files. Same technique.

[extremepayne]

You say space is no longer at such a premium, but these days a 256GB drive can only have 1-2 modern AAA games on it. Optimization is always appreciated

[joshuakirkham]

This is really useful info (for thinking about gaming files). Thank you for the break-down as to why this was chosen.

[mementomori]

I don't understand all this tech mumbo jumbo but enjoying listening to people speak about their passions

[zckhyd]

this is one of my favorite videos of yours because the information is so specific and concrete. What you are saying helps me understand computer hardware better. May I request a thread of videos where you talk about how big-name games use memory and hard drive space and how/why our favorite games perform the way they do in their hardware environments? Also, please do an on-screen introduction to your dog or dogs.

[PhantomV]

I was reading about pointer fix-up tables and how to store game data in a binary file and reading them using their file offset yesterday, but I couldn’t understand why anyone would do this. Then, this video came out… Creepy, but much appreciated!!!

[joaolimaoyt]

Great technical video as always. The number of file descriptors a process can have open at the same time is typically limited by the OS so a process opening too many files at the same time may run out of file descriptors and then that process would need to close files before it can open new ones. Packing assets into DAT files helps with this problem too. Regarding compression, in addition to whole file compression it's also possible to compress individuals entries within an archive even if the whole archive is not compressed. This helps reduce the amount of IO performed. IO is typically slow and there are fast decompression algorithms out there so compression can actually end up speeding up loading times.

[vieleanimations]

really valuable insight. loving those technical explanations

[GeezNutz]

Love hearing The Dog snore through the whole video lolol

[TheLeadZebra]

I remember hearing that Titanfall used uncompressed sound files in order to free up processing for some improvements in graphics. This was probably the first time I considered how things like that might come into play for development, before then I just knew of zip files for improved downloads

[DBagg-zzip]

That’s good info about sectors and seeking. I’ve come across the “how many files to use” question in my own project.

[SianaGearz]

I remember playing some cheaply developed go-kart racing game in 1998, and it took an ETERNITY to load, because of all the tiny files.

Most games that use some sort of pak/big/dat files, they actually have a 2kb placement pitch for individual sub-files within those archive files. Apparently the reason is 2kb block (similar to sector) size on optical media, chosen to be less than a single revolution of a CD, makes seeking or file operations quicker or something on optical media in particular. You don't really care about space wasted on optical media, at least not back then, but you care a lot about placement, since the pickup transport takes almost a second end to end and a third of a second between random spots, it's super slow.

In turn how much space the smallest file takes on computer is not only dependent on sector size but on FS cluster size. If you built your computer in 1995 with a 1300MB hard disk, which was extremely generous at the time, you'd have cluster size of 32KB on FAT16B if it was a single partition. With several partitions or smaller drive, 16KB was a more common value.

[Janonas]

This proves an Arcanum remake is in the works.

[seanothepop]

I remember being on the lookout for an arcanum module that had the player create a "team" instead of just one character, usually 4-6 for some games, though arcanum allowed lots of followers script wise (endless with that bug where you mentally control the npc, then meta magic on them, then cancel the control then meta but they permanently are yours and the limit was when your game crashed) but a good party where you governed the growth of everyone and their origins would be neat. Loved artwork too and was my first and favorite middle modern rpg.

[Pedone_Rosso]

I will always remember when my cousin and I went to visit a friend of a friend of mine,
in the early 90s, when we were around 15 or so.
This guy, who's parents were loaded, had just put together his new top of the line gaming PC,
and he had clearly gone far beyond overkill with the specs.

After a few seconds with our mouths agape,
just trying to grasp the unfathomable amount of space his HDD had,
we asked:
"One GIGABYTE !?"
(Yep, the reference to Back to the Future was still second nature at the time... LOL!)
"... What the hell do you do with 1 GIGABYTE of disk space?"
His answer:
"Well... I just enjoy splashing around in it!"

I mean, that's maybe nothing compared to the jump to Pentium with HDD
from systems like the Commodore 64 with its tape recorder.
But still, hardware related numbers didn't slow down their improvement that much in the last 30 years.

Thanks for your videos!

[michaelbolland]

Love the relaxing iron-lung sounding doggo 🐕🤣