CppCon 2015: Niall Douglas “Racing The File System'

—
—
Almost every programmer knows about and fears race conditions on memory where one strand of execution may concurrently update data in use by another strand of execution, leading to an inconsistent and usually dangerous inconsistent read of program state. Almost every programmer therefore is aware of mutexes, memory ordering, semaphores and the other techniques used to serialise access to memory.

Interestingly, most programmers are but vaguely aware of potential race conditions on the filing system, and as a result write code which assumes that the filing system does not suddenly change out from underneath you when you are working on it. This assumption of a static filing system introduces many potential security bugs never mind ways of crashing your program, and of course creating data loss and corruption.

This workshop will cover some of the ways in which filing system races can confound, and what portable idioms and patterns you should employ to prevent misoperation, even across networked Samba shares. Finally, an introduction of the proposed Boost library AFIO will be made which can help application developers writing filing system race free code portably.
—
Niall Douglas is a lead author of proposed Boost.AFIO, Boost.APIBind and Boost.Outcome. He is currently the lead Google Summer of Code administrator for Boost. He is an Affiliate Researcher with the Waterloo Research Institute for Complexity and Innovation at the University of Waterloo, Canada, and holds postgraduate qualifications in Business Information Systems and Educational and Social Research as well as a second undergraduate degree double majoring in Economics and Management. He has been using Boost since 2002 and was the ISO SC22 (Programming Languages) mirror convener for the Republic of Ireland 2011-2012. He formerly worked for BlackBerry 2012-2013 in their Platform Development group, and was formerly the Chief Software Architect of the Fuel and Hydraulic Test Benches of the EuroFighter defense aircraft. He is a published author in the field of Economics and Power Relations, is the Social Media Coordinator for the World Economics Association and his particular interest lies in productivity, the causes of productivity and the organizational scaling constraints which inhibit productivity. He is presently between contracts as an expert C++ consultant.
—

*-----*
*-----*

[andreaszetterstrom]

A note about the Windows recursive folder deletion; if you try to implement this I would recommend not using %TEMP% as the temp folder. MoveFile can not move folders to different volumes so your algorithm would fail if you try to remove something from for example d: if %TEMP% is on c:

[VioletGiraffe]

I'm only 12 minutes in, and I already think every programmer who is doing any file system manipulation should watch this. I've implemented the naive recursive folder deletion more than once without realizing it can be problematic. Specifically, I had no idea file deletion system API call can be asynchronous. I have this algorithm in commercial software as well as in my cross-platform file manager. And I did get puzzling errors when deleting folders from time to time, wonder if this is exactly the issue I've been hitting. Time to rewrite this stuff!

[StefanoLanzavecchia]

Years ago, I tried to implement the algorithm shown at 12:00 to delete directory trees on Windows and, lo and behold, antivirus software like MicroTrend flag the behaviour as suspicious and quarantine the executable which attempts it. Very very sad... Every now and then I try to run my prototype and even today MicroTrend is very fast in removing the executable from the disk. I repeat: it's not matching an executable signature, because I tried different programming languages, even interpreters: it detects the pattern at execution time and marks the executable as malicious halfway through the first recursive deletion.

[Ariccio]

from a discussion on the LLVM bug tracker, why not just use SHFileOperation?

[UGPepe]

MoveFileEx has a copyAllowed flag which can be used to prevent copying. Doesn't that make it atomic?

[valetprivet]

why x couldn't be increased twice at 3:32 ?

[philg]

Asynchronous file deletion, up there with the dumbest NTFS "features" to come out of Redmond.