EXTRA BITS - Data Security and Flash Memory - Computerphile

Derek McAuley is professor of Digital Economy at University of Nottingham's School of Computer Science.

This video was filmed and edited by Sean Riley.

[ociemitchell]

Brady, these are literally the extra bits. Billions of them!

[armouredhen]

I really enjoyed the guy who spent a few episodes talking about old computing technology . If you can, i would love to see more videos about old technology. Thanks Brady!

[sebbythecrabby]

'Extra bits' ... very clever!

[rocko]

It's just the way the file system is reading the memory. Each step up in byte suffixes (e.g. kilobyte, megabyte, gigabyte) are all by factors of 1000. The file system uses factors of 1024 instead, because that's the standard for RAM and other flash memory types. Hope that helps

[IstasPumaNevada]

It's probably a balance between more robust storage (that then needs less overhead), and more overhead (that then needs to be less robust), and that the exact choice of balance is based on whatever costs the least. As such, I imagine it varies from manufacturer to manufacturer, and perhaps even from model to model.

[boenrobot]

There are in fact already programs that do this for "normal" HDDs. I'm not sure if they'd work on a USB flash drive though.

Keep in mind that there's a reason those are hidden ESPECIALLY on normal HDDs - when a sector becomes unreadable, you want it remapped, and if you end up with no "spare" sectors left, the damaged one would be permanently un-writable, as opposed to the OS/controller remapping it. In that event, you'd have to use the unlock program again to relock the drive to a lesser size.

[TheGNUfan]

There's three reasons for this (afaik):

1. Manufacturers consider a kilobyte to be 1, 000 bytes, while in computing it's 1, 024 bytes. This adds up on large volumes.

2. The manufacturing process isn't perfect.

3. Modern filesystems create a journal that tracks changes that will be made before committing those changes to the filesystem to prevent corruption. Journals can take up a lot of space on large volumes.

[StaffanThomen]

No, filesystem overhead isn't shown like that, this is sort-of what lazyeye79 said. The Capacity is written in bytes, all is well and good, but the giga-prefix used in computing terms is a multiple of 1024 not 1000, hence:

2000158720/(1024^3) = 1.86279296875 rounded down to 1.86 GiB

The abbreviation GiB or Gibibytes (MiB mibibytes, KiB, Kibibytes, etc) is a standard that never caught on to differentiate multiples of 1000 vs. multiples of 1024. The 1024 comes from the binary 2^10.

[ashwith]

Performance drop is fine. For me, it'll be awesome to just see that size increase. I don't intend to do this with every flash drive I get. I will dig deeper into this early next year.

[kght]

how this relates to security is basically because if you try to erase the data from a thumb drive, even if you have it flip everything to 1 then back to 0 (however many times) there will be a corner of it that won't likely be touched. this matters a little but not much if you use encryption from the start because the data in that corner will be encrypted and likely unusable even with the key after you delete the rest of the drive.

[gulllars]

Most SSDs have 7-10% "spare area". This coincides well with the GiB -> GB conversion, so manufacturers can sell a physically 128GiB drive as a 128GB drive. Binary to decimal conversion. 2^10 becomes 10^3 (kilo), 2^20 becomes 10^6 (mega), and 2^30 becomes 10^9 (giga). for every of these steps there is a 2, 4% mismatch which accumulates. 1024 -> 1000. At Giga that means a 9, 26% missmatch.
Some SSDs can come with up to 15-25% spare area, but are then sold as 100GB or 120GB instead of 128GB.

[suit]

it depends on the purpose of the limitation - if it was done just to spare a production line, probably yes - but if it is done for yield optimization (for example they make 8 GiB chips where 2 GiB are defective, they can at least sell it as 4 GiB drive and use the extra space for wear leveling and stuff) you probably will unlock defective spaces

usually its not worth playing around with this kind of stuff sinces smaller flash memory is cheap and bigger stuff (like SSDs) get really complicated

[thebiglightbulb]

because they(the manufacturer) counts to 1000 instead of 1024

[suit]

of course - but in semiconductor manufacturing the yield for high end products is very small - so you have lofts of defective stuff. its pretty sure that, if you get a bigger flash memory IC sold as a smaller one it is not limited just for fun or economic purpoposes, it if often because of defective parts which are disabled

you can't see that from the outside but they are there

[devinpohl]

"They actually have 6 GB flash drives now!"
1 year later
"They actually have 6 TB flash drives now!"

[lazyeye]

While that may be true, what is more likely is: when a manufacture sells a 1gb flash drive they are referring to 1gb as 1000mb, but your computer counts 1gb as 1024mb, so to the computer it is shy by 24mb, which is why it displays it as less that what the manufacture says.

[ShreddySteve]

If you manage to do that, it would be AWESOME! I wish I knew more about programming and stuff. I think I'll speak to some computer engineering friends and see if they can come up with something!

[gulllars]

I seriously doubt that. The inners may LOOK exactly the same, but if you read on the packaging you will find that some packages contain 1 or 2 "dies" (stacked if more than 1), while others can contain up to 8 or even 16 "dies". A die is a piece of silicon with circuitry "printed" on it. This way, the physical devices are almost identical, you just have more stuff in the package for the higher capacity drives. Since the dies are µm thick, the higher capacity packages aren't physically bigger.

[Keelyn]

Thats the way its always worked. Out of a waver you get quite of number of microchips. The quality of the chips decreases the farther away they were from the center of the waver. The better ones are usually sold later for a higher prize or implemented in the premium models of a hardware architecture while the worse once are being sold for a cheaper prize or implemented in low-budget hardware.

[DFXKX]

yeah, they're very rarely exact, some of the filestructure takes up that space, as well. I, for instance, have a 90gb OCZ SSD (basically nothing more then a big, fast flash drive) I can only use 83 gb, though. I'd love to be able to override the microcontroller on such SSDs, because then you could use them for RAID more efficiently.