Tuesday Tech Tip - ZFS Read & Write Caching

Each Tuesday, we will be releasing a tech tip video that will give users information on various topics relating to our Storinator storage servers.

This week, we tackle a commonly asked question here at 45 Drives. Brett talks about a how ZFS does its read and write caching.

Be sure to watch next Tuesday, when we give you another 45 Drives tech tip.

[David_Quinn]

I love these tips, I didn't think I would learn anything from this particular video but I did.

[alekpinchuk]

The key point that's missing here is that ZIL/SLOG is only useful for sync writes (and internal sync operations). It will not be used for async writes so you don't actually need a SLOG if your workload is async.


The other minor detail is ZIL/SLOG is not really a write cache since it's never read if the server doesn't crash. The ZIL/SLOG contents are discarded after the regular TXG mechanics complete for the writes that were stored in the ZIL.

[TechWithYouVee]

Such an easy explanation in layman's terms ❤️

[bertnijhof]

Till December I used 3 HDDs and one 128GB SATA SSD, I used half the SSD as boot device and the other half as ZLOG and L2ARC. SSD and HDDs are LZ4 compressed. L1ARC used 20-25% of memory (max: 4 GB of 16 GB). I had instantaneous response times due to the L1ARC. Due to L2ARC the boot times of my virtual machines were almost equal to the boot time of the host OS, say ~10% more. Both results only were achievable after the caches had been filled sufficiently, so mostly after reloading the program or rebooting the system.

Afterwards I have reorganized my system with a 512 GB NVME SSD (3400/2300 MB/s). I run the Host OS and the Virtual Machines from that SSD. I still use 1.5 TB of 2 HDDs for archives, music, office documents, videos, photos etc. Archives I use once a month. Music, office documents, videos and photos run perfectly from two striped HDD partitions, say at 240 MB/s. Long ago I played that music and those old movies from an USB 2.0 HDD. I have absolutely no need for L2ARC or ZLOG, also because these writes are sequential file IOs and thus asynchronous, bypassing any ZIL!! The difference between booting a VM using L1ARC (basically a reboot) and the initial boot is ~10%, so I reduced my max L1ARC size to 2 GB of 16 GB.

This were of course a valid ways of caching for my situations and use case. That is a single user desktop mainly used with VMs based on a relative slow CPU (Ryzen 3 2200G) and now a fast NVME SSD (Silicon Power (SP), 3400/2300 MB/s). I think, that SP is optimized for program loading and short bursts, but it is less optimal for large file transfers.

[jonathanchevallier]

Thank you for this great explanations. With fun analogies ! ;-)

[dylanp]

Thank you guys for making this information available, I have learned a lot from these videos as well as the documents on 45drives.com. And yes, please make a part 2 on where these would be best implemented. Also, is the SLOG helpful for single (large file) writes? I've heard it only helps with multiple smaller files being written at once. Another thing I have been working on, Is there an easy way to get disk info and alerts from the storage server? I have been using smartmontools. Thanks again!

[olafprzybyszewski]

This is super awesome Brett, your explanation how ZFS and other IT stuff is behave and working. I saw a lot of your great movies about type of Clusters, etc. Very good job! PS. Please if you can do more about performance of GLuster, Ceph compare to hardware (what is needed? RAM cache, SSD cache, NVME cache) for fast reading dozens very small files 4kb-200kb vs bigger files reading like sequence of image files like 12MB-50MB EXR's. I have some experience that there is always some problems to have super performance for big files compare to read small files. Maybe you can show how to configure properly some ZFS storage with cache options and then you will show some benchmarks for good performance to have "both" reading very small files and big ones for many workstation in the network as a clients. And then write performance at same scenario ? :) I'm not sure in my situation is it bed configuration or problem with my hardware (comes not from 45Drives unfortunately...) Best, Olaf Poland

[danielkrajnik]

5:00 so drawer is SSD in this analogy? and are you referring to a swap file or general cache?

[inlandchris]

That’s the idea of zfs cache, now, how do you make a zfs raid with cache. For example, you have 4 1TB SSD’s and 8 (10TB) spinning hard drives. Can you make a zfs raid that when you write large data it goes to the SSD’s first as a cache then somehow goes to the main spinning drives, how?

[attainconsult]

great overview would be good if you could do a simple example like Mitch did in Linux RAID vs ZFS RAID

[beardedgaming]

i ahve 1 x1 slot open. two port m.2 expansion card. i move large files (100+gb) and want to keep my 10gb network saturated. extra SSD for cache or for slog? or both? i also have sata ports open

[tunech]

Example. I have 5 1tb drives in raidz1. If I add 250gb ssd as SLOG would I see improvements in writing many small files ? (Backing up program files folder for example) and do I need a l2arc cache at all? Because main task of that pool it to be written to, hourly backups of databases and daily backups of Windows PCs? My config is FX6300 and 8 GB of RAM.

[OldNorsebrewery]

Where do I set where ZIL should live? I have a log ssd & a cache ssd so do I need another ssd? or is it a setting in Truenas?

[EnricAragorn]

Friend, I have a RAIZ with 4 2tb HD. I would like to give it more speed. Can I use just 1 SSD as a cache? or need 4?

[ming-yuanyu]

Thanks for these great tips! I was curious about putting SLOG and L2ARC on the same NVMe drive. Say I have a 500GB NVMe SSD and I create a small partition for SLOG and use the rest of the space for L2ARC. In what situation would or wouldn't you recommend this setup?

[savagedk]

This is a very poor explanation of how the SLOG works. a SLOG is NOT a write cache. A SLOG is NEVER read! Unless there is a crash, for whatever reason!

Let me explain this to you, in case of a SLOG presence with sync writes:
Data goes from application to ZIL in RAM (yes, there is a ZIL in ram) From there it goes to SLOG, and the write is acknowledged, data still stays in RAM though! Eventually, data is flushed from ZILRAM to pool and data is removed from SLOG and ZILRAM.

What a SLOG really does it prevent the double write that happens if you do not have a SLOG.

In case of no SLOG data goes to RAM, sits there, is writte to ZIL-ON-POOL and is committed! but is not yet written/committed to the pool. Eventually, ZIL-IN-RAM is committed to POOL and the data in ZIL-ON-POOL is removed. And yes, this means double writes to the drives in the pool, but one of the writes is to ZIL on pool and the other is commiting the data.

a SLOG is NOT a write cache! The data on SLOG is NEVER READ! Unless there is a crash of some sort!

ZIL is an intent log, not a cache!

What you are describing is that an application makes a request to write data, this goes to RAM, then to SLOG and eventually it is flushed to POOL from SLOG. This is absolutely obnoxiously incorrect! SLOG is never READ and thus data can never be flushed from SLOG to POOL! Unless there is a crash...

[BloomfieldOscar-di]

Taylor Cynthia Walker Barbara White Betty