Curious Case of Maintaining Sufficient Free Space with ZFS

Sometimes trivial problems turn out to be more interesting than expected.

Curious Case of Maintaining Sufficient Free Space with ZFS
TLDR: ZFS free-space reporting is a lagging indicator. Background I use Proxmox VM server backed by a ZFS array of hard drives for various personal infrastructure. I also have security cameras that upload motion-triggered videos to my server (via FTP!). Problem Description I would like to use 90% of my available space for most-recent security videos. Recipe: Create a dedicated ZFS volume Setup ZFS quota Run a cronjob to free space faster than it gets consumed by video uploads. [Read More]
zfs 

mmap Page Fault Tracing with bpftrace and ext4

Previous blog post on how to trace Firefox IO using bpftrace via official page_fault_user tracepoint left me a bit unsatisfied with how complicated it turned out. Complexity has potential to be error-prone and the syscall-tracing dependency makes it impossible to trace IO within the main executable. I decided to try reimplement the trace using my old approach of tracing ext4 functions that handle page-faults. This turned out to be much more robust. [Read More]
mmap  bpf  io  ext4 

EBPF for Tracing How Firefox Uses Page Faults to Load Libraries

Modern browsers are some of the most complicated programs ever written. For example, the main Firefox library on my system is over 130Mbytes. Doing 130MB of IO poorly can be quite a performance hit, even with SSDs! :). Few people seem to understand how memory-mapped IO works. There are no pre-canned tools to observe it on Linux, thus even even fewer know how to observe it. Years ago, when I was working on Firefox startup performance, I discovered that libraries were loaded backwards (blog1, blog2, paper, GCC bug) on Linux. [Read More]
mmap  bpf  io 

Reading NFS at >=25GB/s using FIO + libnfs

Reading NFS at >=25GB/s using FIO + libnfs
My current employer does a lot of really cool systems work that’s covered by NDAs. I recently did some work to integrate a cool open source tool into our workflow. Felt it deserved a blog post. NFS Testing Requires Parallelism. I work for Pure Storage. One of the products we make is a scale-out NFS1 (and S3-compatible) server called FlashBlade. I was asked to test FlashBlade2 performance scaling. I needed to generate NFS read workloads of 15-300 Gigabytes/second. [Read More]

Firefox's Optimized Zip Format: Reading Zip Files Really Quickly

Firefox's Optimized Zip Format: Reading Zip Files Really Quickly
This post is about minimizing amount of disk IO and CPU overhead when reading Zip files. I recently saw an article about a new format that was faster than zip. This is quite surprising as to my mind, zip is one of the most flexible and low-overhead formats I’ve encountered. Some googling showed me that over past 11 years people have noticed that Firefox uses optimized zip files. This inspired me to document thinking behind the optimized zip format I implemented in Firefox in the pre-pandemic 2010. [Read More]

Motorola bet on faster IO, increasing risk of data-corruptions

I need to follow up on my blog post re Pixel lagging more than Moto Z. Sorry about the click-baity title of the previous blog, but that was a way to get noticed and get some help. I am really thankful for the [small percentage of] thoughtful feedback I received. Turns out Motorola uses the F2FS + nobarrier mount option to speed up writes to NAND. nobarrier does not turn fsync is a no-op. [Read More]

Laggy phones and misleading benchmarks

TLDR: You can predict degree of unresponsiveness of a phone via random-write-4k benchmarks. I wish review websites would fill phones to 80-90% prior to running the benchmark, especially on smaller-capacity phones where users are more likely to run out of space.