Pinboard (jm)

Zip bombs to frustrate AI crawlers

jm — 2025-03-25T10:08:23+00:00

Nifty trick; redirecting abusive AI crawlers to a gzipped file containing 100GB of zeros with a few lines of nginx config:

set $redir_to_gz 1;
if ($host = gz.niko.lgbt) {
    set $redir_to_gz 0;
}
if ($http_user_agent !~* (claudebot|ZoominfoBot|GPTBot|SeznamBot|DotBot|Amazonbot|DataForSeoBot|2ip|paloaltonetworks.com|SummalyBot|incestoma)) {
    set $redir_to_gz 0;
}
if ($redir_to_gz) {
    return 301 https://gz.niko.lgbt/;
}

as for the actual stuff behind gz.niko.lgbt

server {
    # SSL and listen -- snipped

    # static files
    root /var/www/gz.niko.lgbt;
    location / {
        add_header Content-Encoding gzip;
        try_files /42.gz =404;
        gunzip off;
        types { text/html gz; }
    }

    # gunzip off is very important because if the client doesn't support gzip encoding nginx will blow its foot off without that
    # 42.gz is generated with dd if=/dev/zero bs=1M count=102400 | gzip -c - > 42.gz

    # additional config -- snipped
}

Nice one @niko, I'm definitely going to use that :)

Scylla compression benchmarks

jm — 2019-10-08T12:06:43+00:00

ScyllaDB tested out LZ4, Snappy, DEFLATE, and ZStandard at several different levels on a decently real-world-ish workload. tl;dr:

Use compression. Unless you are using a really (but REALLY) fast hard drive, using the default compression settings will be even faster than disabling compression, and the space savings are huge. When running a data warehouse where data is mostly being read and only rarely updated, consider using DEFLATE. It provides very good compression ratios while maintaining high decompression speeds; compression can be slower, but that might be unimportant for your workload. If your workload is write-heavy but you really care about saving disk space, consider using ZStandard on level 1. It provides a good middle-ground between LZ4/Snappy and DEFLATE in terms of compression ratios and keeps compression speeds close to LZ4 and Snappy. Be careful however: if you often want to read cold data (from the SSTables on disk, not currently stored in memory, so for example data that was inserted a long time ago), the slower decompression might become a problem.

How eBay’s Shopping Cart used compression techniques to solve network I/O bottlenecks

jm — 2017-02-09T11:12:22+00:00

compressing data written to MongoDB using LZ4_HIGH --dropped oplog write rates from 150GB/hour to 11GB/hour. Snappy and Gzip didn't fare too well by comparison

lbzip2

jm — 2016-03-08T12:03:43+00:00

a free, multi-threaded compression utility with support for bzip2 compressed file format. lbzip2 can process standard bz2 files in parallel. It uses POSIX threading model (pthreads), which allows it to take full advantage of symmetric multiprocessing (SMP) systems. It has been proven to scale linearly, even to over one hundred processor cores. lbzip2 is fully compatible with bzip2 – both at file format and command line level. Files created by lbzip2 can be decompressed by all versions of bzip2 and other software supporting bz2 format. lbzip2 can decompress any bz2 files in parallel. All bzip2 command-line options are also accepted by lbzip2. This makes lbzip2 a drop-in replacement for bzip2.

research!rsc: Zip Files All The Way Down

jm — 2016-02-05T15:26:51+00:00

quine.zip, quine.gz, and quine.tar.gz. Here's what happens when you mail it through bad AV software: https://twitter.com/FioraAeterna/status/694655296707297281

GZinga

jm — 2015-10-11T07:32:43+00:00

'Seekable and Splittable Gzip', from eBay

Brotli: a new compression algorithm for the internet from Google

jm — 2015-09-22T10:47:11+00:00

While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli. In our study ‘Comparison of Brotli, Deflate, Zopfli, LZMA, LZHAM and Bzip2 Compression Algorithms’ we show that Brotli is roughly as fast as zlib’s Deflate implementation. At the same time, it compresses slightly more densely than LZMA and bzip2 on the Canterbury corpus. The higher data density is achieved by a 2nd order context modeling, re-use of entropy codes, larger memory window of past data and joint distribution codes. Just like Zopfli, the new algorithm is named after Swiss bakery products. Brötli means ‘small bread’ in Swiss German.

Google Fonts recently switched to using Zopfli

jm — 2014-01-14T11:11:31+00:00

Google Fonts recently switched to using new Zopfli compression algorithm: the fonts are ~6% smaller on average, and in some cases up to 15% smaller! [...] What's Zopfli? It's an algorithm that was developed by the compression team at Google that delivers ~3~8% bytesize improvement when compared to gzip with maximum compression. This byte savings comes at a cost of much higher encoding cost, but the good news is, fonts are static files and decompression speed is exactly the same. Google Fonts pays the compression cost once and every clients gets the benefit of smaller download. If you’re curious to learn more about Zopfli: http://bit.ly/Y8DEL4

Compression in Kafka: GZIP or Snappy ?

jm — 2013-04-10T14:36:53+00:00

With Ack: in this mode, as far as compression is concerned, the data gets compressed at the producer, decompressed and compressed on the broker before it sends the ack to the producer. The producer throughput with Snappy compression was roughly 22.3MB/s as compared to 8.9MB/s of the GZIP producer. Producer throughput is 150% higher with Snappy as compared to GZIP. No ack, similar to Kafka 0.7 behavior: In this mode, the data gets compressed at the producer and it doesn’t wait for the ack from the broker. The producer throughput with Snappy compression was roughly 60.8MB/s as compared to 18.5MB/s of the GZIP producer. Producer throughput is 228% higher with Snappy as compared to GZIP. The higher compression savings in this test are due to the fact that the producer does not wait for the leader to re-compress and append the data; it simply compresses messages and fires away. Since Snappy has very high compression speed and low CPU usage, a single producer is able to compress the same amount of messages much faster as compared to GZIP.

Compress data more densely with Zopfli - Google Developers Blog

jm — 2013-03-02T23:10:18+00:00

New compressor from Google, gzip/zip-compatible, slower but slightly smaller results

lrzip

jm — 2012-02-07T17:31:53+00:00

'Lrzip uses an extended version of rzip which does a first pass long distance redundancy reduction. The lrzip modifications make it scale according to memory size. [...] The unique feature of lrzip is that it tries to make the most of the available ram in your system at all times for maximum benefit. It does this by default, choosing the largest sized window possible without running out of memory.'

XZ Utils

jm — 2009-10-23T09:03:09+00:00

15% smaller than bzip, 30% smaller than gzip, and now shipped with Fedora and Ubuntu. uses LZMA2

pigz

jm — 2009-10-21T10:09:18+00:00

'A parallel implementation of gzip for modern multi-processor, multi-core machines', by Mark Adler, no less