Pinboard (jm)

Mic92/strace-macos

jm — 2025-11-19T11:49:32+00:00

"A clone of the strace command for macOS" -- yayyyy, I've been lamenting this loss for years

xcapture and xtop

jm — 2025-09-16T13:59:07+00:00

"0x.Tools: X-Ray vision for Linux systems". Linux Performance Analysis with Modern eBPF and DuckDB; dig into the captured DuckDB files using "xtop": "xtop is like the Linux top tool, but extended with x-ray vision and ability to view your performance data from any chosen angle [..]. This enables dimensional performance analysis on Linux and tools like top for wall-clock time and much more. You can use it for system level overview and drill down into indivual threads’ activity and even into kernel events like lock waits or memory stalls."

Debugging Azure Networking for Elastic Cloud Serverless

jm — 2025-06-10T08:46:06+00:00

Good writeup of fixing a Linux packet loss issue in Azure, using low-level access to the VMs running k8s nodes.

Elastic's Site Reliability Engineering team (SRE) observed unstable throughput and packet loss in Elastic Cloud Serverless running on Azure Kubernetes Service (AKS). After investigation, we identified the primary contributing factors to be RX ring buffer overflows and kernel input queue saturation on SR-IOV interfaces. To address this, we increased RX buffer sizes and adjusted the netdev backlog, which significantly improved network stability.

kafkacat and visidata

jm — 2025-03-04T11:44:29+00:00

Two excellent tools in one blog post. Visidata "is a commandline tool to work with data in all sorts of formats, including from stdin"; in this example it's taking lines of JSONL and producing an instant histogram of values from the stream:

Once visidata is open, use the arrow keys to move to the column on which you want to build a histogram and press Shift-F. Since it works with pipes if you leave the -e off the kafkacat argument you get a live stream of messages from the Kafka topic and the visidata will continue to update as messages arrive (although I think you need to replot the histogram if you want it to refresh).

On top of that, there's kcat, "netcat for Kafka”, "a swiss-army knife of tools for inspecting and creating data in Kafka", even supporting on-the-fly decode of Avro messages. https://github.com/edenhill/kcat

Using dtrace on MacOS with SIP enabled

jm — 2025-02-27T17:26:52+00:00

"On all current MacOS versions (Catalina 10.15.x, Big Sur 11.x) System Integrity Protection (SIP) is enabled by default and prevents most uses of dtrace and other tools and scripts based on it (i.e. dtruss)." Wow this is really complicated. Nice work, Apple (via Tony Finch)

subtrace

jm — 2025-02-26T12:00:30+00:00

Subtrace is "Wireshark for your Docker containers. It lets developers see all incoming and outgoing requests in their backend server so that they can resolve production issues faster."

* Works out-of-the-box * No code changes needed * Supports all languages (Python + Node + Go + everything else) * See full payload, headers, status code, and latency * Less than 100µs performance overhead * Built on Clickhouse * Open source

Looks like it outputs to the Chrome Dev Console's Network tab, or a facsimile of it; "Open the subt.link URL in your browser to watch a live stream of your backend server’s network logs". It may be interesting to try this out. (via LWIA)

How to build highly-debuggable C++ binaries

jm — 2024-12-17T14:43:43+00:00

aka, how to have a modern C++ development environment (for when you still need to do such a thing) -- also, wow, C++ has changed a lot since the last time I was working with it. (Via Tony Finch)

μLA

jm — 2024-07-16T11:14:40+00:00

"Micro Logic Analyzer for RP2040" -- as noted by Alison Parrish at https://friend.camp/@aparrish/112792228639189926 , this makes a really nice logic analyzer/oscilloscope for hardware debugging from a Raspberry Pi Pico or other RP2040 board. If I ever wind up back in hardware hacking, this will be useful

eldadru/ksniff

jm — 2023-07-12T13:58:22+00:00

"A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster. You get the full power of Wireshark with minimal impact on your running pods. When working with micro-services, many times it's very helpful to get a capture of the network activity between your micro-service and it's dependencies. ksniff use kubectl to upload a statically compiled tcpdump binary to your pod and redirecting it's output to your local Wireshark for smooth network debugging experience." This would be an absolutely vital piece of software once you get into the nitty-gritty of debugging TCP issues in K8S; I've been on the verge of needing a packet capture once or twice, but managed to just about avoid it so far. I'll be keeping this one in the back pocket.

Ironies of automation

jm — 2020-01-09T17:32:47+00:00

Wow, this is a great paper recommendation from Adrian Colyer - 'Ironies of automation', Bainbridge, Automatica, Vol. 19, No. 6, 1983.

In an automated system, two roles are left to humans: monitoring that the automated system is operating correctly, and taking over control if it isn’t. An operator that doesn’t routinely operate the system will have atrophied skills if ever called on to take over. Unfortunately, physical skills deteriorate when they are not used, particularly the refinements of gain and timing. This means that a formerly experienced operator who has been monitoring an automated process may now be an inexeperienced one. Not only are the operator’s skills declining, but the situations when the operator will be called upon are by their very nature the most demanding ones where something is deemed to be going wrong. Thus what we really need in such a situation is a more, not a lesser skilled operator! To generate successful strategies for unusual situtations, an operator also needs good understanding of the process under control, and the current state of the system. The former understanding develops most effectively through use and feedback (which the operator may no longer be getting the regular opportunity for), the latter takes some time to assimilate.

(via John Allspaw)

The secret life of DNS packets: investigating complex networks

jm — 2019-05-23T15:04:48+00:00

ah, the joys of building production systems atop AWS and its poorly-documented limits:

One more surprising detail we discovered in the tcpdump data was that the VPC resolver was not sending back responses to many of the queries. During one of the 60-second collection periods the DNS server sent 257,430 packets to the VPC resolver. The VPC resolver replied back with only 61,385 packets, which averages to 1,023 packets per second. We realized we may be hitting the AWS limit for how much traffic can be sent to a VPC resolver, which is 1,024 packets per second per interface.

How to change JVM arguments at runtime to avoid application restart

jm — 2018-06-06T15:27:52+00:00

This is a super nifty feature of the JVM: turn on and off heap class histogram dumps at runtime, for instance.

java -XX:+PrintFlagsFinal -version|grep manageable

Debugging Stuck Ruby Processes — What to do Before You Kill -9

jm — 2018-04-23T10:30:12+00:00

good tips on using gdb to gather backtraces (via Louise)

rr: lightweight recording & deterministic debugging

jm — 2018-03-30T21:19:58+00:00

aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.

(via Kevin Lyda and b0rk)

Native Memory Tracking

jm — 2017-09-18T09:35:48+00:00

Java 8 HotSpot feature to monitor and diagnose native memory leaks

AdoptOpenJDK/jitwatch

jm — 2017-08-24T16:22:31+00:00

Log analyser and visualiser for the HotSpot JIT compiler. Inspect inlining decisions, hot methods, bytecode, and assembly. View results in the JavaFX user interface.

EBS gp2 I/O BurstBalance exhaustion

jm — 2017-07-26T13:14:54+00:00

when EBS volumes in EC2 exhaust their "burst" allocation, things go awry very quickly

Undefined Behavior in 2017

jm — 2017-07-06T11:36:55+00:00

This is an extremely detailed post on the state of dynamic checkers in C/C++ (via the inimitable Marc Brooker):

Recently we’ve heard a few people imply that problems stemming from undefined behaviors (UB) in C and C++ are largely solved due to ubiquitous availability of dynamic checking tools such as ASan, UBSan, MSan, and TSan. We are here to state the obvious — that, despite the many excellent advances in tooling over the last few years, UB-related problems are far from solved — and to look at the current situation in detail.

How-to Debug a Running Docker Container from a Separate Container

jm — 2017-02-20T10:34:01+00:00

arguably this shouldn't be required -- building containers without /bin/sh, strace, gdb etc. is just silly

Debugging Java Native Memory Leaks (evanjones.ca)

jm — 2017-01-10T16:39:09+00:00

Using jemalloc to instrument the contents of the native heap and record stack traces of each chunk's allocators, so that leakers can be quickly identified (GZIPInputStream in this case). See also https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/ , https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md .

The hidden cost of QUIC and TOU

jm — 2016-12-15T10:54:31+00:00

The recent movement to get all traffic encrypted has of course been great for the Internet. But the use of encryption in these protocols is different than in TLS. In TLS, the goal was to ensure the privacy and integrity of the payload. It's almost axiomatic that third parties should not be able to read or modify the web page you're loading over HTTPS. QUIC and TOU go further. They encrypt the control information, not just the payload. This provides no meaningful privacy or security benefits. Instead the apparent goal is to break the back of middleboxes [0]. The idea is that TCP can't evolve due to middleboxes and is pretty much fully ossified. They interfere with connections in all kinds of ways, like stripping away unknown TCP options or dropping packets with unknown TCP options or with specific rare TCP flags set. The possibilities for breakage are endless, and any protocol extensions have to jump through a lot of hoops to try to minimize the damage.

Using jemalloc to get to the bottom of an off-heap Java memory leak

jm — 2016-04-14T11:39:20+00:00

Good technique

The Nyquist theorem and limitations of sampling profilers today, with glimpses of tracing tools from the future

jm — 2016-02-09T12:55:23+00:00

Awesome post from Dan Luu with data from Google:

The cause [of some mystery widespread 250ms hangs] was kernel throttling of the CPU for processes that went beyond their usage quota. To enforce the quota, the kernel puts all of the relevant threads to sleep until the next multiple of a quarter second. When the quarter-second hand of the clock rolls around, it wakes up all the threads, and if those threads are still using too much CPU, the threads get put back to sleep for another quarter second. The phase change out of this mode happens when, by happenstance, there aren’t too many requests in a quarter second interval and the kernel stops throttling the threads. After finding the cause, an engineer found that this was happening on 25% of disk servers at Google, for an average of half an hour a day, with periods of high latency as long as 23 hours. This had been happening for three years. Dick Sites says that fixing this bug paid for his salary for a decade. This is another bug where traditional sampling profilers would have had a hard time. The key insight was that the slowdowns were correlated and machine wide, which isn’t something you can see in a profile.

How to inspect SSL/TLS traffic with Wireshark 2

jm — 2015-12-22T10:50:08+00:00

turns out it's easy enough -- Mozilla standardised a debugging SSL session-key logging file format which Wireshark and Chrome support

Tools for debugging, testing and using HTTP/2

jm — 2015-12-10T14:06:56+00:00

excellent, extensive list from Cloudflare

Byteman

jm — 2015-09-24T10:11:48+00:00

a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.

Benchmarking GitHub Enterprise - GitHub Engineering

jm — 2015-07-27T10:21:09+00:00

Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)

Apache HTrace

jm — 2015-05-12T16:06:27+00:00

a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator

Combining static model checking with dynamic enforcement using the Statecall Policy Language

jm — 2015-03-24T12:55:54+00:00

This looks quite nice -- a model-checker "for regular programmers". Example model for ping(1):

01 automaton ping (int max_count, int count, bool can_timeout) {
02   Initialize;
03   during {
04     count = 0;
05     do {
06       Transmit_Ping;
07       either {
08         Receive_Ping;
09       } or (can_timeout) {
10         Timeout_Ping;
11       };
12       count = count + 1;
13     } until (count >= max_count);
14   } handle {
15     SIGINFO;
16     Print_Summary;
17   };

OpenJDK: jol

jm — 2015-02-16T11:17:41+00:00

'JOL (Java Object Layout) is the tiny toolbox to analyze object layout schemes in JVMs. These tools are using Unsafe, JVMTI, and Serviceability Agent (SA) heavily to decoder the actual object layout, footprint, and references. This makes JOL much more accurate than other tools relying on heap dumps, specification assumptions, etc.' Recommended by Nitsan Wakart, looks pretty useful for JVM devs

8 gdb tricks you should know (Ksplice Blog)

jm — 2015-01-30T22:26:19+00:00

These are very good -- bookmarking for the next time I'm using gdb, probably about 3 years from now

Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale | Engineering Blog | Facebook Code

jm — 2014-11-28T16:47:33+00:00

Excellent real-world war story from Facebook -- a long-running mystery bug was eventually revealed to be a combination of edge-case behaviours across all the layers of the networking stack, from L2 link aggregation at the agg-router level, up to the L7 behaviour of the MySQL client connection pool.

Facebook collocates many of a user’s nodes and edges in the social graph. That means that when somebody logs in after a while and their data isn’t in the cache, we might suddenly perform 50 or 100 database queries to a single database to load their data. This starts a race among those queries. The queries that go over a congested link will lose the race reliably, even if only by a few milliseconds. That loss makes them the most recently used when they are put back in the pool. The effect is that during a query burst we stack the deck against ourselves, putting all of the congested connections at the top of the deck.

Troubleshooting Production JVMs with jcmd

jm — 2014-09-17T22:39:46+00:00

remotely trigger GCs, finalization, heap dumps etc. Handy

Systemd: Harbinger of the Linux apocalypse

jm — 2014-08-18T13:40:05+00:00

While there are many defensible aspects of Systemd, other aspects boggle the mind. Not the least of these was that, as of a few months ago, trying to debug the kernel from the boot line would cause the system to crash. This was because of Systemd's voracious logging and the fact that Systemd responds to the "debug" flag on the kernel boot line -- a flag meant for the kernel, not anything else. That, straight up, is a bug. However, the Systemd developers didn't see it that way and actively fought with those experiencing the problem. Add the fact that one of the Systemd developers was banned by Linus Torvalds for poor attitude and bad design and another was responsible for causing significant issues with Linux audio support, but blamed the problem on everything else but his software, and you have a bad situation on your hands. There's no shortage of egos in the open source development world. There's no shortage of new ideas and veteran developers and administrators pooh-poohing something new simply because it's new. But there are also 45 years of history behind Unix and extremely good reasons it's still flourishing. Tools designed like Systemd do not fit the Linux mold, to their own detriment. Systemd's design has more in common with Windows than with Unix -- down to the binary logging.

The link re systemd consuming the "debug" kernel boot arg is a canonical example of inflexible coders refusing to fix their own bugs. (via Jason Dixon)

Tracedump

jm — 2014-05-29T09:25:20+00:00

a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements: * ptrace monitor - tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses; * pcap sniffer - using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached); * garbage collector - periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses. As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets - readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.

sysdig

jm — 2014-04-04T12:01:42+00:00

open source, system-level exploration: capture system state and activity from a running Linux instance, then save, filter and analyze. Think of it as strace + tcpdump + lsof + awesome sauce. With a little Lua cherry on top.

This sounds excellent. Linux-based, GPLv2.

rr

jm — 2014-03-26T15:42:14+00:00

A cool-looking new debugging tool for C/C++ from Mozilla.

Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.

Low runtime overhead; easy deployability; targeted at 32-bit (?!) Linux; OSS. (via Bryan O'Sullivan)

IntelliJ IDEA 13.1 will support Chronon Debugger

jm — 2014-03-10T11:09:07+00:00

This, IMO, would be a really good reason to upgrade to the payware version of IDEA - Chronon looks cool.

Chronon is a new revolutionary tool keeping track of running Java programs and recording their execution process for later analysis, which can be helpful when you need to thoroughly retrace your steps when dealing with complicated bugs.

A sampling profiler for your daily browsing - Google Groups

jm — 2014-01-15T11:06:16+00:00

via Ilya Grigorik: Chrome Canary now has a built-in, always-on, zero-overhead code profiler. I want this in my server-side JVMs!

flood.io » Convert HAR to a JMeter JMX plan file

jm — 2013-12-02T11:17:00+00:00

this is absolutely fantastic. Thanks flood.io!

Make The Web Fast - The HAR Show: Capturing and Analyzing performance data with HTTP Archive format — Google Developers

jm — 2013-12-02T11:15:34+00:00

Wow, I didn't know about this. Great idea.

Need a flexible format to record, export, and analyze network performance data? Well, that's exactly what the HTTP Archive format (HAR) is designed to do! Even better, did you know that Chrome DevTools supports it? In this episode we'll take a deep dive into the format (as you'll see, its very simple), and explore the many different ways it can help you capture and analyze your sites performance. Join Ilya Grigorik and Peter Lubbers to find out how to capture HAR network traces in Chrome, visualize the data via an online tool, share the reports with your clients and coworkers, automate the logging and capture of HAR data for your build scripts, and even adapt it to server-side analysis use cases

dumping a JVM heap using gdb

jm — 2013-03-23T21:31:17+00:00

now this is a neat trick -- having been stuck having to flip to spares and do other antics while a long-running heap dump took place, this is a winner.

Dumping a JVM’s heap is an extremely useful tool for debugging problems with a J2EE application. Unfortunately, when a JVM explodes, using the standard jmap tool can take an inordinate amount of time to execute for lots of different reasons. This leads to extended downtime when a heap dump is attempted and even then, jmap regularly fails. This blog post is intended to outline an alternate method using [gdb] to achieve a heap dump that only requires mere seconds of additional downtime allowing the slow jmap process to happen once the application is back in service.

Distributed Systems Tracing with Zipkin

jm — 2013-03-14T10:27:38+00:00

Twitter's version of the "canary"/"tracer" request concept

Peek and poke in the age of Linux

jm — 2013-03-06T22:32:52+00:00

Neat demo of using ptrace to inject into a running process, just like the good old days ;)

Some time ago I ran into a production issue where the init process (upstart) stopped behaving properly. Specifically, instead of spawning new processes, it deadlocked in a transitional state. [...] What’s worse, upstart doesn’t allow forcing a state transition and trying to manually create and send DBus events didn’t help either. That meant the sane options we were left with were: restart the host (not desirable at all in that scenario); start the process manually and hope auto-respawn will not be needed. Of course there are also some insane options. Why not cheat like in the old times and just PEEK and POKE the process in the right places? The solution used at the time involved a very ugly script driving gdb which probably summoned satan in some edge cases. But edge cases were not hit and majority of hosts recovered without issues.

Chronon DVR for Java

jm — 2012-05-12T21:47:46+00:00

"record entire execution of your Java app; play it back on any machine". Other features: time-travelling debugger -- step backwards, jump to any point in execution, designed for long running programs; post-execution logging -- add log statements after the program has run, and see what it would have logged. Looks extremely nifty, but I wonder how big those recording files get...

DTrace and Erlang

jm — 2011-11-09T13:54:17+00:00

from Basho, via istvan. DTrace is becoming more compelling as a deep instrumentation/monitoring API -- I didn't realise disabled DTrace probes were virtually 0-overhead (a "2 NOOP instruction placeholder", apparently), that's nifty. Wonder if they've fixed the licensing mess, though

peak6/scala-ssh-shell - GitHub

jm — 2011-10-11T20:25:59+00:00

'Backdoor that gives you a scala shell over ssh on your jvm. The shell is not sandboxed, anyone access the shell can touch anything in the jvm and do anything the jvm can do including modifying and deleting files, etc.' nifty!

ioprofile

jm — 2010-10-16T11:09:45+00:00

wraps strace(1) to summarise and aggregate I/O ops performed by a Linux process. looks pretty nifty (via Jeremy Zawodny)

JSON Format

jm — 2009-11-15T21:03:46+00:00

'your online JSON Formatter'. useful. via JKeyes

Hijack: Get A Live IRB Prompt For Any Existing Ruby Process

jm — 2009-08-25T13:46:08+00:00

injects via gdb. pretty cool, if it works; one comment notes that they couldn't use it on a Rails app