Pinboard (jm)

Mic92/strace-macos

jm — 2025-11-19T11:49:32+00:00

"A clone of the strace command for macOS" -- yayyyy, I've been lamenting this loss for years

Hacking with AI SASTs: An overview of 'AI Security Engineers' / 'LLM Security Scanners' for Penetration Testers and Security Teams | Joshua.Hu | Joshua Rogers' Scribbles

jm — 2025-09-22T10:15:23+00:00

This is actually impressive results from using LLMs to perform security scans on an existing codebase. Daniel Stenberg of curl has given the results of this work a thumbs-up: https://mastodon.social/@bagder/115241241075258997

My general summary is as follows: Multiple AI-native SASTs are already on the market, ready to use today. They work extremely well. They find real vulnerabilities and logic bugs in minutes. They can “think”/”reason” about business logic issues. They can match developer intent with actual code. They aren’t based on static rule-sets and queries. They have low false positive rates. They’re cheap (for now). My results showed that (in order of success), ZeroPath, Corgea, and Almanax, are the top three products on the market right now. I did not test DryRun.

These tools look superb.

Orange-OpenSource/hurl

jm — 2025-06-20T09:42:49+00:00

"Hurl; run and test HTTP requests with plain text". This is pretty nice; a really simple plain-text file format to describe making a HTTP request or set of requests, and performing assertions on their results. The only thing I can spot missing is builtin support for OAuth

Scrappy

jm — 2025-06-18T09:44:50+00:00

"make little apps for you and your friends":

The apps we use are almost exclusively mass-market, sold on an app-store, made for thousands if not millions of users. Or they are enterprise apps that are custom-built for hundreds of thousands of dollars. But there isn’t really any equivalent of home-made software — apps made lovingly by you for your friends and family. Apps that aren’t polished or flashy, but are made to your preference and help you with your particular needs. [...] We ended up creating a research prototype that we call Scrappy — a tool for making scrappy apps for just you and your friends. First and foremost, we aim to contribute a vision of what home-made software could be like. We want to make this vision as concrete as we can, by sharing a working tool and examples of apps made in it. Scrappy, in its current state, is a prototype, not a robust tool, but we hope it paints the picture we carry in our heads — of software as something that can be creative, personal, expressive. Made by anyone, for themselves and their loved ones.

Very Hypercard-ish!

Jetrelay

jm — 2025-05-20T09:18:20+00:00

This is a great little hack: "jetrelay, a pub/sub server compatible with Bluesky’s “jetstream” data feed. Using a few pertinent Linux kernel features, it avoids doing almost any work itself. As a result, it’s highly efficient: it can saturate a 10 Gbps network connection with just 8 CPU cores." Specifically, these are the tricks in question: - Trick #1: Bypassing userspace with sendfile(); - Trick #2: Handling many clients in parallel with io_uring; - Trick #3: Discarding old data with FALLOC_FL_PUNCH_HOLE -- this is a nice way to avoid having to rotate between multiple files, nifty.

uv and PEP 723 for Easy Deployment

jm — 2025-03-28T10:00:08+00:00

By adding metadata comments at the top of the script like this:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "httpx>=0.28.1",
# ]
# ///

uv(1) will automatically handle downloading dependency modules at runtime etc., obviating the need for a requirements.txt file. Fairly neat

My list of useful command line tools

jm — 2025-03-25T09:43:29+00:00

Here's a bunch of fantastic recent CLI tools I hadn't seen before; loads are by one guy, https://github.com/sharkdp , who seems very productive :)

Kicking the tyres on the new DuckDB UI

jm — 2025-03-14T14:55:22+00:00

DuckDB now has a built-in web notebook UI -- very nice

amarlearning/spot-optimizer

jm — 2025-03-14T10:51:20+00:00

"Spot Optimizer is a Python library that helps users select the best AWS spot instances based on their resource requirements, including cores, RAM, storage type (SSD), instance architecture (x86 or ARM), AWS region, EMR version compatibility, and instance family preferences. It replaces complex, in-house logic for finding the best spot instances with a simple and powerful abstraction. No more manual guesswork — just the right instances at the right time." Implemented as a Python lib and CLI tool.

llms and humans unite, you have nothing to lose but your chores

jm — 2025-03-12T11:54:40+00:00

Danny O'Brien posts a nice little automation script co-written with Claude.AI which has a couple of noteworthy angles; (1) instead of scraping the Uber site directly, it co-drives a browser using the Chrome DevTool Protocol and the `playwright` Python package; and (2) it has inline requirements.txt specifications using `uv` comments at the top of the script, which I hadn't seen before. I like the co-driving idea; it's a nice way to automate clicky-clicky boring tasks without using a standalone browser or a scraper client, while being easy to keep an eye on and possibly debug when it breaks. Also good to keep an eye on what LLM-authored code is up to. In the past I've used Browserflow as a no-code app builder for one-off automations of clicky-clicky web flows like this, but next time I might give the vibe-coding+CDP approach a go.

WireMock

jm — 2025-03-06T15:41:51+00:00

I could have done with knowing about this before implementing mock APNs, Huawei, Microsoft and FCM push APIs over the last few years!

An open-source tool for API mock testing, with over 5 million downloads per month. It can help you to create stable test and development environments, isolate yourself from flakey 3rd parties and simulate APIs that don't exist yet.

Nice features include running in-process in a JVM, standalone, or in a Docker container; GraphQL and gRPC support; and fault and latency injection. https://library.wiremock.org/ is a library of pre-built API mocks other people have previously made.

kafkacat and visidata

jm — 2025-03-04T11:44:29+00:00

Two excellent tools in one blog post. Visidata "is a commandline tool to work with data in all sorts of formats, including from stdin"; in this example it's taking lines of JSONL and producing an instant histogram of values from the stream:

Once visidata is open, use the arrow keys to move to the column on which you want to build a histogram and press Shift-F. Since it works with pipes if you leave the -e off the kafkacat argument you get a live stream of messages from the Kafka topic and the visidata will continue to update as messages arrive (although I think you need to replot the histogram if you want it to refresh).

On top of that, there's kcat, "netcat for Kafka”, "a swiss-army knife of tools for inspecting and creating data in Kafka", even supporting on-the-fly decode of Avro messages. https://github.com/edenhill/kcat

Ruff

jm — 2025-03-03T17:54:01+00:00

An extremely fast Python linter and code formatter, written in Rust. Ruff aims to be orders of magnitude faster than alternative tools while integrating more functionality behind a single, common interface. Ruff can be used to replace Flake8 (plus dozens of plugins), Black, isort, pydocstyle, pyupgrade, autoflake, and more, all while executing tens or hundreds of times faster than any individual tool.

subtrace

jm — 2025-02-26T12:00:30+00:00

Subtrace is "Wireshark for your Docker containers. It lets developers see all incoming and outgoing requests in their backend server so that they can resolve production issues faster."

* Works out-of-the-box * No code changes needed * Supports all languages (Python + Node + Go + everything else) * See full payload, headers, status code, and latency * Less than 100µs performance overhead * Built on Clickhouse * Open source

Looks like it outputs to the Chrome Dev Console's Network tab, or a facsimile of it; "Open the subt.link URL in your browser to watch a live stream of your backend server’s network logs". It may be interesting to try this out. (via LWIA)

Artsy's Technology Choices evaluation process

jm — 2025-01-24T10:25:47+00:00

This is a nice way to evaluate new technology options, from Artsy:

We want to accomplish a lot with a lean team, which means we must choose stable technologies. However, we also want to adopt best-of-breed technologies or best-suited tools, which may need work or still be evolving. We've borrowed from ThoughtWorks' Radar to define the following stages for evaluating, adopting, and retiring technologies: - Adopt: Reasonable defaults for most work. These choices have been exercised successfully in production at Artsy and there is a critical mass of engineers comfortable working with them. - Trial: These technologies are being evaluated in limited production circumstances. We don't have enough production experience to recommend them for high-risk or business-critical use cases, but they may be worth consideration if your project seems like a fit. - Assess: Technologies we are interested in and maybe even built proofs-of-concept for, but haven't yet trialed in production. - Hold: Based on our experience, these technologies should be avoided. We've found them to be flawed, immature, or simply supplanted by better alternatives. In some cases these remain in legacy production uses, but we should take every opportunity to retire or migrate away.

(Via Lar Van Der Jagt on the Last Week In AWS slack instance)

Bufferbloat Test

jm — 2025-01-10T10:48:08+00:00

A handy tool to test your internet connection for "bufferbloat", the error condition involving "undesirable high latency caused by other traffic on your network. It happens when a flow uses more than its fair share of the bottleneck. Bufferbloat is the primary cause of bad performance for real-time Internet applications like VoIP calls, video games, and videoconferencing." (My home internet connection is currently rating a C: "your latency increased considerably under load", jumping from a min/mean/p95/max of 10.7, 16.9, 23.7, 30.1ms to 35.3, 98.4, 121.0, 286.0ms under load, yikes, so looks like I need to do some optimising.)

pprof.me

jm — 2024-12-15T11:43:44+00:00

a web app to 'share and visualize .pprof profiles on an intuitive interface'.

ArchiveWeb.page

jm — 2024-10-02T11:40:46+00:00

"Interactive browser-based web archiving from Webrecorder. The ArchiveWeb.page browser extension and standalone application allows you to capture web archives interactively as you browse. After archiving your webpages, your archives can be viewed using ReplayWeb.page — no extension required! For those who need to crawl whole websites with automated tools, check out Browsertrix." This is a nice way to archive a personal dynamic site online in a read-only fashion -- there is a self-hosting form of the replayer at https://replayweb.page/docs/embedding/#self-hosting . As @david302 on the Irish Tech Slack notes: "you can turn on recording, browse the (public) site you want to archive, get the .wacz file and stick that+js on s3/cloudfront."

The Pinecil is the best soldering iron for most people

jm — 2024-09-18T11:52:04+00:00

I need a new soldering iron, this sounds nice

Folk wisdom on visual programming

jm — 2024-08-21T15:05:20+00:00

A (lengthy) summary of third party comments on visual programming environments and tools, from Hacker News (via Tony Finch's retro-links)

Technical post-mortem on the Google/UniSuper account deletion

jm — 2024-05-24T16:24:08+00:00

"Google operators followed internal control protocols. However, one input parameter was left blank when using an internal tool to provision the customer’s Private Cloud. As a result of the blank parameter, the system assigned a then unknown default fixed 1 year term value for this parameter. After the end of the system-assigned 1 year period, the customer’s GCVE Private Cloud was deleted. No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer." Ouch.

ddbsh

jm — 2023-01-30T16:08:20+00:00

'DynamoDB Shell (ddbsh) is an interactive CLI for Amazon DynamoDB', emulating an SQL-like command syntax, from AWS Labs

manojkarthick/pqrs

jm — 2023-01-10T11:00:01+00:00

"Command line tool for inspecting Parquet files", replacement for parquet-tools, written in Rust. Now do Orc!

Visualping

jm — 2022-01-06T10:30:53+00:00

"Website change detection, monitoring and alerts" - with a very friendly UX. nicely done

Mukama

jm — 2021-05-19T11:29:32+00:00

'Essentials for your daily round' -- a really lovely set of EDC products, and not _too_ crazy expensive....

Boosting Dropbox upload speed—and making Windows’ TCP stack resilient to network reordering - Dropbox

jm — 2021-05-18T22:04:37+00:00

Featuring an interesting dig into the current state of Windows system and network-level diagnostic tools:

netsh trace [...] correlates events on the wire with events that happen on the TCP layer, timers, buffer management, socket layer, and even the Windows asyncio subsystem (IOCP).

Tabula

jm — 2021-05-17T13:54:11+00:00

'Extract Tables from PDFs - If you’ve ever tried to do anything with data provided to you in PDFs, you know how painful it is — there's no easy way to copy-and-paste rows of data out of PDF files. Tabula allows you to extract that data into a CSV or Microsoft Excel spreadsheet using a simple, easy-to-use interface. Tabula works on Mac, Windows and Linux.'

SLA & Uptime calculator

jm — 2020-11-26T13:43:11+00:00

aka, 'How much downtime corresponds to 99.9% uptime'. Great utility, plug in your number and see the results

COVID-19 risk calculator

jm — 2020-10-23T08:59:48+00:00

Given the prevalence in your location, and various attributes about your risk level and who will be in the area with you and for how long, this computes a comparative risk level for any given activity.

q - Text as Data

jm — 2020-10-21T09:56:28+00:00

'a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files). q treats ordinary files as database tables, and supports all SQL constructs, such as WHERE, GROUP BY, JOINs etc. It supports automatic column name and column type detection, and provides full support for multiple encodings.' Awesome!

WebPlotDigitizer

jm — 2020-10-01T09:17:17+00:00

Extract data from plots, images, and maps:

It is often necessary to reverse engineer images of data visualizations to extract the underlying numerical data. WebPlotDigitizer is a semi-automated tool that makes this process extremely easy: Works with a wide variety of charts (XY, bar, polar, ternary, maps etc.) Automatic extraction algorithms make it easy to extract a large number of data points Free to use, opensource and cross-platform (web and desktop) Used in hundreds of published works by thousands of users Also useful for measuring distances or angles between various features

Wayback Machine Downloader

jm — 2020-05-28T16:10:40+00:00

'Wayback machine online downloader with CMS. Restore a fully functional copy of the site - 200 files for free!' -- recommended by Damien. It can handle sites of 1000s of files, at $.50 per 1000

The other kind of contact tracing tools

jm — 2020-04-15T12:00:27+00:00

Farzad Mostashari on Twitter: "Last week I posted about automated digital contact tracing apps- lots of discussion since. now lemme talk about the other kind of contact tracing app, tools that increase the efficiency & ease of contact tracing: enhanced directories, multichannel messaging applications, real-time translation services, symptom reporting & isolation monitoring"

BurntSushi/xsv

jm — 2020-01-23T14:23:37+00:00

a command line program for indexing, slicing, analyzing, splitting and joining CSV files. Commands should be simple, fast and composable: Simple tasks should be easy. Performance trade offs should be exposed in the CLI interface. Composition should not come at the expense of performance.

En-ROADS

jm — 2019-12-04T11:34:07+00:00

An excellent global climate simulation tool, to roughly model climate change management strategies and their impacts. (It's not good news.)

databricks/devbox

jm — 2019-02-28T17:51:40+00:00

interesting, a synchronization daemon from Databricks which they use to synch up dev repos with a remote "devbox" in EC2 for heavyweight compilation

Remove Background from Image

jm — 2018-12-18T11:07:46+00:00

A fully automated background-removal tool. Nicely done

Saving a non-profit six figures a year using Squarespace, Airtable and Glitch.com

jm — 2018-07-09T10:15:39+00:00

Airtable in particular sounds like a lovely tool for small-scale users

Thanos: Prometheus at Scale

jm — 2018-05-18T21:54:18+00:00

interesting

Parallelizing S3 Workloads with s5cmd

jm — 2018-04-24T09:32:46+00:00

nice parallel download/upload tool for S3, developed by Peak Games, open source, in Go

tomnomnom/gron

jm — 2018-04-04T15:54:38+00:00

'Make JSON greppable!'

S3 Point In Time Restore

jm — 2017-10-02T15:55:55+00:00

restore a versioned S3 bucket to the state it was at at a specific point in time

Airtable

jm — 2017-09-27T13:35:21+00:00

looks like a nice web-based database, FileMaker Pro-style

AdoptOpenJDK/jitwatch

jm — 2017-08-24T16:22:31+00:00

Log analyser and visualiser for the HotSpot JIT compiler. Inspect inlining decisions, hot methods, bytecode, and assembly. View results in the JavaFX user interface.

How-To: Understand the Wireless Diagnostics app on your Mac to analyze and improve your Wi-Fi network | 9to5Mac

jm — 2017-06-07T10:05:41+00:00

decent enough tool builtin to OSX

pachyderm

jm — 2017-02-20T10:48:24+00:00

'Containerized Data Analytics':

There are two bold new ideas in Pachyderm: Containers as the core processing primitive Version Control for data These ideas lead directly to a system that's much more powerful, flexible and easy to use. To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data (Example: distributed grep). Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, can track changes and diffs, collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click! Version control is also very synergistic with our containerized processing engine. Pachyderm understands how your data changes and thus, as new data is ingested, can run your workload on the diff of the data rather than the whole thing. This means that there's no difference between a batched job and a streaming job, the same code will work for both!

LandSafe.org: if you aren't safe, we'll make noise for you

jm — 2017-02-04T09:34:40+00:00

a Dead Man's Switch for border crossings; if you are detained and cannot make a "checkin", it'll make noise on your behalf so your friends and family know what's happened

Julia Evans reverse engineers Skyliner.io

jm — 2016-11-17T12:34:18+00:00

simple usage of Docker, blue/green deploys, and AWS ALBs

Medium’s DynamoDB Data Source for Apache Spark

jm — 2016-10-13T11:00:48+00:00

nice approach

Rubular

jm — 2016-07-05T14:16:22+00:00

'a Ruby regular expression editor and tester'. Great for prototyping regexps with a little set of test data, providing a neat permalink for the results

plainas/tq

jm — 2016-05-06T12:21:42+00:00

command line utility that performs an HTML element selection on HTML content passed to the stdin. Using css selectors that everybody knows. Since input comes from stdin and output is sent to stdout, it can easily be used inside traditional UNIX pipelines to extract content from webpages and html files. tq provides extra formating options such as json-encoding or newlines squashing, so it can play nicely with everyones favourite command line tooling.

The Make: Weekend Projects Thumbnail Guide To Soldering

jm — 2016-04-27T13:45:02+00:00

man, I wish I had this 30 years ago. now I know what stuff I need to get to make my occasional solders less of a PITA

Qualys SSL Server Test

jm — 2016-03-29T11:09:31+00:00

pretty sure I had this bookmarked previously, but this is the current URL -- SSL/TLS quality report

lbzip2

jm — 2016-03-08T12:03:43+00:00

a free, multi-threaded compression utility with support for bzip2 compressed file format. lbzip2 can process standard bz2 files in parallel. It uses POSIX threading model (pthreads), which allows it to take full advantage of symmetric multiprocessing (SMP) systems. It has been proven to scale linearly, even to over one hundred processor cores. lbzip2 is fully compatible with bzip2 – both at file format and command line level. Files created by lbzip2 can be decompressed by all versions of bzip2 and other software supporting bz2 format. lbzip2 can decompress any bz2 files in parallel. All bzip2 command-line options are also accepted by lbzip2. This makes lbzip2 a drop-in replacement for bzip2.

Plotly

jm — 2016-01-14T14:45:31+00:00

Online chart maker for CSV and Excel data; make charts and dashboards online. One really nice feature is that charts made this way get permalinks, and can be easily inlined as PNGs or HTML5 divs. (See https://www.vividcortex.com/blog/analyzing-sparks-mpp-scalability-with-the-usl for an example.)

Introducing dumb-init, an init system for Docker containers

jm — 2016-01-07T12:20:55+00:00

Yelp fixing one of the sillier shortcomings of Docker

Tools for debugging, testing and using HTTP/2

jm — 2015-12-10T14:06:56+00:00

excellent, extensive list from Cloudflare

Pixelmator

jm — 2015-12-04T12:01:39+00:00

Image editing tool for the Mac, recommended by Oisin

Google Cloud Shell

jm — 2015-10-05T11:14:03+00:00

your command line environment in the [Google] Cloud. This feature enables you to connect to a shell environment on a virtual machine, pre-loaded with the tools you need to easily run commands to develop, deploy and manage your projects. Currently, Cloud Shell is an f1-micro Google Compute Engine machine that exposes a Debian-based development environment. You are also assigned 5 GB of standard persistent disk space as the home disk so you can store files between sessions.

It's also free. This is a great idea -- handy both for beginners getting to grips with GoogCloud and for experts looking for a quite dev env to hack with. I wish AWS had something similar.

CurrencyFair P2P International Money Transfers

jm — 2015-09-30T10:02:09+00:00

recommended by Paul Hickey

httpry

jm — 2015-09-21T11:12:42+00:00

a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.

via Eoin Brazil

GoTTY

jm — 2015-09-01T12:39:42+00:00

'a simple command line tool that turns your CLI tools into web applications'

Miller

jm — 2015-08-24T16:01:47+00:00

'like sed, awk, cut, join, and sort for name-indexed data such as CSV'

Written in "modern C" with zero runtime dependencies. Looks great

peco

jm — 2015-06-25T16:58:21+00:00

'Simplistic interactive filtering tool' -- live incremental-search filtering in a terminal window

sjk

jm — 2015-06-25T16:49:38+00:00

a command line tool for JVM diagnostic troubleshooting and profiling.

Ag: faster than Ack

jm — 2015-03-22T09:01:30+00:00

Some nice performance tricks; I particularly like the use of sljit:

Ag uses Pthreads to take advantage of multiple CPU cores and search files in parallel. Files are mmap()ed instead of read into a buffer. Literal string searching uses Boyer-Moore strstr. Regex searching uses PCRE's JIT compiler (if Ag is built with PCRE >=8.21). Ag calls pcre_study() before executing the same regex on every file. Instead of calling fnmatch() on every pattern in your ignore files, non-regex patterns are loaded into arrays and binary searched.

OpenJDK: jol

jm — 2015-02-16T11:17:41+00:00

'JOL (Java Object Layout) is the tiny toolbox to analyze object layout schemes in JVMs. These tools are using Unsafe, JVMTI, and Serviceability Agent (SA) heavily to decoder the actual object layout, footprint, and references. This makes JOL much more accurate than other tools relying on heap dumps, specification assumptions, etc.' Recommended by Nitsan Wakart, looks pretty useful for JVM devs

yahoo/kafka-manager

jm — 2015-02-13T10:59:11+00:00

A tool for managing Apache Kafka. It supports the following : Manage multiple clusters; Easy inspection of cluster state (topics, brokers, replica distribution, partition distribution); Run preferred replica election; Generate partition assignments (based on current state of cluster); Run reassignment of partition (based on generated assignments)

wrk2

jm — 2014-11-10T15:45:50+00:00

'A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency -- corrects for Coordinated Omission error:

wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.

sferik/t

jm — 2014-10-04T13:53:28+00:00

"A command-line power tool for Twitter." It really is -- much better timeline searchability than the "real" Twitter UI, for example