https://pinboard.in/u:jm/public/ recent bookmarks from jm 2026-06-03T09:26:47+00:00 https://www.coinbase.com/en-fr/blog/a-postmortem-of-our-may-7-2026-outage jmAWS's managed Kafka service failed silently. A significant portion of our event-streaming infrastructure runs on MSK, AWS's managed Kafka offering. The architectural promise of a managed Kafka service is that when individual brokers go down, the service automatically reelects partition leaders and continues to serve traffic out of the remaining brokers. The loss of an entire zone should result in reduced capacity, not unavailability. That is not what happened and this extended the outage. A defect in the AWS MSK control plane prevented automatic partition-leader reelection. Two of our MSK clusters became stuck in a "healing" state with producers unable to write. The cascading effect blocked our fee service, which blocked quoting, which is why most customers experienced this incident as broken trades and quotes rather than as a Kafka outage. Adjacent systems, including portions of our ledger pipeline, payments, and several data pipelines, were affected the same way. Additionally, one of our Kafka clusters was set up in a 2-AZ configuration that increased the blast radius and recovery time, but the MSK control plane defect impacted 2-AZ and 3-AZ Kafka clusters similarly. We worked the recovery in real time with AWS engineering, ultimately performing manual partition reassignments at 3:00 AM ET to migrate topics off the impaired brokers. Priority-zero and priority-one topics were back to full availability by 9:30 AM ET. The remainder cleared by 2:00 PM ET. In fairness, they also had a single-AZ point of failure in their architecture which they also describe there, but still, not great performance from MSK. Disappointing. ]]> msk reliability multi-az aws services kafka resiliency outages post-mortems postmortems coinbase https://pinboard.in/ https://pinboard.in/u:jm/b:7124695a3aec/ 2025-03-24T15:10:54+00:00 https://aws.amazon.com/blogs/compute/handling-billions-of-invocations-best-practices-from-aws-lambda/?ck_subscriber_id=512829374#415:%20Google%20Takes%20Wiz%20-%2017033042 jmDrawing inspiration from the “The Power of Two Random Choices” paper, the Lambda team explored the shuffle-sharding technique for its asynchronous invocations processing. Using this technique, you shuffle-shard tenants into several randomly assigned queues. Upon receiving an asynchronous invocation, you place the message in the queue with the smallest backlog to optimize load distribution. This approach helps to minimize the likelihood of assigning tenants to a busy queue. [....] The shuffle-sharding technique proved remarkably effective. By distributing tenants across shards, the approach ensures that only a very small subset of tenants could be affected by a noisy neighbor. The potential impact is also minimized since each affected tenant maintains access to unaffected queues. As your workloads grow, increasing the number of queues enhances resilience and further reduces the probability of multiple tenants being assigned to the same shard. This significantly lowers the risk of a single point of failure, making shuffle sharding a robust strategy for workload isolation and fault tolerance. Automated Isolation, covered in the next section, is also a neat trick. (via Last Week In AWS)]]> via:lwia sharding architecture services horizontal-scaling shuffle-sharding algorithms load-balancing async queues aws multitenant https://pinboard.in/ https://pinboard.in/u:jm/b:b407ba9caf17/ 2025-03-21T10:30:41+00:00 https://byroot.github.io/ruby/performance/2025/03/04/the-pitchfork-story.html jm ruby performance optimisation optimization heap memory fork forking http services servers monolith rails gc https://pinboard.in/ https://pinboard.in/u:jm/b:2839524a8511/ 2024-07-10T22:46:42+00:00 https://encore.dev/blog/retries jm retries retrying backoff jitter networking soa services interactive demos https://pinboard.in/ https://pinboard.in/u:jm/b:25d687fb2382/ 2024-03-04T10:28:26+00:00 https://answersforaws.com/blog/2024-02-29-results/ jm aws services ops infrastructure architecture sqs sns dynamodb github-actions ecs via:lastweekinaws https://pinboard.in/ https://pinboard.in/u:jm/b:04ea95c6ae80/ 2023-10-12T13:29:48+00:00 https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/single-region-scenarios.html jm via:singer aws reliability architecture availability uptime services ops high-availability https://pinboard.in/ https://pinboard.in/u:jm/b:f9f9b0a1c1c2/ 2022-08-08T10:21:17+00:00 https://www.alexdebrie.com/posts/dynamodb-paper/ jm scalability scaling dynamodb aws storage services architecture https://pinboard.in/ https://pinboard.in/u:jm/b:3f183f470778/ 2021-10-12T12:10:25+00:00 https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/ jm error-handling distributed-systems http aws services soa david-yanacek load load-shedding uptime reliability https://pinboard.in/ https://pinboard.in/u:jm/b:c8ea792bb37a/ 2020-04-22T22:02:21+00:00 https://github.com/Pugmatt/BedrockConnect jm minecraft bedrock hacks interop services xbox ps4 nintendo-switch cross-play https://pinboard.in/ https://pinboard.in/u:jm/b:1ae566b031a8/ 2019-10-02T15:57:31+00:00 https://blog.linuxserver.io/2019/10/01/updating-and-backing-up-docker-containers-with-version-control/ jm docker home ops services upgrades docker-compose version-control https://pinboard.in/ https://pinboard.in/u:jm/b:0af56745e34e/ 2019-01-08T12:19:42+00:00 https://summitroute.github.io/aws_research/service_support.html jmThe goal of this page is to high-light the lack of coverage AWS provides for its services across different security factors. These limitations are not well-understood by many. Further, the "Y" fields are meant to indicate that this service has any capability for the relevant factor. In many cases, this is not full coverage for the service, or there are exceptions or special cases. ]]> amazon aws services slas ops reliability https://pinboard.in/ https://pinboard.in/u:jm/b:ccd898c6775e/ 2018-11-26T16:08:28+00:00 https://blog.linkerd.io/2018/09/18/announcing-linkerd-2-0/ jmThe 2.0 release of Linkerd brings two very significant changes. First, we’ve completely rewritten Linkerd to be orders of magnitude faster and smaller than Linkerd 1.x. Linkerd 2.0’s data plane is comprised of ultralight Rust proxies which consume around 10mb of RSS and have a p99 latency of <1ms. Linkerd’s minimalist control plane (written in Go) is similarly designed for speed and low resource footprint. Second, with the 2.0 release, Linkerd moves beyond the service mesh model to be something not more, but less: Linkerd 2.0 at its core is a service sidecar, running on a single service without requiring cluster-wide installation. This means that if you’re a developer or service owner who doesn’t have access to the whole Kubernetes cluster, you can run Linkerd on your service and get: Instant Grafana dashboards of your service’s success rates, latencies, and throughput; A topology graph of incoming and outgoing dependencies; A live view of requests being made to your service; Improved, latency-aware load balancing; … and much more. ]]> linkerd mesh networking services architecture sidecars https://pinboard.in/ https://pinboard.in/u:jm/b:1d41240fa54d/ 2018-10-22T15:41:47+00:00 http://www.dublincity.ie/main-menu-services-water-waste-and-environment-waste-and-recycling/bulky-household-waste-collection jm waste rubbish household dublin dcc services https://pinboard.in/ https://pinboard.in/u:jm/b:efe79ba5ed1f/ 2018-07-12T21:50:28+00:00 https://medium.com/@NetflixTechBlog/evolution-of-application-data-caching-from-ram-to-ssd-a33d6fa7a690 jmMemcached provides an external storage shim called extstore, that supports storing of data on SSD (I2) and NVMe (I3). extstore is efficient in terms of cost & storage device utilization without compromising the speed and throughput. All the metadata (key & other metadata) is stored in RAM whereas the actual data is stored on flash. ]]> memcached netflix services storage memory ssd nvme extstore caching https://pinboard.in/ https://pinboard.in/u:jm/b:119cb46419e3/ 2018-06-06T09:32:06+00:00 https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ jm amazon aws regions azs services architecture ops https://pinboard.in/ https://pinboard.in/u:jm/b:94e3251bc6d3/ 2018-01-15T11:23:58+00:00 http://www.dwmkerr.com/the-death-of-microservice-madness-in-2018/ jm architecture devops microservices services soa coding monoliths state systems https://pinboard.in/ https://pinboard.in/u:jm/b:0f831a39dc8a/ 2018-01-11T17:23:49+00:00 https://www.reddit.com/r/aws/comments/7phlq2/best_way_designing_a_gdpr_compliant_datalake/ jm gdpr reddit aws tips design services ops https://pinboard.in/ https://pinboard.in/u:jm/b:740cf65fe3c8/ 2018-01-03T10:55:38+00:00 https://www.webdoctor.ie/ jm doctor medicine ireland services health https://pinboard.in/ https://pinboard.in/u:jm/b:7839bb86c629/ 2017-11-30T10:36:11+00:00 https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-the-amazon-time-sync-service/ jm time-sync time aws services ntp ops https://pinboard.in/ https://pinboard.in/u:jm/b:2a60d299c58e/ 2017-11-29T17:09:02+00:00 https://aws.amazon.com/blogs/aws/aws-fargate/?sc_channel=sm&sc_campaign=AWS_reInvent&sc_publisher=TWITTER&sc_country=re:Invent&sc_geo=GLOBAL&sc_outcome=awareness&trk=sm_EC2_Container_Service_9625c5f9_Fargate_TWITTER&sc_content=EC2_Container_Service_9625c5f9_Fargate&sc_category=Amazon_Fargate&linkId=45301695 jm eks ecs fargate aws services ops containers docker https://pinboard.in/ https://pinboard.in/u:jm/b:ea5d427cbdf3/ 2017-06-26T14:00:14+00:00 https://github.com/anaibol/awesome-serverless/ jmA curated list of awesome services, solutions and resources for serverless / nobackend applications. ]]> serverless lambda ops devops saas services architecture lists https://pinboard.in/ https://pinboard.in/u:jm/b:8877523021d9/ 2017-03-20T14:25:19+00:00 https://github.com/atlassian/localstack jmLocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. Currently, the focus is primarily on supporting the AWS cloud stack. LocalStack spins up the following core Cloud APIs on your local machine: API Gateway at http://localhost:4567; Kinesis at http://localhost:4568; DynamoDB at http://localhost:4569; DynamoDB Streams at http://localhost:4570; Elasticsearch at http://localhost:4571; S3 at http://localhost:4572; Firehose at http://localhost:4573; Lambda at http://localhost:4574; SNS at http://localhost:4575; SQS at http://localhost:4576 Additionally, LocalStack provides a powerful set of tools to interact with the cloud services, including a fully featured KCL Kinesis client with Python binding, simple setup/teardown integration for nosetests, as well as an Environment abstraction that allows to easily switch between local and remote Cloud execution. ]]> aws emulation mocking services testing dynamodb s3 https://pinboard.in/ https://pinboard.in/u:jm/b:240a300dcd0c/ 2016-10-12T13:23:54+00:00 http://www.airfixers.com/ jm dublin airbnb management rental services https://pinboard.in/ https://pinboard.in/u:jm/b:e6a3abae80e6/ 2016-10-04T11:26:44+00:00 https://charity.wtf/2016/10/02/the-accidental-dba/ jmYou can’t just go “dudes it’s faster” and jump off a cliff.  This shit is basic.  Test real production workloads. Have a rollback plan.  (Not for *10 days* … try a month or two.) The only thing I'd nitpick on is that it's all very well to say "buy my book" or "come see me talk at Blahcon", but a good blog post or webpage would be thousands of times more useful.]]> databases stateful-services services ops mongodb charity-majors rollback state storage testing dba https://pinboard.in/ https://pinboard.in/u:jm/b:15609640fcf7/ 2015-12-10T14:10:02+00:00 http://blog.lusis.org/blog/2015/12/09/aws-apigateway-for-fun-and-profit/ jm api-gateway aws api http services ops alerting alarming opsgenie signalfx https://pinboard.in/ https://pinboard.in/u:jm/b:a5bbeed7fb8b/ 2015-12-07T12:04:50+00:00 https://medium.com/@AdamRNeary/a-gulp-workflow-for-amazon-lambda-61c2afd723b6#.nbn3nuamp jm lambda aws services testing deployment ops mocha gulp javascript https://pinboard.in/ https://pinboard.in/u:jm/b:04a4a810e7fd/ 2015-12-03T12:52:22+00:00 http://blog.cask.co/2015/12/introducing-netty-http-from-cask/ jmnetty-http library solves [Netty usability issues] by using JAX-RS annotations to build a HTTP path routing layer on top of netty. In addition, the library implements a guava service to manage the HTTP service. netty-http allows users of the library to just focus on writing the business logic in HTTP handlers without having to worry about the complexities of path routing or learning netty pipeline internals to build the HTTP service. We've written something very similar, although I didn't even bother supporting JAX-RS annotations -- just a simple code-level DSL.]]> jax-rs netty http cask java services coding https://pinboard.in/ https://pinboard.in/u:jm/b:c5c91942c05c/ 2015-11-04T16:28:06+00:00 https://github.com/facebook/wangle/blob/master/wangle/concurrent/Codel.h jm wangle facebook codel services capacity reliability queueing https://pinboard.in/ https://pinboard.in/u:jm/b:646392c5785b/ 2014-12-04T10:04:12+00:00 http://john-sheehan.com/blog/ultimate-api-webhook-backend-service-debugging-testing-monitoring-and-discovery-tools-list jm web apis webhooks http hacking services saas https://pinboard.in/ https://pinboard.in/u:jm/b:a4105ae0a14a/ 2014-07-07T11:48:09+00:00 https://hassle.com jm hailo cleaners hassle via:hailo domestic home services b2c https://pinboard.in/ https://pinboard.in/u:jm/b:bec46a2b8077/ 2014-03-25T10:26:55+00:00 http://arnon.me/2014/03/services-microservices-nanoservices/ jm architecture antipatterns nanoservices microservices soa services design esb https://pinboard.in/ https://pinboard.in/u:jm/b:a19528e98dbf/ 2014-03-20T15:19:10+00:00 http://literateprogrammer.blogspot.co.uk/2014/03/the-microservice-declaration-of.html jm microservices soa architecture handwaving http services web deployment https://pinboard.in/ https://pinboard.in/u:jm/b:3c13e569ef49/ 2014-03-10T12:07:09+00:00 http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf jm dapper tracing http services soa google papers request-tracing tracers protobuf devops https://pinboard.in/ https://pinboard.in/u:jm/b:7c7dff9b9866/ 2013-12-07T21:02:00+00:00 http://www.nickhunn.com/smart-metering-is-fcuked/ jmMost utilities don’t want smart metering.  In fact they seem to have used the wrong dictionary.  It is difficult to find anything smart about the UK deployment, until you realise that the utilities use smart in the sense of “it hurts”.  They consider they have a perfectly adequate business model which has no need for new technology.  In many Government meetings, their reluctant support seems to be a veneer for the hope that it will all end in disaster, letting them go back to the world they know, of inflated bills and demands for money with menaces. [...] Even when smart meters are deployed, there is no evidence that any utility will use the resulting data to transform their business, rather than persecute the consumer.  At a recent US conference a senior executive for a US utility which had deployed smart meters, stated that their main benefit was “to give them more evidence to blame the customer”.  That’s a good description of the attitude displayed by our utilities. ]]> smart-metering energy utilities uk services metering consumer https://pinboard.in/ https://pinboard.in/u:jm/b:62a961d0f3e3/ 2013-10-23T21:35:04+00:00 http://nerds.airbnb.com/smartstack-service-discovery-cloud/ jm haproxy services ops load-balancing service-discovery nerve synapse airbnb https://pinboard.in/ https://pinboard.in/u:jm/b:ee71b29b1825/ 2013-08-21T21:25:34+00:00 https://github.com/GeneralElectric/snowizard jman HTTP-based service for generating unique ID numbers at high scale with some simple guarantees. supports returning ID numbers as: JSON and JSONP; Google's Protocol Buffers; Plain text. At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don't have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddable module directly. Odd OSS license, though -- BSDish?]]> java open-source ids soa services snowflake http https://pinboard.in/ https://pinboard.in/u:jm/b:2cdc8b3567a7/ 2013-04-19T20:27:52+00:00 https://speakerdeck.com/mza/latencys-worst-nightmare-performance-tuning-tips-and-tricks jm benchmarks aws ec2 ebs piops services scaling scalability presentations https://pinboard.in/ https://pinboard.in/u:jm/b:0dad472cef4b/ 2013-01-27T21:11:16+00:00 https://github.com/infochimps-labs/ironfan/wiki/walkthrough-web jm chef deployment clusters knife services aws ec2 ops ironfan demo https://pinboard.in/ https://pinboard.in/u:jm/b:ed73e23aae92/ 2013-01-24T21:05:51+00:00 http://smarden.org/runit/index.html jm daemon supervision services unix lsb server ops https://pinboard.in/ https://pinboard.in/u:jm/b:c5a9592af5d4/ 2011-03-27T22:47:53+00:00 http://perspectives.mvdirona.com/2011/03/26/PrioritizingPrinciplasInOnDesigningAndDeployingInternetScaleServices.aspx jm internet software james-hamilton design architecture coding advice services scaling reliability https://pinboard.in/u:jm/b:aae2778d4b3c/ 2010-02-03T12:33:37+00:00 http://arstechnica.com/business/data-centers/2010/02/what-second-life-can-teach-all-companies-about-scaling-web-apps.ars/4 jm linden ian-wilkes scaling datacenters scalability deployment ops services https://pinboard.in/u:jm/b:29fd0403ad9b/