https://pinboard.in/u:jm/public/ recent bookmarks from jm 2025-03-06T11:27:32+00:00 https://www.morling.dev/blog/kip-932-queues-for-kafka/ jm KIP-932 adds a long awaited capability to the Apache Kafka project: queue-like semantics, including the ability to acknowledge messages on a one-by-one basis. This positions Kafka for use cases such as job queuing, for which it hasn’t been a good fit historically. As multiple members of a share group can process the messages from a single topic partition, the partition count does not limit the degree of consumer parallelism any longer. The number of consumers in a group can quickly be increased and decreased as needed, without requiring to repartition the topic. [....] Available as an early access feature as of the [unreleased] Kafka 4.0 release, Kafka queues are not recommended for production usage yet, and there are several limitations worth calling out: most importantly, the lack of DLQ support. More control over retry timing would be desirable, too. As such, I don’t think Kafka queues in their current form will make users of established queue solutions such as Artemis or RabbitMQ migrate to Kafka. It is a very useful addition to the Kafka feature set nevertheless, coming in handy for instance for teams already running Kafka and who look for a solution for simple queuing use cases, avoiding to stand up and operate a separate solution just for these. This story will become even more compelling if the feature gets built out and improved in future Kafka releases. https://jack-vanlightly.com/blog/2023/10/2/the-advantages-of-queues-on-logs is a fantastic article by Jack Vanlightly on the innards of a queue built atop a log, like Kafka's KIP-932.]]> kafka queueing queues architecture logs https://pinboard.in/ https://pinboard.in/u:jm/b:b5eb93efb452/ 2024-08-26T09:47:38+00:00 https://softwaremill.com/mqperf/ jm activemq kafka rabbitmq messaging queueing message-queues sqs postgres storage https://pinboard.in/ https://pinboard.in/u:jm/b:d187c123119b/ 2022-05-21T10:56:37+00:00 https://dublinairportwait.ie/ jm security-queues queueing dublin-airport travel airports dublin ireland holidays https://pinboard.in/ https://pinboard.in/u:jm/b:9d7d46ddadd9/ 2018-11-15T10:40:53+00:00 http://richardstartin.uk/tuning-spark-back-pressure-by-simulation/ jmSpark back pressure, which can be enabled by setting spark.streaming.backpressure.enabled=true, will dynamically resize batches so as to avoid queue build up. It is implemented using a Proportional Integral Derivative (PID) algorithm. This algorithm has some interesting properties, including the lack of guarantee of a stable fixed point. This can manifest itself not just in transient overshoot, but in a batch size oscillating around a (potentially optimal) constant throughput. The overshoot incurs latency; the undershoot costs throughput. Catastrophic overshoot leading to OOM is possible in degenerate circumstances (you need to choose the parameters quite deviously to cause this to happen). Having witnessed undershoot and slow recovery in production streaming jobs, I decided to investigate further by testing the algorithm with a simulator. ]]> backpressure streaming queueing pid-controllers algorithms congestion-control https://pinboard.in/ https://pinboard.in/u:jm/b:eb53955f6313/ 2018-07-11T12:12:36+00:00 https://segment.com/blog/introducing-centrifuge/ jm queueing architecture dead-letter-queue kafka segment multi-tenant isolation https://pinboard.in/ https://pinboard.in/u:jm/b:38ab90c50fc7/ 2017-06-26T11:00:30+00:00 https://atscaleconference.com/videos/cadence-microservice-architecture-beyond-requestreply/ jm swf apis microservices uber cadence asynchronous request-reply distcomp queueing middleware go https://pinboard.in/ https://pinboard.in/u:jm/b:f33a1bf556d4/ 2016-12-14T11:21:39+00:00 https://eng.uber.com/cherami/ jm a competing-consumer messaging queue that is durable, fault-tolerant, highly available and scalable. We achieve durability and fault-tolerance by replicating messages across storage hosts, and high availability by leveraging the append-only property of messaging queues and choosing eventual consistency as our basic model. Cherami is also scalable, as the design does not have single bottleneck. [...] Cherami is completely written in Go, a language that makes building highly performant and concurrent system software a lot of fun. Additionally, Cherami uses several libraries that Uber has already open sourced: TChannel for RPC and Ringpop for health checking and group membership. Cherami depends on several third-party open source technologies: Cassandra for metadata storage, RocksDB for message storage, and many other third-party Go packages that are available on GitHub. We plan to open source Cherami in the near future. ]]> cherami uber queueing tasks queues architecture scalability go cassandra rocksdb https://pinboard.in/ https://pinboard.in/u:jm/b:08be41dbc892/ 2016-11-18T13:13:17+00:00 https://aws.amazon.com/blogs/aws/new-for-amazon-simple-queue-service-fifo-queues-with-exactly-once-delivery-deduplication/ jm fifo queues queueing architecture aws sqs https://pinboard.in/ https://pinboard.in/u:jm/b:41a866c8fba4/ 2016-05-18T10:21:37+00:00 https://groups.google.com/forum/#!msg/hystrixoss/mCCXEwSXb9M/bi_XHKIDAAAJ jm threadpools pooling hystrix capacity queue-theory queueing queues failure resilience soa microservices https://pinboard.in/ https://pinboard.in/u:jm/b:64afc24a5d6c/ 2016-03-07T11:24:20+00:00 https://github.com/conversant/disruptor jm disruptor blocking-queues queues queueing data-structures algorithms java conversant concurrency performance https://pinboard.in/ https://pinboard.in/u:jm/b:502056282b58/ 2016-01-06T12:42:41+00:00 https://blog.engineyard.com/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you jm database mysql queueing queue messaging percona rds locking sql architecture https://pinboard.in/ https://pinboard.in/u:jm/b:5081114f1b88/ 2015-11-04T16:28:06+00:00 https://github.com/facebook/wangle/blob/master/wangle/concurrent/Codel.h jm wangle facebook codel services capacity reliability queueing https://pinboard.in/ https://pinboard.in/u:jm/b:646392c5785b/ 2015-10-29T16:37:01+00:00 http://www.confluent.io/blog/apache-kafka-purgatory-hierarchical-timing-wheels jmIn the new design, we use Hierarchical Timing Wheels for the timeout timer and DelayQueue of timer buckets to advance the clock on demand. Completed requests are removed from the timer queue immediately with O(1) cost. The buckets remain in the delay queue, however, the number of buckets is bounded. And, in a healthy system, most of the requests are satisfied before timeout, and many of the buckets become empty before pulled out of the delay queue. Thus, the timer should rarely have the buckets of the lower interval. The advantage of this design is that the number of requests in the timer queue is the number of pending requests exactly at any time. This allows us to estimate the number of requests need to be purged. We can avoid unnecessary purge operation of the watcher lists. As the result we achieve a higher scalability in terms of request rate with much better CPU usage. ]]> algorithms timers kafka scheduling timing-wheels delayqueue queueing https://pinboard.in/ https://pinboard.in/u:jm/b:60858eeba501/ 2015-06-04T10:04:02+00:00 https://blog.twitter.com/2015/flying-faster-with-twitter-heron jm analytics architecture twitter storm heron backpressure streaming realtime queueing https://pinboard.in/ https://pinboard.in/u:jm/b:8bb89823ffcc/ 2015-05-06T11:19:20+00:00 http://www.developer-tech.com/news/2014/jun/10/why-loggly-loves-apache-kafka-how-unbreakable-infinitely-scalable-messaging-makes-log-management-better/ jm scalability logging loggly kafka queueing ops reliabilty https://pinboard.in/ https://pinboard.in/u:jm/b:61df5b7109a3/ 2015-02-03T00:40:40+00:00 http://tech.forter.com/comparing-message-queue-architectures-on-aws/ jmIf you are light on DevOps and not latency sensitive use SQS for job management and Kinesis for event stream processing. If latency is an issue, use ELB or 2 RabbitMQs (or 2 beanstalkds) for job management and Redis for event stream processing. ]]> amazon architecture aws messaging queueing elb rabbitmq beanstalk kinesis sqs redis kafka https://pinboard.in/ https://pinboard.in/u:jm/b:ae3870806c75/ 2014-11-25T12:06:52+00:00 https://github.com/sclasen/event-shuttle jmUNIX system service [jmason: ie a sidecar] that collects events and reliably delivers them to kafka, relieving other services on the same system from having to do so. Journals events through bolt-db so that in the event of an kafka outage, events can still be accepted, and will be delivered when kafka becomes available. ]]> kafka messaging ruby go events fault-tolerance queueing https://pinboard.in/ https://pinboard.in/u:jm/b:58fdafc74a8c/ 2014-11-17T17:52:27+00:00 http://highscalability.com/blog/2014/11/17/aeron-do-we-really-need-another-messaging-system.html jm aeron messing libraries java martin-thompson performance mechanical-sympathy queueing ipc tcp https://pinboard.in/ https://pinboard.in/u:jm/b:167738c3e8fa/ 2014-05-29T15:35:02+00:00 https://projects.eclipse.org/proposals/moquette-mqtt jma Java implementation of an MQTT 3.1 broker. Its code base is small. At its core, Moquette is an events processor; this lets the code base be simple, avoiding thread sharing issues. The Moquette broker is lightweight and easy to understand so it could be embedded in other projects. ]]> mqtt moquette netty messaging queueing push-notifications iot internet push eclipse https://pinboard.in/ https://pinboard.in/u:jm/b:258866cdaffb/ 2014-04-22T22:20:43+00:00 http://psy-lob-saw.blogspot.ie/2014/04/notes-on-concurrent-ring-buffer-queue.html jm jaq nitsanw atomic concurrency data-structures ring-buffers queueing queues algorithms https://pinboard.in/ https://pinboard.in/u:jm/b:ce4599f4a093/ 2014-02-14T15:36:17+00:00 http://webcache.googleusercontent.com/search?q=cache%3Awww.snookles.com%2Fslf-blog%2F2012%2F01%2F05%2Ftcp-incast-what-is-it%2F&oq=cache%3Awww.snookles.com%2Fslf-blog%2F2012%2F01%2F05%2Ftcp-incast-what-is-it%2F&aqs=chrome..69i57j69i58.795j0j4&sourceid=chrome&espv=210&es_sm=91&ie=UTF-8 jm incast tcp networking bandwidth riak architecture erlang buffering queueing https://pinboard.in/ https://pinboard.in/u:jm/b:895ea7fbed24/ 2013-11-20T12:36:47+00:00 http://reverbrain.com/grape/ jm kafka queue queueing storage realtime fault-tolerance grape cep event-processing https://pinboard.in/ https://pinboard.in/u:jm/b:269fc0b4ff60/ 2013-04-10T22:45:22+00:00 http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/#excursus-topics-partitions-and-replication-in-kafka jm kafka replication queueing distributed ops https://pinboard.in/ https://pinboard.in/u:jm/b:8a53eb32ea1a/ 2013-04-10T22:24:27+00:00 https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ jm performance kafka apache benchmarks ops queueing https://pinboard.in/ https://pinboard.in/u:jm/b:2f67886faaa2/ 2013-02-18T10:36:49+00:00 http://rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics jm heroku rap-genius via:hn networking distcomp distributed load-balancing ip queueing percentiles monitoring https://pinboard.in/ https://pinboard.in/u:jm/b:125334f62bdf/ 2011-11-23T09:52:35+00:00 http://stackoverflow.com/questions/6559308/how-does-lmaxs-disruptor-pattern-work jm via:filippo servers seda queueing concurrency disruptor patterns latency trading performance ring-buffers https://pinboard.in/ https://pinboard.in/u:jm/b:388ee9929e15/ 2011-09-20T21:04:42+00:00 https://github.com/nathanmarz/storm/wiki/Rationale jm data scaling twitter realtime scalability storm queueing https://pinboard.in/ https://pinboard.in/u:jm/b:0781f3875d18/ 2011-07-20T00:43:14+00:00 http://devblog.bu.mp/how-we-use-redis-at-bump jm queueing redis nosql databases storage via:simonw replication bump https://pinboard.in/ https://pinboard.in/u:jm/b:70e64dd1ac5d/ 2010-03-31T11:01:16+00:00 http://lists.openamq.org/pipermail/openamq-dev/2010-March/001598.html jm queueing amqp zeromq imatix mq protocols openamq via:janl https://pinboard.in/u:jm/b:f3c179d80f3e/ 2009-11-05T22:22:53+00:00 http://github.com/blog/542-introducing-resque jm ruby github queueing ipc resque https://pinboard.in/u:jm/b:95ccce78d834/