https://pinboard.in/u:jm/public/ recent bookmarks from jm 2022-09-14T17:05:24+00:00 https://engineering.linkedin.com/blog/2022/real-time-analytics-on-network-flow-data-with-apache-pinot jmInFlow requires storage of tens of TBs of data with a retention of 30 days. To support its real-time troubleshooting use case, the data must be queryable in real-time with sub-second latency so that engineers can query the data without any hassles during outages. For the storage layer, InFlow leverages Apache Pinot. ]]> pinot latency metrics linkedin network-flows realtime analytics storage https://pinboard.in/ https://pinboard.in/u:jm/b:d408bc8a0e1e/ 2017-06-21T10:43:57+00:00 https://engineering.riotgames.com/news/determinism-league-legends-introduction jm clock realtime time determinism testing replay games league-of-legends via:nelson https://pinboard.in/ https://pinboard.in/u:jm/b:14fb1751e281/ 2016-02-10T15:48:45+00:00 http://jvns.ca/blog/2016/02/09/til-clock-skew-exists/ jm clocks clock-skew ntp realtime time bugs distcomp reliability skew https://pinboard.in/ https://pinboard.in/u:jm/b:9eea6afc942d/ 2016-01-03T10:20:18+00:00 https://samsaffron.com/archive/2015/12/29/websockets-caution-required jmThere are very valid technical reasons many of the biggest sites on the Internet have not adopted them. Twitter use HTTP/2 + polling, Facebook and Gmail use Long Polling. Saying WebSockets are the only way and the way of the future, is wrongheaded. HTTP/2 may end up winning this battle due to the huge amount of WebSocket connections web browsers allow, and HTTP/3 may unify the protocols ]]> http realtime websockets long-polling http2 protocols transport web internet https://pinboard.in/ https://pinboard.in/u:jm/b:c7af3e72be13/ 2015-08-31T14:06:02+00:00 https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude/ jm lambda-architecture analytics via:hn redis set-storage storage databases architecture s3 realtime https://pinboard.in/ https://pinboard.in/u:jm/b:cb0a1f26939e/ 2015-06-15T16:04:27+00:00 http://blog.acolyer.org/2015/06/15/twitter-heron-stream-processing-at-scale/ jm storm twitter heron big-data streaming realtime backpressure https://pinboard.in/ https://pinboard.in/u:jm/b:39ffe9705551/ 2015-06-04T10:04:02+00:00 https://blog.twitter.com/2015/flying-faster-with-twitter-heron jm analytics architecture twitter storm heron backpressure streaming realtime queueing https://pinboard.in/ https://pinboard.in/u:jm/b:8bb89823ffcc/ 2015-01-16T22:33:37+00:00 http://betterembsw.blogspot.ie/2014/09/a-case-study-of-toyota-unintended.html?m=1 jmInvestigations into potential causes of Unintended Acceleration (UA) for Toyota vehicles have made news several times in the past few years. Some blame has been placed on floor mats and sticky throttle pedals. But, a jury trial verdict was based on expert opinions that defects in Toyota's Electronic Throttle Control System (ETCS) software and safety architecture caused a fatal mishap.  This talk will outline key events in the still-ongoing Toyota UA litigation process, and pull together the technical issues that were discovered by NASA and other experts. The results paint a picture that should inform future designers of safety critical software in automobiles and other systems. ]]> toyota safety realtime coding etcs throttle-control nasa code-review embedded https://pinboard.in/ https://pinboard.in/u:jm/b:e6e6d56be7de/ 2014-11-12T10:08:56+00:00 http://gotocon.com/dl/goto-berlin-2014/slides/MartinThompson_AeronTheNextGenerationInOpenSourceHighPerformanceMessaging.pdf jm realtime messaging pub-sub ipc queues transports martin-thompson slides latencies open-source java libraries https://pinboard.in/ https://pinboard.in/u:jm/b:63cce56da5d7/ 2014-11-06T20:41:44+00:00 https://www.linkedin.com/pulse/article/20141106180403-2945786-announcing-confluent-a-company-for-apache-kafka-and-realtime-data jm realtime event-processing logs kafka streaming open-source jay-kreps jun-rao confluent https://pinboard.in/ https://pinboard.in/u:jm/b:5d37bc318bf2/ 2014-10-06T15:21:27+00:00 http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/ jmSpark Streaming has been getting some attention lately as a real-time data processing tool, often mentioned alongside Apache Storm. [...] I added an example Spark Streaming application to kafka-storm-starter that demonstrates how to read from Kafka and write to Kafka, using Avro as the data format and Twitter Bijection for handling the data serialization. In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with Spark Streaming. ]]> spark kafka realtime architecture queues avro bijection batch-processing https://pinboard.in/ https://pinboard.in/u:jm/b:2ff12c1b1ca4/ 2014-09-17T14:29:55+00:00 http://sourceforge.net/projects/ccfreaks/files/papers/LeftRight/leftright-extended.pdf jm left-right concurrency multithreading wait-free blocking realtime gc latency reader-writer locking synchronization java https://pinboard.in/ https://pinboard.in/u:jm/b:71e20ba2b8b1/ 2014-08-22T15:13:35+00:00 https://blog.twitter.com/2014/fighting-spam-with-botmaker jmThe main challenges in supporting this type of system are evaluating rules with low enough latency that they can run on the write path for Twitter’s main features (i.e., Tweets, Retweets, favorites, follows and messages), supporting computationally intense machine learning based rules, and providing Twitter engineers with the ability to modify and create new rules instantaneously. ]]> spam realtime scaling twitter anti-spam botmaker rules https://pinboard.in/ https://pinboard.in/u:jm/b:608530e05a81/ 2014-08-09T11:32:45+00:00 http://www.slideshare.net/tantrieuf31/real-time-analytics-with-netty-storm-kafka jm via:trustinlee kafka storm netty architecture ad-tracking ads realtime https://pinboard.in/ https://pinboard.in/u:jm/b:fe33709a254d/ 2014-05-08T13:34:52+00:00 https://news.ycombinator.com/item?id=7711974 jmat higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing. ]]> python languages concurrency go threading gevent scalability disqus realtime hn https://pinboard.in/ https://pinboard.in/u:jm/b:7735324765d8/ 2014-04-30T10:57:02+00:00 https://speakerdeck.com/pyconslides/scaling-realtime-at-disqus-by-adam-hitchcock jm sysctl nginx tuning go disqus realtime push eventsource websockets long-polling iframe python https://pinboard.in/ https://pinboard.in/u:jm/b:92ccc8b3f3ef/ 2013-11-25T11:10:35+00:00 http://yahooeng.tumblr.com/post/65453012905/introducing-samoa-an-open-source-platform-for-mining jm As a library, SAMOA contains state-of-the-art implementations of algorithms for distributed machine learning on streams. The first alpha release allows classification and clustering. For classification, we implemented a Vertical Hoeffding Tree (VHT), a distributed streaming version of decision trees tailored for sparse data (e.g., text). For clustering, we included a distributed algorithm based on CluStream. The library also includes meta-algorithms such as bagging. ]]> storm streaming big-data realtime samoa yahoo machine-learning ml decision-trees clustering bagging classification https://pinboard.in/ https://pinboard.in/u:jm/b:785df99d0cee/ 2013-11-20T12:36:47+00:00 http://reverbrain.com/grape/ jm kafka queue queueing storage realtime fault-tolerance grape cep event-processing https://pinboard.in/ https://pinboard.in/u:jm/b:269fc0b4ff60/ 2013-10-30T10:55:21+00:00 http://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences jm The Camry electronic throttle control system code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable). Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. Barr also discovered inadequate and untracked peer code reviews and the absence of any bug-tracking system at Toyota. On top of this, there was no error-correcting RAM in use; stack-killing recursive code; a quoted 94% stack usage; risks of unintentional RTOS task shutdown; buffer overflows; unsafe casting; race conditions; unchecked error code return values; and a trivial watchdog timer check. Crappy, unsafe coding.]]> firmware horror embedded-systems toyota camry safety acceleration misra-c coding code-verification spaghetti-code cyclomatic-complexity realtime rtos c code-reviews bug-tracking quality https://pinboard.in/ https://pinboard.in/u:jm/b:acba21cb4f78/ 2013-10-29T11:18:21+00:00 http://www.slideshare.net/ashleywbrown/storm-at-spiderio-london-storm-meetup-20130618 jm storm realtime hadoop cascading python cep spider.io anti-spam events architecture distcomp low-latency slides rabbitmq https://pinboard.in/ https://pinboard.in/u:jm/b:3fcd16536c76/ 2013-06-26T09:38:27+00:00 https://www.facebook.com/notes/facebook-engineering/wormhole-pubsub-system-moving-data-through-space-and-time/10151504075843920 jmOver the last couple of years, we have built and deployed a reliable publish-subscribe system called Wormhole. Wormhole has become a critical part of Facebook's software infrastructure. At a high level, Wormhole propagates changes issued in one system to all systems that need to reflect those changes – within and across data centers. Facebook's Kafka-alike, basically, although with some additional low-latency guarantees. FB appear to be using it for multi-region and multi-AZ replication. Proprietary.]]> pub-sub scalability facebook realtime low-latency multi-region replication multi-az wormhole https://pinboard.in/ https://pinboard.in/u:jm/b:c16235547374/ 2013-05-14T21:03:27+00:00 http://www.infoq.com/interviews/ennis-events jm darach-ennis stream-processing messaging architecture qcon interviews erlang cep realtime rx comet events https://pinboard.in/ https://pinboard.in/u:jm/b:262333000223/ 2013-03-20T12:09:20+00:00 https://developers.google.com/drive/realtime/ jm collaboration api realtime google javascript https://pinboard.in/ https://pinboard.in/u:jm/b:69a2edc2ffba/ 2012-10-08T11:06:01+00:00 http://engineering.twitter.com/2012/08/trident-high-level-abstraction-for.html jm Trident is a new high-level abstraction for doing realtime computing on top of Twitter Storm, available in Storm 0.8.0. It allows you to seamlessly mix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar - Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies. ]]> distributed realtime twitter storm trident distcomp stream-processing low-latency nathan-marz https://pinboard.in/ https://pinboard.in/u:jm/b:b7481a5f029d/ 2011-10-22T22:46:38+00:00 http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html jm nathan-marz realtime batch hadoop storm big-data cap https://pinboard.in/ https://pinboard.in/u:jm/b:20004ab62fdd/ 2011-09-20T21:57:37+00:00 http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation jm distcomp distributed realtime storm slides twitter https://pinboard.in/ https://pinboard.in/u:jm/b:9652fbca2b57/ 2011-09-20T21:04:42+00:00 https://github.com/nathanmarz/storm/wiki/Rationale jm data scaling twitter realtime scalability storm queueing https://pinboard.in/ https://pinboard.in/u:jm/b:0781f3875d18/ 2011-05-23T23:13:12+00:00 http://www.slideshare.net/nathanmarz/the-secrets-of-building-realtime-big-data-systems jm hadoop big-data data scalability datamining realtime slides presentations https://pinboard.in/ https://pinboard.in/u:jm/b:cddf36f77523/ 2010-01-12T20:59:02+00:00 http://russ.garrett.co.uk/2010/01/10/semi-realtime-satellite-desktop-backgrounds/ jm weather desktop image satellite realtime backgrounds https://pinboard.in/u:jm/b:b56bda56c3a1/ 2009-07-24T15:48:47+00:00 http://dashes.com/anil/2009/07/the-pushbutton-web-realtime-becomes-real.html jm http-push http feeds atom ping standards messaging pubsubhubbub pubsub async comet realtime web https://pinboard.in/u:jm/b:470f67ac4575/