https://pinboard.in/u:jm/public/ recent bookmarks from jm 2023-10-24T17:07:10+00:00 https://kichik.com/2022/08/31/lessons-learned-from-1tb-dynamodb-import/ jm backfills dynamodb batch scaling aws https://pinboard.in/ https://pinboard.in/u:jm/b:b117711019c1/ 2022-05-10T09:26:37+00:00 https://trino.io/blog/2022/05/05/tardigrade-launch.html jmWhen your long-running queries experience a failure, they don’t have to start from scratch. When queries require more memory than currently available in the cluster they are still able to succeed. When multiple queries are submitted concurrently they are able to share resources in a fair way, and make steady progress. ]]> trino presto sql storage querying etl batch scheduling https://pinboard.in/ https://pinboard.in/u:jm/b:890646400b31/ 2020-01-22T12:14:32+00:00 https://medium.com/bigdatarepublic/a-review-of-netflixs-metaflow-65c6956e168d jm metaflow data-science data batch architecture https://pinboard.in/ https://pinboard.in/u:jm/b:98bc54a96c89/ 2019-02-28T10:44:37+00:00 https://argoproj.github.io/argo jm k8s kubernetes docker containers workflow pipelines architecture batch nightly-jobs ops https://pinboard.in/ https://pinboard.in/u:jm/b:0cd7daacad31/ 2018-11-27T20:38:15+00:00 https://codeascraft.com/2018/11/14/boundary-layer%e2%80%89-declarative-airflow-workflows/ jm airflow python batch cron etl https://pinboard.in/ https://pinboard.in/u:jm/b:5d4305c3f31a/ 2017-12-05T22:09:37+00:00 https://mapzen.com/blog/terrain-tiles-on-aws-batch/ jmUsing this setup on AWS Batch, we are able to generate more than 3.75 million tiles per minute and render the entire world in less than a week! These pre-rendered tiles get stored in S3 and are ready to use by anyone through the AWS Public Dataset or through Mapzen’s Terrain Tiles API. ]]> mapzen mapping tiles batch aws s3 lambda docker https://pinboard.in/ https://pinboard.in/u:jm/b:a932f8829a8e/ 2017-08-08T22:06:18+00:00 https://www.nextflow.io/ jmData-driven computational pipelines Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages. Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows on clouds and clusters. GPLv3 licensed, open source]]> computation workflows pipelines batch docker ops open-source https://pinboard.in/ https://pinboard.in/u:jm/b:52d4c861f8a6/ 2017-07-11T10:19:34+00:00 http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html jm aws batch ecs cron scheduling recurrence ops https://pinboard.in/ https://pinboard.in/u:jm/b:59ea65c27ff1/ 2016-10-19T10:22:04+00:00 https://www.youtube.com/watch?v=dgaoqOZlvEA&feature=youtu.be jm etl airflow batch architecture systems ops https://pinboard.in/ https://pinboard.in/u:jm/b:89b7e8acd127/ 2016-10-17T09:46:07+00:00 https://gtoonstra.github.io/etl-with-airflow/index.html jm etl airflow documentation best-practices batch architecture https://pinboard.in/ https://pinboard.in/u:jm/b:2c8c42ae1e66/ 2016-02-09T11:15:31+00:00 http://blogs.aws.amazon.com/bigdata/post/Tx578UTQUV7LRP/Submitting-User-Applications-with-spark-submit jm emr big-data spark hadoop yarn map-reduce batch https://pinboard.in/ https://pinboard.in/u:jm/b:1a28e3a538b2/ 2015-09-03T20:58:52+00:00 https://medium.com/@foundev/real-time-analytics-with-spark-streaming-and-cassandra-2f90d03342f7 jm spark-streaming kafka analytics cassandra architecture data batch https://pinboard.in/ https://pinboard.in/u:jm/b:b0d85df5ec49/ 2015-08-17T14:09:05+00:00 https://beta.oreilly.com/ideas/the-world-beyond-batch-streaming-101 jmTo summarize, in this post I’ve: Clarified terminology, specifically narrowing the definition of “streaming” to apply to execution engines only, while using more descriptive terms like unbounded data and approximate/speculative results for distinct concepts often categorized under the “streaming” umbrella. Assessed the relative capabilities of well-designed batch and streaming systems, positing that streaming is in fact a strict superset of batch, and that notions like the Lambda Architecture, which are predicated on streaming being inferior to batch, are destined for retirement as streaming systems mature. Proposed two high-level concepts necessary for streaming systems to both catch up to and ultimately surpass batch, those being correctness and tools for reasoning about time, respectively. Established the important differences between event time and processing time, characterized the difficulties those differences impose when analyzing data in the context of when they occurred, and proposed a shift in approach away from notions of completeness and toward simply adapting to changes in data over time. Looked at the major data processing approaches in common use today for bounded and unbounded data, via both batch and streaming engines, roughly categorizing the unbounded approaches into: time-agnostic, approximation, windowing by processing time, and windowing by event time. ]]> streaming batch big-data lambda-architecture dataflow event-processing cep millwheel data data-processing https://pinboard.in/ https://pinboard.in/u:jm/b:ef8048836663/ 2015-06-19T07:47:04+00:00 http://blog.acolyer.org/2015/06/19/discretized-streams-fault-tolerant-stream-computing-at-scale/ jmwe use a data structure called Resilient Distributed Datasets (RDDs), which keeps data in memory and can recover it without replication by tracking the lineage graph of operations that were used to build it. With RDDs, we show that we can attain sub-second end-to-end latencies. We believe that this is sufficient for many real-world big data applications, where the timescale of the events tracked (e.g., trends in social media) is much higher. ]]> rdd spark streaming fault-tolerance batch distcomp papers big-data scalability https://pinboard.in/ https://pinboard.in/u:jm/b:561d8372a2de/ 2015-06-05T09:53:57+00:00 http://nerds.airbnb.com/airflow/ jm airbnb open-source python workflow jobs cron scheduling batch https://pinboard.in/ https://pinboard.in/u:jm/b:b89f8e9c803f/ 2015-02-23T21:54:41+00:00 http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-tips-for-writing-better-spark-programs jm spark performance batch ops tips slides emr https://pinboard.in/ https://pinboard.in/u:jm/b:aec80aa94d61/ 2015-01-17T21:33:33+00:00 http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html jmHere are two helpful guidelines (for largely disjoint populations): If you are going to use a big data system for yourself, see if it is faster than your laptop. If you are going to build a big data system for others, see that it is faster than my laptop. [...] We think everyone should have to do this, because it leads to better systems and better research. ]]> graph coding hadoop spark giraph graph-processing hardware scalability big-data batch algorithms pagerank https://pinboard.in/ https://pinboard.in/u:jm/b:229db78fb862/ 2014-12-22T14:14:17+00:00 http://databricks.com/blog/2014/12/19/announcing-spark-1-2.html jmSpark 1.2 includes several cross-cutting optimizations focused on performance for large scale workloads. Two new features Databricks developed for our world record petabyte sort with Spark are turned on by default in Spark 1.2. The first is a re-architected network transfer subsystem that exploits Netty 4’s zero-copy IO and off heap buffer management. The second is Spark’s sort based shuffle implementation, which we’ve now made the default after significant testing in Spark 1.1. Together, we’ve seen these features give as much as 5X performance improvement for workloads with very large shuffles. ]]> spark sorting hadoop map-reduce batch databricks apache netty https://pinboard.in/ https://pinboard.in/u:jm/b:6d93115441ec/ 2014-10-10T20:33:57+00:00 http://databricks.com/blog/2014/10/10/spark-breaks-previous-large-scale-sort-record.html jmFirst and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks). Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.]]> spark hadoop map-reduce batch parallel sr-iov benchmarks performance netty shuffle algorithms sort-based-shuffle timsort https://pinboard.in/ https://pinboard.in/u:jm/b:c675a1f7b48b/ 2014-07-03T17:07:11+00:00 http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html jm jay-kreps toread architecture data stream-processing batch hadoop storm lambda-architecture https://pinboard.in/ https://pinboard.in/u:jm/b:a8ed8296bab9/ 2014-06-30T10:47:02+00:00 https://blog.twitter.com/2014/tsar-a-timeseries-aggregator jm analytics architecture twitter tsar aggregation event-processing metrics streaming hadoop batch https://pinboard.in/ https://pinboard.in/u:jm/b:663be2fcb029/ 2013-12-21T20:50:48+00:00 http://www.slideshare.net/cpwatson/cpn302-yourlinuxamioptimizationandperformance jm netflix aws tuning ami perf systemtap tunables sched_batch batch hadoop optimization performance https://pinboard.in/ https://pinboard.in/u:jm/b:5c1292b5cc15/ 2013-02-07T11:49:20+00:00 http://highscalability.com/blog/2013/1/7/analyzing-billions-of-credit-card-transactions-and-serving-l.html jmThe optimal histogram is computed using a random-restart hill climbing approximated algorithm. The algorithm has been shown very fast and accurate: we achieved 99% accuracy compared to an exact dynamic algorithm, with a speed increase of one factor. [...] The amount of information to serve in Voldemort for one year of BBVA's credit card transactions on Spain is 270 GB. The whole processing flow would run in 11 hours on a cluster of 24 "m1.large" instances. The whole infrastructure, including the EC2 instances needed to serve the resulting data would cost approximately $3500/month. ]]> scalability scaling voldemort hadoop batch algorithms histograms statistics bucketing percentiles https://pinboard.in/ https://pinboard.in/u:jm/b:8012838d5d72/ 2011-10-22T22:46:38+00:00 http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html jm nathan-marz realtime batch hadoop storm big-data cap https://pinboard.in/ https://pinboard.in/u:jm/b:20004ab62fdd/