https://pinboard.in/u:jm/public/ recent bookmarks from jm 2018-06-14T10:29:17+00:00 https://www.scylladb.com/2018/06/12/scylla-leverages-control-theory/ jm At any given moment, a database like ScyllaDB has to juggle the admission of foreground requests with background processes like compactions, making sure that the incoming workload is not severely disrupted by compactions, nor that the compaction backlog is so big that reads are later penalized. In this article, we showed that isolation among incoming writes and compactions can be achieved by the Schedulers, yet the database is still left with the task of determining the amount of shares of the resources incoming writes and compactions will use. Scylla steers away from user-defined tunables in this task, as they shift the burden of operation to the user, complicating operations and being fragile against changing workloads. By borrowing from the strong theoretical background of industrial controllers, we can provide an Autonomous Database that adapts to changing workloads without operator intervention. ]]> scylladb storage settings compaction automation thresholds control-theory ops cassandra feedback https://pinboard.in/ https://pinboard.in/u:jm/b:0924f70f896e/ 2016-12-14T11:21:39+00:00 https://eng.uber.com/cherami/ jm a competing-consumer messaging queue that is durable, fault-tolerant, highly available and scalable. We achieve durability and fault-tolerance by replicating messages across storage hosts, and high availability by leveraging the append-only property of messaging queues and choosing eventual consistency as our basic model. Cherami is also scalable, as the design does not have single bottleneck. [...] Cherami is completely written in Go, a language that makes building highly performant and concurrent system software a lot of fun. Additionally, Cherami uses several libraries that Uber has already open sourced: TChannel for RPC and Ringpop for health checking and group membership. Cherami depends on several third-party open source technologies: Cassandra for metadata storage, RocksDB for message storage, and many other third-party Go packages that are available on GitHub. We plan to open source Cherami in the near future. ]]> cherami uber queueing tasks queues architecture scalability go cassandra rocksdb https://pinboard.in/ https://pinboard.in/u:jm/b:08be41dbc892/ 2016-09-15T09:52:28+00:00 http://rockthecode.io/blog/highly-available-counters-using-cassandra/ jm crdts algorithms data-structures cassandra ha counters https://pinboard.in/ https://pinboard.in/u:jm/b:c45847cfbdbb/ 2016-03-24T13:21:13+00:00 http://prometheus.io/blog/2016/03/23/interview-with-life360/ jm'Prometheus has been known to us for a while, and we have been tracking it and reading about the active development, and at a point (a few months back) we decided to start evaluating it for production use. The PoC results were incredible. The monitoring coverage of MySQL was amazing, and we also loved the JMX monitoring for Cassandra, which had been sorely lacking in the past.' ]]> metrics monitoring time-series prometheus testimonials life360 cassandra jmx mysql https://pinboard.in/ https://pinboard.in/u:jm/b:c4405c9925a7/ 2015-12-09T11:35:37+00:00 https://labs.spotify.com/2015/11/17/monitoring-at-spotify-introducing-heroic/ jm cassandra elasticsearch spotify monitoring metrics heroic https://pinboard.in/ https://pinboard.in/u:jm/b:cdb427d86615/ 2015-10-15T10:49:07+00:00 http://www.scylladb.com/2015/10/13/cluster-benchmark/ jm scylla databases storage cassandra nosql https://pinboard.in/ https://pinboard.in/u:jm/b:aa43216c9e0e/ 2015-09-28T15:30:07+00:00 http://blog.threatstack.com/scaling-cassandra-lessons-learned jm cassandra configuration tuning scale ops https://pinboard.in/ https://pinboard.in/u:jm/b:41bfcab2a82b/ 2015-09-03T20:58:52+00:00 https://medium.com/@foundev/real-time-analytics-with-spark-streaming-and-cassandra-2f90d03342f7 jm spark-streaming kafka analytics cassandra architecture data batch https://pinboard.in/ https://pinboard.in/u:jm/b:b0d85df5ec49/ 2015-05-06T16:34:59+00:00 https://timharris.uk/papers/2015-hotos.pdf jm blade via:adriancolyer papers gc distsys algorithms distributed java jvm latency spark cassandra https://pinboard.in/ https://pinboard.in/u:jm/b:bf80614a2fb9/ 2015-04-29T15:42:34+00:00 https://issues.apache.org/jira/browse/CASSANDRA-7486 jm cassandra tuning ops g1gc cms gc java jvm production performance memory https://pinboard.in/ https://pinboard.in/u:jm/b:0b4b3d1abe38/ 2015-04-06T20:37:04+00:00 http://www.slideshare.net/WrathOfChris/cassandra-meetup-20150331-46523636 jm cassandra spotify date-tiered-compaction metrics graphite cyanite chris-maxwell time-series-data https://pinboard.in/ https://pinboard.in/u:jm/b:7af91a493440/ 2015-04-01T15:11:51+00:00 http://mail-archives.apache.org/mod_mbox/cassandra-user/201504.mbox/%3CCALamADJu4yo%3DcO8HgA6NpgFc1wQN_VNqpkMn-3SZwhPq9foLBw%40mail.gmail.com%3E jmUnder its default configuration, Cassandra binds an unauthenticated JMX/RMI interface to all network interfaces. As RMI is an API for the transport and remote execution of serialized Java, anyone with access to this interface can execute arbitrary code as the running user. ]]> cassandra jmx rmi java ops security https://pinboard.in/ https://pinboard.in/u:jm/b:f1a9c3b5500d/ 2015-03-27T09:36:04+00:00 http://muratbuffalo.blogspot.co.uk/2015/03/paper-review-simple-testing-can-prevent.html jm race-conditions startup bugs failure fault-tolerance hbase redis reliability ops papers concurrency exception-handling cassandra hdfs mapreduce https://pinboard.in/ https://pinboard.in/u:jm/b:3dd7b48e5fed/ 2015-02-03T22:09:12+00:00 http://tobert.github.io/tldr/cassandra-java-huge-pages.html jm jvm performance tuning huge-pages vm ops cassandra java https://pinboard.in/ https://pinboard.in/u:jm/b:d448b5d53e34/ 2015-01-12T22:29:19+00:00 https://labs.spotify.com/2015/01/09/personalization-at-spotify-using-cassandra/ jm via:dehora spotify cassandra replication storage ops https://pinboard.in/ https://pinboard.in/u:jm/b:13abf373f291/ 2015-01-05T17:56:26+00:00 http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html jmif you just set the "ThreadPriorityPolicy" to something else than the legal values 0 or 1, [...] a slight logic bug in Sun's JVM code kicks in, and thus sets the policy to be as if running with root - thus you get exactly what one desire. The operating system, Linux, won't allow priorities to be heightened above "Normal" (negative nice value), and thus just ignores those requests (setting it to normal instead, nice value 0) - but it lets through the requests to set it lower (setting the nice value to some positive value). ]]> cassandra thread-priorities threads java jvm linux nice hacks https://pinboard.in/ https://pinboard.in/u:jm/b:abed46e6d700/ 2014-11-04T17:12:53+00:00 http://engineering.onlive.com/2013/12/12/didnt-use-kafka/ jmWe came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren’t in both. It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn’t fix every failure case. Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn’t going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype. ]]> cassandra java kafka scala network-partitions availability multi-region multi-az aws replication onlive https://pinboard.in/ https://pinboard.in/u:jm/b:6bbae76b03db/ 2014-09-19T09:58:07+00:00 https://github.com/tobert/pcstat jmA common question when tuning databases and other IO-intensive applications is, "is Linux caching my data or not?" pcstat gets that information for you using the mincore(2) syscall. I wrote this is so that Apache Cassandra users can see if ssTables are being cached. ]]> linux page-cache caching go performance cassandra ops mincore fincore https://pinboard.in/ https://pinboard.in/u:jm/b:4b9a6fb3fd3a/ 2014-09-19T09:55:25+00:00 http://rustyrazorblade.com/2014/09/cassandra-summit-recap-diagnosing-problems-in-production/ jm cassandra ops storage https://pinboard.in/ https://pinboard.in/u:jm/b:3f0c6dff42d2/ 2014-06-16T12:56:53+00:00 https://github.com/comeara/pillar jmManages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery. Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework. ]]> migrations database ops pillar cassandra activerecord scala continuous-delivery automation build https://pinboard.in/ https://pinboard.in/u:jm/b:acc70894611d/ 2014-04-28T11:35:23+00:00 http://aws.amazon.com/solutions/case-studies/hailo/ jm hailo cassandra ubuntu mysql rds aws ec2 haproxy architecture https://pinboard.in/ https://pinboard.in/u:jm/b:210362e2cd0c/ 2014-04-03T12:59:08+00:00 https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale jm manhattan consistency database twitter eventual-consistency nosql voldemort cassandra riak time-series https://pinboard.in/ https://pinboard.in/u:jm/b:5ce9c084548e/ 2014-02-12T10:49:07+00:00 http://www.fullcontact.com/blog/mongo-to-cassandra-migration/ jm cassandra luks raid linux tuning ops blockdev disks sdd https://pinboard.in/ https://pinboard.in/u:jm/b:9e9bd44bf027/ 2014-01-28T22:51:37+00:00 http://www.datastax.com/dev/blog/improving-compaction-in-cassandra-with-cardinality-estimation jm hyperloglog hll algorithms cassandra bloom-filters sstables cardinality https://pinboard.in/ https://pinboard.in/u:jm/b:cca5dc02afd5/ 2014-01-24T10:14:24+00:00 http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads jmThe cluster we tuned is hosted on AWS and is comprised of 6 hi1.4xlarge EC2 instances, with 2 1TB SSDs raided together in a raid 0 configuration. The cluster’s dataset is growing steadily. At the time of this writing, our dataset is 341GB, up from less than 200GB a few months ago, and is growing by 2-3GB per day. The workload on this cluster is very read heavy, with quorum reads making up 99% of all operations. Some careful GC tuning here. Probably not applicable to anyone else, but good approach in general.]]> java performance jvm scaling gc tuning cassandra ops https://pinboard.in/ https://pinboard.in/u:jm/b:bc4cff6803b7/ 2013-12-09T10:08:22+00:00 https://github.com/pyr/cyanite jma metric storage daemon, exposing both a carbon listener and a simple web service. Its aim is to become a simple, scalable and drop-in replacement for graphite's backend. Pretty alpha for now, but definitely worth keeping an eye on to potentially replace our burgeoning Carbon fleet... ]]> graphite carbon cassandra storage metrics ops graphs service-metrics https://pinboard.in/ https://pinboard.in/u:jm/b:d96de514fb8f/ 2013-10-19T20:19:28+00:00 http://techblog.netflix.com/2013/10/introducing-chaos-to-c.html jm ops autoremediation outages remediation cassandra storage netflix chaos-monkey https://pinboard.in/ https://pinboard.in/u:jm/b:08c65b108c70/ 2013-10-14T09:45:54+00:00 http://aphyr.com/posts/299-the-trouble-with-timestamps jmTimestamps, as implemented in Riak, Cassandra, et al, are fundamentally unsafe ordering constructs. In order to guarantee consistency you, the user, must ensure locally monotonic and, to some extent, globally monotonic clocks. This is a hard problem, and NTP does not solve it for you. When wall clocks are not properly coupled to the operations in the system, causal constraints can be violated. To ensure safety properties hold all the time, rather than probabilistically, you need logical clocks. ]]> clocks time distributed databases distcomp ntp via:fanf aphyr vector-clocks last-write-wins lww cassandra riak https://pinboard.in/ https://pinboard.in/u:jm/b:dcc3c0071909/ 2013-10-05T20:56:35+00:00 http://www.datastax.com/dev/blog/rapid-read-protection-in-cassandra-2-0-2 jm cassandra nosql replication distcomp latency storage https://pinboard.in/ https://pinboard.in/u:jm/b:74b80ef24320/ 2013-10-02T12:55:23+00:00 http://attentionshard.wordpress.com/2013/09/30/why-tellybug-moved-from-cassandra-to-amazon-dynamodb/ jm aws dynamodb cassandra nosql storage tellybug counters scalability reliability latency https://pinboard.in/ https://pinboard.in/u:jm/b:8b0a38474b92/ 2013-09-11T21:40:35+00:00 https://blog.twitter.com/2013/observability-at-twitter jmThere are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data. ]]> twitter monitoring metrics service-metrics tsd time-series storage architecture cassandra https://pinboard.in/ https://pinboard.in/u:jm/b:3f55dd143a82/ 2013-09-03T22:29:45+00:00 https://issues.apache.org/jira/browse/CASSANDRA-5582 jm disruptor cassandra java p99 latency speed performance concurrency via:kellabyte https://pinboard.in/ https://pinboard.in/u:jm/b:70d101c721e7/ 2013-09-02T15:19:32+00:00 http://blueflood.io/ jm cassandra tsd storage time-series data open-source java rackspace https://pinboard.in/ https://pinboard.in/u:jm/b:9991bb557631/ 2013-06-09T21:47:43+00:00 http://planetcassandra.org/blog/post/instagram-making-the-switch-to-cassandra-from-redis-75-instasavings jma 12 node cluster of EC2 hi1.4xlarge instances; we store around 1.2TB of data across this cluster. At peak, we're doing around 20,000 writes per second to that specific cluster and around 15,000 reads per second. We've been really impressed with how well Cassandra has been able to drop into that role. ]]> ram ssd cassandra databases nosql redis instagram storage ec2 https://pinboard.in/ https://pinboard.in/u:jm/b:971263030ef5/ 2013-04-09T09:43:42+00:00 http://spyced.blogspot.ie/2008/12/couchdb-not-drinking-kool-aid.html jmHere are some reasons you should think twice and do careful testing before using CouchDB in a non-toy project: Writes are serialized.  Not serialized as in the isolation level, serialized as in there can only be one write active at a time.  Want to spread writes across multiple disks?  Sorry. CouchDB uses a MVCC model, which means that updates and deletes need to be compacted for the space to be made available to new writes.  Just like PostgreSQL, only without the man-years of effort to make vacuum hurt less. CouchDB is simple.  Gloriously simple.  Why is that a negative?  It's competing with systems (in the popular imagination, if not in its author's mind) that have been maturing for years.  The reason PostgreSQL et al have those features is because people want them.  And if you don't, you should at least ask a DBA with a few years of non-MySQL experience what you'll be missing.  The majority of CouchDB fans don't appear to really understand what a good relational database gives them, just as a lot of PHP programmers don't get what the big deal is with namespaces. A special case of simplicity deserves mention: nontrivial queries must be created as a view with mapreduce.  MapReduce is a great approach to trivially parallelizing certain classes of problem.  The problem is, it's tedious and error-prone to write raw MapReduce code.  This is why Google and Yahoo have both created high-level languages on top of it (Sawzall and Pig, respectively).  Poor SQL; even with DSLs being the new hotness, people forget that SQL is one of the original domain-specific languages.  It's a little verbose, and you might be bored with it, but it's much better than writing low-level mapreduce code. ]]> cassandra couch nosql storage distributed databases consistency https://pinboard.in/ https://pinboard.in/u:jm/b:ea7fb334af3e/ 2013-03-12T21:48:41+00:00 http://www.datastax.com/dev/blog/metric-collection-and-storage-with-cassandra jm datastax nosql metrics analytics cassandra tsd time-series storage https://pinboard.in/ https://pinboard.in/u:jm/b:307f55e4ee22/ 2013-03-06T10:34:40+00:00 http://techo-ecco.com/blog/monitoring-apache-hadoop-cassandra-and-zookeeper-using-graphite-and-jmxtrans/ jm graphite monitoring ops zookeeper cassandra hadoop jmx jmxtrans graphs https://pinboard.in/ https://pinboard.in/u:jm/b:b5aa3112e5b1/ 2013-03-06T09:46:52+00:00 http://techblog.netflix.com/2013/02/netflix-queue-data-migration-for-high.html jmThere will come a time in the life of most systems serving data, when there is a need to migrate data to [another] data store while maintaining or improving data consistency, latency and efficiency. This document explains the data migration technique we used at Netflix to migrate the user’s queue data between two different distributed NoSQL storage systems [SimpleDB to Cassandra]. ]]> cassandra netflix migrations data schema simpledb storage https://pinboard.in/ https://pinboard.in/u:jm/b:d3773b000a99/ 2013-02-25T22:07:17+00:00 http://www.odbms.org/blog/2013/02/big-data-analytics-at-netflix-interview-with-christos-kalantzis-and-jason-brown/ jm cassandra netflix user-stories testimonials nosql storage ec2 mongodb https://pinboard.in/ https://pinboard.in/u:jm/b:979bc48e6cf8/ 2013-02-25T15:35:01+00:00 http://blog.markedup.com/2013/02/cassandra-hive-and-hadoop-how-we-picked-our-analytics-stack/ jm riak mongodb cassandra hbase performance analytics hadoop hive big-data storage databases nosql https://pinboard.in/ https://pinboard.in/u:jm/b:b335abec7a75/ 2012-03-14T00:08:44+00:00 http://www.slideshare.net/adrianco/arch-tutoriallo2of3 jm cassandra api design oo object-model java adrian-cockroft slides qcon scaling aws netflix https://pinboard.in/ https://pinboard.in/u:jm/b:047c60818c5c/ 2011-11-03T22:58:25+00:00 http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html jm ec2 aws cassandra scaling benchmarks netflix performance https://pinboard.in/ https://pinboard.in/u:jm/b:c5f9c3bbde44/ 2011-04-13T21:45:40+00:00 http://news.ycombinator.com/item?id=2434187 jm algorithms database data-structures b-trees hacker-news cassandra https://pinboard.in/u:jm/b:5576a3613ad1/ 2010-04-22T10:33:14+00:00 http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010 jm twitter analytics cassandra databases hadoop pdf logs metrics number-crunching nosql pig presentation slides scribe https://pinboard.in/u:jm/b:29181b7623c2/ 2010-04-15T11:14:59+00:00 http://ewh.ieee.org/r6/scv/computer//nfic/2009/IBM-Jun-Rao.pdf jm via:jzawodny mail cassandra database data ibm nosql performance presentation pdf https://pinboard.in/u:jm/b:6e9057ce7983/