Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmsuperfly/corrosion2023-09-24T21:13:24+00:00
https://github.com/superfly/corrosion
jmIn a nutshell, Corrosion:
Maintains a SQLite database on each node
Gossips local changes throughout the cluster
Uses CR-SQLite for conflict resolution with CRDTs
Uses Foca to manage cluster membership using a SWIM protocol
Periodically synchronizes with a subset of other cluster nodes, to ensure consistency
This is very cool stuff for configuration distribution across a large network, where eventually consistent config is doable....]]>eventual-consistency configuration corrosion sqlite cr-sqlite crdts distributed-systemshttps://pinboard.in/https://pinboard.in/u:jm/b:8d001e0517d8/Don't Settle For Eventual Consistency2017-06-30T13:01:55+00:00
https://yokota.blog/2017/02/17/dont-settle-for-eventual-consistency/
jmWith an AP system, you are giving up consistency, and not really gaining anything in terms of effective availability, the type of availability you really care about. Some might think you can regain strong consistency in an AP system by using strict quorums (where the number of nodes written + number of nodes read > number of replicas). Cassandra calls this “tunable consistency”. However, Kleppmann has shown that even with strict quorums, inconsistencies can result.10 So when choosing (algorithmic) availability over consistency, you are giving up consistency for not much in return, as well as gaining complexity in your clients when they have to deal with inconsistencies.
]]>cap-theorem databases storage cap consistency cp ap eventual-consistencyhttps://pinboard.in/https://pinboard.in/u:jm/b:f658ec7b1871/Existential Consistency: Measuring and Understanding Consistency at Facebook2015-10-21T09:39:26+00:00
http://blog.acolyer.org/2015/10/19/existential-consistency-measuring-and-understanding-consistency-at-facebook/
jmThe metric is termed φ(P)-consistency, and is actually very simple. A read for the same data is sent to all replicas in P, and φ(P)-consistency is defined as the frequency with which that read returns the same result from all replicas. φ(G)-consistency applies this metric globally, and φ(R)-consistency applies it within a region (cluster). Facebook have been tracking this metric in production since 2012.
]]>facebook eventual-consistency consistency metrics papers cap distributed-computinghttps://pinboard.in/https://pinboard.in/u:jm/b:7214e8685f31/Race conditions on Facebook, DigitalOcean and others2015-04-27T11:23:22+00:00
http://josipfranjkovic.blogspot.ie/2015/04/race-conditions-on-facebook.html?m=1
jmattacks exploits race-conditions bugs eventual-consistency distributed-transactions http facebook digitalocean via:aphyrhttps://pinboard.in/https://pinboard.in/u:jm/b:c75313846d35/S3's "s3-external-1.amazonaws.com" endpoint2015-04-07T09:41:46+00:00
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
jmaws s3 eventual-consistency consistency us-east replication workarounds legacyhttps://pinboard.in/https://pinboard.in/u:jm/b:4c98f4560655/What is maximum Amazon S3 replication time on file upload? - Stack Overflow2015-04-07T09:40:20+00:00
http://stackoverflow.com/questions/23786609/what-is-maximum-amazon-s3-replication-time-on-file-upload/23794474#23794474
jmnetflix aws s3 consistency eventual-consistency bugs outageshttps://pinboard.in/https://pinboard.in/u:jm/b:b75e8fa81cb1/When S3's eventual consistency is REALLY eventual2015-04-07T09:38:38+00:00
http://www.stackdriver.com/eventual-consistency-really-eventual/
jms3 eventual-consistency aws consistency read-after-writes bugs outages stackdriverhttps://pinboard.in/https://pinboard.in/u:jm/b:336db21a6346/If Eventual Consistency Seems Hard, Wait Till You Try MVCC2014-12-09T16:42:44+00:00
http://www.xaprb.com/blog/2014/12/08/eventual-consistency-simpler-than-mvcc/
jmSince I am not ready to assert that there’s a distributed system I know to be better and simpler than eventually consistent datastores, and since I certainly know that InnoDB’s MVCC implementation is full of complexities, for right now I am probably in the same position most of my readers are: the two viable choices seem to be single-node MVCC and multi-node eventual consistency. And I don’t think MVCC is the simpler paradigm of the two.
]]>nosql concurrency databases mysql riak voldemort eventual-consistency reliability storage baron-schwartz mvcc innodb postgresqlhttps://pinboard.in/https://pinboard.in/u:jm/b:0dddeb567400/Great comment from the "banking uses eventual consistency" thread...2014-04-07T10:26:38+00:00
https://twitter.com/peat/statuses/452984415837093888
jmbanking banks insurance eventual-consistency databases consistency transactions nosqlhttps://pinboard.in/https://pinboard.in/u:jm/b:5a13ae1263cd/Manhattan, our real-time, multi-tenant distributed database for Twitter scale | Twitter Blogs2014-04-03T12:59:08+00:00
https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale
jmmanhattan consistency database twitter eventual-consistency nosql voldemort cassandra riak time-serieshttps://pinboard.in/https://pinboard.in/u:jm/b:5ce9c084548e/Mathematical Purity in Distributed Systems: CRDTs Without Fear2014-01-15T21:36:16+00:00
http://codedependents.com/2014/01/13/mathematical-purity-in-distributed-systems-crdts-without-fear/
jmstorage crdts semilattice idempotency commutativity data-structures distcomp eventual-consistencyhttps://pinboard.in/https://pinboard.in/u:jm/b:3060bd5cc9c5/Non-blocking transactional atomicity2013-09-02T13:07:32+00:00
http://www.bailis.org/blog/non-blocking-transactional-atomicity/#putting_it_all_together
jmalgorithms nbta transactions databases storage distcomp distributed atomic coding eventual-consistency crdtshttps://pinboard.in/https://pinboard.in/u:jm/b:5b1708428e3f/Information on Google App Engine's recent US datacenter relocations - Google Groups2013-08-25T21:12:20+00:00
https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/T_e7lVg7QNY/discussion
jmgae google app-engine outages ops paxos eventual-consistency replication storage hrdhttps://pinboard.in/https://pinboard.in/u:jm/b:282e914f9180/"Scalable Eventually Consistent Counters over Unreliable Networks" [paper, pdf]2013-08-23T20:27:44+00:00
http://arxiv.org/pdf/1307.3207v1.pdf
jm
Counters are an important abstraction in distributed computing, and
play a central role in large scale geo-replicated systems, counting events such as web page impressions or social network "likes". Classic distributed counters, strongly consistent, cannot be made both available and partition-tolerant, due to the CAP Theorem, being unsuitable to large scale scenarios.
This paper defines Eventually Consistent Distributed Counters (ECDC) and presents an implementation of the concept, Handoff Counters, that is scalable and works over unreliable networks. By giving up the sequencer aspect of classic distributed counters, ECDC implementations can be made AP in the CAP design space, while retaining the essence of counting. Handoff Counters are the first CRDT (Conflict-free Replicated Data Type) based mechanism that overcomes the identity explosion problem in naive CRDTs, such as G-Counters (where state size is linear in the number of independent actors that ever incremented the counter), by managing identities towards avoiding global propagation, and garbage collecting temporary entries. The approach used in Handoff Counters is not restricted to counters, being more generally applicable to other data types with associative and commutative operations.
]]>pdf papers eventual-consistency counters distributed-systems distcomp cap-theorem ecdc handoff-counters crdts data-structures g-countershttps://pinboard.in/https://pinboard.in/u:jm/b:dfa596e513f2/_Bolt-On Causal Consistency_ [slides]2013-06-27T23:34:26+00:00
http://www.bailis.org/talks/bolton-sigmod2013.pdf
jmeventual-consistency state cap-theorem storage peter-bailishttps://pinboard.in/https://pinboard.in/u:jm/b:4b2971b7b6b8/Riak, CAP, and eventual consistency2013-04-20T22:03:34+00:00
https://gist.github.com/macintux/5418295
jmAs Brewer's CAP theorem established, distributed systems have to make hard choices. Network partition is inevitable. Hardware failure is inevitable. When a partition occurs, a well-behaved system must choose its behavior from a spectrum of options ranging from "stop accepting any writes until the outage is resolved" (thus maintaining absolute consistency) to "allow any writes and worry about consistency later" (to maximize availability). Riak leans toward the availability end of the spectrum, but allows the operator and even the developer to tune read and write requests to better meet the business needs for any given set of data.]]>riak cap eventual-consistency distcomp distributed-systems partition last-write-wins voldemort allow_multhttps://pinboard.in/https://pinboard.in/u:jm/b:1bded8180d0b/Eventual Consistency Today: Limitations, Extensions, and Beyond - ACM Queue2013-04-19T20:10:29+00:00
http://queue.acm.org/detail.cfm?id=2462076
jmeventual-consistency data storage horizontal-scaling research distcomp distributed-systems via:martin-thompson crdts calm acid caphttps://pinboard.in/https://pinboard.in/u:jm/b:dc79d334a4c6/Boundary Techtalk - Large-scale OLAP with Kobayashi2013-04-10T17:12:32+00:00
http://boundary.com/blog/2012/08/21/boundary-techtalk-large-scale-olap-with-kobayashi/
jmDietrich Featherston, Engineer at Boundary, walks through the process of designing Kobayashi, the time-series analytics database behind our network metrics. He goes through the false-starts and lessons learned in effectively using Riak as the storage layer for a large-scale OLAP database. The system is ultimately capable of answering complex, ad-hoc queries at interactive latencies.
]]>video boundary tsd riak eventual-consistency storage kobayashi olap time-serieshttps://pinboard.in/https://pinboard.in/u:jm/b:a71181b0a65d/