https://pinboard.in/u:jm/public/ recent bookmarks from jm 2025-11-06T10:38:19+00:00 https://wthhyb.sacha.house/ jm shitposting funny architecture riak redis mongodb ouch scalability https://pinboard.in/ https://pinboard.in/u:jm/b:0420d47b34a3/ 2022-04-19T15:01:30+00:00 https://github.com/WeTransfer/prorate jm...Using a Lua script, because is the only language available which runs inside Redis. Thanks to the speed benefits of Lua the script runs fast enough to apply it on every throttle call. Using a Lua script in Prorate helps us achieve the following guarantees: The script will run atomically. The script is evaluated as a single Redis command. This ensures that the commands in the Lua script will never be interleaved with another client: they will always execute together. Any usages of time will use the Redis time. Throttling requires a consistent and monotonic time source. The only monotonic and consistent time source which is usable in the context of Prorate, is the TIME result of Redis itself. We are throttling requests from different machines, which will invariably have clock drift between them. This way using the Redis server TIME helps achieve consistency. ]]> redis rate-limiting throttling apis via:cian lua https://pinboard.in/ https://pinboard.in/u:jm/b:c9b0eb6ee68c/ 2018-10-18T09:22:16+00:00 http://antirez.com/news/123 jm antirez coding art georg-nees redis lolwut https://pinboard.in/ https://pinboard.in/u:jm/b:db63ad41a3f8/ 2017-03-30T10:01:30+00:00 http://tech.trivago.com/2017/01/25/learn-redis-the-hard-way-in-production/ jm redis scalability ops architecture horror trivago php https://pinboard.in/ https://pinboard.in/u:jm/b:86839e5457c2/ 2016-11-01T11:31:38+00:00 https://aws.amazon.com/blogs/aws/amazon-elasticache-for-redis-update-sharded-clusters-engine-improvements-and-more/?sc_channel=em&sc_campaign=global_LA_elasticache_2016_cluster_1_20161020_ampat.launch_la_ot-elasticache_2016_cluster_1_send&sc_publisher=aws&sc_medium=em_23618&sc_content=launch_la_ot&sc_country=global&sc_geo=global&sc_category=elasticache&sc_outcome=launch&trk=em_23618&mkt_tok=eyJpIjoiT1dSaU56UmhOR013WXpsaiIsInQiOiIxUnY0cWRSaHlQSFhpc0RzUVdhbGxRY0c5a09iekpuZ3lGVmlyNHMrRkwyM0NVeVhDQllZeDZlT1N1dVBZZlIxeVd1aVpoRmo5YmhsZWR4VDlrc0tWQk1LZlwvdkhmek9QRVJaU1hMQXBjVHc9In0%3D jm elasticache sharding storage aws databases redis ops https://pinboard.in/ https://pinboard.in/u:jm/b:10191f450066/ 2016-02-09T10:08:40+00:00 http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html jm distributed locking redis algorithms coding distcomp redlock martin-kleppman zookeeper https://pinboard.in/ https://pinboard.in/u:jm/b:8a6991e635e7/ 2016-01-29T12:06:11+00:00 https://pbs.twimg.com/media/BNELF1GCUAExynU.png jm mongodb redis funny architecture gifs png reactiongifs https://pinboard.in/ https://pinboard.in/u:jm/b:072d8b46b5b9/ 2016-01-12T09:58:54+00:00 https://nchan.slact.net/ jmNchan is a scalable, flexible pub/sub server for the modern web, built as a module for the Nginx web server. It can be configured as a standalone server, or as a shim between your application and tens, thousands, or millions of live subscribers. It can buffer messages in memory, on-disk, or via Redis. All connections are handled asynchronously and distributed among any number of worker processes. It can also scale to many nginx server instances with Redis. Messages are published to channels with HTTP POST requests or websockets, and subscribed also through websockets, long-polling, EventSource (SSE), old-fashioned interval polling, and more. Each subscriber can listen to up to 255 channels per connection, and can be optionally authenticated via a custom application url. An events meta channel is also available for debugging. Also now supports HTTP/2. This used to be called the Nginx HTTP Push Module, and I used it with great results in that form. This is the way to do HTTP push in all its forms....]]> nginx pubsub websockets sse http http-push http2 redis long-polling nchan https://pinboard.in/ https://pinboard.in/u:jm/b:07bf4fd5598d/ 2015-08-31T14:06:02+00:00 https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude/ jm lambda-architecture analytics via:hn redis set-storage storage databases architecture s3 realtime https://pinboard.in/ https://pinboard.in/u:jm/b:cb0a1f26939e/ 2015-03-27T09:36:04+00:00 http://muratbuffalo.blogspot.co.uk/2015/03/paper-review-simple-testing-can-prevent.html jm race-conditions startup bugs failure fault-tolerance hbase redis reliability ops papers concurrency exception-handling cassandra hdfs mapreduce https://pinboard.in/ https://pinboard.in/u:jm/b:3dd7b48e5fed/ 2015-03-23T14:00:34+00:00 https://shk.io/2015/03/22/transparent-huge-pages/ jmjemalloc(3) extensively uses madvise(2) to notify the operating system that it's done with a range of memory which it had previously malloc'ed. The page size on this machine is 2MB because transparent huge pages are in use. As such, a lot of the memory which is being marked with madvise(..., MADV_DONTNEED) is within substantially smaller ranges than 2MB. This means that the operating system never was able to evict pages which had ranges marked as MADV_DONTNEED because the entire page has to be unneeded to allow a page to be reused. Despite initially looking like a leak, the operating system itself was unable to free memory because of madvise(2) and transparent huge pages. This led to sustained memory pressure on the machine and redis-server eventually getting OOM killed. ]]> oom-killer oom linux ops thp jemalloc huge-pages madvise redis memory https://pinboard.in/ https://pinboard.in/u:jm/b:9b3c5ea145c2/ 2015-02-13T11:10:44+00:00 http://vaurien.readthedocs.org/en/1.8/index.html jmVaurien is basically a Chaos Monkey for your TCP connections. Vaurien acts as a proxy between your application and any backend. You can use it in your functional tests or even on a real deployment through the command-line. Vaurien is a TCP proxy that simply reads data sent to it and pass it to a backend, and vice-versa. It has built-in protocols: TCP, HTTP, Redis & Memcache. The TCP protocol is the default one and just sucks data on both sides and pass it along. Having higher-level protocols is mandatory in some cases, when Vaurien needs to read a specific amount of data in the sockets, or when you need to be aware of the kind of response you’re waiting for, and so on. Vaurien also has behaviors. A behavior is a class that’s going to be invoked everytime Vaurien proxies a request. That’s how you can impact the behavior of the proxy. For instance, adding a delay or degrading the response can be implemented in a behavior. Both protocols and behaviors are plugins, allowing you to extend Vaurien by adding new ones. Last (but not least), Vaurien provides a couple of APIs you can use to change the behavior of the proxy live. That’s handy when you are doing functional tests against your server: you can for instance start to add big delays and see how your web application reacts. ]]> proxy tcp vaurien chaos-monkey testing functional-testing failures sockets redis memcache http https://pinboard.in/ https://pinboard.in/u:jm/b:c2b6d44f05b0/ 2015-02-03T00:40:40+00:00 http://tech.forter.com/comparing-message-queue-architectures-on-aws/ jmIf you are light on DevOps and not latency sensitive use SQS for job management and Kinesis for event stream processing. If latency is an issue, use ELB or 2 RabbitMQs (or 2 beanstalkds) for job management and Redis for event stream processing. ]]> amazon architecture aws messaging queueing elb rabbitmq beanstalk kinesis sqs redis kafka https://pinboard.in/ https://pinboard.in/u:jm/b:ae3870806c75/ 2014-11-04T18:05:19+00:00 https://groups.google.com/a/jclarity.com/forum/#!topic/friends/QPXl_8ce5Fg jm redis memory defrag huge-pages linux kernel ops latency performance transparent-huge-pages https://pinboard.in/ https://pinboard.in/u:jm/b:02aad4ff8ea8/ 2014-10-30T16:03:54+00:00 https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJgrFOsXFd jm fork xen redis bugs performance latency p99 https://pinboard.in/ https://pinboard.in/u:jm/b:097ee6064211/ 2014-10-28T20:52:23+00:00 https://stripe.com/blog/game-day-exercises-at-stripe jmWe’ve started running game day exercises at Stripe. During a recent game day, we tested failing over a Redis cluster by running kill -9 on its primary node, and ended up losing all data in the cluster. We were very surprised by this, but grateful to have found the problem in testing. This result and others from this exercise convinced us that game days like these are quite valuable, and we would highly recommend them for others. Excellent post. Game days are a great idea. Also: massive Redis clustering fail]]> game-days redis testing stripe outages ops kill-9 failover https://pinboard.in/ https://pinboard.in/u:jm/b:0de011811783/ 2014-10-28T13:49:09+00:00 https://github.com/orbitz/roshiak jm riak roshi crdt redis storage time-series-data https://pinboard.in/ https://pinboard.in/u:jm/b:90d10e6b1b00/ 2014-09-09T10:52:12+00:00 http://highscalability.com/blog/2014/9/8/how-twitter-uses-redis-to-scale-105tb-ram-39mm-qps-10000-ins.html jm twitter redis caching memcached yao-yu scaling https://pinboard.in/ https://pinboard.in/u:jm/b:b7585fb83de9/ 2014-04-24T09:16:58+00:00 http://comcast.github.io/sirius/overview.html jmAt Comcast, our applications need convenient, low-latency access to important reference datasets. For example, our XfinityTV websites and apps need to use entertainment-related data to serve almost every API or web request to our datacenters: information like what year Casablanca was released, or how many episodes were in Season 7 of Seinfeld, or when the next episode of the Voice will be airing (and on which channel!). We traditionally managed this information with a combination of relational databases and RESTful web services but yearned for something simpler than the ORM, HTTP client, and cache management code our developers dealt with on a daily basis. As main memory sizes on commodity servers continued to grow, however, we asked ourselves: How can we keep this reference data entirely in RAM, while ensuring it gets updated as needed and is easily accessible to application developers? The Sirius distributed system library is our answer to that question, and we're happy to announce that we've made it available as an open source project. Sirius is written in Scala and uses the Akka actor system under the covers, but is easily usable by any JVM-based language. Also includes a Paxos implementation with "fast follower" read-only slave replication. ASL2-licensed open source. The only thing I can spot to be worried about is speed of startup; they note that apps need to replay a log at startup to rebuild state, which can be slow if unoptimized in my experience. Update: in a twitter conversation at https://twitter.com/jon_moore/status/459363751893139456 , Jon Moore indicated they haven't had problems with this even with 'datasets consuming 10-20GB of heap', and have 'benchmarked a 5-node Sirius ingest cluster up to 1k updates/sec write throughput.' That's pretty solid!]]> open-source comcast paxos replication read-only datastores storage memory memcached redis sirius scala akka jvm libraries https://pinboard.in/ https://pinboard.in/u:jm/b:54736b003116/ 2014-04-02T10:38:41+00:00 https://news.ycombinator.com/item?id=7506774 jm hll bloom-filters hyperloglog redis data-structures estimation cardinality probabilistic probability hashing random https://pinboard.in/ https://pinboard.in/u:jm/b:1231febb74e0/ 2014-02-20T21:20:18+00:00 http://www.roguelazer.com/2013/12/a-rant-on-redis/ jmI want to emphasize that if you use redis as intended (as a slightly-persistent, not-HA cache), it's great. Unfortunately, more and more shops seem to be thinking that Redis is a full-service database and, as someone who's had to spend an inordinate amount of time maintaining such a setup, it's not. If you're writing software and you're thinking "hey, it would be easy to just put a SET key value in this code and be done," please reconsider. There are lots of great products out there that are better for the overwhelming majority of use cases. Ouch. (via Aphyr)]]> redis storage architecture memory caching ha databases https://pinboard.in/ https://pinboard.in/u:jm/b:aa7e82ae343e/ 2013-12-07T21:18:08+00:00 https://groups.google.com/forum/#!msg/redis-db/Oazt2k7Lzz4/-7kDmWJHXLMJ jm Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn't exist, because sometimes it doesn't. In you're fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can't consider Redis a CP system. It can't be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_. ]]> kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp https://pinboard.in/ https://pinboard.in/u:jm/b:07f669c99101/ 2013-09-16T22:48:55+00:00 http://blog.meldium.com/home/2013/9/13/benchmarking-redis-on-aws-elasticache jm latency redis measurement benchmarks ec2 elasticache aws storage tests https://pinboard.in/ https://pinboard.in/u:jm/b:19ca978874ad/ 2013-07-25T10:07:30+00:00 http://www.twilio.com/blog/2013/07/billing-incident-post-mortem.html jmAt 1:35 AM PDT on July 18, a loss of network connectivity caused all billing redis-slaves to simultaneously disconnect from the master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time. Receiving full sync requests from each redis-slave caused the master to suffer extreme load, resulting in performance degradation of the master and timeouts from redis-slaves to redis-master. By 2:39 AM PDT the host’s load became so extreme, services relying on redis-master began to fail. At 2:42 AM PDT, our monitoring system alerted our on-call engineering team of a failure in the Redis cluster. Observing extreme load on the host, the redis process on redis-master was misdiagnosed as requiring a restart to recover. This caused redis-master to read an incorrect configuration file, which in turn caused Redis to attempt to recover from a non-existent AOF file, instead of the binary snapshot. As a result of that failed recovery, redis-master dropped all balance data. In addition to forcing recovery from a non-existent AOF, an incorrect configuration also caused redis-master to boot as a slave of itself, putting it in read-only mode and preventing the billing system from updating account balances. See also http://antirez.com/news/60 for antirez' response. Here's the takeaways I'm getting from it: 1. network partitions happen in production, and cause cascading failures. this is a great demo of that. 2. don't store critical data in Redis. this was the case for Twilio -- as far as I can tell they were using Redis as a front-line cache for billing data -- but it's worth saying anyway. ;) 3. Twilio were just using Redis as a cache, but a bug in their code meant that the writes to the backing SQL store were not being *read*, resulting in repeated billing and customer impact. In other words, it turned a (fragile) cache into the authoritative store. 4. they should probably have designed their code so that write failures would not result in repeated billing for customers -- that's a bad failure path. Good post-mortem anyway, and I'd say their customers are a good deal happier to see this published, even if it contains details of the mistakes they made along the way.]]> redis caching storage networking network-partitions twilio postmortems ops billing replication https://pinboard.in/ https://pinboard.in/u:jm/b:2e374b3b528b/ 2013-07-17T10:24:54+00:00 http://engineering.pinterest.com/post/55272557617/building-a-follower-model-from-scratch jmWe run our cluster in a Redis master-slave configuration, and the slaves act as hot backups. Upon a master failure, we failover the slave as the new master and either bring up a new slave or reuse the old master as the new slave. We rely on ZooKeeper to make this as quick as possible. Each master Redis instance (and slave instance) is configured to write to AOF on Amazon EBS. This ensures that if the Redis instances terminate unexpectedly then the loss of data is limited to 1 second of updates. The slave Redis instances also perform BGsave hourly which is then loaded to a more permanent store (Amazon S3). This copy is also used by Map Reduce jobs for analytics. As a production system, we need many failure modes to guard ourselves. As mentioned, if the master host is down, we will manually failover to slave. If a single master Redis instance reboots, monit restart restores from AOF, implying a 1 second window of data loss on the shards on that instance. If the slave host goes down, we bring up a replacement. If a single slave Redis instance goes down, we rely on monit to restart using the AOF data. Because we may encounter AOF or BGsave file corruption, we BGSave and copy hourly backups to S3. Note that large file sizes can cause BGsave induced delays but in our cluster this is mitigated by smaller Redis data due to the sharding scheme. ]]> graph redis architecture ha high-availability design redundancy sharding https://pinboard.in/ https://pinboard.in/u:jm/b:78c92555c7b0/ 2013-07-09T09:01:05+00:00 http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html jmTwitter is primarily a consumption mechanism, not a production mechanism. 300K QPS are spent reading timelines and only 6000 requests per second are spent on writes. * their approach of precomputing the timeline for the non-search case is a good example of optimizing for the more frequently-exercised path. * MySQL and Redis are the underlying stores. Redis is acting as a front-line in-RAM cache. they're pretty happy with it: https://news.ycombinator.com/item?id=6011254 * these further talks go into more detail, apparently (haven't watched them yet): http://www.infoq.com/presentations/Real-Time-Delivery-Twitter http://www.infoq.com/presentations/Twitter-Timeline-Scalability http://www.infoq.com/presentations/Timelines-Twitter * funny thread of comments on HN, from a big-iron fan: https://news.ycombinator.com/item?id=6008228]]> scale architecture scalability twitter high-scalability redis mysql https://pinboard.in/ https://pinboard.in/u:jm/b:5bddc42e545c/ 2013-06-09T21:47:43+00:00 http://planetcassandra.org/blog/post/instagram-making-the-switch-to-cassandra-from-redis-75-instasavings jma 12 node cluster of EC2 hi1.4xlarge instances; we store around 1.2TB of data across this cluster. At peak, we're doing around 20,000 writes per second to that specific cluster and around 15,000 reads per second. We've been really impressed with how well Cassandra has been able to drop into that role. ]]> ram ssd cassandra databases nosql redis instagram storage ec2 https://pinboard.in/ https://pinboard.in/u:jm/b:971263030ef5/ 2013-05-23T09:04:10+00:00 http://www.crashlytics.com/blog/reducing-mongodb-traffic-by-78-with-redis-caching/ jm crashlytics fail roflscale rails caching redis ruby via:oisin https://pinboard.in/ https://pinboard.in/u:jm/b:684457d967c4/ 2013-05-20T16:28:49+00:00 http://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions jm distributed network databases cap nosql redis mongodb postgresql riak crdt aphyr https://pinboard.in/ https://pinboard.in/u:jm/b:821665f78174/ 2013-04-15T21:17:02+00:00 http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html jma [Cassandra-style] Cluster Management Algorithm is a SPOF. If there’s a bug it impacts every node. This took them down 4 times. yeah, so, eek ;)]]> clustering sharding architecture aws scalability scaling pinterest via:matt-sergeant redis mysql memcached https://pinboard.in/ https://pinboard.in/u:jm/b:94eb7274d2de/ 2013-01-15T22:44:02+00:00 https://github.com/aaw/hyperloglog-redis#readme jm cardinality sets redis algorithms ruby gems hyperloglog https://pinboard.in/u:jm/b:a5e139c07bf4/ 2013-01-07T11:51:37+00:00 http://www.devco.net/archives/2013/01/06/solving-monitoring-state-storage-problems-using-redis.php jm redis storage https://pinboard.in/ https://pinboard.in/u:jm/b:8a21db3f95fb/ 2013-01-01T15:36:05+00:00 http://www.slideshare.net/crashlytics/crashlytics-on-redis-analytics jm analytics redis presentation metrics https://pinboard.in/u:jm/b:15778a5cc0f6/ 2012-11-26T13:06:38+00:00 http://redis.io/topics/memory-optimization jm memory redis performance big-o hash-tables storage coding cache arrays https://pinboard.in/ https://pinboard.in/u:jm/b:f7c066b4367c/ 2012-11-21T21:07:08+00:00 http://highscalability.com/blog/2012/11/19/gone-fishin-tumblr-architecture-15-billion-page-views-a-mont.html jm tumblr scalability web finagle redis kafka https://pinboard.in/ https://pinboard.in/u:jm/b:c26da999fe5c/ 2012-04-19T20:21:48+00:00 http://whilefalse.blogspot.com/2012/04/scaling-its-not-what-it-used-to-be.html jm scaling nginx memcached haproxy redis https://pinboard.in/ https://pinboard.in/u:jm/b:3c6a1dbdfeb9/ 2011-09-29T21:10:15+00:00 http://petermblair.com/fbl-n-gram-analyzer/ jm anti-spam spam fbl feedback filtering n-grams similarity hashing redis searching https://pinboard.in/ https://pinboard.in/u:jm/b:00bea3b79665/ 2011-07-20T00:43:14+00:00 http://devblog.bu.mp/how-we-use-redis-at-bump jm queueing redis nosql databases storage via:simonw replication bump https://pinboard.in/ https://pinboard.in/u:jm/b:70e64dd1ac5d/ 2011-02-27T23:31:09+00:00 http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/ jm redis sharding scaling craigslist https://pinboard.in/u:jm/b:25694b34dca4/ 2010-05-28T21:58:07+00:00 http://github.com/blog/655-scheduled-maintenance-today-22-00-pst jm upgrades redis spof nosql databases github deployment https://pinboard.in/u:jm/b:0740082abf7f/ 2010-05-05T17:08:53+00:00 http://playnice.ly/blog/2010/05/05/a-fast-fuzzy-full-text-index-using-redis/ jm metaphone sounds-like indexing python redis search full-text fuzzy https://pinboard.in/u:jm/b:1a3b4637d2f5/ 2009-10-22T14:21:35+00:00 http://simonwillison.net/2009/Oct/22/redis/ jm python storage databases schemaless nosql redis simon-willison data-store https://pinboard.in/u:jm/b:a2024135fb8e/