https://pinboard.in/u:jm/public/ recent bookmarks from jm 2025-11-28T13:01:42+00:00 https://pocketbase.io/ jm golang opensource database pocketbase web-apps sqlite https://pinboard.in/ https://pinboard.in/u:jm/b:7e63b498d320/ 2021-09-29T09:09:08+00:00 https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ jm github mysql architecture database https://pinboard.in/ https://pinboard.in/u:jm/b:8c92e3000937/ 2021-05-28T08:57:02+00:00 https://github.com/nocodb/nocodb jm airtable database sql mysql nocodb spreadsheets ui web https://pinboard.in/ https://pinboard.in/u:jm/b:61352fd4e436/ 2021-04-06T13:39:59+00:00 https://rbranson.medium.com/10-things-i-hate-about-postgresql-20dbab8c2791 jmOn a particularly large deployment, I eventually had to layer in a second pgbouncer tier. One tier ran on the application servers and another tier on the database servers. Altogether it aggregated connections for around 1 million client processes. Tuning it was 40% dark art, 40% brute force, and 10% pure luck. Amazing to see that these issues are still something that Postgres users have to worry about :)]]> database postgresql coding postgres pgbouncer ops rick-branson https://pinboard.in/ https://pinboard.in/u:jm/b:325436d797ce/ 2020-10-21T09:56:28+00:00 http://harelba.github.io/q/ jm csv database sql cli data tools unix tsv https://pinboard.in/ https://pinboard.in/u:jm/b:bbf7da485984/ 2019-12-16T11:07:54+00:00 https://github.com/simonw/datasette jmDatasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world. ]]> database api json python sqlite data exploring csv tsv https://pinboard.in/ https://pinboard.in/u:jm/b:d1af06975552/ 2019-04-08T09:39:48+00:00 https://github.com/ankane/strong_migrations jm Strong Migrations detects potentially dangerous operations in [Rails database] migrations, prevents them from running by default, and provides instructions on safer ways to do what you want. ]]> database migrations rails releases ops databases mysql ruby gems https://pinboard.in/ https://pinboard.in/u:jm/b:11703c588753/ 2019-02-13T14:20:18+00:00 https://blog.cryptographyengineering.com/2019/02/11/attack-of-the-week-searchable-encryption-and-the-ever-expanding-leakage-function/ jmIn all seriousness: database encryption has been a controversial subject in our field. I wish I could say that there’s been an actual debate, but it’s more that different researchers have fallen into different camps, and nobody has really had the data to make their position in a compelling way. There have actually been some very personal arguments made about it. The schools of thought are as follows: The first holds that any kind of database encryption is better than storing records in plaintext and we should stop demanding things be perfect, when the alternative is a world of constant data breaches and sadness. To me this is a supportable position, given that the current attack model for plaintext databases is something like “copy the database files, or just run a local SELECT * query”, and the threat model for an encrypted database is “gain persistence on the server and run sophisticated statistical attacks.” Most attackers are pretty lazy, so even a weak system is probably better than nothing. The countervailing school of thought has two points: sometimes the good is much worse than the perfect, particularly if it gives application developers an outsized degree of confidence of the security that their encryption system is going to provide them. If even the best encryption protocol is only throwing a tiny roadblock in the attacker’s way, why risk this at all? Just let the database community come up with some kind of ROT13 encryption that everyone knows to be crap and stop throwing good research time into a problem that has no good solution. I don’t really know who is right in this debate. I’m just glad to see we’re getting closer to having it. (via Jerry Connolly) ]]> cryptography attacks encryption database crypto security storage ppi gdpr search databases via:ecksor https://pinboard.in/ https://pinboard.in/u:jm/b:1935af4cab15/ 2018-11-08T14:56:53+00:00 https://dev.to/jaredsilver/how-do-you-populate-your-development-databases-e8e jm database data testing system-tests dev https://pinboard.in/ https://pinboard.in/u:jm/b:286cf87eafef/ 2017-09-27T13:35:21+00:00 https://airtable.com/ jm filemaker collaboration database tools web sharing teams https://pinboard.in/ https://pinboard.in/u:jm/b:02166edb17fd/ 2017-04-05T15:00:38+00:00 https://blog.timescale.com/when-boring-is-awesome-building-a-scalable-time-series-database-on-postgresql-2900ea453ee2 jm database postgresql postgres timeseries tsd storage state via:nelson https://pinboard.in/ https://pinboard.in/u:jm/b:9956c3efa969/ 2016-07-27T09:47:20+00:00 https://eng.uber.com/mysql-migration/ jm database mysql postgres postgresql uber architecture storage sql https://pinboard.in/ https://pinboard.in/u:jm/b:bb13fe501b54/ 2016-06-15T16:01:27+00:00 https://clickhouse.yandex/ jm yandex analytics database storage sql clickhouse https://pinboard.in/ https://pinboard.in/u:jm/b:5defba0ab614/ 2016-03-25T15:48:55+00:00 http://www.codeproject.com/Articles/33052/Visual-Representation-of-SQL-Joins jm sql joins mysql reference database https://pinboard.in/ https://pinboard.in/u:jm/b:da83d142d2e3/ 2016-01-06T12:42:41+00:00 https://blog.engineyard.com/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you jm database mysql queueing queue messaging percona rds locking sql architecture https://pinboard.in/ https://pinboard.in/u:jm/b:5081114f1b88/ 2015-04-22T10:08:00+00:00 http://www.slideshare.net/mikebrittain/mbrittain-continuous-deploymentalm3public jm cd deploy etsy slides migrations database schema ops ci version-control feature-flags https://pinboard.in/ https://pinboard.in/u:jm/b:7b09e64e7d8f/ 2015-03-09T23:08:39+00:00 https://github.com/soundcloud/lhm jm The basic idea is to perform the migration online while the system is live, without locking the table. In contrast to OAK and the facebook tool, we only use a copy table and triggers. The Large Hadron is a test driven Ruby solution which can easily be dropped into an ActiveRecord or DataMapper migration. It presumes a single auto incremented numerical primary key called id as per the Rails convention. Unlike the twitter solution, it does not require the presence of an indexed updated_at column. ]]> migrations database sql ops mysql rails ruby lhm soundcloud activerecord https://pinboard.in/ https://pinboard.in/u:jm/b:7319dc67d62e/ 2014-06-16T12:56:53+00:00 https://github.com/comeara/pillar jmManages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery. Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework. ]]> migrations database ops pillar cassandra activerecord scala continuous-delivery automation build https://pinboard.in/ https://pinboard.in/u:jm/b:acc70894611d/ 2014-05-08T16:53:32+00:00 http://www.brunton-spall.co.uk/post/2014/05/06/database-migrations-done-right/ jmThe rule is simple. You should never tie database migrations to application deploys or vice versa. By minimising dependencies you enable faster, easier and cleaner deployments. A solid description of why this is a good idea, from an ex-Guardian dev.]]> migrations database sql mysql postgres deployment ops dependencies loose-coupling https://pinboard.in/ https://pinboard.in/u:jm/b:2e8db5bfa149/ 2014-04-03T12:59:08+00:00 https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale jm manhattan consistency database twitter eventual-consistency nosql voldemort cassandra riak time-series https://pinboard.in/ https://pinboard.in/u:jm/b:5ce9c084548e/ 2013-12-07T21:18:08+00:00 https://groups.google.com/forum/#!msg/redis-db/Oazt2k7Lzz4/-7kDmWJHXLMJ jm Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn't exist, because sometimes it doesn't. In you're fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can't consider Redis a CP system. It can't be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_. ]]> kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp https://pinboard.in/ https://pinboard.in/u:jm/b:07f669c99101/ 2013-10-07T21:01:01+00:00 http://www.bailis.org/blog/non-blocking-transactional-atomicity/ jm algorithms database distributed scalability storage peter-bailis distcomp https://pinboard.in/ https://pinboard.in/u:jm/b:b97a35baf620/ 2013-06-09T21:42:47+00:00 http://henryr.github.io/cap-faq/ jmNo subject appears to be more controversial to distributed systems engineers than the oft-quoted, oft-misunderstood CAP theorem. The purpose of this FAQ is to explain what is known about CAP, so as to help those new to the theorem get up to speed quickly, and to settle some common misconceptions or points of disagreement. ]]> database distributed nosql cap consistency cap-theorem faqs https://pinboard.in/ https://pinboard.in/u:jm/b:598f6bda75fe/ 2013-04-12T16:56:57+00:00 http://www.infoq.com/presentations/Project-Voldemort-at-Gilt-Groupe jmGeir Magnusson explains how Gilt Groupe is using Project Voldemort to scale out their e-commerce transactional system. The initial SQL solution had to be replaced because it could not handle the transactional spikes the site is experiencing daily due to its particular way of selling their inventory: each day at noon. Magnusson explains why they chose Voldemort and talks about the architecture. via Filippo ]]> via:filippo database architecture nosql data voldemort gilt-groupe ops storage presentations https://pinboard.in/ https://pinboard.in/u:jm/b:44f4ba430e85/ 2013-04-10T17:11:17+00:00 http://research.microsoft.com/apps/pubs/default.aspx?id=178758 jmThe emergence of new hardware and platforms has led to reconsideration of how data management systems are designed. However, certain basic functions such as key indexed access to records remain essential. While we exploit the common architectural layering of prior systems, we make radically new design decisions about each layer. Our new form of B tree, called the Bw-tree achieves its very high performance via a latch-free approach that effectively exploits the processor caches of modern multi-core chips. Our storage manager uses a unique form of log structuring that blurs the distinction between a page and a record store and works well with flash storage. This paper describes the architecture and algorithms for the Bw-tree, focusing on the main memory aspects. The paper includes results of our experiments that demonstrate that this fresh approach produces outstanding performance. ]]> bw-trees database paper toread research algorithms microsoft sql sql-server b-trees data-structures storage cache-friendly mechanical-sympathy https://pinboard.in/ https://pinboard.in/u:jm/b:6ffba3a01727/ 2013-03-06T09:44:53+00:00 https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932 jmSome ALTER TABLE statements take too long form the perspective of some MySQL users. The fast index create feature for the InnoDB plugin in MySQL 5.1 makes this less of an issue but this can still take minutes to hours for a large table and for some MySQL deployments that is too long.   A workaround is to perform the change on a slave first and then promote the slave to be the new master. But this requires a slave located near the master. MySQL 5.0 added support for triggers and some replication systems have been built using triggers to capture row changes. Why not use triggers for this? The openarkkit toolkit did just that with oak-online-alter-table. We have published our version of an online schema change utility (OnlineSchemaChange.php aka OSC). ]]> facebook mysql sql schema database migrations ops alter-table https://pinboard.in/ https://pinboard.in/u:jm/b:f2bd9f37b00f/ 2012-12-12T13:28:36+00:00 http://codeascraft.etsy.com/2012/04/20/two-sides-for-salvation/ jm database etsy mysql replication schema availability downtime https://pinboard.in/ https://pinboard.in/u:jm/b:3662d0721dbe/ 2012-09-15T21:32:32+00:00 http://research.google.com/archive/spanner.html jm Abstract: Spanner is Google's scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: non-blocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner. To appear in: OSDI'12: Tenth Symposium on Operating System Design and Implementation, Hollywood, CA, October, 2012. ]]> database distributed google papers toread pdf scalability distcomp transactions cap consistency https://pinboard.in/ https://pinboard.in/u:jm/b:7dec089086fc/ 2011-12-19T21:49:50+00:00 http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html jm mysql twitter scalability gizzard innodb performance database https://pinboard.in/ https://pinboard.in/u:jm/b:bf5a9b9e5b85/ 2011-04-13T21:45:40+00:00 http://news.ycombinator.com/item?id=2434187 jm algorithms database data-structures b-trees hacker-news cassandra https://pinboard.in/u:jm/b:5576a3613ad1/ 2010-04-15T11:14:59+00:00 http://ewh.ieee.org/r6/scv/computer//nfic/2009/IBM-Jun-Rao.pdf jm via:jzawodny mail cassandra database data ibm nosql performance presentation pdf https://pinboard.in/u:jm/b:6e9057ce7983/ 2010-04-08T11:18:28+00:00 http://engineering.twitter.com/2010/04/introducing-gizzard-framework-for.html jm twitter gizzard database nosql storage sharding scalability scala replication https://pinboard.in/u:jm/b:d31219a6d2c7/ 2009-07-22T08:56:42+00:00 http://technology.timesonline.co.uk/tol/news/tech_and_web/the_web/article6718560.ece jm privacy uk law hacking phishing fraud crime police database identity-theft lucid-intelligence data-protection security colin-holder https://pinboard.in/u:jm/b:882475f1ee04/