https://pinboard.in/u:jm/public/ recent bookmarks from jm 2025-11-19T10:37:18+00:00 https://blog.cloudflare.com/18-november-2025-outage/ jm classic @devops_borat. "To make error is human. To propagate error to all server in automatic way is #devops"]]> postmortem cloudflare outages configuration deployment cloud error-handling rollouts via:itc https://pinboard.in/ https://pinboard.in/u:jm/b:ed67da858e59/ 2021-11-25T10:48:23+00:00 https://fly.io/blog/stuff-your-pi-hole-from-anywhere/ jm fly.io ops deployment cdn docker containers https://pinboard.in/ https://pinboard.in/u:jm/b:cdcabf745817/ 2021-11-23T09:58:27+00:00 https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh jm cloud outages tragic google race-conditions gclb patching deployment ops https://pinboard.in/ https://pinboard.in/u:jm/b:e167312e4892/ 2020-06-23T22:53:17+00:00 https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/ jmEach team needs to balance the safety of small-scoped deployments with the speed at which we can deliver changes to customers in all Regions. Deploying changes to 24 Regions or 76 Availability Zones through the pipeline one at a time has the lowest risk of causing broad impact, but it could take weeks for the pipeline to deliver a change to customers globally. We have found that grouping deployments into “waves” of increasing size, as seen in the previous sample prod pipeline, helps us achieve a good balance between deployment risk and speed. Each wave’s stage in the pipeline orchestrates deployments to a group of Regions, with changes being promoted from wave to wave. New changes can enter the production phase of the pipeline at any time. After a set of changes is promoted from the first step to the second step in wave 1, the next set of changes from gamma is promoted into the first step of wave 1, so we don’t end up with large bundles of changes waiting to be deployed to production. The first two waves in the pipeline build the most confidence in the change: The first wave deploys to a Region with a low number of requests to limit the possible impact of the first production deployment of the new change. The wave deploys to only one Availability Zone (or cell) at a time within that Region to cautiously deploy the change across the Region. The second wave then deploys to one Availability Zone (or cell) at a time in a Region with a high number of requests where it is highly likely that customers will exercise all the new code paths and where we get good validation of the changes. After we have higher confidence in the safety of the change from the initial pipeline waves’ deployments, we can deploy to more and more Regions in parallel in the same wave. For example, the previous sample prod pipeline deploys to three Regions in wave 3, then to up to 12 Regions in wave 4, then to the remaining Regions in wave 5. The exact number and choice of Regions in each of these waves and the number of waves in a service team’s pipeline depend on the individual service’s usage patterns and scale. The later waves in the pipeline still help us achieve our objective to prevent negative impact to multiple Availability Zones in the same Region. When a wave deploys to multiple Regions in parallel, it follows the same cautious rollout behavior for each Region that was used in the initial waves. Each step in the wave only deploys to a single Availability Zone or cell from each Region in the wave. ]]> automation ops devops amazon aws deployment waves az multi-region ci cd https://pinboard.in/ https://pinboard.in/u:jm/b:7ecaa0ea15ea/ 2020-02-05T21:17:42+00:00 https://azure.microsoft.com/en-gb/blog/advancing-safe-deployment-practices/ jm pilot phase with hardware diversity -> early regions, building up to larger regions -> full deployment.]]> deployment azure microsoft rollout ops safety outages https://pinboard.in/ https://pinboard.in/u:jm/b:006afc2918bd/ 2019-07-15T12:18:36+00:00 https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ jm cloudflare outages regex postmortems regexps deployment rollback via:jgc https://pinboard.in/ https://pinboard.in/u:jm/b:1ddbe47b29c6/ 2019-05-24T10:47:21+00:00 https://nixery.appspot.com/ jmNixery provides the ability to pull ad-hoc container images from a Docker-compatible registry server. The image names specify the contents the image should contain, which are then retrieved and built by the Nix package manager. Nix is also responsible for the creation of the container images themselves. e.g. "docker run -ti nixery.appspot.com/shell/htop bash" ]]> docker containers nix nixpkgs packaging deployment ops https://pinboard.in/ https://pinboard.in/u:jm/b:415edb4724cb/ 2019-01-08T16:34:24+00:00 https://serverless.pub/lambda-utility-layers/ jmThe basic AWS Lambda container is quite constrained, and until recently it was relatively difficult to include additional binaries into Lambda functions. Lambda Layers make that easy. A Layer is a common piece of code that is attached to your Lambda runtime in the /opt directory. You can reuse it in many functions, and deploy it only once. Individual functions do not need to include the layer code in their deployment packages, which means that the resulting functions are smaller and deploy faster. For example, at MindMup, we use Pandoc to convert markdown files into Word documents. The actual lambda function code is only a few dozen lines of JavaScript, but before layers, each deployment of the function had to include the whole Pandoc binary, larger than 100 MB. With a layer, we can publish Pandoc only once, so we use significantly less overall space for Lambda function versions. Each code change now requires just a quick redeployment. ]]> serverless lambda dependencies deployment packaging ops https://pinboard.in/ https://pinboard.in/u:jm/b:355dc75951ce/ 2018-11-19T11:56:28+00:00 https://docs.opsmop.io/ jmUses: Web-scale configuration management of all Linux/Unix systems; Application deployment; Immutable systems build definition; Maintaining stateful services such as database and messaging platforms; Automating one-off tasks & processes; Deployment and management of the undercloud. Features: Python 3 DSL; Declarative resource model with imperative capabilities; Type / Provider plugin seperation; Implicit ordering (with handler notification); Formalized “Plan” vs “Apply” evaluation stages; Early validation prior to runtime; Programatically scoped variables; Strong object-orientation ]]> opsmop ops configuration-management deployment build https://pinboard.in/ https://pinboard.in/u:jm/b:2f28aa217ebe/ 2018-10-03T10:57:19+00:00 http://www.doxsey.net/blog/kubernetes--the-surprisingly-affordable-platform-for-personal-projects jmAt the beginning of the year I spent several months deep diving on Kubernetes for a project at work. As an all-inclusive, batteries-included technology for infrastructure management, Kubernetes solves many of the problems you're bound to run into at scale. However popular wisdom would suggest that Kubernetes is an overly complex piece of technology only really suitable for very large clusters of machines; that it carries a large operational burden and that therefore using it for anything less than dozens of machines is overkill. I think that's probably wrong. Kubernetes makes sense for small projects and you can have your own Kubernetes cluster today for as little as $5 a month. (via Tony Finch)]]> via:fanf deployment howto kubernetes ops projects hacks clustering https://pinboard.in/ https://pinboard.in/u:jm/b:fa2e19a6200f/ 2018-02-19T11:01:41+00:00 https://l0rd.github.io/containerspatterns/#1 jm docker containers design-patterns coding packaging deployment via:fanf https://pinboard.in/ https://pinboard.in/u:jm/b:0e99583f73a7/ 2018-01-01T23:45:31+00:00 https://www.cs.columbia.edu/~smb/blog/2017-12/2017-12-30.html jmWhen you engineer a system for deployment you build it to meet certain real-world goals. You may find that there are tradeoffs, and that you can't achieve all of your goals, but that's normal; as I've remarked, "engineering is the art of picking the right trade-off in an overconstrained environment". For any computer-based financial system, one crucial parameter is the transaction rate. For a system like Bitcoin, another goal had to be avoiding concentrations of power. And of course, there's transaction privacy. There are less obvious factors, too. These days, "mining" for Bitcoins requires a lot of computations, which translates directly into electrical power consumption. One estimate is that the Bitcoin network uses up more electricity than many countries. There's also the question of governance: who makes decisions about how the network should operate? It's not a question that naturally occurs to most scientists and engineers, but production systems need some path for change. In all of these, Bitcoin has failed. The failures weren't inevitable; there are solutions to these problems in the acdemic literature. But Bitcoin was deployed by enthusiasts who in essence let experimental code escape from a lab to the world, without thinking about the engineering issues—and now they're stuck with it. Perhaps another, better cryptocurrency can displace it, but it's always much harder to displace something that exists than to fill a vacuum. ]]> steven-bellovin bitcoin tech software systems engineering deployment cryptocurrency cypherpunks https://pinboard.in/ https://pinboard.in/u:jm/b:4f78659ed032/ 2017-11-07T10:25:09+00:00 https://www.theregister.co.uk/2017/11/06/coreos_kubernetes_v_world/ jmThat doesn't mean Polvi is a fan. "Lambda and serverless is one of the worst forms of proprietary lock-in that we've ever seen in the history of humanity," said Polvi, only partly in jest, referring to the most widely used serverless offering, AWS Lambda. "It's seriously as bad as it gets." He elaborated: "It's code that tied not just to hardware – which we've seen before – but to a data center, you can't even get the hardware yourself. And that hardware is now custom fabbed for the cloud providers with dark fiber that runs all around the world, just for them. So literally the application you write will never get the performance or responsiveness or the ability to be ported somewhere else without having the deployment footprint of Amazon." Absolutely agreed...]]> lambda amazon aws containers coreos deployment lockin proprietary serverless alex-polvi kubernetes https://pinboard.in/ https://pinboard.in/u:jm/b:8417fb02298a/ 2017-07-17T10:48:03+00:00 https://medium.com/netflix-techblog/developer-experience-lessons-operating-a-serverless-like-platform-at-netflix-a8bbd5b899a0 jm serverless dependencies packaging deployment versioning devex netflix developer-experience dev testing staging scripting https://pinboard.in/ https://pinboard.in/u:jm/b:a6089d93b40f/ 2017-05-10T11:06:53+00:00 http://pushtrain.club/ jmIt’s notable that almost all of the hard things we dealt with were social problems. Some of these solutions involved writing code, but the hard part was the human organization. The hard parts in were maintaining a sense of community ownership over the state of the whole system. ]]> etsy ci cd deployment devops deploys dan-mckinley mcfunley presentations https://pinboard.in/ https://pinboard.in/u:jm/b:023c5d8b0b45/ 2017-04-28T12:42:17+00:00 https://abe-winter.github.io/blues/2017/04/27/config-vs-containers.html jm docker containers complaining whinge networking swarm deployment architecture build packaging https://pinboard.in/ https://pinboard.in/u:jm/b:d1e61e83e357/ 2016-11-17T12:34:18+00:00 http://jvns.ca/blog/2016/11/16/ideas-about-how-to-use-aws/ jm docker alb aws ec2 blue-green-deploys deployment ops tools skyliner via:jgilbert https://pinboard.in/ https://pinboard.in/u:jm/b:f2861c3cf680/ 2016-11-01T11:59:27+00:00 https://www.percona.com/blog/2016/02/11/measuring-docker-io-overhead/ jm docker percona overhead mysql deployment performance ops containers https://pinboard.in/ https://pinboard.in/u:jm/b:c324d2498cee/ 2016-09-30T09:10:36+00:00 https://twitter.com/r39132/status/781655649268731905 jm ml packaging airflow asg ami deployment ops infrastructure rollback https://pinboard.in/ https://pinboard.in/u:jm/b:d38d7335c227/ 2016-05-30T08:56:17+00:00 http://fbrnc.net/blog/2016/05/green-blue-deployments-with-aws-lambda-and-cloudformation jm asg elb aws lambda deployment ops blue-green-deploys https://pinboard.in/ https://pinboard.in/u:jm/b:b11b95720b3a/ 2016-05-24T07:17:17+00:00 https://about.gitlab.com/2016/05/23/gitlab-container-registry/ jm gitlab github git ci cd containers docker deployment coding https://pinboard.in/ https://pinboard.in/u:jm/b:5378add3a162/ 2016-03-27T10:20:43+00:00 https://jenkins.io/2.0/#pipelines jm via:eric jenkins ci cd deployment pipelines testing automation build https://pinboard.in/ https://pinboard.in/u:jm/b:435381483deb/ 2016-01-29T14:12:23+00:00 http://labs.unacast.com/2016/01/27/three-lessons-from-running-k8s-in-production/ jm deployment ops kubernetes production https://pinboard.in/ https://pinboard.in/u:jm/b:e8d8083dd7ab/ 2015-12-07T12:04:50+00:00 https://medium.com/@AdamRNeary/a-gulp-workflow-for-amazon-lambda-61c2afd723b6#.nbn3nuamp jm lambda aws services testing deployment ops mocha gulp javascript https://pinboard.in/ https://pinboard.in/u:jm/b:04a4a810e7fd/ 2015-11-02T16:30:59+00:00 http://queue.acm.org/detail.cfm?ref=rss&id=2839461 jmA "move-fast" mentality does not have to be at odds with reliability. To make these philosophies compatible, Facebook's infrastructure provides safety valves. This is full of interesting techniques. * Rapidly deployed configuration changes: Make everybody use a common configuration system; Statically validate configuration changes; Run a canary; Hold on to good configurations; Make it easy to revert. * Hard dependencies on core services: Cache data from core services. Provide hardened APIs. Run fire drills. * Increased latency and resource exhaustion: Controlled Delay (based on the anti-bufferbloat CoDel algorithm -- this is really cool); Adaptive LIFO (last-in, first-out) for queue busting; Concurrency Control (essentially a form of circuit breaker). * Tools that Help Diagnose Failures: High-Density Dashboards with Cubism (horizon charts); What just changed? * Learning from Failure: the DERP (!) methodology, ]]> ben-maurer facebook reliability algorithms codel circuit-breakers derp failure ops cubism horizon-charts charts dependencies soa microservices uptime deployment configuration change-management https://pinboard.in/ https://pinboard.in/u:jm/b:f84c5f60c6e9/ 2015-10-10T20:07:25+00:00 https://support.stripe.com/questions/outage-postmortem-2015-10-08-utc jmThere was a breakdown in communication between the developer who requested the index migration and the database operator who deleted the old index. Instead of working on the migration together, they communicated in an implicit way through flawed tooling. The dashboard that surfaced the migration request was missing important context: the reason for the requested deletion, the dependency on another index’s creation, and the criticality of the index for API traffic. Indeed, the database operator didn’t have a way to check whether the index had recently been used for a query. Good demo of how the Etsy-style chatops deployment approach would have helped avoid this risk.]]> stripe postmortem outages databases indexes deployment chatops deploy ops https://pinboard.in/ https://pinboard.in/u:jm/b:464d897d2cd9/ 2015-09-18T16:37:49+00:00 http://www.eightypercent.net/post/layers-in-the-stack.html jm architecture ops stack docker containerization deployment containers rkt coreos prod monitoring xooglers https://pinboard.in/ https://pinboard.in/u:jm/b:6929918b25c5/ 2015-07-29T12:52:39+00:00 http://sirupsen.com/production-docker/ jm docker containers devops deployment releases linux ops https://pinboard.in/ https://pinboard.in/u:jm/b:f942eafba4ab/ 2015-07-28T19:28:48+00:00 https://www.facebook.com/notes/kent-beck/taming-complexity-with-reversibility/1000330413333156 jmDevelopment servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren't obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people's primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). "We expect this feature to affect this metric. If it doesn't, it's gone." Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old. We do a bunch of these in work, and the rest are on the to-do list. +1 to these!]]> software deployment complexity systems facebook reversibility dark-releases releases ops cd migration https://pinboard.in/ https://pinboard.in/u:jm/b:6a3089426e1e/ 2015-07-15T10:41:15+00:00 https://blog.newrelic.com/2015/06/18/zero-to-docker/ jm docker ops deployment packaging new-relic https://pinboard.in/ https://pinboard.in/u:jm/b:6206878c472e/ 2015-06-23T13:13:29+00:00 http://www.slideshare.net/Docker/docker-at-shopify-from-thislooksfun-to-production-by-simon-eskildsen-shopify jm docker ops deployment packaging shopify slides https://pinboard.in/ https://pinboard.in/u:jm/b:8a84896037d6/ 2015-06-23T11:21:50+00:00 http://googlecloudplatform.blogspot.ie/2015/06/Container-Engine-Container-Registry-Updates-New-Features-Pricing.html jmGoogle Container Registry helps make it easy for you to store your container images in a private and encrypted registry, built on Cloud Platform. Pricing for storing images in Container Registry is simple: you only pay Google Cloud Storage costs. Pushing images is free, and pulling Docker images within a Google Cloud Platform region is free (Cloud Storage egress cost when outside of a region). Container Registry is now ready for production use: * Encrypted and Authenticated - Your container images are encrypted at rest, and access is authenticated using Cloud Platform OAuth and transmitted over SSL * Fast - Container Registry is fast and can handle the demands of your application, because it is built on Cloud Storage and Cloud Networking. * Simple - If you’re using Docker, just tag your image with a gcr.io tag and push it to the registry to get started.  Manage your images in the Google Developers Console. * Local - If your cluster runs in Asia or Europe, you can now store your images in ASIA or EU specific repositories using asia.gcr.io and eu.gcr.io tags. ]]> docker registry google gcp containers cloud-storage ops deployment https://pinboard.in/ https://pinboard.in/u:jm/b:dd814f15bb49/ 2015-05-24T08:18:52+00:00 https://medium.com/fabric8-io/kubernetes-for-developers-2a9c7202fcd3 jm kubernetes ops docker containers rocket deployment packaging https://pinboard.in/ https://pinboard.in/u:jm/b:2bae4eb96313/ 2015-05-19T10:19:06+00:00 http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html jm docker aws ec2 beanstalk deployment ops containers https://pinboard.in/ https://pinboard.in/u:jm/b:892d57be519a/ 2015-05-18T21:46:37+00:00 https://medium.com/s-c-a-l-e/google-systems-guru-explains-why-containers-are-the-future-of-computing-87922af2cf95 jmWhat is the relationship between Kubernetes, Borg and Omega (the two internal resource-orchestration systems Google has built)? I would say, kind of by definition, there’s no shared code but there are shared people. You can think of Kubernetes — especially some of the elements around pods and labels — as being lessons learned from Borg and Omega that are, frankly, significantly better in Kubernetes. There are things that are going to end up being the same as Borg — like the way we use IP addresses is very similar — but other things, like labels, are actually much better than what we did internally. I would say that’s a lesson we learned the hard way. ]]> google architecture kubernetes docker containers borg omega deployment ops https://pinboard.in/ https://pinboard.in/u:jm/b:7ab8ae0d6508/ 2015-05-09T08:22:24+00:00 https://docs.docker.com/registry/deploying/#understand-production-deployment jm docker registry ops deployment s3 https://pinboard.in/ https://pinboard.in/u:jm/b:1bdb65947ad0/ 2015-05-06T14:59:03+00:00 http://www.slideshare.net/nathariel/patterns-for-building-resilient-and-scalable-microservices-platform-onaws-full jm deployment ops devops hailo microservices platform patterns slides https://pinboard.in/ https://pinboard.in/u:jm/b:b424fcec24ec/ 2015-04-28T18:13:52+00:00 https://hashicorp.com/blog/vault.html jm hashicorp deployment security secrets authentication vault storage keys key-rotation https://pinboard.in/ https://pinboard.in/u:jm/b:6dc7702df3f8/ 2015-04-26T21:16:00+00:00 http://www.hokstad.com/docker/patterns jm tips docker ops deployment https://pinboard.in/ https://pinboard.in/u:jm/b:8eb79d23e20e/ 2015-04-22T10:17:42+00:00 http://www.slideshare.net/mikebrittain/full-stackengmeetupreleasemanagementmikebrittain jm deployment etsy deploy deployinator princess staging ops testing devops smoke-tests production jenkins https://pinboard.in/ https://pinboard.in/u:jm/b:7463b5d3bfc9/ 2015-04-16T16:06:38+00:00 http://www.slideshare.net/nathariel/microservices-and-elastic-resource-pools-with-amazon-ec2-container-service jm versioning microservices hailo aws ec2 ecs docker containers scheduling allocation deployment provisioning qos https://pinboard.in/ https://pinboard.in/u:jm/b:6721d7ee3f45/ 2015-04-16T13:52:19+00:00 http://research.google.com/pubs/pub43438.html jmGoogle's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it. (via Conall)]]> via:conall clustering google papers scale to-read borg cluster-management deployment packing reliability redundancy https://pinboard.in/ https://pinboard.in/u:jm/b:e76acc614892/ 2015-04-15T15:36:36+00:00 https://corner.squareup.com/2015/04/keywhiz.html jm'a secret management and distribution service [from Square] that is now available for everyone. Keywhiz helps us with infrastructure secrets, including TLS certificates and keys, GPG keyrings, symmetric keys, database credentials, API tokens, and SSH keys for external services — and even some non-secrets like TLS trust stores. Automation with Keywhiz allows us to seamlessly distribute and generate the necessary secrets for our services, which provides a consistent and secure environment, and ultimately helps us ship faster. [...] Keywhiz has been extremely useful to Square. It’s supported both widespread internal use of cryptography and a dynamic microservice architecture. Initially, Keywhiz use decoupled many amalgamations of configuration from secret content, which made secrets more secure and configuration more accessible. Over time, improvements have led to engineers not even realizing Keywhiz is there. It just works. Please check it out.' ]]> square security ops keys pki key-distribution key-rotation fuse linux deployment secrets keywhiz https://pinboard.in/ https://pinboard.in/u:jm/b:c4ff3d9449a6/ 2015-03-23T12:25:15+00:00 http://sandervanderburg.blogspot.ie/2014/07/managing-private-nix-packages-outside.html jm nix packaging deployment ops build ci nixpkgs https://pinboard.in/ https://pinboard.in/u:jm/b:fe5585e44425/ 2015-03-11T23:51:36+00:00 http://concourse.ci/ jmConcourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off builds, either locally or in another CI system. ]]> ci concourse-ci build deployment continuous-integration continuous-deployment devops https://pinboard.in/ https://pinboard.in/u:jm/b:ae3dccf71ef0/ 2015-03-07T21:21:38+00:00 http://blog.matthewskelton.net/2015/03/06/hp-is-trying-to-patent-continuous-delivery-here-is-how-you-can-help-block-this-madness/ jmOn 1st March 2015 I discovered that in 2012 HP had filed a patent (WO2014027990) with the USPO for ‘Performance tests in a continuous deployment pipeline‘ (the patent was granted in 2014). [....] HP has filed several patents covering standard Continuous Delivery (CD) practices. You can help to have these patents revoked by providing ‘prior art’ examples on Stack Exchange. In fairness, though, this kind of shit happens in most big tech companies. This is what happens when you have a broken software patenting system, with big rewards for companies who obtain shitty troll patents like these, and in turn have companies who reward the engineers who sell themselves out to write up concepts which they know have prior art. Software patents are broken by design!]]> cd devops hp continuous-deployment testing deployment performance patents swpats prior-art https://pinboard.in/ https://pinboard.in/u:jm/b:a16ec2c9f13d/ 2015-02-25T16:45:11+00:00 http://blogs.aws.amazon.com/application-management/post/Tx33XKAKURCCW83/Automatically-Deploy-from-GitHub-Using-AWS-CodeDeploy jm github aws ec2 codedeploy deployment ops https://pinboard.in/ https://pinboard.in/u:jm/b:2235bc32057f/ 2014-12-21T23:18:28+00:00 http://azure.microsoft.com/blog/2014/12/17/final-root-cause-analysis-and-improvement-areas-nov-18-azure-storage-service-interruption/ jm root-cause azure outages postmortem cloud microsoft deployment https://pinboard.in/ https://pinboard.in/u:jm/b:9e35cdbf126f/ 2014-12-09T22:36:38+00:00 https://wiki.gnome.org/Projects/OSTree jm"git for operating system binaries". OSTree is a tool for managing bootable, immutable, versioned filesystem trees. It is not a package system; nor is it a tool for managing full disk images. Instead, it sits between those levels, offering a blend of the advantages (and disadvantages) of both. You can use any build system you like to place content into it on a build server, then export an OSTree repository via static HTTP. On each client system, "ostree admin upgrade" can incrementally replicate that content, creating a new root for the next reboot. This provides fully atomic upgrades. Any changes made to /etc are propagated forwards, and all local state in /var is shared. A key goal of the project is to complement existing package systems like RPM and Debian packages, and help further their evolution. In particular for example, RPM-OSTree (linked below) has as a goal a hybrid tree/package model, where you replicate a base tree via OSTree, and then add packages on top. ]]> os gnome git linux immutable deployment packaging via:fanf https://pinboard.in/ https://pinboard.in/u:jm/b:7fcc1fbae602/ 2014-11-25T12:03:56+00:00 http://www.shopify.com/technology/15934308-docker-at-shopify-how-we-built-containers-that-power-over-100-000-online-shops jm docker devops deployment ops shopify containers production https://pinboard.in/ https://pinboard.in/u:jm/b:14b1174b98f7/ 2014-11-20T14:53:47+00:00 http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption/ jmAs part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues. I'm really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won't.]]> change-management cm microsoft outages postmortems azure deployment multi-region flighting azure-storage https://pinboard.in/ https://pinboard.in/u:jm/b:8acd095f17d5/ 2014-11-19T11:51:12+00:00 http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath/219801#219801 jm java jvm globbing classpath uberjars jars deployment https://pinboard.in/ https://pinboard.in/u:jm/b:985e11ba3303/ 2014-10-21T09:54:34+00:00 https://t37.net/is-docker-ready-for-production-feedbacks-of-a-2-weeks-hands-on.html jmFrom my point of view, Docker is probably the best thing I’ve seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they’re up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don’t feel like it’s production ready in a complex environment, because it adds too much complexity. And I’m not even sure that’s what it was designed for. ]]> docker complexity devops ops production deployment soa web-services provisioning networking logging https://pinboard.in/ https://pinboard.in/u:jm/b:7e2b93489b1c/ 2014-10-10T21:15:54+00:00 http://techblog.netflix.com/2013/11/preparing-netflix-api-for-deployment.html jmIf a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments.  ]]> devops deployment feature-flags release testing integration-tests uat qa production ops gating netflix https://pinboard.in/ https://pinboard.in/u:jm/b:2a3b3c58309f/ 2014-09-23T13:43:29+00:00 http://forty9ten.com/post/64676776664/avoiding-chef-suck-with-auto-scaling-groups jm infrastructure chef ops asg auto-scaling ec2 provisioning deployment https://pinboard.in/ https://pinboard.in/u:jm/b:2c44711b992b/ 2014-08-18T10:39:24+00:00 https://rclayton.silvrback.com/failing-at-microservices jm microservices devops collaboration architecture fail team deployment soa https://pinboard.in/ https://pinboard.in/u:jm/b:84d4dbc41eb6/ 2014-06-30T10:07:41+00:00 http://www.meetup.com/DevOps-Ireland/events/190786232/ jmThis month, we'll have Tomas Doran from Yelp talking about Docker, service discovery, and deployments. 'There are many advantages to a container based, microservices architecture - however, as always, there is no silver bullet. Any serious deployment will involve multiple host machines, and will have a pressing need to migrate containers between hosts at some point. In such a dynamic world hard coding IP addresses, or even host names is not a viable solution. This talk will take a journey through how Yelp has solved the discovery problems using Airbnb’s SmartStack to dynamically discover service dependencies, and how this is helping unify our architecture, from traditional metal to EC2 ‘immutable’ SOA images, to Docker containers.' ]]> meetups talks dublin deployment smartstack ec2 docker yelp service-discovery https://pinboard.in/ https://pinboard.in/u:jm/b:c729894d5fa6/ 2014-06-16T14:46:25+00:00 http://blog.cloudbees.com/2012/09/continuous-deployment-for-mobile-apps.html jm build deployment ios jenkins iphone continuous-deployment https://pinboard.in/ https://pinboard.in/u:jm/b:12bfa12e5424/ 2014-05-08T16:53:32+00:00 http://www.brunton-spall.co.uk/post/2014/05/06/database-migrations-done-right/ jmThe rule is simple. You should never tie database migrations to application deploys or vice versa. By minimising dependencies you enable faster, easier and cleaner deployments. A solid description of why this is a good idea, from an ex-Guardian dev.]]> migrations database sql mysql postgres deployment ops dependencies loose-coupling https://pinboard.in/ https://pinboard.in/u:jm/b:2e8db5bfa149/ 2014-04-28T11:40:53+00:00 http://peter.bourgon.org/go-in-production/ jm go tips deployment best-practices soundcloud ops https://pinboard.in/ https://pinboard.in/u:jm/b:e4292e28734a/ 2014-03-21T16:04:25+00:00 http://www.meetup.com/London-Continuous-Delivery/events/100034862/ jmLonely Planet and Dr Foster Intelligence both make heavy use of ETL in their products, and both organisations have applied the principles of Continuous Delivery to their delivery process. Some of the Continuous Delivery norms need to be adapted in the context of ETL, and some interesting patterns emerge, such as running Continuous Integration against data, as well as code.]]> etl video presentations lonely-planet dr-foster-intelligence continuous-delivery deployment pipelines https://pinboard.in/ https://pinboard.in/u:jm/b:e178b4690b79/ 2014-03-20T15:19:10+00:00 http://literateprogrammer.blogspot.co.uk/2014/03/the-microservice-declaration-of.html jm microservices soa architecture handwaving http services web deployment https://pinboard.in/ https://pinboard.in/u:jm/b:3c13e569ef49/ 2014-02-11T12:26:54+00:00 https://github.com/oleiade/trousseau jm gpg encryption crypto secrets key-distribution pki devops deployment https://pinboard.in/ https://pinboard.in/u:jm/b:62b9738d3540/ 2014-01-20T21:51:10+00:00 https://github.com/baudehlo/deploy_to_runit jm github node.js runit deployment git continuous-deployment devops ops https://pinboard.in/ https://pinboard.in/u:jm/b:9aeabd119bbf/ 2014-01-04T21:30:16+00:00 http://rc3.org/2013/03/03/dont-get-stuck/ jm continuous-deployment coding agile deployment devops etsy rafe-colburn https://pinboard.in/ https://pinboard.in/u:jm/b:d8dec5f4bf39/ 2013-12-30T00:22:40+00:00 http://engineering.thetrainline.com/2013/12/02/moving-to-multiple-deployments-per-week-at-thetrainline-com/ jm thoughtworks continuous-delivery continuous-deployment devops deployment https://pinboard.in/ https://pinboard.in/u:jm/b:8bb0abb6492f/ 2013-11-28T17:42:04+00:00 https://blogs.atlassian.com/2013/11/docker-all-the-things-at-atlassian-automation-and-wiring/ jm docker infrastructure devops ops deployment lxc containers linux https://pinboard.in/ https://pinboard.in/u:jm/b:cabd127a5fe5/ 2013-11-26T23:14:06+00:00 http://pl.atyp.us/2013-11-shared-libraries.html jm shared-libraries unix linux linker deployment https://pinboard.in/ https://pinboard.in/u:jm/b:738b282d2a97/ 2013-11-04T14:17:49+00:00 http://www.feld.com/wp/archives/2013/07/why-every-company-needs-a-devops-team-now.html jm "Make sure we have environments available early in the Development process"; enforce a policy that the code and environment are tested together, even at the earliest stages of the project; “Wake up developers up at 2 a.m. when they break things"; and "Create reusable deployment procedures". ]]> devops work ops deployment testing pager-duty https://pinboard.in/ https://pinboard.in/u:jm/b:53e5a44c03d1/ 2013-09-26T21:19:18+00:00 http://mesosphere.io/2013/09/26/docker-on-mesos/ jm mesos docker deployment ops images virtualization containers linux https://pinboard.in/ https://pinboard.in/u:jm/b:4327725c5c80/