Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmShould you use a Lambda Monolith, aka Lambdalith, for your API?2023-11-14T12:39:20+00:00
https://rehanvdm.com/blog/should-you-use-a-lambda-monolith-lambdalith-for-the-api
jm
The argument to limit the blast radius on a per route level by default is too fine-grained, adds bloat and optimizes too early. The boundary of the blast radius should be on the whole API/service level, just as it is and always has been for traditional software.
Use a Lambdalith if you are not using any advance features of AWS REST API Gateway and you want the highest level of portability to other AWS gateways or compute layer. There are also many escape hatches to fill some of the promises that single-purpose functions offer.
]]>lambda monolith api design architecture aws serverlesshttps://pinboard.in/https://pinboard.in/u:jm/b:ccf4ade953bc/Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region2023-10-31T10:59:34+00:00
https://aws.amazon.com/message/061323/?ck_subscriber_id=512829374
jmaws outages fail lambda sts security us-east-1https://pinboard.in/https://pinboard.in/u:jm/b:a9ab0d0bca2f/Container Loading in AWS Lambda2023-05-23T22:10:31+00:00
https://brooker.co.za/blog/2023/05/23/snapshot-loading.html
jmThis system gets performance by doing as little work as possible (deduplication, caching, lazy loading), and then gets resilience by doing slightly more work than needed (erasure coding, salted deduplication, etc). This is a tension worth paying attention to in all system designs.
]]>architecture aws lambda marc-brooker performance storage caching containers cacheshttps://pinboard.in/https://pinboard.in/u:jm/b:18da95343673/Erasure Coding versus Tail Latency - Marc's Blog2023-05-23T22:05:23+00:00
https://brooker.co.za/blog/2023/01/06/erasure.html
jmarchitecture cache storage tail-latencies performance marc-brooker lambda erasure-coding algorithms latencyhttps://pinboard.in/https://pinboard.in/u:jm/b:dbbb79c2955e/AWS Lambda Function URLs2022-04-08T10:46:45+00:00
https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/
jmaws http lambda serverless microservices web-serviceshttps://pinboard.in/https://pinboard.in/u:jm/b:373bac9edd30/doctorray117/minecraft-ondemand2021-09-07T08:27:16+00:00
https://github.com/doctorray117/minecraft-ondemand
jm
Instead of paying a minecraft hosting service for a private server for you and your friends, host it yourself. By utilizing several AWS services, a minecraft server can automatically start when you're ready to use it, and shut down when you are done. The final cost will depend on use but can be as little as a a dollar or two per month. The cost estimate breakdown is below. This is a reasonably cost effective solution for someone that doesn't need their server running 24/7. If that's you, read on!
The process works as follows:
Open Minecraft Multiplayer, let it look for our server, it will time out.
The DNS lookup query is logged in Route 53 on our public hosted zone.
CloudWatch forwards the query to a Lambda function.
The Lambda function modifies an existing ECS Fargate service to a desired task count of 1.
Fargate launches two containers, Minecraft and a watchdog, which updates the DNS record to the new IP
The watchdog optionally sends a text message through Twilio when the server is ready.
Refresh Minecraft server list, server is ready to connect.
After 10 minutes without a connection or 20 minutes after the last client disconnects (customizable) the watchdog sets the desired task count to zero and shuts down.
This is a very neat hack, actually quite potentially usable, and a good illustration of how viable Fargate+EFS are at hosting transient but not transitory workloads!
]]>minecraft fargate efs aws lambda hacks hostinghttps://pinboard.in/https://pinboard.in/u:jm/b:4e1150b1f48c/Lambda vs EC2 costs2021-08-10T09:10:29+00:00
https://www.bbva.com/en/economics-of-serverless/
jm
With traffic profiles where requests arrive in at periodic intervals, and a low total amount of requests, serverless architecture seems to be a great architecture in terms of cost, speed of delivery and effort. Thus, Lambda is probably the way to go if our application has sufficiently large periods of inactivity.
Once the break-even point is reached, when EC2 is more cost-effective than Lambda, the cost difference grows rapidly, making Lambda less and less attractive in terms of cost. Thus, it is of great importance to know if the expected amount of traffic will be around the break-even point.
Be aware of the CPU throttling you will get with the smaller memory flavors of Lambda. If your code is CPU-bound, choosing the smaller memory flavors might not be an option, since execution times, and thus latency, might grow beyond your requirements. On the other hand, if your code is I/O bound, the CPU throttling might not affect you significantly.
Break-even point (if there is one, that is) strongly depends on the application itself. Without measuring the target application code, knowing the intended usage of the service, the SLA and the capabilities of the team in charge of building the application it is almost impossible to know for sure which service, Lambda or EC2, is more convenient.
IMO there are still significant costs in organisational and infrastructure terms around replacing a working EC2 infrastructure with a Lambda-based one; deployment and other integration points with AWS are extremely tricky to deal with. But this is good data on the $ point alone.
]]>serverless aws lambda costs cost-control ec2 hosting architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:2ff2f72827fa/A Shared File System for Your Lambda Functions2020-06-17T10:31:30+00:00
https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/
jmaws serverless lambda storage efs nfs opshttps://pinboard.in/https://pinboard.in/u:jm/b:6ee6431a5ef6/Examining how AWS builds their own serverless apps2019-10-21T15:00:21+00:00
https://www.trek10.com/blog/examining-aws-serverless-apps/
jmserverless lambda aws architecture codinghttps://pinboard.in/https://pinboard.in/u:jm/b:ec338d6368d6/The average lifetime of a Lambda run-time between AWS support for it to EOL is 2 years and 23 days2019-10-20T20:41:06+00:00
https://twitter.com/0xdabbad00/status/1185595610608484357
jmaws lambda support ops platformshttps://pinboard.in/https://pinboard.in/u:jm/b:fafa681287dd/Serverless for data scientists2019-04-09T09:49:10+00:00
https://mike.place/talks/serverless/
jmzappa talks presentations python data-science serverless lambda api-gateway awshttps://pinboard.in/https://pinboard.in/u:jm/b:6c2732f96b2c/aws-lambda-container-image-converter2019-01-28T15:56:02+00:00
https://github.com/awslabs/aws-lambda-container-image-converter/
jmlambda docker aws layers filesystem coding containers serverlesshttps://pinboard.in/https://pinboard.in/u:jm/b:b673d7cf4082/Serverless Computing: One Step Forward, Two Steps Back - Speaker Deck2019-01-15T16:47:05+00:00
https://speakerdeck.com/jhellerstein/serverless-computing-one-step-forward-two-steps-back
jmfaas lambda serverless fail slides architecture aws dynamodb s3 cloudhttps://pinboard.in/https://pinboard.in/u:jm/b:84a40ec73c2d/FFmpeg, SOX, Pandoc and RSVG for AWS Lambda2019-01-08T16:34:24+00:00
https://serverless.pub/lambda-utility-layers/
jmThe basic AWS Lambda container is quite constrained, and until recently it was relatively difficult to include additional binaries into Lambda functions. Lambda Layers make that easy. A Layer is a common piece of code that is attached to your Lambda runtime in the /opt directory. You can reuse it in many functions, and deploy it only once. Individual functions do not need to include the layer code in their deployment packages, which means that the resulting functions are smaller and deploy faster. For example, at MindMup, we use Pandoc to convert markdown files into Word documents. The actual lambda function code is only a few dozen lines of JavaScript, but before layers, each deployment of the function had to include the whole Pandoc binary, larger than 100 MB. With a layer, we can publish Pandoc only once, so we use significantly less overall space for Lambda function versions. Each code change now requires just a quick redeployment.
]]>serverless lambda dependencies deployment packaging opshttps://pinboard.in/https://pinboard.in/u:jm/b:355dc75951ce/A Serverless Journey: AWS Lambda Under the Hood (SRV409-R1) - AWS re:…2018-12-01T12:27:29+00:00
https://www.slideshare.net/AmazonWebServices/a-serverless-journey-aws-lambda-under-the-hood-srv409r1-aws-reinvent-2018
jmlambda aws marc-brooker ops slideshttps://pinboard.in/https://pinboard.in/u:jm/b:317c2befada0/15 Key Takeaways from the Serverless Talk at AWS Startup Day2018-07-16T14:45:50+00:00
https://www.jeremydaly.com/15-key-takeaways-from-the-serverless-talk-at-aws-startup-day/
jmaws lambda serverless ops hacks amazonhttps://pinboard.in/https://pinboard.in/u:jm/b:8e575f5419bf/Using AWS Batch to Generate Mapzen Terrain Tiles · Mapzen2017-12-05T22:09:37+00:00
https://mapzen.com/blog/terrain-tiles-on-aws-batch/
jmUsing this setup on AWS Batch, we are able to generate more than 3.75 million tiles per minute and render the entire world in less than a week! These pre-rendered tiles get stored in S3 and are ready to use by anyone through the AWS Public Dataset or through Mapzen’s Terrain Tiles API.
]]>mapzen mapping tiles batch aws s3 lambda dockerhttps://pinboard.in/https://pinboard.in/u:jm/b:a932f8829a8e/'Lambda and serverless is one of the worst forms of proprietary lock-in we've ever seen in the history of humanity' • The Register2017-11-07T10:25:09+00:00
https://www.theregister.co.uk/2017/11/06/coreos_kubernetes_v_world/
jmThat doesn't mean Polvi is a fan. "Lambda and serverless is one of the worst forms of proprietary lock-in that we've ever seen in the history of humanity," said Polvi, only partly in jest, referring to the most widely used serverless offering, AWS Lambda. "It's seriously as bad as it gets."
He elaborated: "It's code that tied not just to hardware – which we've seen before – but to a data center, you can't even get the hardware yourself. And that hardware is now custom fabbed for the cloud providers with dark fiber that runs all around the world, just for them. So literally the application you write will never get the performance or responsiveness or the ability to be ported somewhere else without having the deployment footprint of Amazon."
Absolutely agreed...]]>lambda amazon aws containers coreos deployment lockin proprietary serverless alex-polvi kuberneteshttps://pinboard.in/https://pinboard.in/u:jm/b:8417fb02298a/srcecde/aws-lambda-cheatsheet2017-10-16T11:04:23+00:00
https://github.com/srcecde/aws-lambda-cheatsheet
jmaws lambda ops serverless reference quick-referenceshttps://pinboard.in/https://pinboard.in/u:jm/b:142d536042b8/How to operate reliable AWS Lambda applications in production2017-10-12T10:57:33+00:00
https://www.concurrencylabs.com/blog/how-to-operate-aws-lambda/
jmrunning a reliable Lambda application in production requires you to still follow operational best practices. In this article I am including some recommendations, based on my experience with operations in general as well as working with AWS Lambda.
]]>aws cloud lambda ops amazonhttps://pinboard.in/https://pinboard.in/u:jm/b:e1434327aa56/AWS Lambda Deployment using Terraform – Build ACL – Medium2017-08-13T20:09:12+00:00
https://medium.com/build-acl/aws-lambda-deployment-with-terraform-24d36cc86533
jmA complete picture of what it takes to deploy your Lambda function to production with the same diligence you apply to any other codebase using Terraform. [...] There are many cases where frameworks such as SAM or Serverless are not enough. You need more than that for a highly integrated Lambda function. In such cases, it’s easier to simply use Terraform.
]]>infrastructure aws lambda serverless ops terraform samhttps://pinboard.in/https://pinboard.in/u:jm/b:27404d4a39f4/BinaryAlert: Serverless, Real-time & Retroactive Malware Detection2017-08-03T09:51:18+00:00
https://github.com/airbnb/binaryalert
jmaws serverless lambda airbnb malware yara binaryalert architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:2e486eeef221/anaibol/awesome-serverless: A curated list of awesome services, solutions and resources for serverless / nobackend applications.2017-06-26T14:00:14+00:00
https://github.com/anaibol/awesome-serverless/
jmA curated list of awesome services, solutions and resources for serverless / nobackend applications.
]]>serverless lambda ops devops saas services architecture listshttps://pinboard.in/https://pinboard.in/u:jm/b:8877523021d9/How Did I “Hack” AWS Lambda to Run Docker Containers?2017-06-26T09:07:00+00:00
https://hackernoon.com/how-did-i-hack-aws-lambda-to-run-docker-containers-7184dc47c09b
jmdocker lambda aws serverless ops hacks udockerhttps://pinboard.in/https://pinboard.in/u:jm/b:801adc7942f3/How does language, memory and package size affect cold starts of AWS Lambda?2017-06-19T15:32:19+00:00
https://read.acloud.guru/does-coding-language-memory-or-package-size-affect-cold-starts-of-aws-lambda-a15e26d12c76?__s=gf36pf8g1gjugcqh6ppo
jmjava cold-start lambda serverless awshttps://pinboard.in/https://pinboard.in/u:jm/b:4bb0f99fa5ce/lambci/docker-lambda2017-06-14T10:13:04+00:00
https://github.com/lambci/docker-lambda
jmA sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.
(via og-aws)]]>docker lambda images testing aws serverlesshttps://pinboard.in/https://pinboard.in/u:jm/b:382e8c25cbbe/AWS Greengrass2017-04-25T13:22:24+00:00
https://aws.amazon.com/greengrass/
jmAWS Greengrass is software that lets you run local compute, messaging & data caching for connected devices in a secure way. With AWS Greengrass, connected devices can run AWS Lambda functions, keep device data in sync, and communicate with other devices securely – even when not connected to the Internet. Using AWS Lambda, Greengrass ensures your IoT devices can respond quickly to local events, operate with intermittent connections, and minimize the cost of transmitting IoT data to the cloud.
AWS Greengrass seamlessly extends AWS to devices so they can act locally on the data they generate, while still using the cloud for management, analytics, and durable storage. With Greengrass, you can use familiar languages and programming models to create and test your device software in the cloud, and then deploy it to your devices. AWS Greengrass can be programmed to filter device data and only transmit necessary information back to the cloud. AWS Greengrass authenticates and encrypts device data at all points of connection using AWS IoT’s security and access management capabilities. This way data is never exchanged between devices when they communicate with each other and the cloud without proven identity.
]]>aws cloud iot lambda devices offline synchronization architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:2b27152c0ac8/Lessons Learned in Lambda2017-04-10T14:10:33+00:00
https://collectiveidea.com/blog/archives/2017/03/31/lessons-learned-in-lambda
jmlambda aws shitshow architecture serverlesshttps://pinboard.in/https://pinboard.in/u:jm/b:7c48422ee06d/The Occasional Chaos of AWS Lambda Runtime Performance2017-03-05T22:40:50+00:00
https://blog.symphonia.io/the-occasional-chaos-of-aws-lambda-runtime-performance-880773620a7e#.hq8y0jxeq
jmIf our code has modest resource requirements, and can tolerate large changes in performance, then it makes sense to start with the least amount of memory necessary. On the other hand, if consistency is important, the best way to achieve that is by cranking the memory setting all the way up to 1536MB.
It’s also worth noting here that CPU-bound Lambdas may be cheaper to run over time with a higher memory setting, as Jim Conning describes in his article, “AWS Lambda: Faster is Cheaper”. In our tests, we haven’t seen conclusive evidence of that behavior, but much more data is required to draw any strong conclusions.
The other lesson learned is that Lambda benchmarks should be gathered over the course of days, not hours or minutes, in order to provide actionable information. Otherwise, it’s possible to see very impressive performance from a Lambda that might later dramatically change for the worse, and any decisions made based on that information will be rendered useless.
]]>aws lambda amazon performance architecture ops benchmarkshttps://pinboard.in/https://pinboard.in/u:jm/b:f9249af7133f/Eric Jonas: Extracting 25 TFLOPS from AWS Lambda, or #TheCloudIsTooDamnHard2017-03-03T14:23:59+00:00
http://ericjonas.com/pywren.html
jmpython aws lambda ops serverless architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:26aef07dfdcc/AWS Step Functions – Build Distributed Applications Using Visual Workflows2016-12-02T17:30:21+00:00
https://aws.amazon.com/blogs/aws/new-aws-step-functions-build-distributed-applications-using-visual-workflows/
jmswf lambda workflows aws architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:fb51c9b8ea7b/AWS latency comparison: API Gateway vs Lambda vs Bare EC22016-10-12T11:04:22+00:00
https://www.prerender.cloud/lambda-latency
jmaws latency lambda api-gateway architecture httphttps://pinboard.in/https://pinboard.in/u:jm/b:afa22c5b2f10/Green/Blue Deployments with AWS Lambda and CloudFormation - done right2016-05-30T08:56:17+00:00
http://fbrnc.net/blog/2016/05/green-blue-deployments-with-aws-lambda-and-cloudformation
jmasg elb aws lambda deployment ops blue-green-deployshttps://pinboard.in/https://pinboard.in/u:jm/b:b11b95720b3a/3 Reasons AWS Lambda Is Not Ready for Prime Time2016-05-23T13:03:53+00:00
https://www.datawire.io/3-reasons-aws-lambda-not-ready-prime-time/
jmWhen we at Datawire tried to actually use Lambda for a real-world HTTP-based microservice [...], we found some uncool things that make Lambda not yet ready for the world we live in:
Lambda is a building block, not a tool;
Lambda is not well documented;
Lambda is terrible at error handling
Lung skips these uncool things, which makes sense because they’d make the tutorial collapse under its own weight, but you can’t skip them if you want to work in the real world. (Note that if you’re using Lambda for event handling within the AWS world, your life will be easier. But the really interesting case in the microservice world is Lambda and HTTP.)
]]>aws lambda microservices datawire http api-gateway apis https python opshttps://pinboard.in/https://pinboard.in/u:jm/b:43995416c92c/How to host Hugo static website generator on AWS Lambda2015-12-18T16:27:09+00:00
http://bezdelev.com/post/hugo-aws-lambda-static-website/
jmaws lambda hosting hugo blogs website-generatorshttps://pinboard.in/https://pinboard.in/u:jm/b:5db60b23bde7/A Gulp Workflow for Amazon Lambda2015-12-07T12:04:50+00:00
https://medium.com/@AdamRNeary/a-gulp-workflow-for-amazon-lambda-61c2afd723b6#.nbn3nuamp
jmlambda aws services testing deployment ops mocha gulp javascripthttps://pinboard.in/https://pinboard.in/u:jm/b:04a4a810e7fd/It's an Emulator, Not a Petting Zoo: Emu and Lambda2015-10-30T21:44:15+00:00
https://blog.fugue.co/2015-10-29-aws-lambda-emulator.html
jmlambda aws coding unit-tests devhttps://pinboard.in/https://pinboard.in/u:jm/b:3be9cc831ef2/(ARC308) The Serverless Company: Using AWS Lambda2015-10-18T22:00:01+00:00
http://www.slideshare.net/AmazonWebServices/arc308-the-serverless-company-using-aws-lambda
jmops lambda aws reinvent slides architecturehttps://pinboard.in/https://pinboard.in/u:jm/b:f1020fef464d/The Totally Managed Analytics Pipeline: Segment, Lambda, and Dynamo2015-10-09T10:11:41+00:00
https://segment.com/blog/the-totally-managed-analytics-pipeline/
jmaws pipelines data streaming lambda dynamodb analytics terraform opshttps://pinboard.in/https://pinboard.in/u:jm/b:131e7334d502/How We Use AWS Lambda for Rapidly Intensifying Workloads · CloudSploit2015-09-16T04:16:54+00:00
http://blog.cloudsploit.com/2015/09/15/how-we-use-lambda/
jmlambda aws ec2 autoscaling cloudsploithttps://pinboard.in/https://pinboard.in/u:jm/b:5b91285e9a68/Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)2015-05-26T16:28:58+00:00
https://alestic.com/2015/05/aws-lambda-recurring-schedule/
jmThe Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses.
You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == "00") or once a day (e.g., when hour == "00" and minute == "00") or any other series of intervals. You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it’s after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.
]]>alestic aws lambda cron time clock periodic-tasks recurrence hackshttps://pinboard.in/https://pinboard.in/u:jm/b:e3cb4ef5d5c9/Lambda: Bees with Frickin' Laser Beams2015-05-07T16:30:32+00:00
http://log.roadtrippers.com/lambda-bees-with-frickin-laser-beams.html
jmlambda aws node javascript hacks http load-testinghttps://pinboard.in/https://pinboard.in/u:jm/b:467110c5993b/awslabs/aws-lambda-redshift-loader2015-05-05T16:45:05+00:00
https://github.com/awslabs/aws-lambda-redshift-loader
jmlambda aws ec2 redshift s3 loaders etl pipelinehttps://pinboard.in/https://pinboard.in/u:jm/b:16697c14e9af/Kappa2015-04-28T13:31:05+00:00
https://github.com/garnaat/kappa
jmaws lambda mitch-garnaat coding testing cli kappahttps://pinboard.in/https://pinboard.in/u:jm/b:b5c7fbfe5ff1/Running Arbitrary Executables in AWS Lambda2015-04-20T20:25:45+00:00
https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/
jmlambda aws architecture ops node.js javascript unix linuxhttps://pinboard.in/https://pinboard.in/u:jm/b:17e3177786d6/Subscribing AWS Lambda Function To SNS Topic With aws-cli2015-04-14T10:03:26+00:00
http://alestic.com/2015/04/aws-cli-sns-lambda
jmaws aws-cli cli lambda sns hackshttps://pinboard.in/https://pinboard.in/u:jm/b:f2eed264e7ce/AWS Lambda Event-Driven Architecture With Amazon SNS2015-04-13T11:18:11+00:00
http://alestic.com/2015/04/aws-lambda-sns
jmAny message posted to an SNS topic can trigger the execution of custom code you have written, but you don’t have to maintain any infrastructure to keep that code available to listen for those events and you don’t have to pay for any infrastructure when the code is not being run. This is, in my opinion, the first time that Amazon can truly say that AWS Lambda is event-driven, as we now have a central, independent, event management system (SNS) where any authorized entity can trigger the event (post a message to a topic) and any authorized AWS Lambda function can listen for the event, and neither has to know about the other.
]]>aws ec2 lambda sns events cep event-processing coding cloud hacks eric-hammondhttps://pinboard.in/https://pinboard.in/u:jm/b:feb4d70146df/A Zero-Administration Amazon Redshift Database Loader - AWS Big Data Blog2015-03-07T08:12:09+00:00
http://blogs.aws.amazon.com/bigdata/post/Tx24VJ6XF1JVJAA/A-Zero-Administration-Amazon-Redshift-Database-Loader
jmlambda amazon aws redshift etlhttps://pinboard.in/https://pinboard.in/u:jm/b:9fefa35280f8/Lambda, Javascript Micro-Services on AWS2014-12-18T22:32:02+00:00
http://anders.janmyr.com/2014/12/lambda-javascript-micro-services-on-aws.html
jmaws lambda ec2 node exampleshttps://pinboard.in/https://pinboard.in/u:jm/b:9c83af8f0867/Collection Pipeline2014-07-28T13:02:16+00:00
http://martinfowler.com/articles/collection-pipeline/
jmmartin-fowler patterns coding ruby clojure streams pipelines pipes unix lambda fp java languageshttps://pinboard.in/https://pinboard.in/u:jm/b:3294217f4efc/