On The Bike Shed, hosts Joël Quenneville and Stephanie Minn discuss development experiences and challenges at thoughtbot with Ruby, Rails, JavaScript, and whatever else is drawing their attention, admiration, or ire this week.
…
continue reading
Player FM - Internet Radio Done Right
32 subscribers
Checked 1y ago
Added seven years ago
Content provided by Jon Christensen. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Jon Christensen or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Similar to Mobycast
Hanselminutes is Fresh Air for Developers. A weekly commute-time podcast that promotes fresh technology and fresh voices. Talk and Tech for Developers, Life-long Learners, and Technologists.
…
continue reading
This is the audio podcast version of Troy Hunt's weekly update video published here: https://www.troyhunt.com/tag/weekly-update/
…
continue reading
Exploring React Native Together
…
continue reading
Developer Tea exists to help driven developers connect to their ultimate purpose and excel at their work so that they can positively impact the people they influence. With over 13 million downloads to date, Developer Tea is a short podcast hosted by Jonathan Cutrell (@jcutrell), co-founder of Spec and Director of Engineering at PBS. We hope you'll take the topics from this podcast and continue the conversation, either online or in person with your peers. Twitter: @developertea :: Email: deve ...
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
A weekly talk show taking a pragmatic look at the art and business of Software Development and the world of technology.
…
continue reading
It takes more than great code to be a great engineer. Soft Skills Engineering is a weekly advice podcast for software developers about the non-technical stuff that goes into being a great software developer.
…
continue reading
Software's best weekly news brief, deep technical interviews & talk show.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
))
Daily Deals
Podcasts Worth a Listen
SPONSORED
<
<div class="span index">1</div> <span><a class="" data-remote="true" data-type="html" href="/series/all-about-change">All About Change</a></span>
How do we build an inclusive world? Hear intimate and in-depth conversations with changemakers on disability rights, youth mental health advocacy, prison reform, grassroots activism, and more. First-hand stories about activism, change, and courage from people who are changing the world: from how a teen mom became the Planned Parenthood CEO, to NBA player Kevin Love on mental health in professional sports, to Beetlejuice actress Geena Davis on Hollywood’s role in women’s rights. All About Change is hosted by Jay Ruderman, whose life’s work is seeking social justice and inclusion for people with disabilities worldwide. Join Jay as he interviews iconic guests who have gone through adversity and harnessed their experiences to better the world. This show ultimately offers the message of hope that we need to keep going. All About Change is a production of the Ruderman Family Foundation. Listen and subscribe to All About Change wherever you get podcasts. https://allaboutchangepodcast.com/
Mobycast
Mark all (un)played …
Manage series 2155284
Content provided by Jon Christensen. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Jon Christensen or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
A Podcast About Cloud Native Software Development, AWS, and Distributed Systems
…
continue reading
160 episodes
Share
Mark all (un)played …
Manage series 2155284
Content provided by Jon Christensen. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Jon Christensen or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
A Podcast About Cloud Native Software Development, AWS, and Distributed Systems
…
continue reading
160 episodes
All episodes
×
1 Hands On AWS - Massively Scalable Image Hosting Using S3 and CloudFront - Part 2 41:13
41:13 
Play Later 
Play Later 
Lists 
Like 
Liked41:13
Play Later
Play Later
Lists
Like
LikedIn this episode, we cover the following topics: We discuss the features and limitations of serving files directly from S3. We then talk about how CloudFront can address many of S3's limitations. In particular, CloudFront is performant, inexpensive and allows us to use custom CNAMEs with TLS encryption. How to create a secure CloudFront distribution for files hosted in S3. What is OAI (Origin Access Identity), why we need it and how to set it up. We show how you can configure your CloudFront distribution to use TLS and redirect HTTP to HTTPS. We finish up by discussing "byte-range requests" and how to enable them for our image hosting solution. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ End Song Beauty in Rhythm by Roy England More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Hands On AWS - Massively Scalable Image Hosting Using S3 and CloudFront - Part 1 43:25
43:25 Play Later Play Later Lists Like Liked43:25
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: A common feature for web apps is image upload. And we all know the "best practices" for how to build this feature. But getting it right can be tricky. We start off by discussing the problem space, and what we want to solve. A key goal is to have a solution that is massively scalable while being cost-effective. We outline the general architecture of the solution, with separate techniques for handling image uploading and downloading. We then dive deep into how to handle image uploading, highlighting various techniques for controlling access over who can perform uploads. Two common techniques for securing uploads when using AWS are presigned URLs and presigned POSTs. We discuss how each works and when to use one over the other. We finish up by putting everything together and detailing the steps involved with uploading an image. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Lazy Sunday by Roy England More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Replay of Ep 43 - The Birth of NoSQL and DynamoDb – Part 5 42:42
42:42 Play Later Play Later Lists Like Liked42:42
Play Later Play Later Lists Like LikedShow Details Jon Christensen and Chris Hickman of Kelsus and Rich Staats of Secret Stache conclude their series on the birth of NoSQL and DynamoDB. They compare the NoSQL database, Leviathan, created by Chris’s startup in the late 1990s to today’s DynamoDB. A lot of things haven’t changed, even though technology has evolved. It’s cyclical. There are patterns and problems that continue to dominate. Some of the highlights of the show include: Reason for Creation of NoSQL Database: How to scale database with Internet-scale applications to have a virtual pool of infinite storage that can be scaled out Main Architecture Components of Leviathan: API client Update distributor (UD) Base server (storage node) Shepherd (housekeeping management system) Additional core components included smart IP and storage abstraction layer (SAL) Leviathan mostly used C code and minimal Java code to support users Big difference between DynamoDB and Leviathan is request router and partition metadata system living on the server vs. living on the edge Leviathan was a closed system with an instance for every network or data center; not designed to run as a software as a service, like DynamoDB Leviathan was strongly consistent, unlike DynamoDB’s eventually consistent model Definition and Different Types of Transactions Shepherd was used to identify and address consistency, synchronous, and timing issues Rather than using a file system, Leviathan used relational databases Links and Resources DynamoDB Microsoft SQL Oracle DB AWS IoT Greengrass Kelsus Secret Stache Media Quotes: “We had the same kind of problems that DynamoDB had - how do you scale your database dealing with Internet-scale applications and have this virtual pool of infinite storage that can be scaled out.” Chris Hickman “This system and this technology went through many iterations.” Chris Hickman “You can’t have a 100% consistent state across everything. It’s just impossible. How do you do the right thing?” Chris Hickman “The big difference between DynamoDB and Leviathan...is the request router and partition metadata system living on the server vs. living out at the edge.” Jon Christen…
1 Replay of Ep 42 - The Birth of NoSQL and DynamoDb – Part 4 41:05
41:05 Play Later Play Later Lists Like Liked41:05
Play Later Play Later Lists Like LikedShow Details What’s under the hood of Amazon’s DynamoDB? Jon Christensen and Chris Hickman of Kelsus continue their discussion on DynamoDB, specifically about it’s architecture and components. They utilize a presentation from re:Invent titled, Amazon DynamoDB Under the Hood: How we built a hyper-scale database . Some of the highlights of the show include: Partition keys and global secondary indexes determine how data is partitioned across a storage node; allows you to scale out, instead of up Understand how a database is built to make architecture/component definitions less abstract DynamoDB has four components: 1. Request Router: Frontline service that receives and handles requests 2. Storage Node: Services responsible for persisting and retrieving data 3. Partition Metadata System: Keeps track of where data is located 4. Auto Admin: Handles housekeeping aspects to manage system What level of uptime availability do you want to guarantee? Replication: Strongly Consistent vs. Eventually Consistent Walkthrough of Workflow: What happens when, what does it mean when… DynamoDB architecture and components are designed to improve speed and scalability Split Partitions: Longer times that your database is up and the more data you put into it, the more likely you’re going to get a hot partition or partitions that are too big Links and Resources DynamoDB re:Invent Amazon DynamoDB Under the Hood: How we built a hyper-scale database Paxos Algorithm Amazon S3 Amazon Relational Database Service (RDS) MongoDB JSON Kelsus Secret Stache Media Quotes: “Keep in mind that data is partitioned across storage node, and that’s a key feature of being able to scale out, as opposed to scaling up.” Jon Christensen “Amazon was opening up the kimono...how DynamoDB has been architected and constructed and how it works.” Chris Hickman “Managed Service - they get to decide how it’s architected...because they also have to keep it up and live up to their SLA.” Chris Hickman “The longer the time that your database is up and the more data you put into it, the more likely that you’re going to get a hot partition or partitions are just going to get too big.” Chris Hickman…
1 Replay of Ep 41 - The Birth of NoSQL and DynamoDb – Part 3 29:40
29:40 Play Later Play Later Lists Like Liked29:40
Play Later Play Later Lists Like LikedShow Details Jon Christensen and Chris Hickman of Kelsus and Rich Staats of Secret Stache continue their discussion on the birth of NoSQL and DynamoDB. They examine DynamoDB’s architecture and popularity as a solution for Internet-scale databases. Some of the highlights of the show include: Challenges, evolution, and reasons associated with Internet-scale data DynamoDB has been around a long time, but people are finally using it DynamoDB and MongoDB are document or key value stores that offer scalability and event-driven programming to reduce complexity Techniques for keeping NoSQL database’s replicated data in sync Importance of indexes to understand query patterns DynamoDB’s Table Concept: Collection of documents/key value items; must have partition key to uniquely identify items in table and determine data distribution Sort keys create indexes (i.e. global/local secondary index) to support items within partitioning Query a DynamoDB database based on what’s being stored and using keys; conduct analysis on queries to determine their effectiveness Links and Resources AWS re:Invent DynamoDB NoSQL MongoDB Groupon JSON PostgreSQL Kelsus Secret Stache Media Quotes: “Kind of what drove this evolution from SQL to NoSQL - realizing that the constraints were now different, the economics of the resources that were being used.” Chris Hickman “People are realizing that Dynamo is not an ugly stepchild.” Jon Christensen “Event-driven programming...it’s very popular, and it’s going to become even more popular.” Chris Hickman End Song Benirrás Nights by Roy England ft. Dovetracks…
1 Replay of Ep 40 - The Birth of NoSQL and DynamoDb – Part 2 33:23
33:23 Play Later Play Later Lists Like Liked33:23
Play Later Play Later Lists Like LikedShow Details Jon Christensen and Rich Staats learn about Chris Hickman’s first venture-backed startup (circa 1998) and its goal to build a database for Internet-scale applications. His story highlights what software is all about – history repeating itself because technology/software is meant to solve problems via new tools, techniques, and bigger challenges at bigger scales. Some of the highlights of the show include: Why Chris left Microsoft and how much it cost him; yet, he has no regrets Chris’s concept addressed how to build a scalable database layer; how to partition, chart, and cluster; and how to make it highly available and a completely scale-out architecture Chris couldn’t use the code he had created for it while at Microsoft; but from that, he learned what he wouldn’t do again Chris let the file system be the database at Microsoft, and the project was named, Internet File Store (IFS); it used backend code and was similar to S3 Chris named his startup Viathan; had to do copyright, trademark, and domain name searches Data for the Microsoft project could be stored in files/XML documents; Viathan took a different approach and used relational databases instead of a file system Companies experienced problems at the beginning of the Internet; rest of ecosystem wasn’t developed and there weren’t enough people needing Internet solutions yet Viathan went through several iterations that led to patents being issued and being considered as Prior art Viathan’s technology couldn’t just be plugged in and turned on, applications had to be modified – a tough sell Chris did groundbreaking work for what would become DynamoDB Links and Resources AWS DynamoDB AWS re:Invent 2018 – Keynote with Werner Vogels re:Invent DeepRacer JSON Moby Dick MongoDB Acid Compliance Prior Art Kelsus Secret Stache Media…
1 Replay of Ep 39 - The Birth of NoSQL and DynamoDB 33:01
33:01 Play Later Play Later Lists Like Liked33:01
Play Later Play Later Lists Like LikedChris Hickman and Jon Christensen of Kelsus and Rich Staats from Secret Stache offer a history lesson on the unique challenges of data at “Internet scale” that gave birth to NoSQL and DynamoDB. How did AWS get to where it is with DynamoDB? And, what is AWS doing now? Some of the highlights of the show include: Werner’s Worst day at Amazon: Database system crashes during Super Saver Shipping Amazon strives to prevent problems that it knows will happen again by realizing relational database management systems aren’t built/designed for the Internet/Cloud Internet: Scale up vs. scale out via databases or servers; statefulness of databases prevents easy scalability Need sharding and partitioning of data to have clusters that can be scaled up individually Amazon’s Aha Moment: Realization that 90% of data accessed was simplistic, rather than relational; same thing happened at Microsoft - recall the Internet Tidal Wave memo? Challenge of building applications using CGI bin when Internet was brand new Solution: Build your own Internet database; optimize for scalability Links AWS re:Invent DynamoDB NoSQL AWS re:Invent 2018 - Keynote with Andy Jassy AWS re:Invent 2018 - Keynote with Werner Vogels Oracle Database Bill Gates’ Internet Tidal Wave CGI Bin Kelsus Secret Stache Media End Song Whisper in a Dream by Uskmatu More Info We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Replay of Ep 14. Stop Worrying About Cloud Lock-in 27:29
27:29 Play Later Play Later Lists Like Liked27:29
Play Later Play Later Lists Like LikedOriginal Show Notes: At the recent Gluecon event, a popular topic centered around how to prevent Cloud Lock-in. Chris Hickman and Jon Christensen of Kelsus and Rich Staats from Secret Stache discuss why you your time is better spent focusing on one cloud provider. If/when Cloud Lock-in becomes an issue, you will have the resources to deal with it. Some of the highlights of the show include: AWS Fargate is ‘serverless ECS’. You don’t need to manage your own cluster nodes. This sounds great, but we’ve found the overhead of managing your own cluster to be minimal. Fargate is more expensive than ECS, and you have greater control if you manage your own cluster. Cloud lock-in was a huge concern among people at Gluecon 2018. People from large companies talked about ‘being burned’ in the past with vendor lock-in. The likely risks are (1) price gouging and (2) vendors going out of business. Cloud allows people to deploy faster and more cheaply than running their own hardware, as long as you don’t have huge scale. Few businesses get large enough to need their own data center on-prem to save money. Small and startup companies often start off in the Cloud. Big companies often have their own data centers and they are now migrating to the Cloud. AWS does allow you to run their software in your own data center, but this ties you to AWS. There is huge complication and risk to architecting a system to run in multiple cloud environments, and it almost certainly wouldn’t run optimally in all clouds. We think the risk of AWS hiking prices drastically, or going out of business, is essentially zero. If you were building a microservice-based multi-cloud system, some of the difficulties include: Which cloud hosts the database? How do I spread my services across 2 clouds? What about latency between cloud providers networks? How do I maintain security? How do I staff people who are experts at operating in both clouds? It’s clear that lock-in is a real fear for many companies, regardless of our opinion that it shouldn’t be such a concern. Jon thinks the fear of lock-in may drive cloud providers toward standardization; Chris thinks AWS doesn’t have a compelling reason to standardize since they’re the industry leader. Our advice: as a small or medium size company, don’t worry about cloud lock in. If you get big enough that it’s really a concern, we recommend building abstractions for the provider-specific parts of your system, and having a backup of your system ready to run in a 2nd cloud provider, but don’t try to run them concurrently. Links and Resources Kelsus Secret Stache Media AWS Fargate re:Invent Gluecon Kubernetes…
1 Learn cloud native software development by podcast 2:15
2:15 Play Later Play Later Lists Like Liked2:15
Play Later Play Later Lists Like Liked 1 Automate all the things - Updating container secrets using CloudWatch Events + Lambda 1:08:15
1:08:15 Play Later Play Later Lists Like Liked1:08:15
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: Developing a system for automatically updating containers when secrets are updated is a two-part solution. First, we need to be notified when secrets are updated. Then, we need to trigger an action to update the ECS service. CloudWatch Events can be used to receive notifications when secrets are updated. We explain CloudWatch Events and its primary components: events, rules and targets. Event patterns are used to filter for the specific events that the rule cares about. We discuss how to write event patterns and the rules of matching events. The event data structure will be different for each type of emitter. We detail a handy tip for determining the event structure of an emitter. We discuss EventBridge and how it relates to CloudWatch Events. We explain how to create CloudWatch Event rules for capturing update events emitted by both Systems Manager Parameter Store and AWS Secrets Manager. AWS Lambda can be leveraged as a trigger of CloudWatch Events. We explain how to develop a Lambda function that invokes the ECS API to recycle all containers. We finish up by showing how this works for a common use case: using the automatic credential rotation feature of AWS Secrets Manager with a containerized app running on ECS that connects to a RDS database. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Night Sea Journey by Derek Russo More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Database Soup - Explaining ACID, BASE, CAP - Part 3 49:21
49:21 Play Later Play Later Lists Like Liked49:21
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: In this new series, we are discussing database consistency models explained in three acts. This episode is "Act III: Eventual consistency saves the web (circa early 2000s)". We explain eventual consistency and the motivation behind the philosophy. The BASE acronym stands for three key properties of a distributed system that utilizes eventual consistency. We define and explain these BASE attributes: Basically available Soft state Eventual consistency We share the story of Werner Vogel's keynote at re:Invent 2018, where he outlined the reasons why DynamoDB was created. In particular, DynamoDB allows for an eventual consistency data model. Interestingly, the DynamoDB story closely parallels what happened when Chris was at Microsoft. It just happened at least 6 years earlier. We then wrap up everything we have learned about ACID, CAP, and BASE by providing some guidelines on when to choose ACID vs. BASE systems. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Whisper In A Dream (Feathericci Remix) by Uskmatu More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Database Soup - Explaining ACID, BASE, CAP - Part 2 45:27
45:27 Play Later Play Later Lists Like Liked45:27
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: In this new series, we are discussing database consistency models explained in three acts. This episode is "Act II: The arrival of the Internet creates new challenges (circa 1998)". Problems with building large scale-out systems led to the "discovery" of the CAP theorem (by Eric Brewer of Inktomi). We explain what the CAP theorem postulates and break it down in understandable terms. The three properties of the CAP theorem are consistency, availability and partition tolerance. What exactly is meant by "partition tolerance"? A key implication of the CAP theorem is that must choose your priorities. As a system scales, it cannot be both available and consistent. We discuss Physalia, a technology developed by AWS for making Elastic Block Service (EBS) more resilient. The design of Physalia was inspired by the principles of the CAP theorem. We then take a personal story detour that is (mostly) related to the CAP theorem. It turns out, Eric Brewer and Chris share a common experience during the first Internet bubble. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Disruption by Miquel Salla More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 Database Soup - Explaining ACID, BASE, CAP - Part 1 41:19
41:19 Play Later Play Later Lists Like Liked41:19
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: In this new series, we are discussing database consistency models explained in three acts. This episode is "Act I: Transaction processing (circa 1973)". We start with the motivation behind talking about database soup - why are ACID, CAP, and BASE important to understand? We define transaction processing and its origins. What exactly is a "transaction"? Transactions are governed by ACID semantics. We define and explain the four characteristics of the ACID acronym:: Atomicity Consistency Isolation Durability The computer scientist, Jim Gray, came up with the idea of ACID semantics in the late 1970's. We discuss some of the history behind this, along with a bizarre and tragic ending to his story. We also share a personal story about another important player in transaction processing, Phil Bernstein. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Talcum by Lost Lake More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
Oh by the way, buy girl scout cookies from my daughter here! GIRL SCOUT COOKIEEEEES! In this episode, we cover the following topics: Technology is changing at an increasing rate, with a constant stream of new things to learn. We discuss how innovation has changed the rules of the game. "Life moves pretty fast. If you don't stop and look around once in a while, you could miss it." - Ferris Bueller Chris recounts a personal story that emphasizes the importance of continual learning and growth. During preparation for a previous Mobycast mini-series, he relies on a trusted solution pattern for accessing private subnets. But during final fact checking, he discovers entirely new options that may be better/easier solutions. We break down some important lessons learned from this experience. Why do we think the following Stephen King quote is sage advice for all of us? "Kill your darlings, kill your darlings, even when it breaks your egocentric little scribbler’s heart, kill your darlings." We then turn our attention to discussing what is your most important skill. What is it? How can you cultivate it? Why does this matter? Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Disco Pigs (Rave Mix) by Roy England More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
1 The Future of Containers - Part 3 - Unikernels 57:27
57:27 Play Later Play Later Lists Like Liked57:27
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: We continue our discussion of microVMs with a look at Kata Containers. Kata Containers formed by the merger of two projects: Intel Clear Containers and Hyper runV. How does Kata Containers integrate with existing container tooling? How mature are Kata Containers - are they ready for production? We then take a look at unikernels, which take a dramatically different approach to solving the problem of providing high security with blazing performance. The benefits of unikernels along with a comparison on how they differ from containers. We discuss some of the most popular unikernel implementations, including OSv and MirageOS. Does the future point to a deathmatch between containers and unikernels, or will there be a need for both approaches to cloud-native apps? DETAILED SHOW NOTES Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ SUPPORT MOBYCAST https://glow.fm/mobycast END SONG Palm of Your Hand by Blynkwth MORE INFO For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 The Future of Containers - Part 2 - Making Sense of MicroVMs (continued) 1:00:52
1:00:52 Play Later Play Later Lists Like Liked1:00:52
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: We revisit a misunderstanding from last week's show to find out exactly what the Firecracker team means when they list "Single VM per Firecracker process" as a security benefit. We discuss what's next on the Firecracker product roadmap, with particular emphasis on support for snapshot/restore. We learn how AWS uses Firecracker in production today with AWS Lambda. AWS is currently working on updating Fargate to use Firecracker. We look at why they are doing this and the design details of updating Fargate to use Firecracker. We finish by looking at how you can use Firecracker for your own containers, by incorporating Firecracker-aware tooling into your container infrastructure. Specifically, we look at firecracker-containerd and Weave Ignite. DETAILED SHOW NOTES Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ SUPPORT MOBYCAST https://glow.fm/mobycast END SONG Thing Is by Public Address MORE INFO For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 The Future of Containers - Part 1 - Making Sense of MicroVMs 1:08:23
1:08:23 Play Later Play Later Lists Like Liked1:08:23
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: We review virtual machines (full virtualization) and their benefits and tradeoffs. We then revisit containers (OS-level virtualization) and briefly recap how they use OS kernel features to enable virtualization. Containers provide great performance and resource efficiency, but at the cost of losing strong isolation. Can we have the performance and efficiency benefits of containers but with the strong isolation of VMs? There are some promising technologies that aim to combine the best of both VM and container worlds: microVMs, unikernels and container sandboxes. What are microVMs? What are unikernels? What are container sandboxes? AWS Firecracker is one of the most talked about microVMs. We discuss what it is, and the key benefits of using Firecracker. DETAILED SHOW NOTES Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ SUPPORT MOBYCAST https://glow.fm/mobycast END SONG Smooth Modulator by aMIGAaMIGO MORE INFO For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Psst... Secrets Handling for Cloud-Native Apps - Part 2 46:42
46:42 Play Later Play Later Lists Like Liked46:42
Play Later Play Later Lists Like LikedIn this episode, we cover the following topics: AWS offers not one, but two, managed services for secrets management. Systems Manager Parameter Store and AWS Secrets Manager have similar functionality, making it sometimes confusing to know which to use. We compare and contrast the two services to help guide your choice. The three types of sensitive data injection supported by Elastic Container Service (ECS). Understanding when sensitive data is injected into the container and how to handle updates to secrets (such as credential rotation). The required configuration changes and IAM permissions you need to enable ECS integration with Parameter Store and Secrets Manager. A walkthrough of the specific steps you need to take to update your ECS application to support secrets integration. Detailed Show Notes Want the complete episode outline with detailed notes? Sign up here: https://mobycast.fm/show-notes/ Support Mobycast https://glow.fm/mobycast End Song Straddling by Derek Russo More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Psst... Secrets Handling for Cloud-Native Apps - Part 1 55:49
55:49 Play Later Play Later Lists Like Liked55:49
Play Later Play Later Lists Like LikedSupport Mobycast -> https://glow.fm/mobycast <- In this episode, we cover the following topics: What is secrets management and why we need it for our cloud-native applications. Guidelines for best practices when handling secrets. We walkthrough a simple, roll-your-own approach to secrets management using encryption (KMS) and an object store (S3). Although this is a simple technique, it does provide a very secure (and auditable) approach to secrets handling. But, for most situtations, you'll want to leverage an off-the-shelf secrets management solution. We discuss 3 popular choices, including Hashicorp Vault, AWS Systems Manager Parameter Store and Amazon Secrets Manager. What are the features you should expect from a secrets management solution. We take a closer look at Vault, Parameter Store and Secrets Manager, and discuss the features that each provides. We finish with some guidance on how to make the right choice of secrets management solution for your applications. Links Secrets Management for Cloud-Native Applications Vault - Unlocking the Cloud Operating Model: Security AWS Systems Manager Parameter Store How AWS Systems Manager Parameter Store Uses AWS KMS Introducing AWS Secrets Manager AWS Secrets Manager How AWS Secrets Manager Uses AWS KMS Rotating Your AWS Secrets Manager Secrets Tutorial: Specifying Sensitive Data Using Secrets Manager Secrets AWS Secrets Manager now supports VPC endpoint policies How to Manage Secrets for Amazon EC2 Container Service–Based Applications by Using Amazon S3 and Docker End Song Warming Trend by Aphreaq More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 VPC Ninja - Part 3 - Moving an ECS Application to Private Subnets 53:48
53:48 Play Later Play Later Lists Like Liked53:48
Play Later Play Later Lists Like LikedSupport Mobycast -> https://glow.fm/mobycast <- In this episode, we explain how to move an existing ECS application to private subnets. We cover the following topics: We describe the existing application, which is a typical two-tier web application, with a web service fronted by an Application Load Balancer (ALB) and database hosted on MySQL using RDS. The current application is containerized and running under ECS. Everything (the load balancer, ECS cluster, RDS instance) is running on public subnets. The goal is to leave only the ALB public-facing, with all other resources protected on private subnets. There are two phases to moving the application to private subnets. First, we need to move the ECS cluster to private subnets. Then, we can move the RDS instance to private subnets. We detail step-by-step two deployment approaches for moving our ECS cluster to private subnets, both of which involve zero downtime. Rolling deployment, which updates the existing cluster in-place. Blue/green deployment, which creates a new cluster to replace the existing one. We discuss the steps on moving the database instance to private subnets, including application downtime considerations. As a bonus, we explain how to add encryption-at-rest to the RDS instance during the migration. Links VPC with Public and Private Subnets (NAT) Changing the Launch Configuration for an Auto Scaling Group Add Encryption to an Unencrypted RDS DB Instance Amazon ECS-optimized AMIs End Song The Runner (Lost Lake Remix) by Fax More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 That's a Wrap - AWS re:Invent 2019 Takeaways - Part 2 45:52
45:52 Play Later Play Later Lists Like Liked45:52
Play Later Play Later Lists Like LikedSupport Mobycast -> https://glow.fm/mobycast <- In this episode, we cover the following topics: Recap and analysis of Andy Jassy's keynote, including: The theme of this year's keynote is transformation , presented via 6 theme songs. "The hunger keeps on growing" (Dave Matthews Band, "Too Much") Storage performance is growing much faster than compute/memory (6x faster since 2012). This is enabling new innovations like AQUA for Redshift, making it 10x faster than any other cloud data warehouse. How the new Ultrawarm for Elasticsearch tier reduces cost by 90%. AWS helps again with undifferentiated heavy lifting by offering a managed Cassandra service. "I would walk 500 miles" (The Proclaimers, "I'm Gonna Be (500 Miles)") AWS is one of the best places for AI/ML across all layers of the stack. SageMaker is a vibrant ML platform that is rapidly evolving into the next generation developer desktop. Can AWS really automate code reviews with its new Code Guru service? Using the power of AI to search for enterprise data with Kendra. "Break on through to the other side" (The Doors, "Break on Through") If you can't come to AWS, AWS is coming to you! AWS infrastructure is everywhere with VMware Cloud on AWS, AWS Outposts, AWS Local Zones and AWS Wavelength. Recap and analysis of Werner Vogel's keynote, including: The problems and limitations of classic virtualization (Xen-based). How the Nitro System takes a microservices approach to computer system design. The importance of Firecracker as a next generation VM. A case-study of the Elastic Block Service (EBS) architecture and how AWS is using a new technology called Physalia to battle the CAP theorem. You can now learn more about how AWS designs and develops software with the new Amazon Builders' Library. Key takeaways The Nitro System is a key technology that is enabling AWS to break through barriers. It will provide us with powerful new capabilities to tackle previously unsolvable problems. AWS is everywhere, extending all the way out to the edge with IoT/Greengrass, and everywhere in between with on-premise (AWS Outposts), near-premise (Local Zones) and even integrated on the 5G network with AWS Wavelength. AWS is a leader in AI/ML, with Sagemaker becoming a next generation developer platform. Analytics is a big priority, with focus on Redshift and data lakes. Quantum computing is no longer the stuff of science fiction. It will be here sooner than you think. Links AWS re:Invent AWS re:Invent 2019 Session Videos AWS Nitro System Elastic Fabric Adapter Firecracker firecracker-containerd The Amazon Builders' Library Amazon Braket Amazon Braket – Get Started with Quantum Computing End Song Walking Dub For Black Queens, by Place More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 That's a Wrap - AWS re:Invent 2019 Takeaways - Part 1 51:32
51:32 Play Later Play Later Lists Like Liked51:32
Play Later Play Later Lists Like LikedSupport Mobycast -> https://glow.fm/mobycast <- In this episode, we cover the following topics: re:Invent 2019 by the numbers: 65,000 attendees, 3,000+ sessions, 4 keynotes, 6 venues. Recap and analysis of Monday Night Live keynote with Peter DeSantis, including: What is high performance computing (HPC)? How AWS is reinventing the supercomputer. Why everyone should care about HPC, not just the scientists. How networking advancements are paving the way forward for cluster computing and enabling entirely new types of problem solving. A discussion of the Elastic Fabric Adapter (EFA) and the new Scalable Reliable Datagram (SRD) networking protocol as a replacement for TCP for high performance networking. Using the Nitro System to enable new instance types for machine learning infrastructure, such as the P3dn, G4dn and Inf1 instance types. Utilizing custom silicon to make the "Inferentia" processor, which is a high-performance ML inference chip. Recap and analysis of Andy Jassy's keynote, including: The theme of this year's keynote is transformation , presented via 6 theme songs. "Don't wait until tomorrow" (Van Halen, "Right Now") Your transformation needs to start today. The problems will only get harder, deeper tomorrow. "Don't stop me now, I'm having such a good time" (Queen, "Don't Stop Me Now") Developers love AWS and its capabilities (175+ services, breadth & depth). AWS is rapidly innovating its compute capabilities with new instance types (driven by Nitro System) and new ways of running containers (including the just announced Fargate for EKS). "Is that all you get for your money?" (Billy Joel, "Movin' Out (Anthony's Song)") You need to modernize your technology stack. Get off mainframes, migrate away from the old guard databases with their licensing tricks, and switching from Windows to Linux. "The hunger keeps on growing" (Dave Matthews Band, "Too Much") Data is exploding, and customers are moving from data silos to data lakes, with S3 the most popular choice for data lakes. New feature, Amazon S3 Access Points, helps make giving access to S3 data easier on a per user/application basis. AWS is a leader in analytics infrastructure with Athena, EMR, Redshift, ElastiSearch, Kinesis, and QuickSight. AWS is investing heavily in its Redshift platform, with many new features announced including the ability to now manage compute and storage separately using the new Redshift RA3 instances. Links AWS re:Invent AWS re:Invent 2019 Session Videos AWS Nitro System Elastic Fabric Adapter AWS Inferentia End Song You Just Can’t, by Roy England More Info For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 VPC Ninja - Part 2 - Private subnets with VPN (continued) 1:01:44
1:01:44 Play Later Play Later Lists Like Liked1:01:44
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast In this episode, we cover the following topics: Before we get started, a CAVEAT. There are other (potentially BETTER) ways of accessing resources on private subnets. We'll talk about these (such as AWS Client VPN or AWS Systems Manager Session Manager) in future episodes. But a great choice (with the most flexibility/power) remains our current choice: a third-party software-only VPN solution. There are many options for third-party software VPNs, both commercial and open source. Some of the options we considered include: SoftEther Openswan OpenVPN (* our choice) Discussion of the different flavors and pricing models for OpenVPN Access Server. Step-by-step walkthrough of installing OpenVPN Access Server via the AWS Marketplace. Including how to setup TLS for your VPN server. We detail the process of how to create private subnets within a VPC. Create new subnets to be used as private subnets, keeping in mind a multi-AZ design. Routing table considerations. Setting up a NAT gateway to forward Internet traffic for private subnets. Some pro tips to keep in mind when building out your cloud network. CIDR block considerations (the "Goldilocks" approach to sizing). Did you know that NAT gateways are SPOFs? We discuss how to improve availability. Links VPC with Public and Private Subnets (NAT) Software VPN OpenVPN SoftEther Openswan Amazon Web Services EC2 BYOL appliance quick start guide AWS Certificate Manager ZeroSSL End Song Tachyon, by Roy England For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 VPC Ninja - Part 1 - Private Subnets with VPN 57:50
57:50 Play Later Play Later Lists Like Liked57:50
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast Show Details In this episode, we cover the following topics: Subnet 101 Public subnets Used for public facing resources which allow inbound connections from the public Internet Private subnets What are they? Used for resources that should not be exposed to open Internet Do not allow direct access from open Internet Require use of network address translation (NAT) for egress-only Internet access Why use private subnets? Protect your cloud servers from script kiddies Limit exposure Security groups and routing tables allow resources on public subnets to communicate with private subnets NAT (network address translation) deep dive What is NAT? Remaps one IP address space into another Done by modifying network address information in IP header of packets while in transit across routing device Tool to deal with IPv4 address exhaustion Only need single public IP address for NAT, which hides entire private network behind it Note: Actual role of NAT device is both address translation and port address translation How does it work? IP header consists of: Source IP Source port Destination IP Destination port Routing device modifies IP address in packets Outgoing packets (from private-to-public) Source IP and port changed to NAT values I.e. packets appear to originate from NAT (instead of private IP itself) Incoming packets (public-to-private) Dest IP and port changes to private values For TCP/UDP NAT keeps in memory table that maps traffic to private IPs Table includes each active connection (particularly the destination address and port) When reply comes back to router, uses table to determine private IP that reply should be forwarded to Port numbers are changed so combination of IP and port on returned packet can be unambiguously mapped to corresponding private destination Note: conversation to open Internet has to originate in private network! This is because initial message establishes required information in translation table How can a single computer have both public and private IP addresses? A quick primer on IP addresses and network interface cards MAC (media access control) address Physical address Unique ID assigned to NIC IP address Logical address Network switches maintain Address Resolution Protocol (ARP) tables that map IP addresses to MAC addresses ARP table used to know which MAC address to attach to packet Single NIC can have multiple IP addresses Alas, private subnets are less convenient than public subnets. Instances on private subnet won't be publicly accessible, they can only be accessed from inside the network. This leads to the problem of how to connect to an instance on a private subnet from a remote location? Three broad categories of solutions: Direct Connect Dedicated network connection over private lines straight into AWS backbone Requires network equipment on customer side Cons: Requires dedicated hardware Expensive Applicable only when you have an on-prem location that needs to be physically connected to VPC Bastion host (jump host) Public-facing server running SSH daemon Once connected to bastion host, users can then ssh to machines on private subnet Typically have a single instance on public subnet Minimizes surface area to be protected Cons: Adds an extra layer of indirection ssh key management is more complicated SPOF Security risk of protecting the bastion host VPN (virtual private network) Many different options, ranging in cost and equipment requirements For both connecting on-prem location, as well as general remote user access VPN Available options Managed VPN Managed IPsec VPN connection over existing internet Quick and usually simple method for making secure connection to VPC Can be used as redundant link for Direct Connect Supports static routes or BGP peering/routing How to setup: Designate an appliance to act as your customer gateway (usually the on-prem router) Create VPN connection in AWS and download config file for your customer gateway Configure customer gateway with config file VPN CloudHub Connect locations in hub and spoke manner using Virtual Private Gateway Allows remote locations to communicate with each other via the hub (Virtual Private Gateway in AWS) Each remote location uses Site-to-Site VPN connection to connect to hub Reuses existing internet connection Supports BGP routes to direct traffic e.g. use MPLS first then CloudHub VPN as backup How to setup: Assign multiple Customer Gateways to a Virtual Private Gateway, each with their own BGP ASN and unique IP ranges Third-party software VPN You provide your own VPN endpoint/software Use this option if you must manage both ends of VPN connection How to setup: Install VPN software via Marketplace appliance or on EC2 instance TIL... AWS has increased the options Managed VPN is now known as "AWS Site-to-Site VPN" New option: "AWS Client VPN" Fully managed, highly available software-only VPN Supports OpenVPN-based clients We'll discuss "AWS Client VPN" in-depth in a future episode Our choice for this episode: let's setup a third-party software VPN solution Rationale: Not too much $$$ Pretty sophisticated solution that's easy to manage Links VPC with Public and Private Subnets (NAT) Network-to-Amazon VPC Connectivity Options Network address translation RFC 1631 - The IP Network Address Translator (NAT) Multiple IP Addresses AWS VPN Introducing AWS Client VPN to Securely Access AWS and On-Premises Resources End Song Zero Gravity by Roy England For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email:…
M
Mobycast
Support Mobycast https://glow.fm/mobycast In this episode, we cover the following topics: AWS re:Invent general overview December 2nd thru December 6th 2,500+ sessions, spread over 6 venues, spanning 2.5 miles of the Las Vegas Strip Discuss the 4 primary types of content and the pros/cons of each Sessions, chalk talks, workshops and builders sessions Our general observations of themes to expect this year Hint: Kubernetes is hot We point out some of the sessions we are particularly looking forward to re:Invent is not all work, you get to play too re:play Other parties and where to find them Our predictions for new product/service announcements at re:Invent 2019 We also talk about Apple's recent launch of a new MacBook Pro (goodbye butterfly keyboard!) Links AWS re:Invent AWS re:Invent Twitter feed AWS re:Invent 2019 - How to re:Invent YouTube channel Startup Central is Headed to re:Invent! A Cloud Guru Guide to re:Invent Linux Academy Guide to re:Invent Las Vegas Monorail Unofficial - re:Invent Parties Twitter feed Unofficial - List of AWS re:Invent Conference and Vendor Parties Apple launches 16-inch MacBook Pro with 6 speakers and 'Magic Keyboard' End Song Flowerchild (Lost Lake Remix) by Owen Ni For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Serverless Containers with ECS Fargate - Part 3 58:34
58:34 Play Later Play Later Lists Like Liked58:34
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast Show Details In this episode, we cover the following topics: Container networking ECS networking mode Configures the Docker networking mode to use for the containers in the task Specified as part of the task definition Valid values: none Containers do not have external connectivity and port mappings can't be specified in the container definition bridge Utilizes Docker's built-in virtual network which runs inside each container instance Containers on an instance are connected to each other using the docker0 bridge Containers use this bridge to communicate with endpoints outside of the instance using primary ENI of instance they are running on Containers share networking properties of the primary ENI, including the firewall rules and IP addressing Containers are addressed by combination of IP address of primary ENI and host port to which they are mapped Cons: You cannot address these containers with the IP address allocated by Docker It comes from pool of locally scoped addresses You cannot enforce finely grained network ACLs and firewall rules host Bypass Docker's built-in virtual network and maps container ports directly to the EC2's NIC directly You can't run multiple instantiations of the same task on a single container instance when port mappings are used awsvpc Each task is allocated its own ENI and IP address Multiple applications (including multiple copies of same app) can run on same port number without conflict You must specify a NetworkConfiguration when you create a service or run a task with the task definition Default networking mode is bridge host and awsvpc network modes offer the highest networking performance They use the Amazon EC2 network stack instead of the virtualized network stack provided by the bridge mode Cannot take advantage of dynamic host port mappings Exposed container ports are mapped directly... host: to corresponding host port awsvpc: to attached elastic network interface port Task networking (aka awsvpc mode networking) Benefits Each task has its own attached ENI With primary private IP address and internal DNS hostname Simplifies container networking No host port specified Container port is what is used by task ENI Container ports must be unique in a single task definition Gives more control over how tasks communicate With other tasks Containers share a network namespace Communicate with each other over localhost interface e.g. curl 127.0.0.1:8080 With other services in VPC Note: containers that belong to the same task can communicate over the localhost interface Take advantage of VPC Flow Logs Better security through use of security groups You can assign different security groups to each task, which gives you more fine-grained security Limitations The number of ENIs that can be attached to EC2 instances is fairly small E.g. c5.large EC2 may have up to 3 ENIs attached to it 1 primary, and 2 for task networking Therefore, you can only host 2 tasks using awsvpc mode networking on a c5.large However, you can increase ENI density using "VPC trunking" VPC trunking Allows for overcoming ENI density limits Multiplexes data over shared communication link How it works Two ENIs are attached to the instance Primary ENI Trunk ENI Note that enabling trunking consumes an additional IP address per instance Your account, IAM user, or role must opt in to the awsvpcTrunking account setting Benefits Up to 5x-17x more ENIs per instance E.g. with trunking, c5.large goes from 3 to 12 ENIs 1 primary, 1 trunk, and 10 for task networking Migrating a container from EC2 to Fargate IAM roles Roles created automatically by ECS Amazon ECS service-linked IAM role, AWSServiceRoleForECS Gives permission to attach ENI to instance Task Execution IAM Role (ecsTaskExecutionRole) Needed for: Pulling images from ECR Pushing logs to CloudWatch Create a task-based IAM role Required because we don't have an ecsInstanceRole anymore Create a IAM policy that gives minimal privileges needed by task Remember two categories of policies: AWS Managed Customer Managed We are going to create a new customer managed policy that contains only the permissions our app needs KMS Decrypt, S3 GETs from specific bucket IAM -> Policies -> Create Policy -> JSON See IAM Policy example below Create role based on "Elastic Container Service Task" service role This service role gives permission to ECS to use STS to assume role (sts:AssumeRole) and perform actions on its behalf IAM -> Roles -> Create Role "Select type of trusted entity": AWS Service Choose "Elastic Container Service", and then "Elastic Container Service Task" use case Next, then attach IAM policy we created to the role and save Task definition file changes Task-level parameters Add FARGATE for requiredCompatibilities Use awsvpc as the network mode Specify cpu and memory limits at the task level Specify Task Execution IAM Role (executionRoleARN) Allows task to pull images from ECR and send logs to CloudWatch Logs Specify task-based IAM role (taskDefinitionArn) Needed to give task permissions to perform AWS API calls (such as S3 reads) Container-level parameters Only specify containerPort (do not specify hostPort) See Task Definition example below Create ECS service Choose cluster Specify networking VPC, subnets Create a security group for this task Security group is attached to the ENI Allow inbound port 80 traffic Auto-assign public IP Attach to existing application load balancer Specify production listener (port/protocol) Create a new target group When creating target group, you specify "target type" Instance IP Lambda function For awsvpc mode (and by default, Fargate), you must use the IP target type Specify path pattern for ALB listener, health check path Note: you cannot specify host-based routing through the console You can update that after creating the service through the ALB console Update security groups Security group for ALB Allow outbound port 80 to the security group we attached to our ENI Security group for RDS Allow inbound port 3306 from the security group for our ENI Create Route 53 record ALIAS pointing to our ALB Log integration with SumoLogic Update task to send logs to stdout/...…
M
Mobycast
1 Bonus Episode! Docker Is Kind of Acquired By ... Who Is Mirantis? 23:06
23:06 Play Later Play Later Lists Like Liked23:06
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast Links Techcrunch article Mirantis Docker End Song La Place by Iwa For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Serverless Containers with ECS Fargate - Part 2 57:25
57:25 Play Later Play Later Lists Like Liked57:25
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast In this episode, we cover the following topics: Identity and access management for ECS Primary roles ECS Container Instance IAM Role ecsInstanceRole IAM policy and role required by ECS agent to make ECS API calls on your behalf ECS Service Scheduler IAM Role ecsServiceRole ECS service scheduler makes calls to EC2 and ELB APIs on your behalf Register/deregister container instances with load balancers ECS Task Execution IAM Role ecsTaskExecutionRole Also used by ECS agent to make AWS API calls on your behalf Typical use cases Your task uses Fargate and is... pulling a container image from Amazon ECR uses the awslogs log driver Your tasks uses either Fargate or EC2 launch type and... pulls images from private registry the task definition is referencing sensitive data using Secrets Manager or Parameter Store Secondary roles ECS Service Auto Scaling IAM Role ecsAutoscaleRole Used by Application Auto Scaling service to describe CloudWatch alarms and registered services Updates ECS services's desired count CloudWatch Events IAM Role ecsEventsRole Required role when you have ECS scheduled tasks Interacts with CloudWatch Events rules and targets This IAM policy and role gives CloudWatch permissions to run ECS tasks on your behalf ECS CodeDeploy IAM Role ecsCodeDeployRole Required when doing blue/green deployments (powered by CodeDeploy) Best practice: Using task-based IAM roles IAM role for Amazon ECS tasks Allows you to specify an IAM role that can be used by the containers in a task IAM role for task is specified using the taskRoleArn setting in task definition Prefer more granular task-based IAM roles over instance roles Each specific task definition or service should have its own role Benefits of task-based IAM roles Least privilege By specifying access at the task-level (instead of at the instance-level), we can have fine-grained control Only give the minimum required permissions for the tasks to operate Credential isolation Container can only use credentials assigned to it Auditability Access and event logging available via CloudTrail CloudTrail logs show taskArn Creating a task-based IAM role First create IAM policy that specifies the minimal permissions needed by your containers Or use an existing managed policy Next create an IAM role for your task Create role based on Amazon Elastic Container Service Task Role service role Then attach your IAM policy to the task role Example: Container needs to make S3 calls Create a new IAM role for the task, and attach the "AmazonS3ReadOnlyAccess" policy to the role Then use the role ARN in task definition Links Amazon Elastic Container Service AWS Fargate - Product Page ECS Fargate - Developer Guide IAM Roles for Tasks End Song Beauty in Rhythm (Fredy Grogan Remix) - Roy England For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Serverless Containers with ECS Fargate - Part 1 1:04:42
1:04:42 Play Later Play Later Lists Like Liked1:04:42
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast In this episode, we cover the following topics: Amazon Elastic Container Service (ECS) basics Orchestration system for containers Well integrated with all the other Amazon services – More bang for your buck ECS components Cluster Logical grouping of tasks or services For EC2 launch type, set of EC2 instances that are defined and managed by: Launch Configuration Auto Scale Group Service Allows you to run and maintain a specified number of instances of a task definition simultaneously For long-running applications Task Defines a collection of containers that you want to run together Specifies resource quotas needed to run (e.g. memory, CPU, disk volumes) Simple deployment with ECS Build image, publish image, create task definition revision, update ECS service Running containers Three methods Create a long running task ECS service, service scheduler, integration with ELB Run a single task Create a scheduled task We are going to focus on the most typical use case - ECS services You have to choose a launch type EC2 or Fargate Fargate Announced at re:Invent 2017 Generally available since 2018 What is it? Allows you to run containers without having to manage servers or clusters of EC2s Don't need to choose server types, decide when to scale your clusters, or optimize cluster packing You get complete control over task placement within your own VPC But underlying infrastructure is managed by Fargate Benefits No clusters to manage Seamless scaling Only pay for when you are running tasks Ideal for batch jobs, cron jobs and other on-and-off workloads Running cluster of instances constantly incurs costs, but Fargate stops billing when containers stop Specifics Each Fargate task has its own isolation boundary It does not share the underlying kernel, CPU resources, memory resources, or ENI Leverages Firecracker microVM Increases efficiency (e.g. approximately 50% price cut for Fargate in January 2019 due to Firecracker) Tasks must be launched into a cluster Cluster is logical infrastructure and permissions boundary for isolating groups of tasks Clusters support running both EC2 and Fargate launch types (mix-n-match) Fargate tasks require awsvpc network mode Provides each task with an ENI You must specify one or more subnets You must specify one or more security groups Decide on whether to assign public IP address to ENI If on public subnet, you must assign public IP to pull images If on private subnet, just requires NAT gateway You must specify CPU and memory at the task level You can also optionally specify CPU and memory at container level Only supports the following log drivers awslogs Sends log information to CloudWatch Logs splunk Pricing Based on amount of CPU and memory used Charged by the second, with minimum charge of 1 minute Example costs for running a blog server 24x7 Note: costs for us-west-2 region Fargate, 0.25 VCPU, 0.5GB memory per vCPU per hour: $0.04048 per GB per hour: $0.004445 Memory = $1.60 (30 days * 24 hours * 0.5 GB * 0.004445) CPU = $7.29 (30 days * 24 hours * 0.25VCPU * 0.04048) Total = $8.89 / month t2.micro, 1 VCPU, 1GB memory per hour: $0.0116 Total = $8.35 (30 days * 24 hours * 0.0116) t3.nano, 2 VCPU, 0.5GB memory per hour: $0.0052 Total = $3.74 (30 days * 24 hours * 0.0052) Links Amazon Elastic Container Service ECS Fargate AWS Fargate Price Reduction – Up to 50% End Song ERRE - Lamictal For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Reddit: https://reddit.com/r/mobycast…
M
Mobycast
1 Virtual Machines vs. Containers Revisited - Part 4 56:46
56:46 Play Later Play Later Lists Like Liked56:46
Play Later Play Later Lists Like LikedSupport Mobycast https://glow.fm/mobycast In this episode, we cover the following topics: Container runtimes Responsible for: Setting up namespaces and cgroups for containers Running commands inside those namespaces and cgroups Types of runtimes Low-level Handles tasks related to containers such as: Creating a container Attaching a process to an existing container High-level Handles "high level" tasks such as: Image creation Image management Defers container tasks to "low level" runtime Container standards Open Container Initiative (OCI) Established in June 2015 by Docker and others Contains two specifications: Runtime Specification (runtime-spec) runc is an implementation of OCI runtime-spec Image Specification (image-spec) Runc Low-level container runtime CLI tool for spawning and running containers according to the OCI specification Container runtime originally developed as part of Docker Later lifted out as separate open source project Containerd High-level container runtime Provides abstraction layer for the syscalls and OS-specific functionality required to run container Platforms can build on top of abstraction layer without having to drop down to kernel Much nicer to work with abstractions (Container, Task, Snapshot) than to manage calls to clone() and mount() Available as daemon for Linux and Windows Designed to be embedded into a larger system (like Docker) Used by container platforms like Docker and Kubernetes Under the hood, containerd uses runc Only deals with core container management Push/pull functionality Image management Container lifecycle APIs Create, execute and manage containers and their tasks Snapshot management Out of scope Networking Very complicated, much more platform specific than abstracting Linux calls Instead, containerd exposes events so consumers can subscribe to events they care about E.g. hook events to add interfaces to network namespace Exposes functionality via GRPC API Listens on Linux socket Runtime platforms Docker Uses containerd, plus shim Shim allows for daemon-less containers e.g. You can upgrade Docker daemon without restarting containers rkt Application container engine, part of CoreOS (RedHat) rkt has no centralized "init" daemon Instead it launches containers directly from client commands Was part of CNCF, but archived in August 2019 Reasons cited by CNCF: "end user adoption has severely declined" "project activity and the number of contributors has also steadily declined over time" containerd and CRI-O are now the CNCF container runtime projects CRI-O Implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI runtimes Lightweight alternative to using Docker, Moby or rkt as the runtime for k8s Allows k8s to use any OCI-compliant runtime as the container runtime for running pods Supports runc and Kata Containers Revisit pseudo code example of creating a container Steps: Create root filesystem for container Spin up busybox in Docker container, and then export filesystem Run "launcher" process that sets up "child" namespace Launcher process forks new child process (now under new namespaces) Child process then forks new process for container chroot (to our root filesystem) mount any other FS set cgroups (e.g. apply CPU constraints) Links Open Container Initiative - OCI runc containerd What is containerd? CoreOS rkt rkt vs Docker End Song Miquel Salla - All is Coming Back (Focalist Remix) For a full transcription of this episode, please visit the episode webpage . We'd love to hear from you! You can reach us at: Web: https://mobycast.fm Voicemail: 844-818-0993 Email: ask@mobycast.fm Twitter: https://twitter.com/hashtag/mobycast Support Mobycast: https://glow.fm/mobycast…
Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.
Daily Deals
Similar to Mobycast
On The Bike Shed, hosts Joël Quenneville and Stephanie Minn discuss development experiences and challenges at thoughtbot with Ruby, Rails, JavaScript, and whatever else is drawing their attention, admiration, or ire this week.
…
continue reading
Hanselminutes is Fresh Air for Developers. A weekly commute-time podcast that promotes fresh technology and fresh voices. Talk and Tech for Developers, Life-long Learners, and Technologists.
…
continue reading
This is the audio podcast version of Troy Hunt's weekly update video published here: https://www.troyhunt.com/tag/weekly-update/
…
continue reading
Exploring React Native Together
…
continue reading
Developer Tea exists to help driven developers connect to their ultimate purpose and excel at their work so that they can positively impact the people they influence. With over 13 million downloads to date, Developer Tea is a short podcast hosted by Jonathan Cutrell (@jcutrell), co-founder of Spec and Director of Engineering at PBS. We hope you'll take the topics from this podcast and continue the conversation, either online or in person with your peers. Twitter: @developertea :: Email: deve ...
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
A weekly talk show taking a pragmatic look at the art and business of Software Development and the world of technology.
…
continue reading
It takes more than great code to be a great engineer. Soft Skills Engineering is a weekly advice podcast for software developers about the non-technical stuff that goes into being a great software developer.
…
continue reading
Software's best weekly news brief, deep technical interviews & talk show.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
