React Native allows developers to build native applications for iOS and Android using components written in the React JavaScript framework. These ReactJS components render to native application code by going over a JavaScript bridge, a message bus that communicates between JavaScript code and native iOS or Android runtimes. For most mobile appl ...…

In the early days of YouTube, there were scalability problems with the MySQL database that hosted the data model for all of YouTube’s videos. The state of the art solution to scaling MySQL at the time was known as “application-level sharding.” To scale a database using application-level sharding, you break up the database into shards–disjoint r ...…

Zoox is a full-stack self-driving car company. Zoox engineers work on everything a self-driving car company needs, from the physical car itself to the algorithms running on the car to the ride hailing system which the company plans to use to drive around riders. Since starting in 2014, Zoox has grown to over 500 employees. Ethan Dreyfuss is a s ...…

DoorDash is a food delivery company where users find restaurants to order from. When a user opens the DoorDash app, the user can search for types of food or specific restaurants from the search bar or they can scroll through the feed section and look at recommendations that the app gives them within their local geographic area. Recommendations ...…

The nature of software projects is changing. Projects are using a wider variety of cloud providers and SaaS tools. Projects are being broken up into more git repositories, and the code in those repositories are being deployed into small microservices. With the increased number of tools, repositories, and deployment targets, it can become diffic ...…

Ethereum allows developers to run decentralized applications. But the tooling for building and managing those decentralized applications is immature. Experienced software engineers have difficulty getting started with writing Ethereum applications because the stack of tools is so unfamiliar and different than traditional software tools. Whether ...…

A Kubernetes cluster presents multiple potential attack surfaces: the cluster itself, a node running on the cluster, a pod running in the node, a container running in a pod. If you are managing your own Kubernetes cluster, you need to be aware of the security settings on your etcd, your API server, and your container build pipeline. Many of the ...…

Cloud computing has been popular for less than twenty years. Large software companies have existed for much longer. If your company was started before the cloud became popular, you probably have a large, data center on your companies premises. The shorthand term for this software environment is “on-prem”. Deploying software to your own on-prem ...…

Uber manages the car rides for millions of people. The Uber system must remain operational 24/7, and the app involves financial transactions and the safety of passengers. Uber infrastructure runs across thousands of server instances and produce terabytes of monitoring data. The monitoring data is used to understand the health of the software sy ...…

Data infrastructure is advancing beyond the days of Hadoop MapReduce, single-node databases, and nightly reporting. Companies are adopting modern data warehouses, streaming data systems, and cloud-specific data tools like BigQuery. Every company with a large amount of data wants to aggregate that data into a data lake and make the data availabl ...…

Infrastructure software is having a renaissance. Cloud providers offer a wide range of deployment tools, including virtual machines, managed Kubernetes clusters, standalone container instances, and serverless functions. Kubernetes has standardized the container orchestration layer and created a thriving community. The Kubernetes community gives ...…

Virtualization software allows companies to get better utilization from their physical servers. A single physical host can manage multiple virtual machines using a hypervisor. VMware brought virtualization software to market, creating popular tools for allowing enterprises to deploy virtual machines throughout their organization. Containers pro ...…

Real estate is an asset that is not straightforward to invest in. Real estate can generate excellent returns for investors, but can require much more time and expertise than stocks. Cadre is a company that allows users to invest in real estate more easily and intelligently. Cadre provides users with lots of data about potential investments and ...…

RocksDB is a storage engine based on the log-structured merge tree data structure. RocksDB was developed at Facebook to provide a tool for embedded databases. The code for RocksDB is a fork of LevelDB, an embedded database built by Google for the Chrome browser. Every database has a storage engine. The storage engine is the low-level data struc ...…

HashiCorp was founded seven years ago with the goal of building infrastructure tools for automating cloud workflows such as provisioning, secret management, and service discovery. Hashicorp’s thesis was that operating cloud infrastructure was too hard: there was a need for new tools to serve application developers. Hashicorp founders Mitchell H ...…

Tyler Cowen’s book Stubborn Attachments outlines a framework that individuals can use to make decisions grounded in economic philosophy. In his previous books, Tyler examined recent economic history. Stubborn Attachments gives his perspective for navigating the future. Tyler is a professor of economics at George Mason University. He is also the ...…

Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how ...…

Mars is a cold, inhospitable planet far from earth. It presents one of the most complex challenges faced by engineers: how can we create a new world? To create a new world, first we have to get there. We can build new rockets with improved propulsion systems. We can build ships that allow us to survive the long, grueling trip from Earth to Mars ...…

Social media has transformed our lives. It has also transformed how wars are fought. P.W. Singer’s new book “Likewar: The Weaponization of Social Media” describes the far-reaching impact of social media on the tactics and strategies used by military, business, and everyday citizens. We have all read about stories such as Russian bots and Cambri ...…

Infrastructure software can be a great business. An infrastructure software company sells core technology to a large enterprise such as a bank or insurance company. This software has near zero marginal cost and generates a large annuity for the infrastructure software company. Once a bank has purchased your infrastructure software, the bank is ...…

When TensorFlow came out of Google, the machine learning community converged around it. TensorFlow is a framework for building machine learning models, but the lifecycle of a machine learning model has a scope that is bigger than just creating a model. Machine learning developers also need to have a testing and deployment process for continuous ...…

When a user interacts with an application to order a ride with a ridesharing app, the data for that user interaction is written to a “transactional” database. A transactional database is a database where specific rows need to be written to and read from quickly and consistently. Speed and consistency are important for applications like a user o ...…

React, Vue, and Angular are the most popular frontend JavaScript frameworks. Each of these frameworks lets frontend developers build components. A component is a high level visual abstraction that is used to compose a user interface. Frontend development has moved towards component-driven-development. At a typical technology company, a designer ...…

Netflix has thousands of service instances communicating with each other. When a Netflix client on a smartphone makes a request for a movie, that request hits Netflix’s backend, where the request is fulfilled by a chain of requests through different services. Services and clients communicate using several different interaction patterns. A servi ...…

Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry. Prometheus gath ...…

When a developer provisions a cloud server, that server is called an “instance”. These instances can be used for running whatever workload a developer has, whether it is a web application, a database, or a set of containers. The cloud is cheap to get started on. New applications with few users can often be hosted on infrastructure that is less ...…

If a business has been operating successfully for a few years, that business has accumulated a high volume of data. That data exists in spreadsheets, CSV files, log files, and balance sheets. Data might be spread across local files on a user’s laptop, databases in the cloud, or storage systems in an on-premise data center. Older businesses have ...…

Robots are making their way into every area of our lives. Security robots roll around industrial parks at night, monitoring the area for intruders. Amazon robots tirelessly move packages around in warehouses, reducing the time and cost of logistics. Self-driving cars have become a ubiquitous presence in cities like San Francisco. For a hacker i ...…

Netflix has petabytes of data and thousands of workloads running across that data every day. These workloads generate movie recommendations for users, create dashboards for data analysts to study, and reshape data in ETL jobs, to make it more accessible across the organization. Over the last ten years, data engineering has become a key componen ...…

Chinese Internet companies operate at a massive scale. WeChat has over a billion users and is widely used as the primary means of payment by urban Chinese consumers. Alibaba ships 12 million packages per day, which is four times the amount of Amazon. JD.com, a Chinese ecommerce company, has perhaps the largest production Kubernetes installation ...…

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provi ...…

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Jus ...…

Functions-as-a-service allow developers to run their code in a “serverless” environment. A developer can provide a function to a cloud provider and the code for that function will be scheduled onto a container and executed whenever an event triggers that function. An “event” can mean many different things. It is a signal that something has chan ...…

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between diff ...…

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of thei ...…

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, a ...…

Augmented reality glasses will let us walk through a world where the digital blends together with the physical. 3-D objects will be rendered and superimposed onto our field of vision, creating an environment for people to build applications we can hardly dream of today. These augmented reality glasses are probably three to five years away from ...…

Cloud providers created the ability for developers to easily deploy their applications to servers on data centers. In the early days of the cloud, most of the code that a developer wrote for their application could run on any cloud provider, whether it was Amazon, Google, or Microsoft. These cloud providers were giving developers the same Linux ...…

Originally posted on 13 September 2017. Machines understand the world through mathematical representations. In order to train a machine learning model, we need to describe everything in terms of numbers. Images, words, and sounds are too abstract for a computer. But a series of numbers is a representation that we can all agree on, whether we ar ...…

Originally posted on 28 July 2017. Self-driving cars are here. Fully autonomous systems like Waymo are being piloted in less complex circumstances. Human-in-the-loop systems like Tesla Autopilot navigate drivers when it is safe to do so, and lets the human take control in ambiguous circumstances. Computers are great at memorization, but not yet ...…

Originally posted on 1 May 2018. Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. ...…

Originally posted on 16 June 2017. John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engi ...…

Originally posted on 14 February 2017. Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardiza ...…

At Facebook, Venkat Venkataramani saw how large volumes of data were changing software infrastructure. Applications such as logging servers and advertising were creating fast moving, semi-structured data. The user base was growing, the traffic was growing, and the volume of data was growing. And the popular methods for managing this data were i ...…

Ten years ago, there was a distinction between “backend” and “frontend” developers. A backend developer would be managing the business logic and database transactions using Ruby on Rails or Java. A frontend developer would be responsible for implementing designs and arranging buttons using raw HTML and JavaScript. Today, developers can build en ...…

Software products are distributed across more and more servers as they grow. With the proliferation of cloud providers like AWS, these large infrastructure deployments have become much easier to create. With the maturity of Kubernetes, these distributed applications are more reliable. Developers and operators can use a service mesh to manage th ...…

Market strategy defines how a company is positioning itself to be successful. This strategy encompasses engineering, sales, marketing, recruiting, and everything else within a company. Herb Cunitz has led teams at Hortonworks, VMware, SpringSource, and several other companies over his 30 year career in software. After working as president of Ho ...…

Software companies today rely on group chat applications. The world of startups and small businesses is dominated by Slack. But for some large enterprises, regulatory constraints prevent them from using Slack. Slack is a web application that is hosted in the cloud, and regulated industries such as banking often need to run their applications on ...…

Wes Bos has created popular courses on React, GraphQL, and JavaScript. With hundreds of thousands of students, Wes has earned a cult following for his fun, practical lessons on web development. The courses produced by Wes teach developers how to build useful applications such as a complete e-commerce store. Wes has built a career around studyin ...…

A bank account is a platform for apps to be built on top of. If that sounds like a weird idea, think about the features of a bank account. Most users only have a single bank account, making it a tool for identity and authentication. The series of transactions in a bank account provides a data set that can be used for analyzing payment history a ...…