Best Edward Yang Podcasts (2024)

1
TORCH_TRACE and tlparse 15:28

2M ago15:28

15:28

TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.By PyTorch

1
Higher order operators 17:10

2M ago17:10

17:10

Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also u…

1
Inductor - Post-grad FX passes 24:07

3M ago24:07

24:07

The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes th…

1
CUDA graph trees 20:50

3M ago20:50

20:50

CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA grap…

1
Min-cut partitioner 15:56

3M ago15:56

15:56

The min-cut partitioner makes decisions about what to save for backwards when splitting the forward and backwards graph from the joint graph traced by AOTAutograd. Crucially, it doesn't actually do a "split"; instead, it is deciding how much of the joint graph should be used for backwards. I also talk about the backward retracing problem.…

1
AOTInductor 17:30

4M ago17:30

17:30

AOTInductor is a feature in PyTorch that lets you export an inference model into a self-contained dynamic library, which can subsequently be loaded and used to run optimized inference. It is aimed primarily at CUDA and CPU inference applications, for situations when your model export once to be exported once while your runtime may still get continu…

1
Tensor subclasses and PT2 13:25

4M ago13:25

13:25

Tensor subclasses allow you to add extend PyTorch with new types of tensors without having to write any C++. They have been used to implement DTensor, FP8, Nested Jagged Tensor and Complex Tensor. Recent work by Brian Hirsh means that we can compile tensor subclasses in PT2, eliminating their overhead. The basic mechanism by which this compilation …

1
Compiled autograd 18:07

4M ago18:07

18:07

Compiled autograd is an extension to PT2 that permits compiling the entirety of a backward() call in PyTorch. This allows us to fuse accumulate grad nodes as well as trace through arbitrarily complicated Python backward hooks. Compiled autograd is an important part of our plans for compiled DDP/FSDP as well as for whole-graph compilation.…

1
PT2 extension points 15:54

5M ago15:54

15:54

We discuss some extension points for customizing PT2 behavior across Dynamo, AOTAutograd and Inductor.By PyTorch

1
Inductor - Define-by-run IR 12:06

5M ago12:06

12:06

Define-by-run IR is how Inductor defines the internal compute of a pointwise/reduction operation. It is characterized by a function that calls a number of functions in the 'ops' namespace, where these ops can be overridden by different handlers depending on what kind of semantic analysis you need to do. The ops Inductor supports include regular ari…

1
Unsigned integers 13:07

5M ago13:07

13:07

Traditionally, unsigned integer support in PyTorch was not great; we only support uint8. Recently, we added support for uint16, uint32 and uint64. Bare bones functionality works, but I'm entreating the community to help us build out the rest. In particular, for most operations, we plan to use PT2 to build anything else. But if you have an eager ker…

1
Inductor - IR 18:00

5M ago18:00

18:00

Inductor IR is an intermediate representation that lives between ATen FX graphs and the final Triton code generated by Inductor. It was designed to faithfully represent PyTorch semantics and accordingly models views, mutation and striding. When you write a lowering from ATen operators to Inductor IR, you get a TensorBox for each Tensor argument whi…

1
Dynamo - VariableTracker 15:55

6M ago15:55

15:55

I talk about VariableTracker in Dynamo. VariableTracker is Dynamo's representation of the Python. I talk about some recent changes, namely eager guards and mutable VT. I also tell you how to find the functionality you care about in VariableTracker (https://docs.google.com/document/d/1XDPNK3iNNShg07jRXDOrMk2V_i66u1hEbPltcsxE-3E/edit#heading=h.i6v7gq…

1
#57: Asif Suria On The Virtues Of Event-Driven Investing 1:07:56

1y ago1:07:56

1:07:56

This episode is long overdue. I've been following the work of Asif Suria for over a decade now and it has proven to be one the best sources of investment ideas I have come across. Asif tracks and writes about everything from predictive insider trading patterns to "ubercannibals," or companies that consistently use excess free cash flow to buyback s…

1
#56: Joseph Wang On How The Fed Forever Altered The Investing Landscape 1:03:03

1y ago1:03:03

1:03:03

My guest for this episode is Joseph Wang. Joseph is one of the most knowledgeable people I’ve come across when it comes to the world of central banking and how it intersects with markets. His experience on the open markets desk at the New York Fed gives him unique insight into the inner workings of the financial system and how it has evolved in rec…

1
#55: Ben Hunt On The Big 'Bust Out' 1:06:18

1y ago1:06:18

1:06:18

My guest for this episode is Ben Hunt. This is Ben’s second appearance on the podcast. I had the pleasure of first interviewing him a few years ago and if you are not familiar with his work, I encourage you to go listen to that episode in which he discusses his background and the key concepts he utilizes in analyzing markets. It’s not at all necess…

1
Unbacked SymInts 21:31

1+ y ago21:31

21:31

This podcast goes over the basics of unbacked SymInts. You might want to listen to this one before listening to https://pytorch-dev-podcast.simplecast.com/episodes/zero-one-specialization Some questions we answer (h/t from Gregory Chanan): - Are unbacked symints only for export? Because otherwise I could just break / wait for the actual size. But m…

1
Zero-one specialization 21:07

1+ y ago21:07

21:07

Mikey Dagistes joins me to ask some questions about the recent recent composability sync https://www.youtube.com/watch?v=NJV7YFbtoR4 where we discussed 0/1 specialization and its implications on export in PT2. What's the fuss all about? What do I need to understand about PT2 to understand why 0/1 specialization is a thing?…

1
torchdynamo 25:35

1+ y ago25:35

25:35

What is torchdynamo? From a bird's eye view, what exactly does it do? What are some important things to know about it? How does it differ from other graph capture mechanisms? For more reading, check out https://docs.google.com/document/d/13K03JN4gkbr40UMiW4nbZYtsw8NngQwrTRnL3knetGM/edit#By PyTorch

1
PyTorch 2.0 17:51

1+ y ago17:51

17:51

Soumith's keynote on PT2.0: https://youtu.be/vbtGZL7IrAw?t=1037 PT2 Manifesto: https://docs.google.com/document/d/1tlgPcR2YmC3PcQuYDPUORFmEaBPQEmo8dsh4eUjnlyI/edit# PT2 Architecture: https://docs.google.com/document/d/1wpv8D2iwGkKjWyKof9gFdTf8ISszKbq1tsMVm-3hSuU/edit#By PyTorch

1
History of functorch 19:10

1+ y ago19:10

19:10

Join me with Richard Zou to talk about the history of functorch. What was the thought process behind the creation of functorch? How did it get started? JAX’s API and model is fairly different from PyTorch’s, how did we validate that it would work in PyTorch? Where did functorch go after the early user studies? Where is it going next?…

1
#54: Tian Yang On The Virtues Of Variant Perception Versus Unconscious Conformity 1:04:13

2y ago1:04:13

1:04:13

Variant Perception is an investment research firm founded on the idea that man plus machine beats machine (or man) alone. In an era dominated by passive investing and other purely quantitative strategies with relatively short look back periods on the one hand and rampant speculation in meme stocks and other moon shots on the other, nothing could be…

1
#53: Edward Chancellor On What History Can Teach Us About The True Cost Of Easy Money 1:07:43

2y ago1:07:43

1:07:43

Known as "one of the greatest financial historians alive," Edward Chancellor has been working in the markets for three decades now; his study of finance theory, speculative bubbles and economics goes back even further. 'The Price of Time,' Edward's latest book, is clear evidence of his passion for the subject of financial history and his ability to…

1
Season 3 Episode 2 - Miniseries Cigar Talk - Part 2 27:05

2y ago27:05

27:05

Featured Cigar - E. P. Carrillo Pledge Prequel Cigar Maker: Ernesto Perez Carrillo https://en.wikipedia.org/wiki/Ernesto_Perez-Carrillo . Cigar Size and Charts https://bespokeunit.com/cigars/vitolas/ . Cigar Talk Part 1 https://podcasts.apple.com/us/podcast/season-3-episode-1-cigar-talk-with-nyu-executive-mba/id1484596168?i=1000570539812 . . Rex Ne…

1
Season 3 Episode 1 Cigar Talk with NYU Executive MBA Team about Cigar 101 37:29

2y ago37:29

37:29

What is a Cigar? What impression do you associate cigars with? Is it a wise older man reflecting on his past on a porch? Or father sharing a moment with the sons? Do you find it odd that a cigar is one of the most presented items in all major life events and celebrations but is also one of the best companions with a whisky during a heartbreak? Mark…

1
Learning rate schedulers 19:35

2y ago19:35

19:35

What’s a learning rate? Why might you want to schedule it? How does the LR scheduler API in PyTorch work? What the heck is up with the formula implementation? Why is everything terrible?By PyTorch

1
Weak references 16:46

2y ago16:46

16:46

What are they good for? (Caches. Private fields.) C++ side support, how it’s implemented / release resources. Python side support, how it’s implemented. Weak ref tensor hazard due to resurrection. Downsides of weak references in C++. Scott Wolchok’s release resources optimization. Other episodes to listen to first: https://pytorch-dev-podcast.simpl…

1
Strides 20:31

2y ago20:31

20:31

Mike Ruberry has an RFC about stride-agnostic operator semantics (https://github.com/pytorch/pytorch/issues/78050), so let's talk about strides. What are they? How are they used to implement views and memory format? How do you handle them properly when writing kernels? In what sense are strides overspecified, and therefore, not worth slavishly reim…

1
#52: Fred Hickey On 'Keeping Your Head When All About You Are Losing Theirs' 1:23:48

2y ago1:23:48

1:23:48

For over 35 years now, Fred Hickey has consistently performed in-depth research into a wide variety of individual stocks and what drives them over the full market cycle. What's more, during that entire span he's distilled his findings into concise and actionable research for subscribers to his newsletter, The High Tech Strategist. In the process, h…

1
AOTAutograd 19:12

2y ago19:12

19:12

AOTAutograd is a cool new feature in functorch for capturing both forward and backward traces of PyTorch operators, letting you run them through a compiler and then drop the compiled kernels back into a normal PyTorch eager program. Today, Horace joins me to tell me how it works, what it is good to use for, and what our future plans for it are.…

1
Dispatcher questions with Sherlock 18:36

2y ago18:36

18:36

Sherlock recently joined the PyTorch team, having previously worked on ONNX Runtime at Microsoft, and Sherlock’s going to ask me some questions about the dispatcher, and I’m going to answer them. We talked about the history of the dispatcher, how to override dispatching order, multiple dispatch, how to organize various dispatch keys and torch funct…

1
New CI 16:12

2y ago16:12

16:12

PyTorch recently moved all of its CI from CircleCI to GitHub Actions. There were a lot of improvements in the process, making my old podcast about CI obsolete! Today, Eli Uriegas joins me to talk about why we moved to GitHub Actions, how the new CI system is put together, and what some cool features about our new CI.…

1
Python exceptions 14:47

2y ago14:47

14:47

C++ has exceptions, Python has exceptions. But they’re not the same thing! How do exceptions work in CPython, how do we translate exceptions from C++ to Python (hint: it’s different for direct bindings versus pybind11), and what do warnings (which we also translate from C++ to Python) have in common with this infrastructure?…

1
#51: Diego Parrilla On The Three Levels Of The Investment Game 1:30:22

2y ago1:30:22

1:30:22

This is the third conversation with Diego Parrilla I have published as a podcast but the fourth time I have actually recorded a discussion of ours. The first time Diego was kind of enough to share many of the facets of his investment process, I found only afterwards that, due to a technology issue, the audio recording was sadly unusable. So I was v…

1
Torch vs ATen APIs 15:03

2y ago15:03

15:03

PyTorch’s torch API is the Python API everyone knows and loves, but there’s also another API, the ATen API, which most of PyTorch’s internal subsystems are built on. How to tell them apart? What implications do these have on our graph mode IR design? Also, a plug for PrimTorch, a new set of operators, not designed for eager mode, that is supposed t…

1
#50: Kiril Sokoloff On The Virtues Of Just Watching The River 49:21

2y ago49:21

49:21

For long-time Superinvestors listeners, Kiril Sokoloff needs no introduction. Last summer, he was kind enough to host me at his home in Sun Valley, Idaho where we discussed much of the framework that makes up the foundation of Kiril's research and investment process. In this conversation, he shares how he applies this framework to the inflation deb…

1
#49: Todd Harrison On The 'Trade Of A Lifetime' 1:01:06

2+ y ago1:01:06

1:01:06

Todd Harrison is truly one of the good guys on Wall Street. After a career that took him from the Morgan Stanley options trading desk to running a major hedge fund and then starting a financial media business, Todd founded CB1 Capital several years ago. The firm, an investment adviser and manager, was born out of Todd's belief in cannabis as a rela…

1
All about NVIDIA GPUs 19:29

3y ago19:29

19:29

PyTorch is in the business of shipping numerical software that can run fast on your CUDA-enabled NVIDIA GPU, but it turns out there is a lot of heterogeneity in NVIDIA’s physical GPU offering and when it comes to what is fast and what is slow, what specific GPU you have on hand matters quite a bit. Yet there are literally hundreds of distinct NVIDI…

1
Tensor subclasses and Liskov substitution principle 19:13

3y ago19:13

19:13

A lot of recent work going in PyTorch is all about adding new and interesting Tensor subclasses, and this all leads up to the question of, what exactly is OK to make a tensor subclass? One answer to this question comes from an old principle from Barbara Liskov called the Liskov substitution principle, which informally can be stated as S is a subtyp…

1
Half precision 18:00

3y ago18:00

18:00

In this episode I talk about reduced precision floating point formats float16 (aka half precision) and bfloat16. I'll discuss what floating point numbers are, how these two formats vary, and some of the practical considerations that arise when you are working with numeric code in PyTorch that also needs to work in reduced precision. Did you know th…

1
DataLoader with multiple workers leaks memory 16:38

3y ago16:38

16:38

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these th…

1
Batching 13:37

3y ago13:37

13:37

PyTorch operates on its input data in a batched manner, typically processing multiple batches of an input at once (rather than once at a time, as would be the case in typical programming). In this podcast, we talk a little about the implications of batching operations in this way, and then also about how PyTorch's API is structured for batching (hi…

1
Multiple dispatch in __torch_function__ 14:20

3y ago14:20

14:20

Python is a single dispatch OO language, but there are some operations such as binary magic methods which implement a simple form of multiple dispatch. torch_function__ (through its Numpy predecessor __array_function) generalizes this mechanism so that invocations of torch.add with different subclasses work properly. This podcast describes how this…

1
Multithreading 18:34

3y ago18:34

18:34

Writing multithreading code has always been a pain, and in PyTorch there are buckets and buckets of multithreading related issues you have to be aware about and deal with when writing code that makes use of it. We'll cover how you interface with multithreading in PyTorch, what goes into implementing those interfaces (thread pools!) and also some mi…

1
#48: Jim Stack On Charting A Course Through Stock Market Rapids 1:14:01

3y ago1:14:01

1:14:01

Jim Stack has been assiduously studying markets in an effort to develop and hone a full-cycle investment discipline since he was first introduced to stock market risk back in the 1973-74 bear market. That experience instilled in him the modicum of skepticism needed to successfully navigate major market turning points, both tops and bottoms, as well…

1
Asynchronous versus synchronous execution 15:03

3y ago15:03

15:03

CUDA is asynchronous, CPU is synchronous. Making them play well together can be one of the more thorny and easy to get wrong aspects of the PyTorch API. I talk about why non_blocking is difficult to use correctly, a hypothetical "asynchronous CPU" device which would help smooth over some of the API problems and also why it used to be difficult to i…

1
gradcheck 16:58

3y ago16:58

16:58

We talk about gradcheck, the property based testing mechanism that we use to verify the correctness of analytic gradient formulas in PyTorch. I'll talk a bit about testing in general, property based testing and why gradcheck is a particularly useful property based test. There will be some calculus, although I've tried to keep the math mostly to int…

1
#47: David Hay On Balancing Risk Management With Timely Opportunism 1:03:37

3y ago1:03:37

1:03:37

David Hay has been managing money for over 40 years using a tried and true approach steeped in classic value investing paired with an appreciation for technical analysis that can only be developed through vast experience in the markets. What's more, David also has a unique ability to balance a healthy respect for risk management with an openness to…

1
torch.use_deterministic_algorithms 10:50

3y ago10:50

10:50

torch.use_deterministic_algorithms lets you force PyTorch to use deterministic algorithms. It's very useful for debugging! There are some errors in the recording: the feature is called torch.use_deterministic_algorithms, and there is not actually a capability to warn (this was in an old version of the PR but taken out), we just error if you hit non…

1
Reference counting 15:14

3y ago15:14

15:14

Reference counting is a common memory management technique in C++ but PyTorch does its reference counting in a slightly idiosyncratic way using intrusive_ptr. We'll talk about why intrusive_ptr exists, the reason why refcount bumps are slow in C++ (but not in Python), what's up with const Tensor& everywhere, why the const is a lie and how TensorRef…

Podcasts Worth a Listen

Edward Yang Podcasts

Podcasts Worth a Listen

Quick Reference Guide