Speaker A
All right, Ben, are you ready to go?
Ben Wellington discusses complex feature engineering at Two Sigma, focusing on AI-driven quantitative investing and the evolving role of features.
Key Takeaways
- Feature engineering is central to gaining an edge in quantitative investing as raw data becomes widely accessible.
- AI and large language models are reshaping how features are created, making the process cheaper and more scalable.
- Human intuition remains crucial to hypothesize and create meaningful features rather than relying solely on automation.
- Quantitative investors must adapt their skills to leverage AI tools effectively and understand the economic context of features.
- Diverse data sources and broad applicability of features enhance predictive power and investment opportunities.
Summary
- Ben Wellington, head of complex feature engines at Two Sigma, shares his journey from NLP PhD to quantitative investing.
- The conversation centers on the concept of 'features' in quantitative finance, ranging from simple price changes to complex textual data.
- Feature engineering is critical as raw data becomes commoditized; the edge lies in what is built from the data.
- Ben explains how AI, especially large language models, is transforming feature creation and quantitative modeling.
- The discussion covers the balance between human intuition and automation in feature generation.
- Ben highlights the diversity of data sources and the importance of economic meaning in features.
- They explore challenges such as overfitting, horizon awareness, and the evolving skill set needed for quants.
- The democratization of data and AI tools is seen as both an opportunity and a challenge for maintaining competitive edges.
- Ben emphasizes the value of features that apply across many companies rather than just one.
- The episode concludes with advice on building skills that compound in a career dominated by AI.
Chapters
- 00:00Introduction and guest background
- 03:15Ben's career path and early roles at Two Sigma
- 06:38Defining 'feature' in quantitative investing
- 09:25Portfolio perspective and modeling approach
- 12:55Data sourcing and vendor interactions
- 16:22Diversity of approaches and collaboration
- 19:23Feature relevance and forecasting horizons
- 25:27AI, automation, and the future of feature engineering
- 30:11Challenges with overfitting and model robustness
- 36:50Career advice and evolving skills in AI-driven quant investing
Full Transcript — Download SRT & Markdown
Speaker A
I'm ready.
Speaker A
All right, three, two, one, let's jam.
Speaker A
Hello and welcome, everyone. I'm Corey Hoffstein, and this is Flirting with Models, the podcast that pulls back the curtain to discover the human factor behind the quantitative strategy.
Speaker A
Corey Hoffstein is the co-founder and chief investment officer of Newfound Research. [music] Due to industry regulations, he will not discuss any of Newfound Research's funds on this podcast. All opinions expressed by podcast participants are solely their own opinion and do not reflect the
Speaker A
opinion of Newfound Research. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Newfound Research may maintain positions in securities discussed in this podcast. For more information, [music] visit thinknewfound.com.
Speaker A
My guest this episode is Ben Wellington, head of complex feature engines at Two Sigma. In a modern quant process, one might argue that edge can live in three places: the data you can get your hands on, what you do with that data, and how
Speaker A
you forecast from it. Ben lives squarely in the middle layer, feature generation, which also happens to be the place AI is reshaping the world fastest. So that's where we spend our time. We start with what a feature even is and why. As raw
Speaker A
data gets commoditized, the edge increasingly comes from what you build out of it. Then we follow the thread running through the whole conversation:
Speaker A
large language models. Ben has a line I keep coming back to that anything can be language now, and we trace what that unlocks and whether making feature creation this cheap just democratizes the edge away. We close on what a quant
Speaker A
starting out today should be building toward and which skills compound most in a career where AI is the dominant tool.
Speaker A
Please enjoy my conversation with Ben Wellington. Ben, welcome to the show. This is going to be an exciting one for me to dive into. When I found out I was going to get the opportunity to chat with you,
Speaker A
it was great because often I've had a lot of people come on to talk about equities and equity factors, but I think some of the work you're doing is really novel in the conversation where to dive in here really gets to the cutting edge
Speaker A
of what it means to be a systematic quantitative equity investor today. So, I really appreciate you joining me.
Speaker A
Thank you very much. I'm happy to be here. Thanks for having me. So before we dive in, maybe we can give the listeners a little bit of lay of the land and your background of who you are and would love to get a sort of
Speaker A
sense of where you went to school and how you ended up at Two Sigma and what the beginning of your career looked like and take us, you know, the quick run through to where you are today in the
Speaker A
complex feature engines team.
Speaker A
Yeah, I did my PhD in natural language processing at New York University with really a focus in machine translation. So, you have to kind of go back in time a little bit to think about
Speaker A
the fact that it was still quite odd to be, say, speaking to a computer or using translation or before Siri, before Alexa, all this stuff. And it just sounded like a cool field. So, when I was looking to go to grad school because I
Speaker A
didn't know what to do with my life, I read some research areas and it just seemed, it just seemed fun. So dove into what I guess fast forward to today turned out to be a pretty central part of the growing economy, but that was
Speaker A
mostly luck, right? I did my PhD and my first real job after my PhD was a place in Soho, New York called Two Sigma with about 125 or so people in sort of a loft building. And when I met people on the
Speaker A
interview, I was like, these are interesting, cool, fun people. And so I jumped right in and actually my first role there was on the data engineering team. So that team was responsible for starting to collect and ingest and clean
Speaker A
up the data that we were collecting at the time. I joined in 2007 and I had my background in natural language processing but I was more just really being an engineer more broadly at the time. And as I did it, I started to
Speaker A
look around and I noticed all these different data sets at Two Sigma that they collected that were really, really interesting. And at one point, I came across some data that had been collected. And I did this analysis and I
Speaker A
was curious. It was sort of a political analysis of different news places and whether they were leaning one way or the other. And I posted this in an internal message board. And I got this call from someone at the company who said, "Hey, I
Speaker A
like what you did. Maybe not the best to send everyone everything they like this, but what else can you do?" And it turned out that was somebody over in modeling at Two Sigma and from there I kind of shifted my work towards what began in
Speaker A
the news space, news data, and eventually the textual space more broadly and spent a good 10 plus years just studying how text predicts markets in various ways. So that's kind of how I went from grad school to an engineering role and then
Speaker A
eventually to a more modeling focused role where our goal is to predict using as much data as we can how the things that we create are going to move.
Speaker A
Are they going to go up or down in value over different time horizons and that's a big part of the journey.
Speaker A
So your title today is head of complex feature engines and right in there is that word feature and that word gets thrown around a lot in the world of quantitative investing but at least anecdotally when I talk to many people
Speaker A
it means different things to different people and so I would love to maybe just start with you. What does the phrase feature mean? How does Two Sigma really use that word? Maybe if you can give us some examples that span the spectrum
Speaker A
from the very familiar perhaps to the very alternative.
Speaker A
We've had many debates over the definition of the word feature. It's central to what we do. And I think we've landed in different areas, but let me start with the examples. It's easier to
Speaker A
give an example than a definition. I think an example is something like the change in stock price over the last week of a company, right? It's a fact about something we trade that's economically meaningful. At least at Two Sigma, the
Speaker A
economically part is important, but you could have features in other industries where that's not part of the definition.
Speaker A
And so really, we look at it as saying, okay, what is a fact about the world that is worth self-noting? And so the fact that a stock went up 5%,
Speaker A
right? The stock's price is sort of a raw piece of data. It goes up and down, but you can pull different features.
Speaker A
Well, what did it do since last year? What happened? How much did it move?
Speaker A
What was the volatility? Those are more technical common features that you might hear about. But then features can get more interesting like did an analyst recently upgrade the stock? It's kind of interesting. It's a feature about a stock or a feature about an analyst
Speaker A
depending how you look at it. And then it can get progressively more specific. You know, from my background in textual data, how much news coverage is there about a particular company compared to the history? That's a feature about that company, right? We're
Speaker A
understanding that it's growing in its news coverage or reducing or when did they release this filing or what day of the week did the press release come out?
Speaker A
Did it come out on a Friday after hours or did it come on an earlier time? So, it's really a way to encapsulate these ideas that are usually driven by some hypothesis that's innate like, hey, the fact that it might come out on a Friday
Speaker A
afternoon is itself interesting enough that I'm going to make a feature out of it. The nice thing about making a feature is that you can then study that feature instead of just saying, "I'm going to study all press releases in a
Speaker A
giant pile and throw it at a computer and see what comes out." By layering human intelligence and intuition over it, I can star
Speaker A
In a year or two, we're going to have to revisit that feature example about Friday afternoons and see if it survives a 247 market environment.
Speaker A
That's true. [laughter] As things start to change, that's a good point. So in the world of quant equity, one of the big evolutions and big adoptions has been the pod shop model and two sigma in contrast to that operates on I guess we
Speaker A
would call a shared platform model. It's it's one core portfolio rather than a series of competing pods. How does that structural choice shape what you and your team focus on and optimize for dayto-day? And how do you think that
Speaker A
differs from what a researcher at a pod shop would be focused on and optimizing for?
Speaker A
I'll first say that it's not that we have literally a single portfolio. We have many products and portfolios, but as modelers at two sigma who are trying to find alpha. We are not looking at it through the lens of any portfolio
Speaker A
specifically, right? Our job is to kind of make these predictions and we don't control ultimately how these portfolios trade. So, we have teams who are out there trying to say, "Okay, here's what's going to happen to the IBM price
Speaker A
over the next week. Here's what's going to happen to Nvidia and each coming from a different angle and they're not competing against each other because in the end of the day, all of these predictions are being centralized on
Speaker A
some of the firm's larger portfolios and being aggregated and decisions are going to be made based on the combination of all these individuals. So, that creates a lot of interesting opportunity, right?
Speaker A
In a world where you have pods competing, there is reason why you might say have a technology that you don't want to share with another group or invest truly locally because you feel like you're trying to beat out another
Speaker A
team. At Two Sigma, we're thinking, how can we globally optimize instead of locally optimize? So, we want to elevate some of the best stuff we do up to a shared platform so others can use it.
Speaker A
And so that might be a set of features like we discussed earlier that are so interesting I'm going to put it out there for other teams to study or it might be a really clever machine learning algorithm that somebody
Speaker A
discovered and they're going to build a tool around it and share it for others to use. And this really allows our platform to scale in ways that is really remarkable because there's so many people contributing kind of globally to
Speaker A
make the overall platform better as opposed to just a platform that exists that others kind of attach to and pull from. I think in a normal pod shop there's a centralized platform that's mostly pulled from. It's the team
Speaker A
utilizing the platform and here it's cyclical. The platform is an iterative process and our teams are constantly pushing up into the platform for others to share because we all are aligned in making two sigma make the best predictions possible.
Speaker A
At the risk of perhaps oversimplifying the world, when I look at the quant process, I sort of see three main silos that alpha can really live in. The first is in the data itself, right? This might be getting your hands on alternative
Speaker A
data that nobody else has. The second is feature generation. It might be that people have the same data, but you're able to transform it into observations or unique observations in a way that nobody else has. That can be insightful.
Speaker A
And the third would be sort of the forecasting layer itself that everyone maybe has the same data and the same features, but you can apply some new method, some new machine learning algorithm that extracts more information from those features in a predictive way
Speaker A
that gives you an edge. from where you're sitting today curious how you would weight where alpha lives in those three buckets and has your answer changed over the last 5 or 10 years probably over 10 years it's changed a
Speaker A
little I think the first bucket data access I mean it is an important part of what we do it would be naive not to realize that some of that has been commoditized over the last 15 to 20 years right that when I started doing
Speaker A
for example modeling within the news space or really across most vendors. I mean, you would reach out and they would say things like, "Well, which company do you want data on?" And we'd say things like, "All of them." And they'd say,
Speaker A
"What do you mean all of them?" And you'd say, "We want to track all the companies." How does that doesn't make any sense? And so, clearly, we were the first people interacting with many of these vendors thinking about a different
Speaker A
way to package their data, you know, teaching them. And so in those early days, I would say data access as a relative weight was probably larger than it is today. Though it's still important. I just think that with more
Speaker A
commoditization, it puts pressure on that proprietary data piece. We still have a lot of proprietary data. We have a long history that others might not have, which is very rich. We collect our own trading ideas from crowds and things
Speaker A
like that. So we do have a lot of proprietary data. I'm not underelling it. I just think its proportional weight has probably dropped. your second part which was what you do with it in terms of building features ideas. I mean I
Speaker A
think that is just huge. I think that the ability to think about in a clever and unique way how to take data and come up with really clever ideas of how to use it is just a huge edge. And I could
Speaker A
say that with confidence because I see how we build successful models that are predictive and it's the creative process that we say, "Oh, that's really clever." That, you know, that drives a lot of what we do. And so that suggests to me
Speaker A
that a good part of what's creating this alpha is certainly the human ingenuity layer of how to convert piles of bits into features that are interesting, meaningful, clever, and maybe that you think others haven't thought of.
Speaker A
I'll give you just a flavor of that. For example, I like to talk about a classic one of an analyst. And you might say, okay, an analyst is making some decisions. And there's a lot of papers out there and others about whether an
Speaker A
analyst target price moves markets or they're upgrade or downgrade. But you know, how many different ways can you study an analyst? And at first it's like it's not that much data, right? And then when you start to think about, well,
Speaker A
where does an analyst live? and do they live, you know, what market are they in compared to who they're covering? And how long have they been in this position? And do they go to the same school as the CEO of a company when
Speaker A
they're covering them? I mean, these are small points, but in aggregate, if you can ask 200 of those questions and you've created 200 new features, it's that long list of just kind of in Oh, yeah. Oh, that's cool. I hadn't thought
Speaker A
of that. It's a long list that in aggregate starts to make a difference because any one of those things I'm talking about now it's not it's like you know you're right 50.001% 00 1% of the time, right? I mean, these are tiny tiny
Speaker A
signals in the way we act, but it's the aggregation of all of that interesting ideas that I think really makes a huge difference and that's why we have these teams at two sigma that are centered around features. It's like it's the core
Speaker A
mission. Your third part of your question was these machine learning algorithms and how you know the science there that is also a very big part of it. We have a large team called techniques that really focuses on those
Speaker A
algorithms themselves and how to make the most out of them. And so you kind of have these feature focused teams where they're thinking about all right what are all the cool clever data I can make and you have these techniques teams who
Speaker A
are like okay given cool data what are all the cool algorithms I can put on it to make things work and it's at the intersectionality of those two areas where we see just amazing things happening. It's experts from different
Speaker A
areas with different interests coming together to make predictions and that diversity of approach I think really strengthens what we do.
Speaker A
I want to talk about the feature side a bit because you know you gave this example of the analyst and did they go to the same school as the CEO? Do they live in the market that they cover? But
Speaker A
given that you're trying to come up with all these unique and interesting features, I can imagine a world where I can sit and ideulate a whole bunch of features that I would presume probably are meaningless. Like how tall is the
Speaker A
analyst? What color are the analyst's eyes? You know, like do they and the CEO have the same name? Like it's either N is too small or it's just frankly probably irrelevant. Can you walk me a little bit through what the process
Speaker A
actually looks like in practice in ideulating these features coming from largely commoditized data sets? Like how is your team really taking a piece of widely available data and pulling something out of it that the rest of the market hasn't? I think persistence is a
Speaker A
part of it. You're right that probably eye color is going to have a prior that's pretty low and probably uninteresting, right? I'm not going to spend much time on that. And in fact, I'd probably avoid it as an indicator
Speaker A
because I think it would add noise, right? And so there is a human layer of what's possible there. Our goal is not to just come up with any feature ever we come up of a human. If we're talking
Speaker A
about analysts, there's a lot of features about humans, right? But rather which ones do like, oh yeah, where these ideas come from is varied, right? I mean, obviously there's some academic work which is inspirational at times. I actually saw a paper where they oddly
Speaker A
enough looked at this academic paper actually took analyst and mapped them to post to try to determine whether they had children or not and then they use that as an indicator and I won't tell you which direction it goes cuz I have
Speaker A
children so I'm not going to tell you but we know whether or not having kids affects your ability to predict [laughter] stock prices and so you know that's an example that the academic did that just makes you understand how far
Speaker A
these things can go And we really stop and think and we approach it by diversity of thought. So like people from different backgrounds come to these problems. You know the physicist comes with the mathematician comes with a computer scientist because they come
Speaker A
from different backgrounds. They come up with different ways to ask these questions and solve these questions. And so that's part of it. I think also we see things happen in the world. There's a you know train derailment somewhere
Speaker A
and there's some article written and it says hey this stock is down off this.
Speaker A
And you say oh that's interesting. How do I generalize that? How do I take this one instance that's happened in the world and every time something happens to a company somewhere, we're kind of like, is that generalizable? Can we take
Speaker A
this instance and think about how it how you would write it up and play it back over a decade to all companies or at least all companies in an industry or something like that? So, we're kind of constantly on the lookout for just
Speaker A
clever ways to build out these features. And once you do, what we don't want to do is just throw everything in some giant algorithm, run it, say go and we're done, right? I mean there are we have priors. So if
Speaker A
your prior is that something that a train collision is a bad thing that's your prior and then you go and study it and it's the opposite sign. I mean I that's not great, right? It kind of breaks with your it's you want you feel
Speaker A
like oh well maybe it's because people over Yeah. You come up you can always post hawk rationalize anything but if it doesn't meet your priors that's sort of the scientific method. You you state your hypothesis you test it and if it
Speaker A
doesn't go the way you said you can't just be like well I meant the opposite direction right? So we're approaching things scientifically that way and we have to verify our priors when we can and that's part of the process. So we
Speaker A
have clean features, clean hypotheses, clean usage and ultimately we feel like that ends up with a set of data that we can use to be predictive. One of the things that strikes me here is if you have a team that is just constantly
Speaker A
searching for features, right? there's the risk that they do identify novel interesting features, but those features may have a high degree of collinearity with other features that it may not seem like they do, right? And so you're generating hundreds, if not thousands of
Speaker A
features that are independently interesting on their own, but actually make sort of the portfolio construction process far more complicated because you get this colinearity issue. I'm curious how you deal with that as you get this expanding both potentially graveyard of
Speaker A
features as well as like actual features you're putting into an implementation. I mean, who knows? Maybe the color of an analyst's eyes correlates very highly to I don't know the weather of who knows like I'm just I'm trying to think of two
Speaker A
very random things that actually in practice like they came up with independently but are highly colinear in what they predict.
Speaker A
Yeah. So our job is at its core every day to come up with orthogonal and novel predictions. It's kind of easy to come up with things that have correlated to what we've done in the past and that's the lowest hanging fruit, right? And so
Speaker A
almost by definition, we wake up every day as an alpha modeler searching for orthogonality. So colinearity is the death of a signal. Right? If you found something that someone else is trading through a different data set, I mean, I
Speaker A
give a classic example of trading news data. Now, you could say, okay, I'm going to trade news sentiment and I'm going to, you know, say, okay, this is my new sentiment model, but what if it's just picking up post- earnings stories?
Speaker A
And so, what you call new sentiment is just chasing earnings numbers. And so, you're sitting there saying, I got this novel data source and all it is is a dirty version of earnings numbers. You might get excited and think you found
Speaker A
something clever, but by the time you've run through the tools, you've explored correlation structures, which is just a part of the job. If you can't do that, then you're not going to make it. And so, we have certain rules and thresholds
Speaker A
where we are comfortable only when new alpha signals have a certain amount of orthogonality because we just don't believe that throwing a bunch of things in that are equally correlated to things in the past is useful. So, yeah, it's
Speaker A
kind of core to what we do. It's like you're driven by orthogonality. you're driven to avoid correlation and your goal is to come up with the next signal that's not so correlated with our previous ones. And it's so it's like
Speaker A
it's kind of where you start. It's not even just like an after the fact thing.
Speaker A
How aware do the analysts have to be about the horizon over which they're trying to ultimately forecast returns and sort of back out which features would be relevant to that horizon. I can imagine there are some features that are
Speaker A
highly relevant to the, you know, the high frequency space or the super slow frequency space, the mid-frequency space. those are going to be different features. How much is that in the back of their mind when they're thinking about these novel features?
Speaker A
It's a big part of it, right? If your hypothesis is that you're studying, I don't know, people visiting stores, you probably think that and you have a good indication that people have been visiting the stores a lot this week or
Speaker A
whatever for going to Walmart. You probably don't think that that's a signal that's going to be priced in in the next 5 seconds, right? It's probably it's got to make its way through. it's got to eventually show up on some same
Speaker A
source sales release and it's going to be months away. And so different signals have different hypotheses of how they're going to find a way in. And so yeah, models are definitely thinking about that and aware it's kind of built into
Speaker A
what they're studying often is what horizon I want to talk about that example before I move on to talking about just people in general. But that same store sales data example or feature is really interesting to me because you would
Speaker A
imagine when that data maybe first comes out and there's not a lot of people using that data, the horizon over which it forecasts may be slower and then as more and more people get access to that data and act on it faster, you could
Speaker A
actually imagine it getting the market moving on it much much faster, right? and it's sort of the same feature, but the forecast horizon continues to speed up until maybe the point it becomes so efficient it it's no longer relevant.
Speaker A
How does that play through like the life cycle of features for you? I mean, it is the life cycle of features. Your goal is to create something so creative that it doesn't come under that pressure. That's not always going to happen. There are
Speaker A
going to be other firms that study things and so they're going to put pressure on and so yeah, you do see exactly like you described. you see things speed up and sometimes to the point where yeah the signal is no longer
Speaker A
viable. We're all trying to be as creative as possible so that we don't get that pressure and in the end of the day we have monitoring in place that kind of tracks what's happening with a particular model and at the firm level
Speaker A
the portfolio level is watching that and deciding how hard to trade any signal based on their own analysis. Right? We as modelers are kind of making these predictions. the portfolio itself, those that run on the portfolio, which is a
Speaker A
whole another, you know, quant function, are going to be monitoring for that type of thing and they're going to be setting looking for alpha decay and then appropriately setting weights to match that.
Speaker A
I want to talk about the people side here because what you're talking about sounds like a very creative role and I think when people talk about quants or hiring for quants they typically think about the hard technical skills the math
Speaker A
the statistics the coding a lot of the features you're talking about require a degree of ingenuity and creative thinking and you know lateral thinking to a certain degree. How do you go about finding those people? How do you go
Speaker A
about assembling this team and screening candidates to determine whether they're going to be good at identifying novel and useful features?
Speaker A
I think for a team like mine that looks for features during an interview process, right? You have conversations that are open-ended and you ask, "Hey, here's some data.
Speaker A
Here's some if you had access to a fire hose from social media, tell me all the cool things you could predict." very open-ended questions that are not, you know, solve this math equation, but rather things that don't have a right or
Speaker A
wrong answer. And we walk through and and the question is, how many clever ways and how many things that I've never heard anyone else say are they saying?
Speaker A
That's a good indicator if somebody is coming up with just interesting things that are novel during that process. I think it's a good thing. I mean, it reminds me, look, frankly, back in when I was in graduate school, I I was
Speaker A
looking at an interview at Microsoft, and I remember they I guess they had these product testers was one of the roles that I remember being interviewed for at some point. And the question to me was just like, you have a vending
Speaker A
machine. Your goal is to test it. Like, how would you test the vending machine?
Speaker A
Seems like a silly question, but like there's no right or wrong answer. And I just started talking for, you know, however long about, you know, don't if you stick your hand in, does it cut off your hand? What happens if you unplug
Speaker A
it? like can you put a piece of paper in, can you put a coin on a on a rope?
Speaker A
What if you shake it? And I can go on and on. You know, it's kind of I just way I think I'm always like, "Oh, what's next? What's next?" So, we're looking for that kind of spark of people who
Speaker A
like to think and solve problems in sort of a continuous basis and can always find the next question and a series of questions for a particular problem. And so, that I don't know that interview question, although I don't think it goes
Speaker A
for anything like the same role and I've never asked that specific question, it is inspirational in its openness and it's obviously been around a while since somebody asked it to me 20 years ago.
Speaker A
Let's pivot to AI, which is I think where we're going to spend the majority of the rest of this conversation. In our precall, you said something to me that really stuck with me, which was, and I'm quoting here, NLP was about turning
Speaker A
language into numbers. With LLMs, I can turn numbers into language. Anything can be language. Now, I'd love you to just unpack what you meant by that. What does that shift mean practically? And maybe what does that mean for the types of
Speaker A
features that can be built today that couldn't be done 3 to 5 years ago?
Speaker A
I think for a firm like two Sigma that's been trading off of textual data for over 15 years, getting more textual data is pretty exciting. Obviously, we've been collecting all sorts of text about the world and there is a limited amount
Speaker A
of things that you can collect out there, right? The internet has its bounds. Vendors have their bounds. And so you can only analyze what you can get your hands on. And I think what's really exciting about LMS is that they generate
Speaker A
textual data. And so this idea of what is proprietary data? Maybe it's a little wider tent than we thought. Right?
Speaker A
Before it was just data that we went out and collected or that we had unique access to, but now it's data that we can generate. And what does that look like?
Speaker A
Right? The every output of an LLM is another textual data set. And so for the group and the team and the company that's been studying that stuff for so long and then here comes this technology that just just like as if you know how
Speaker A
to use oil and you've run low on oil and then somebody has this giant oil hose that's coming out. It's exciting, right?
Speaker A
So it's kind of this revolution of opportunity. And what I meant about numbers into text was simply that you can have a conversation about anything with an LLM, right? You can upload a Excel sheet or your personal finances at
Speaker A
home and you can say tell me about it and what you're getting out is is text right and so that text if it has meaning is interesting in itself. I just find that to be a really exciting moment for
Speaker A
those of us who have been studying the use of text this way because it feels like the suddenly this onslaught of new information that is bigger than anything we've ever seen. And so it's a huge opportunity.
Speaker A
you gave this really interesting example during our pre-all about measuring I don't know what I would call micro expressions during CEO earnings calls like you know measuring how often they blink or where their eyes go when they say certain words and years ago as you
Speaker A
pointed out that undertaking a project to research that may have been a massive you would have had to have a dedicated computer vision team and it really would have been a very difficult process to explore something like that today, you
Speaker A
know, with the tools that are available via an LLM to rapidly prototype and explore those ideas, the costs have collapsed quite a bit. How has that changed the types of features that your team is now willing to explore or ideas
Speaker A
they're willing to try versus the prelim world? Yeah. I mean, it's massively massively changed things, right? Everything was kind of return on investment. you have a prior does the fact that the CEO is blinking three times more than they used
Speaker A
to in this call, right? We have a baseline for every CEO, this is hypothetical, by the way. I'm just saying it like if you're studying this and you study CEOs historically and you look at how much they're blinking and
Speaker A
you could create, you know, a zcore of well, how many standard deviations up or down is their blinking going? Let's just say we had that. Is that meaningful? And like part of you is probably saying, "Yeah, that's kind of weird. Why are
Speaker A
they spiking and blinking?" And so part of me says, "Yes, like that's an interesting signal. I want to look at that. I bet you there there might be something there. But then if someone said to you, well it's going to take you
Speaker A
six months of your team's time to study because you have to download thing and do run vision, you know, bake these features. You're going to say, well, I mean, it's a good idea, but it's not that good idea. I have a lot to do here
Speaker A
and I'd rather look at earnings numbers than CEO blinkings and I have higher priors on this data set. But if you can really bring that ROI, the return on your investment, if that investment plummets, it really changes some of the
Speaker A
calculation. And so all these ideas you had swimming around like I wish I could see this. I wish that would be cool. You know, suddenly they're viable. And so it's a really exciting time as well because if you have 2,000 ideas in your
Speaker A
list of things that you're never going to get to and suddenly you can look at them and say, "Hey, I can get to these all this cool stuff I've been wanting to do." So I think it really changes it
Speaker A
frees the mind from having to worry so much about implementation as a constraint. And as a scientist, it's kind of like you're a scientist and you work in a lab and suddenly there's no budget for your equipment, right? It's
Speaker A
like it changes things when you don't have to worry about building the particle collider. You have a thousand in front of you. Similarly, if you don't have to worry about a giant investment to answer every question, it just frees
Speaker A
you to reorder your things and not worry so much about the investment part. Does that mean that everything come falls out for free out of an LLM? Like, no. I'm not suggesting that this is all solved, but it certainly can be 100 times faster
Speaker A
than it used to be for a lot of these projects. And so that opens up a lot of opportunity.
Speaker A
One of the counters here might be that if LLMs help make feature creation much cheaper and the technology in and of itself is no longer a moat, it's you know you have fairly democratized access. It might be that edge gets
Speaker A
eroded much more quickly that even in in those you know harder edge cases where edge was over this wall of difficulty that wall has come down and you might see edge erode much more quickly as all this sort of data gets explored quite
Speaker A
rapidly and everyone converges on exploring features much more quickly. How do you think about that sort of critique or risk? It's absolutely the case that people can build features faster not just at two singla but everywhere if they want to right so
Speaker A
there is the lowering of that moat on the counter side I would say we have such deep experience building these things at scale that we're shovel ready for this change right like it's our DNA and so when you have this 10x increase
Speaker A
and you've been set up to systematically understand how to use these things it's just really exciting when you have a 10x increase and you're saying, "Oh, wow.
Speaker A
How do we sort through this? We have all this data now." So, it is true that people can make a lot of data. Don't get me wrong. I think we can make it at home on our laptops. We can make it right
Speaker A
there. You can generate as much as you want, but the then what is still really key. And so, while the ability to create a data set is not the moat, the how, the what is very much part of that. And it's
Speaker A
experience is a big part of that, right? The fact that we've been doing this for so long just makes us ready for that moment in time. In our precall, you raised a concern about AI tooling specifically and you had a great quote
Speaker A
that I want to use here. You described the risk as AI of AI as potentially quote lowering the entropy of the output. And I'd love for you to expand on what you meant by that. What is the risk and how can you fight back?
Speaker A
Yeah, this is really interesting, right? Everyone says automation, let's automate everything. And it sounds right. Like I would love every we all want to be on the beach with tools doing our jobs for us, right? Why not, right? Hey, but
Speaker A
automation is a nuance thing. So if I'm building a cardboard box, automation, I want the output of that thing to be the same every time. I want my cardboard box to look like it is. I'm on an assembly
Speaker A
line and that's fine. The thing about adding alpha to a portfolio is you want the opposite. As we talked about earlier, we want orthogonality. We want originality. And so if you overautomate, you could end up with a system that it
Speaker A
doesn't matter who's using it, it creates the same thing. And if everyone starts creating the same thing, your portfolio is not going to look nearly as good. What might strengthen a portfolio is diversity. We all know that's a
Speaker A
classic investing component. And at a place like two sigma, it's the hundreds and thousands of models, each with their own data, hypothesis, researcher, this is key, each with their own person behind it that leads to that orthogonality that when these things
Speaker A
come together, the correlation is low enough that they can make really strong predictions. And so the fear is that if you give everyone a tool, a push button tool, and the tool is throw all your data in and make a model, they all hit
Speaker A
the button and they all point at the same place, they're all going to be putting in the same output and wow, I did that really fast. That's wonderful.
Speaker A
But we're all doing the same thing. And that's terrifying to me for AI in an industry like ours where diversity is the point. How do we fight back? Look, we see AI as an amplification tool of our originality. Going back to that
Speaker A
correlation question you asked earlier, I want to build tools that allow our people to multiply what's unique about them. So when I use the tool and you use the tool and we point to the same data set because we approach it differently
Speaker A
because we historically do different things, we have different thoughts, we have different ideas, the tools need to understand that and you can do that in AI via things like context that allows them to navigate the pathways and come
Speaker A
up with different answers even though we're using the same tool. And so if a tool doesn't allow two people to come up with different answers, if those two people have different backgrounds, I have a physicist and a computer
Speaker A
scientist try to solve the problem and they end up with the same answer. I don't like that. That makes me nervous.
Speaker A
So we really need to think as these tools as amplification of what we're good at as humans, which is sort of originality and orthogonality. And that's kind of maybe front to our AI strategy when it comes to empowering our
Speaker A
modeling teams. Years and years ago, I heard of this idea called rational inattention, which is that it can make complete sense to ignore something. And that's been front of mind for me with LLMs where the pace of capability
Speaker A
improvement has been so rapid and exponential. Spending time trying to fight against the LLM may not make sense when it could be a solved problem in 12 to 18 months of what's capable. Right? So, you know, what's painful or just barely possible
Speaker A
today might be easy in the future. So I would imagine a team like yours faces sort of this real choice for any given idea. You can try an idea today and maybe fight against the technology if it's just barely possible and you
Speaker A
know deal with the rough edges or you know you can wait 12 to 18 months when the idea is much more trivial to explore. Let capability catch up but it also makes it sort of frankly easier for everyone else to explore that idea as
Speaker A
well. How do you think about that trade-off? when do you sort of lean into the difficulty and when do you say let's wait and see what's capable in six months?
Speaker A
So I'll say first of all the difficulty is important to lean into just to understand what it is. Like the very act of being unable to solve a problem because the technology is not there yet.
Speaker A
It's like having a flashlight seeing the wall that's blocking you. Like if you can't see that wall and then someone comes and they're like I can blow up a wall and you're like I don't know where the wall is so let's start looking for
Speaker A
walls. It's too late. So I think it's healthy to push the boundaries faster than the tools come so that when those tools appear if you haven't solved that problem which you might have but if you haven't you know exactly where that tool
Speaker A
goes because you've been darn it hitting that wall over and over again for 6 months and it's like if only I could solve this and it drops this new technology you're like aha this is what I need. So there's really really it's
Speaker A
really important as a technology kind of first company to get out ahead of what's happening to understand the limits. I'll also say that you mentioned 12 to 18 months. I mean look 12 18 months beating any competitor to a market like as
Speaker A
competitive as trying to trade on alpha signals. I mean 18 months can be an eternity if you are the only one in the market taking certain approaches and that means that your alpha signals are just much more impactful when you're out
Speaker A
first. So I think waiting 18 months is you might save yourself some engineering headache, but those 18 months can be quite exciting to be out coming up with ideas before the market is caught up.
Speaker A
And so you want to take advantage of that while you can if you think you're out ahead.
Speaker A
I want to talk about the infrastructure side of this for a moment, right? Because there's sort of maybe two major choices you can make. You can use models that are living in someone else's data center which introduces maybe some IP
Speaker A
concerns or just general dependency concerns about a vendor's versioning or pricing or just roadmap decisions they make versus you could bring everything on premise and that's going to come with its own costs both quite literally but also maybe in the capabilities that are
Speaker A
delivered there versus you know having a vendor delivered to you and the pace of change and upgrades there. How is two sigma thinking about this set of trade-offs?
Speaker A
I think in the end it's about diversification. You don't want to lean into any one way of doing things because then you're held hostage by the limitations there. So what's that mean?
Speaker A
It means you have to build in a way that will allow for growth. You don't want to make assumptions about the models that you're utilizing. Right? If you're going to use a model, be it something from a vendor, some external frontier model, or
Speaker A
something you've built internally, you want to make sure that your stack has the ability to swap that around when needed and figure out what that mean is, what that means to you. Because ultimately, you don't want to ever be
Speaker A
stuck on just one thing. And so, you're right, there are all sorts of trade-offs. I would say maybe the one that you didn't mention, but I think is actually the most important is just control of the life cycle of a model.
Speaker A
And what happens is if you use a Frontier model and then let's just say you're using something from Enthropic and then they say okay well we're turning this off in six months and you're like but I just tuned all these
Speaker A
models on it. You've created a situation where you have to now upgrade your thing to the next one or the next one. And that's fine but it is a moving target that you have no control over if it's
Speaker A
externally hosted. So if you want control certainly internally hosted models help a lot there. The other thing to be aware of is probably just look ahead bias. fact that if you don't understand what model's trained on, you risk having it kind of fool you into
Speaker A
thinking something that's not real. These models as they get more complex, it's going to be harder and harder to understand what's deep in their psyche.
Speaker A
The fact that like does it connect the company Enron with negativity inherently in the back of the LLM and such that when you ask it anything about Enron in some point in time basis, you go back in time, you say, "Okay, well, what do you
Speaker A
think? Bad." It's like, "Tell me about the stock. Bad, bad, bad." and you're like, I haven't even asked the question yet. So, you have to be careful and I think control is important. And so, all else being equal, obviously, you would
Speaker A
prefer to have full control. However, it's also hard to not recognize that the frontier labs are pushing boundaries at a rate that is impressive. And so, you need to balance both that control so you can understand what's happening with the
Speaker A
state-of-the-art. And figuring out when to use both is part of the journey. But the trade-offs you mentioned are exactly right. And it's just part of getting it right.
Speaker A
with more data and more capability to research ideas both that are more novel as well as potentially do it cheaper and faster. I sort of feel like I have to ask the cliche question about the risk of P hacking and overfitting. I'm
Speaker A
curious how you're dealing with this institutionally, right? How are you keeping researchers focused not on rediscovering the same signals or overfitting signals or how are you dealing with this right now?
Speaker A
I'll say a couple things. First of all, if you as a researcher overfit terribly and somehow we don't catch it during the process, and we'll get to that in a second, then ultimately your model is not going to do very well. And
Speaker A
ultimately that's going to catch up with you in your career, in your success, in what you're doing. Right? Your job is to create models that can trade into the future. And if you've over fit on the past, so much so that if you can't do
Speaker A
that, you can't overfit the future and succeed. You can and fail, right? But there is a natural point that anybody building a model should be self-checking. That being said, humans can get excited and fool themselves when they see data and they can come up with,
Speaker A
hey, this is great. I found this thing. I'm so excited. And so, how do we check for that? Like, this is a problem we've been facing for a very long time. We've been doing scaled experiments with machine learning and large data sets for
Speaker A
a long time. And so, questions about Packing are kind of implicit in how we do things. And so, we've developed tools to run tests and statistical tests. And we have priors. And so there's just a lot of just best practices that we've
Speaker A
employed to protect us from too much be hacking and too much overfitting. Now it's important to note that everybody overfits by definition because the future is not the same as the past. If you've trained on the past and you're
Speaker A
predicting the future, you overfit. Congratulations. Right? The world changed since yesterday, right? And so it's not about not overfitting. Let's not pretend we don't overfit. We everything is overfit to some degree.
Speaker A
It's about overfitting to a degree that you're comfortable with and that you can measure and make sure that despite the fact that there's going to be some noise that the future is going to change from the past that you still have a signal
Speaker A
that you're proud of and that's robust to the trading environment and you can measure that right we can at two sigma monitor our models mon our portfolios and we can measure if we're overfitting how much how do we adjust for that and
Speaker A
systematically just do that at scale and so I think it's just part of the role it's inherent your question kind of premise on AI. Does that make that more scary? Yes, because if you have a thousand people running a thousand
Speaker A
experiments, it's a lot more scary when you have three people running a thousand experiments. So, we do need to make sure the education's there, that we're doing the checks, that we're doing robustness, that we're not letting our guard down as
Speaker A
scale increases. So, it is putting pressure. You've hit the exact point. It is putting pressure on those worries, but we're experienced. We feel robust to it, and we're ready for the fact that like things are scaling faster than
Speaker A
we're used to, but we know how to study that, and we're okay with that.
Speaker A
We've been spending a lot of time talking about the feature engineering side of things. I do want to as we draw towards the close here ask a question or two about the data side of things. And in our pre-call you made what I thought
Speaker A
was a really interesting comment which is these data libraries that come out aren't necessarily fixed. There's often you can get upgrades to the data. Better coverage, cleaner data, a longer history. And I think a lot of people would intuitively rebuild their model
Speaker A
with new data. And you push back on that idea. you said that's actually not always the optimal choice. Talk me through that view.
Speaker A
It's an interesting nuance question. I mean I sometimes say one of the hardest parts about moving from academia to a place like two sigma is that in academia you are focused on your PhD thesis if you're sorry if you're a PhD student
Speaker A
you're focused on your thesis. You're focused on this result and your goal is to be the best thing you can ever do at this one thing. And so you're putting all your things into this one way of doing things. And if you can just ek out
Speaker A
a new finding in this very specific thing, you know, that's often part of that journey. And that leads you to say, okay, how can I make this better? How can I make this better? How on an infinite loop until you succeed? And
Speaker A
there's nothing wrong with that. It's good. But if your job is to make a diverse portfolio, you might believe that the last 10% of gains in that journey is 90% of the work. That's the most novel stuff. And so there's
Speaker A
diminishing returns as you push in one area to make it improving. And so what we really believe is that like that time might be better spent doing 10 things at 90% capacity than one thing at 95% capacity. And ultimately if you're
Speaker A
looking for a diverse portfolio looking at it from 10 different ways not necessarily the best you could ever do with another 5 years of work cuz any one of our models by the way could be a PhD thesis like could be a three-month
Speaker A
project or it could be a fiveyear project. We all want to make things better. We're all it's built in all of us. It's like ah I can do this better and better and better. And what makes a good researcher is like okay wait no no
Speaker A
stop. That's enough. Either it looks good enough to walk away and ship it. That's hard for people cuz they want to do better. Or this isn't working, walk away, which is also hard for people.
Speaker A
That skill, I think, is what makes a great quant at a place like Two Sigma.
Speaker A
It's like the ability to know when to move on. Which brings us to your question about the fact that things change in the world. You have to ask yourself when data gets better or a when an LLM improves in its ability or
Speaker A
there's a new algorithm. In a perfect world, you could hit a button and everything would get retrained. And if you had like the stack and everything was done in this sort of smaller scale thing, yeah, I mean, obviously, we would
Speaker A
love to automatically upgrade everything at a push of a button and anything we've ever done in any software. Like, sure, that's great. But in the real world, things are a little bit more nuanced, right? Because you've built a model with
Speaker A
certain assumptions. And if you go and you restart that process, see my previous comment about a PhD student, like you might be improving it and you might make it 10% better, but what is the time trade-off of making your thing
Speaker A
10% better versus going and building something completely different? And so we really believe that it's that diversity of approach that we kind of put first. Now, does that mean we don't ever update things? No, we do all the
Speaker A
time. This is obviously we're constantly monitoring. If we see opportunity, we will chase that opportunity down. But we do it with a business lens like what is the ROI of the time versus how much better do I really think this thing can
Speaker A
get with this new data feed. So a vendor turns from V3 to V4 and I could retrain my whole project. Or I could say, look, I've already got something that uses that type of data. It's doing pretty well. Why not keep going on my agenda
Speaker A
doing something else? And yeah, I will maybe go back and visit it if I think it's the best use of my time. But often it's not just because there's so much of the world we haven't yet captured. So if
Speaker A
you're looking for orthogonality, it's rarely just going to be in like, oh, this vendor made something slightly better.
Speaker A
The intersection of data and AI. One of the ideas you brought up on our prep call was this idea of idiosyncrasy at scale. There are these examples of sort of data libraries where maybe you get consumer spending information about
Speaker A
three companies in Japan that historically may have been hard to use, but maybe this idea you brought up was AI allows you to do something hyperco company specific, but simultaneously scale hypercomp specific much more easily. And I wanted to sort of get you
Speaker A
to expand upon that thought and how you see sort of the frontier of feature creation with this idea of idiosyncratic information in mind.
Speaker A
Yeah. I mean I I use that term a lot at scale to inspire people who have generally maybe shied away from the very companyspecific measurement because they're looking for larger sample sizes with more companies and larger opportunity, right? And so you can
Speaker A
believe that if you trade one company, it's less opportunity than if you trade a 100. And so if you have a feature with one company, it's less exciting than if you have one about 100 companies, right?
Speaker A
And that's kind of natural if you think about it. But the thing about AI is that it opens up this unique ability to dig in on a specific company and build categories of features that like might be specific to that company, but might
Speaker A
also be generalizable to other companies. And so you can reason your way down to the very very specific things about a company and then you could go back and generalize that and call that. Okay, well what's my outlook
Speaker A
on sales or I don't know and the path there might be very very idiosyncratic to a company and so if you think about what a discretionary investor does right I mean a lot of it is just deep knowledge about some of the things that
Speaker A
they're looking at and these things are hard to pick out at scale when looking at you know large earnings numbers right they're usually very company specific opportunities that arise that aren't going to show up in that space and so if
Speaker A
you want to expand to be more competitive in that type of analysis, but also have this kind of scale attitude of two sigma. I just deep down believe that AI is a big part of that opportunity because we can use AI to do
Speaker A
something at 3,000 companies at once while the thing it's doing for those 3,000 companies is itself unique. And so that's kind of an interesting thing to both think about scale and specificity at the same time. And so, yeah, big part
Speaker A
of some of the exciting expansions that we're thinking about here and how to use AI to just go deeper and deeper into the more bespoke and hard to model parts of company data.
Speaker A
One of sort of the I'll say jokes the longtime listeners will have heard me sort of talk about before was that when I went to grad school in 2009 the capstone project of my grad school is pricing credit default swaps which
Speaker A
became a totally irrelevant skill on in Wall Street within you know one year of me graduating. It seems like for feature engineers and researchers today, the landscape is moving very quickly. And so as you're working with junior feature
Speaker A
engineers on your team and they're thinking about their careers right now and where the world is moving in 10 years, what do you think they should be thinking about? What sort of skills or instincts do you think are going to
Speaker A
compound the most for them over the next decade? I think with the change of AI, it's that the technical skills of being able to code up your idea or like put it into a particular format to execute it
Speaker A
are obviously going to drop in relative value to the idea to the skill of like telling those things what how to do or what to do. And so whereas you might have had somebody who is very adept at
Speaker A
building the thing that somebody tells them to build, I feel like that as a skill is growing less valuable because if you can ask an AI to build it, you know that person's additive value is less. And so where does that leave us? I
Speaker A
mean I think ultimately it is a question of first of all getting comfortable with AI and its capabilities, right?
Speaker A
Whoever's in school right now is going to be going into a world with more AI.
Speaker A
Okay. And what you find with each generation of people that are coming out of, you know, into Sigma right now as we hire from schools is that they're more and more AI native, they're more and more expecting things to look a certain
Speaker A
way because that's what they have at school. And it's happening very quickly. I think first of all, you need to be comfortable, as I said before, how to use AI to bring your own skills forward.
Speaker A
And maybe not the skill of just quote build a website, right? But like what's unique to you? What does your knowledge and your education bring that you can do that others can't? And how do you use AI to amplify that? Because to succeed, you
Speaker A
need to both stand out. See my orthogonality comment before, but also scale. And I think AI as a tool, many people think, oh, it's going to replace me or it's going to replace this role.
Speaker A
And I'm saying, no, no, no. Let it amplify you. Let like like don't look at it as, oh, it's doing the thing I did.
Speaker A
say, "Wow, it can help me do what I did way better and faster, and I can use my ideas and just scale up what I want to do before, but much faster." I'd say maybe that's the main takeaway that
Speaker A
getting comfortable with how to use this next generation of algorithms to do really unique and clever and creative things is going to be the skill that's going to carry over because ultimately it's going to be a world where there's
Speaker A
going to be a lot of interacting through human language. The amount that your people are coding is going to go down.
Speaker A
the amount of people that are prompting is going to go up and therefore it's the ideas that are going to carry the day.
Speaker A
It's the how do you use this in the most effective and creative way to get what you need done? And so a focus on that I think is kind of key.
Speaker A
And I want to end with the same question I've asked everyone this season which is what is something that you are currently obsessed with and ideally this is outside of work. So this could be an idea, could be literature, art, movie,
Speaker A
just something you can't get out of your mind right now. What is your obsession and why?
Speaker A
I am kind of obsessed with public data and the government's really so I know this is kind it's you know it's related to my work because I'm a data person but you know done personally and so I think
Speaker A
it's fascinating that there's all this information out there you know the parket tickets that we get the who gets you whether or not people get stopped for speeding health inspections at restaurants and there's so much interesting stuff in there that can be
Speaker A
done and I've really enjoy exploring the opportunity there how we can kind of make our cities and society run better by utilizing the data that's already there. I think the nice thing about AI is that it eventually will it
Speaker A
used to be that the people like only data scientists could go in and use this data, right? And you had to have the skill to like utilize it. But if you don't need that skill anymore and you can just ask questions, suddenly that
Speaker A
data is going to be like driving power for you know like urban planners and lawyers and anyone. It just democratizes the access to information for a public institution. And so I I just find that really exciting and inspiring that you
Speaker A
don't need a p there's no more you know tech requirement or there won't be soon I guess between you and the questions you might have on public data. So that's an area that I study and explore and nerd out on related to my work but you
Speaker A
know also on a personal basis. I would suggest that the more and more people who get tuned into that, the more it look makes uh local or federal government look less efficient, the quicker that data will be pulled, right?
Speaker A
[laughter] We'll be able to track how useful that information is over time. Well, look, Ben, this has been a fantastic chat. I really appreciate you taking the time out of your day to share all this with the listeners. Thank you so much.
Speaker A
Yeah, a great time and a great [music] conversation. Thanks so much for your time.
Topics:feature engineeringquantitative investingTwo Sigmamachine learningnatural language processinglarge language modelsAI in financequantitative researchalternative datasystematic investing
![[CC]FULL《炽热的夏天》EP01：寻找最合拍搭子！包上恩周柯宇礼物同频默契拉满｜#包上恩 #周柯宇 #炽… — Transcript](https://i.ytimg.com/vi/oHa4dHs7-2A/maxresdefault.jpg)
