Speaker A
Hi, my name's John. I lead the search and machine learning teams at Google. I think it's amazingly inspiring that people all over the world turn to search engines to ask trivial questions and incredibly important questions.
How search engines work: spiders crawl the web to build a search index, then ranking algorithms like PageRank and machine learning pick the most relevant results.
Key Takeaways
- Search speed comes from pre-indexing the web rather than crawling it at query time.
- Ranking determines which of millions of matching pages users see first, and it is the core competitive challenge.
- Link-based signals like PageRank were a foundational innovation, but ranking now blends many factors and machine learning.
- Users should still verify that sources are trustworthy by checking web addresses.
What the video covers
- Search engines do not search the live web for each query; they rely on a pre-built search index.
- Over a billion websites exist, with hundreds more created every minute, making real-time crawling impractical.
- Spider programs follow hyperlinks across pages and record information into a search index.
- A query's words are looked up in the index to retrieve all pages containing them, which can number in the millions.
- Each search engine uses its own ranking algorithm, checking factors such as title matches and word proximity.
- Google's PageRank ranks pages partly by how many other pages link to them, named after co-founder Larry Page.
- Spammers try to game rankings, so search engines regularly update algorithms to demote fake or untrustworthy sites.
- Modern engines use implicit context such as location; a dog park search returns nearby parks without the user typing a location.
- Machine learning helps engines understand meaning, distinguishing 'fast pitcher' (athlete) from 'large pitcher' (kitchenware).
- Speakers from Google and Bing emphasize the societal responsibility of providing accurate answers to users worldwide.
Chapters
Full Transcript — Download SRT & Markdown
Speaker A
So it's a huge responsibility to give them the best answers that we can. Hi, my name's Akshaya and I work on the Bing search team.
Speaker A
There are many times where we will start looking into artificial intelligence and machine learning, but we have to address how are the users going to use this, because at the end of the day, we want to make an impact to society.
Speaker A
Let's ask a simple question. How long does it take to travel to Mars? Where did these results come from and why was this listed before the other one?
Speaker A
Okay, let's dive in and see how the search engine turned your request into a result.
Speaker A
The first thing you need to know is when you do a search, the search engine isn't actually going out to the World Wide Web to run your search in real time.
Speaker A
And that's because there's over a billion websites on the internet and hundreds more are being created every single minute.
Speaker A
So if the search engine had to look through every single site to find the one you wanted, it would just take forever.
Speaker A
So to make your search faster, search engines are constantly scanning the web in advance to record the information that might help with your search later.
Speaker A
That way, when you search about travel to Mars, the search engine already has what it needs to give you an answer in real time.
Speaker A
Here's how it works. The internet is a web of pages connected to each other by hyperlinks.
Speaker A
Search engines are constantly running a program called a Spider that cross through these web pages to collect information about them.
Speaker A
Each time it finds a hyperlink, it follows it until it has visited every page it can find on the entire internet.
Speaker A
For each page the spider visits, it records any information it might need for a search by adding it to a special database called a search index.
Speaker A
Now, let's go back to that search from earlier and see if we can figure out how the search engine came up with the results.
Speaker A
When you ask how long does it take to travel to Mars, the search engine looks in each of those words in the search index to immediately get a list of all the pages on the internet containing those words.
Speaker A
But just looking for these search terms could return millions of pages, so the search engine needs to be able to determine the best matches to show you first.
Speaker A
This is where it gets tricky because the search engine may need to guess what you're looking for.
Speaker A
Each search engine uses its own algorithm to rank the pages based on what it thinks you want.
Speaker A
The search engine's ranking algorithm might check if your search term shows up in the page title, it might check if all of the words show up next to each other, or any number of other calculations that help it better determine
Speaker A
which pages you'll want to see and which you won't. Google invented the most famous algorithm for choosing the most relevant results for a search by taking into account how many other Web pages linked to a given page.
Speaker A
The idea is that if lots of websites think that a web page is interesting, then it's probably the one you're looking for.
Speaker A
This algorithm is called page rank, not because it ranks web pages, but because it was named after its inventor, Larry Page, who's one of the founders of Google.
Speaker A
Because a website often makes money when you visit it, spammers are constantly trying to find ways to game the search algorithm so that their pages are listed higher in the results.
Speaker A
Search engines regularly update their algorithms to prevent fake or untrustworthy sites from reaching the top.
Speaker A
Ultimately, it's up to you to keep an eye out for these pages that are untrustworthy by looking at the web address and making sure it's a reliable source.
Speaker A
Search programs are always evolving to improve the algorithms wo they return better results, faster results than their competitors.
Speaker A
Today's search engines even use information that you haven't explicitly provided to help you narrow down your search.
Speaker A
So, for example, if you did a search for dog parks, many search engines would give you results for all the dog parks nearby, even though you didn't type in your location.
Speaker A
Modern search engines also understand more than just the words on a page, but what they actually mean in order to find the best one that matches what you're looking for.
Speaker A
For example, if you search for fast pitcher, it will know you're looking for an athlete.
Speaker A
But if you search for large pitcher, it will find you options for your kitchen.
Speaker A
To understand the words better, we use something called machine learning, a type of artificial intelligence.
Speaker A
It enables search algorithms to search out not just individual letters or words in the page, but understand the underlying meaning of the words.
Speaker A
The internet is growing exponentially, but if the teams that design search engines do our jobs right, the information you want should always be just a few keystrokes away.
Topics:search engineweb crawlingsearch indexPageRankranking algorithmmachine learningSEO spamhyperlinksGoogleBing
