Mac Studio CLUSTER vs M3 Ultra 🤯

Even with four Mac Studios hooked up, I still can't get to the green. Exo, you're tough. Mac Studio cluster, ladies and gents. And today, I'll be doing some experiments. So, you might know that machine learning models need a ton of memory. Usually, you'd reach for a high-end GPU because, well, they're faster at running parallel tasks than CPUs. But dedicated professional GPUs like H100s by Nvidia, for example, that Elon Musk buys, well, those are really expensive. They're power hungry and still have limited RAM. Even consumer-grade GPUs like the RTX 5090 here, they're still expensive, power hungry, and have even less memory. This goes up to 32 GB. And good luck finding them, by the way. On the other hand, Apple silicon machines, while not as fast as Nvidia GPUs, they're available, they're ultra power efficient, and they have lots of memory, which can be shared with the GPU, allowing me to run things like this 70 billion parameter model with ease. Boom. Load it up. 65 GB out of 128 used. Write a JavaScript function. Don't care what it is. Just do it. Calculate the area of a perimeter and a rectangle. There it is. It's going. It's moving. How fast is it going? 10 tokens per second. I told you it wasn't the fastest thing in the world, but it works. That was the M4 Max, 128 GB machine. 70 billion is okay, but today's open-source large language models that you can run locally can go higher than that. They require a ridiculous amount of memory. 405 billion parameters for Llama, 671 billion parameters for Deepseek R1. The more parameters, the more capable they are. If you have a 1 billion parameter model, chances are it's going to give you gibberish. 671 billion parameters is going to give you decent results. So, Apple dropped this Mac Studio on us recently with 512 GB of unified memory. And, uh, at $10,000 is definitely not cheap, even though it is cheaper than the equivalent of Nvidia GPUs would be. Now, in this cluster, all these Mac Studios are maxed out at 124 GB of memory each, and they're all M4 Max's. Just got to get to that green. What if I add the M3 Ultra to the cluster? Oh, still in the yellow. So close. Oh, and if you're curious about the power usage, check this out. So, we've got four Mac Studios here. All four of them are plugged into this using 48 watts of power while just pretty much just sitting there. I have a few programs open, but it's still sipping that power. So, we're going to kick things off with this cluster and see how it does on its own before it meets the final boss, the M3 Ultra Max Studio for a final showdown. Now, EXO is one way to do clustering. The other way that I talked about briefly in my other video is MLX distributed. MLX is Apple's answer to what CUDA can do on Nvidia GPUs. This is specific for Apple Silicon and that's why I'm going to demonstrate to you it works a little bit faster. So here I've loaded up a model called Deepc Coder V2 light instruct and this is the GGUF version offloaded completely to the GPU. Write a JavaScript function and there it is. Pretty fast. 115 tokens per second. Really nice. Again, just to make sure. 113 tokens per second. Same ballpark. That's already really good. Now we're going to load up the MLX version of the exact same model. DC coderv2 light instruct, but this time it's MLX. Load model. Write another JavaScript function. Boom. 168 tokens per second. Once more, 163. You can see where we're going with this. MLX is really good and optimized on Apple Silicon. And MLX distributed is a way that you can run machine learning models across a cluster of Macs. Down in the description, I'm going to link this post right here by AI who is working on MLX. And he wrote out how you can set this up locally for yourself. There are a few extra steps that are needed beyond this. This, uh, post assumes a few things. So, I created another repository where I give a more detailed guide as well as some scripts. Now, these scripts also make some assumptions, so make sure you read the instructions carefully. But here's what you need. You want to make sure that all your machines have SSH enabled and they're allowed for passwordless login between all the machines. Now, I'm using, which is a way to configure your Python environments, so you can isolate the Python environments. I made videos about this before. I'll link a tutorial on how to set that up down below as well. So you want to make sure you have conda environment and it's the exact same environment on all these machines. One easy way to do this is create the environment on one machine, set everything up, make sure everything works including the clustering instructions. Basically, you have a cluster of one working, right? So here I'm going to have a definition in my host file of one host name. And then I'm going to launch the MLX instructions with a command with a prompt. I mean boom, there it is. I'm using the same model by the way. Deep sea coder v2 light instruct 4-bit MLX. This is just pure MLX right here. And here I got 173 tokens per second. So even better than LM Studio. And keep that number in mind because I'm going to be adding more machines and we'll see how that affects this performance. Let me run that again. And while that's happening, we're going to take a look at Mactop to see that we are using 100% of the GPU. So that's really, really good. Memory. This machine has 128 gigs. We got plenty to go. That's why I'll run bigger models. So, you got and your project file set up on one machine. All that is described in that doc. Now, to find your environment, you go to your home folder, find your minion folder, environments, and there's mine right there. MLX disc. All I got to do is just copy this folder over to the environments folder on all the other machines. And boom, I suddenly have my environments configured exactly the same way on all the machines. I would do the exact same thing for my project folder. What's this project folder I'm talking about? Project folder is something you can clone from my repository or you can create it and it's pretty easy to create. All you need in that project folder is two files. One file is hosts.json. This describes which specific hosts are going to be in your cluster. Hosts, what the heck is that? Well, hosts are the machine names. I name mine pretty simply. AMS1, a2, a3, and a4. And these are discoverable as those host names on the network. And this is a local network. Well, if these two were plugged in. I was doing an experiment. So, they're all plugged in here to this network switch down here. It's a 10 gigabit network switch. And I've discovered that it actually is not making much of a difference whether you're using a network switch or you're using Thunderbolt. I'll get into that momentarily. But this is a local network. So, all these machines can talk to each other directly at 10 gigs. They all have 10 gig Ethernet cards now. And you can check the names of these hosts and rename them to how you like by going to your system settings. Then general sharing remote login. Make sure that's on. That's SSH. And then down here, your local host name. You can edit it. So here's mine. A-MS-1.local. From this machine, I can now say SSH ams-2. And now I'm on this machine from here without a password. Exit. And of course, you want to be able to get into every one of these machines that same way. Now, if we check the speed between these machines, I'm going to set one up as a server. IPF would let us check the speed. Iperf-C is going to be the client. And I'm going to point to this other machine. AMS-1. And check it out. We're getting speeds of 9.4 GBs per second. So, that's going through our Ethernet ports locally. If I wasn't using the Ethernet ports, I'm going to unplug those. Now, we're using Thunderbolt Bridge. Now, to use Thunderbolt Bridge, I connected this machine. This is the first one. It's like a hub and spoke kind of situation. So, I connected it to this one, to this one, and then to that one. But that's not all. Yo

memory. Usually, you'd reach for a high-end GPU because, well, they're faster at running parallel tasks and CPUs. But dedicated professional GPUs like H100s by Nvidia, for example, that Elon Musk buys, well, those are really expensive. They're power hungry and

still have limited RAM. Even consumer grade GPUs like the RTX5090 here, they're still expensive, power hungry, and have even less memory. This goes up to 32 GB. And good luck finding them, by the way. On the other hand, Apple

silicon machines, while not as fast as Nvidia GPUs, they're available, they're ultra power efficient, and they have lots of memory, which can be shared with the GPU, allowing me to run things like this 70 billion parameter model with

ease. Boom. Load it up. 65 GB out of 128 used. Write a JavaScript function. Don't care what it is. Just do it. Calculate the area of a perimeter and a rectangle.

There it is. It's going. It's moving. How fast it's going? 10 tokens per second. I told you it wasn't the fastest thing in the world, but it works. That was the M4 Max, 128 GB machine. 70 billion is okay, but today's open-source

large language models that you can run locally can go higher than that. They require a ridiculous amount of memory.

405 billion parameters for Llama, 671 billion parameters for Deepseek R1. The more parameters, the more capable they are. If you have a 1 billion parameter model, chances are it's going to give you gibberish. 671 billion parameters is going to give you decent results. So,

Apple dropped this Mac Studio on us recently with 512 GB of unified memory. And uh at $10,000 is definitely not cheap, even though it is cheaper than the equivalent of Nvidia GPUs would be.

Now, in this cluster, all these Mac Studios are maxed out at 124 GB of memory each, and they're all M4 Max's.

Just got to get to that green. What if I add the M3 Ultra to the cluster? Oh, still in the yellow. So close. Oh, and if you're curious about the power usage, check this out. So, we've got four Mac

Studios here. All four of them are plugged into this using 48 watts of power while just pretty much just sitting there. I have a few programs open, but it's still sipping that power.

So, we're going to kick things off with this cluster and see how it does on its own before it meets the final boss, the M3 Ultra Max Studio for a final showdown. Now, EXO is one way to do

clustering. The other way that I talked about briefly in my other video is MLX distributed. MLX is Apple's answer to what CUDA can do on Nvidia GPUs. This is specific for Apple Silicon and that's why I'm going to demonstrate to you it

works a little bit faster. So here I've loaded up a model called Deepc Coder V2 light instruct and this is the GGUF version offloaded completely to the GPU.

Write a JavaScript function and there it is. Pretty fast. 115 tokens per second. Really nice. again just to make sure.

113 tokens per second. Same ballpark. That's already really good. Now we're going to load up the MLX version of the exact same model. DC coderv2 light instruct, but this time it's MLX. Load model. Write another JavaScript function.

Boom. 168 tokens per second. Once more, 163. You can see where we're going with this. MLX is really good and optimized on Apple Silicon. And MLX distributed is a way that you can run machine learning models across a cluster

of Macs. Down in the description, I'm going to link this post right here by AI who is working on MLX. And he wrote out how you can set this up locally for yourself. There are a few extra steps

that are needed beyond this. This uh post assumes a few things. So, I created another repository where I give a more detailed guide as well as some scripts.

Now, these scripts also make some assumptions, so make sure you read the instructions carefully. But here's what you need. You want to make sure that all your machines have SSH enabled and they're allowed for passwordless login between all the machines. Now, I'm

using, which is a way to configure your Python environments, so you can isolate the Python environments. I made videos about this before. I'll link a tutorial on how to set that up down below as well. So you want to make sure you have

cond environment and it's the exact same environment on all these machines. One easy way to do this is create the environment on one machine, set everything up, make sure everything works including the clustering instructions. Basically you have a

cluster of one working, right? So here I'm going to have a definition in my host file of one host name. And then I'm going to launch the MLX instructions with a command with a prompt. I mean boom, there it is. I'm using the same

model by the way. Deep sea coder v2 light instruct 4-bit MLX. This is just pure MLX right here. And here I got 173 tokens per second. So even better than LM Studio. And keep that number in mind because I'm going to be adding more

machines and we'll see how that affects this performance. Let me run that again. And while that's happening, we're going to take a look at Mactop to see that we are using 100% of the GPU. So that's really, really good. Memory. This

machine has 128 gigs. We got plenty to go. That's why I'll run bigger models.

So, you got and your project file set up on one machine. All that is described in that doc. Now, to find your environment, you go to your home folder, find your minion folder, environments, and there's mine right there. MLX disc. All I got to

do is just copy this folder over to the environments folder on all the other machines. And boom, I suddenly have my environments configured exactly the same way on all the machines. I would do the exact same thing for my project folder.

What's this project folder I'm talking about? Project folder is something you can clone from my repository or you can create it and it's pretty easy to create. All you need in that project folder is two files. One file is

hosts.json. This describes which specific hosts are going to be in your cluster. Hosts, what the heck is that?

Well, hosts are the machine names. I name mine pretty simply. AMS1, a2, a3, and a4. And these are discoverable as those host names on the network. And this is a local network. Well, if these two were plugged in. I was doing an

experiment. So, they're all plugged in here to this network switch down here. It's a 10 gigabit network switch. And I've discovered that it actually is not making much of a difference whether you're using a network switch or you're

using Thunderbolt. I'll get into that momentarily. But this is a local network. So, all these machines can talk to each other directly at 10 gigs. They all have 10 gig Ethernet cards now. And you can check the names of these hosts

and rename them to how you like by going to your system settings. Then general sharing remote login. Make sure that's on. That's SSH. And then down here, your local host name. You can edit it. So here's mine.

A-MS-1.local. From this machine, I can now say SSH ams-2. And now I'm on this machine from here without a password. exit. And of course, you want to be able to get into every one of these machines that same

way. Now, if we check the speed between these machines, I'm going to set one up as a server. IPF would let us check the speed. Iperf- C is going to be the client. And I'm going to point to this

other machine. AMS-1. And check it out. We're getting speeds of 9.4 GBs per second. So, that's going through our Ethernet ports locally. If I wasn't using the Ethernet ports, I'm going to unplug those. Now, we're using Thunderbolt Bridge. Now, to

use Thunderbolt Bridge, I connected this machine. This is the first one. It's like a hub and spoke kind of situation.

So, I connected it to this one, to this one, and then to that one. But that's not all. You then have to go to system settings, network, and then Thunderbolt bridge, and then details. Under TCP IP, you want to make sure you configure it

manually, and that way you can provide your own IP address. I did 192168.210 2.20 230 24.40 up here. And then change your subnet mask accordingly. Now we're using Thunderbolt Bridge for networking as well. But we're not using host names now. We're using IP

addresses. You can set it up to use host names if you want to, but I just have IP addresses. For our purposes, it actually does not matter. So these are speeds that we're getting with Thunderbolt 5.

And you can see that it's considerably faster. 65 Gbits per second here. Now, you'd think, "Wow, that should be a lot faster." It's not. It would be faster if you're transferring files back and forth. But with MLX distributed, you

already have all the full files of the models on each of the machines. So, it's not taking advantage of this. Now, I want to add one more host. So, I can have two machines in my cluster. I go to

my host file and I just make a copy of that line and change that to ams. Your host names will obviously be different. Why would you need a in yours? That's my initials. Let's go in here and run that command again. By the

way, I did write scripts that encapsulate these commands, but I just happen to be running the command here.

Why not? And there it is. It's coming out pretty fast. I asked it to summarize the theory of relativity for me, and we got 107 tokens per second this time. So, when I'm using two machines, it slowed down a bit. Let's take a look at that

again. All right, now it's generating. And here we go. So, you can see it's not reaching quite 100% on both machines.

It's only going up to about 50 to 60%. That's all it needs for this one, apparently. There you go. It got the job done. I kind of wish it used a little bit more. I'm not sure how it works

underneath the hood, but I know that if it's not up to 100%, it's not fully utilizing what's available, but of course, we know there's going to be some overhead with MLX distributed and how network communication works with MPI.

NPI is what's working underneath to do the communication. Cool. Should we add one more machine? This is a very small model, so it's better to experiment with small models instead of big models until you get everything tuned and then you

start with a big model. Now, what the heck? Let's just do all four of them. And there we go. All four of them are working. We got GPU usage. We got memory usage on all four. And we're down

to 79 tokens per second. Definitely taking a hit. And you can kind of estimate how the scaling is going here.

It's It's taking a lot of a hit on those tokens per second the more machines you add. I bet you're wondering, what about a larger model? What about DeepSeek R1 with all 671 billion parameters? What does that look like? First, let's take a

look at how that would look running on one host. Oh, in case you're wondering, if you search LM Studio for Deep Seek 1 and then select instead of GGUF MLX up here, you'll find it. Here it is. It

says it's likely too large for this machine. Yeah, cuz it's 420 GB. Let's try running it. Instead of this Deep Sea Coder V2 light, I'm going to do Deep Seekar1 4bit. And boom. Keep an eye on that memory right

there. We're still in the green, but it's going up. We're in the orange and we're in the red. We're up to 120 GB. And suddenly, we hit a wall. And boom. Process killed.

We give up. The memory is back. No can do. model is too big. Can we do two hosts? These two right here. If you run this command now and the model is not on one of the machines, it's going to have

to download to that machine first. So, you're going to have to wait until that model is fully on all the machines that's in your cluster. So, the way to do this faster is to go into your cache directory. By the way, it's a hidden

directory. To reveal it, you go shift command period and you can toggle uh hidden files and folders in Mac OS. So you want to go to cache in your home directory hugging face hub and that's where all these models are going to

live. There's my deepsear 4-bit model and it's 420 GB in size. The fastest way to transfer this on machines that are equipped with Thunderbolt 5 is to use a Thunderbolt 5 drive. I've shown videos on how fast these things are. I'll link

to some of those videos down below. But it was basically about a minute to transfer all 420 GB from the Mac Studio to the hard drive and then another minute from the hard drive. Why am I saying hard drive? It's an SSD. They're

fast. Okay, I got them on all the machines now. Let's run this thing. By the way, this is the first time I'm running this. I'm curious about this. This may not work at all. Actually, I I don't even know cuz I've never tried this.

This is the first time. Serious. Nothing seems to be happening. It showed more signs of life when I ran it on one machine than trying this on the cluster.

Now, there's that memory creeping up. 70 GB. And that's on one machine. It means it's double for both of the machines.

102 used on this one, 107 on this one, and we're getting into the orange, folks. And now in the red. Basic calculations can tell you that this is not going to fit into two machines that are 128 GB, right? But I don't know.

Let's Let's try it. Pretend I can't do math, which most of the time I can't. to use to calculators these days. Okay, it broke to be expected. Let's add the other machines in here. At least I know that it's working. 76 GB 81. It's going

up. The memory on all these is going up. Here we go. 102 GB on each machine now at this point. Oh, look at that little ladder. Still in the green.

That's very nice. Are we going to start using the GPU? It's still filling up the memory. Nope. Not printing anything yet.

There it goes. There's the GPU. Or is that just a little blip? I'm like this, you know, like, please work. Come on.

I'm doing a demo for these nice people over here. You have to try and work here for me. Okay. Okay. GPU is going 40% on each machine. About 40%. And it's printing. It's printing. It's not too slow, but it's significantly slower than

before. Of course, this is the full 671 parameter model running on the cluster. Very nice. 15 tokens per second. Now, what if we try it on the M3 Ultra? Can we beat that 15 tokens per second on the

cluster here? 128 * 4 is 512, which is exactly how much RAM is in this machine.

Look at that memory just completely deflated after it was done processing. So, if we want to run this again, got to fill up that memory one more time with each query. And that's the oneshot MLX thing. If you want to keep things in

memory, you got to use other commands or other tools like for example LM Studio.

Let's start with that comparison. So I'm going to kick off just like before Deep Sea Coder Light, the small one. This is the MLX version. And I'm going to compare it to here. Of course, got to download it first. Write a C method to

balance a budget. Whatever the heck that means. I don't know. Let's Let's go for it. Boom. Wow. That is extremely fast.

Look at that. 166 tokens per second and it's balancing our budget, I guess. I don't know. I'm not going to check it.

So, now we're kind of comparing the M4 Max, which you've seen on the channel before. I have an M4 Max MacBook Pro that I use as my daily driver. That's what's in here, too. And that's what's capable of those kinds of speeds. Now,

what about M3 Ultra? It should be faster, even though it's a M3 chip. Single core operations are a little bit slower there than the M4s, but the multi-core operations should be quite a bit faster. Write a C# method to balance

a budget. And boom. Still pretty darn fast. I I can't even tell the difference. Oh, 146. Come on. That can't be right. That's slower again. 140. I'm baffled. Why is it faster on the M4 Max than the M3 Ultra?

What did I do? That's not the same. I really expected that one to be faster.

Something's off. Let's check out llama 3.370 billion. And we're going to offload all the layers to the GPU. Write a JavaScript function to find a prime number. Any prime number will do. That's fine. And here we go. This is

the 70 billion parameter model, so it's going to be quite a bit slower, but it's still kind of acceptable, I suppose. And I'm going to do the exact same thing over here. And boom, there it goes. It uh looks a little bit slower. This is

the one that we ran earlier, if you remember. And I think we got like 10 tokens per second here. Still doing it.

Great. I'm going to stop it right there. We got 9.93. So about 10 tokens per second here. And 13.27 27 over here on the M3 Ultra. And that aligns pretty much with my expectations because of what we've seen with Geekbench.

Multi-core results put the M4 Max at about 25 26,000 and it put the M3 Ultra at about 29,000. Not that much performance gain for multi-core there, even though the price is way up higher.

And that's because of the memory. But what if we run a very large model on this, which is what it's really going to be meant for, right?

Deepseek R1 4bit. It's got to suck up all that data, that whole model into memory. Oh. Oh, there it goes. There it goes. The pressure is building up. Oh, it's writing. It's going pretty fast.

Look at that. Memory used 407 GB. We're not even in the orange or red yet. Has a lot to say about the theory of relativity. That's good. Love seeing that green chart right there. Smooth as butter. And we're at 19 tokens per

second. Oh yeah, that is actually a very very nice rate. Full Deepseek R1. Pretty good. So here they are running at the exact same time on the cluster and on the big boy here. Big boy taking up 251

watts of power. Cluster taking up 150 watts of power. Feels like I'm forgetting something. Oh yeah, running a model on these M4 Maxes together with the M3 Ultra. So, I started digging around for an 8bit quantization of Deepseek R1, and there is one, but it's

been a few weeks since I recorded the rest of the video. A couple of things have happened. One is Deepseek V3 is now out. It has an 8bit quantized version, and this model is about 750 GB in size.

So, if I run this on all five machines, I have a bit of a problem. 750 / 5 is 150 GB. Yeah, I needed the calculator for that. If you go to MLXLM, this tool that I'm using on GitHub, and you take a

look at models, you'll see DeepSseek V3. And yeah, this has been updated since I started this video. It can now distribute the load dynamically among the machines, which means it only needs part of the model on each machine. But

there's still the issue of equal distribution versus whatever the machine is capable of. Even if we have the 512 GB monster over here, it's still going to use only 150 out of that, and it's going to try to use 150 out of each one

of these, which is not going to work. So I reached out to Anie who was working on this at Apple and he recommended I take a look at this function right here. This is the definition of the pipeline

function. If we update this to make it static or detect to the machines, we can have the 128 GB machines get let's say 120 GB and give more of the model to the big machine to process. So after working

on this for days, it seems like uh I finally got something that worked. But I didn't test the big model yet. I started with a smaller model because if it can do the 4bit model, then it should be

able to do the 8-bit model, but then I don't need to wait as long in between my runs. And I specify how many layers I want to be processed by the big boy over there versus the other machines just as

a test. Here we go. We can take a look at what's happening with the 512 GB machine. And you can see it's using right now 305 GB of memory. Nice little bump right there in the memory pressure chart. So, 305 GB, which is great.

That's definitely more than any of these have. And this one got up to about 73 GB, and that's perfect. You can also tell from that little memory pressure chart on this machine. It's right in the middle there. Nice little bump. And it

looks just like this one. Even though they have very different memory profiles, the chart looks exactly the same. And that's kind of how I want it to look. Now, I did get this working at one point where it explained the theory

of relativity pretty well to me, but right now it's giving me not the best results. This definitely needs to be tweaked. This is just an experiment to get the split correct and to see the tokens per second.

16.482 is a pretty nice tokens per second for DeepS v3. Now, it's time to run the 8-bit version. Okay, it's been a few more days and I got something else to report. takes a while to download these gigantic models. And if something

is off and you need to do it again, then it takes a while more. But I'm still wearing the same clothes because continuity, huh? Never did I think I was going to be working on a video for so

many months. By the way, this video doesn't have a sponsor, but I do want to thank the members of the channel for your continued support. Thank you so much. Members get extra videos. There's a join button right down below.

Subscribing and liking the video is also very much appreciated, and that's free. There's a couple of other things I found out. Of course, for the large model that's 750 GB in size, I had to use all five clustered together, which I was

really hopeful to get working. The big challenge is the uneven distribution, but uh I didn't solve it the way I thought I was going to solve it. MLX distributed uses MPI. MPI is message passing interface. I didn't know this. I

just looked it up. Basically, it's a lower level interface for the computers to talk to each other and to be able to control each other through SSH. Pretty cool stuff. But MPI does not take into account the order that we've specified

the machines in the host file. So even though I specified a MS5, the 512 GB machine as the first thing, which means index zero, it does not automatically mean that it's going to get rank zero when the model pipeline code is hit.

Here I'm assigning rank zero the majority of the layers and I'm giving about six layers to each of these guys.

But when this code runs it's not guaranteed that zero will be the big boy. So I had to dig around some more and alter the command that we're running. I spent quite a bit of time doing this stuff. And finally I have

this thing called a rank file. Binds the host name a5 to rank zero. So now our command is slightly different. This is not documented anywhere. Maybe I'll add it to the documentation. I probably should do that. But here is the final

command. MLX launch. You pass it the host file. You specify MPI as the back end, but then you pass in MPI ARG map by rank file and then you give it the file name. And everything else is pretty much

the same. Here's the 8bit model. Let's observe the split. I'm going to run this and look at the memory usage here. All these four machines are using just enough memory. We're at 87 GB here, 92 over here, 51 over here, 92 over here.

Perfect. And then the big boy. I don't know why I keep calling him the big boy.

That's weird. We're waiting on that one. It's getting layers 28 through 60. And it's getting pipeline rank zero. There's the memory growing. 380 GB, 394, 400 GB.

Still plenty of room on this machine. You can see the memory usage is up to almost 80% now. And there is that memory pressure going up. But then, uhoh, we've got some problems. And because I don't know what the heck I'm doing, I cannot

figure out how to get past this memory issue. So now the memory split is even figured out how to pass the right ranks to hardcode it. But at this point, I've kind of hit a wall. I don't know

anything about this stuff. I had to dig into the code for days trying to figure out how to even get anywhere. But I think this video was taking a little bit too long to put out and I wanted to let

you know how it's progressing. However, I did ask my Twitter buddy Ivan to run this model because he's got two of the 512 GB M3 Ultras. And even though this thing took probably half a day to download, he came through and he did it.

1 GB worth of machines, same as here, but evenly split. 12 tokens per second for the 8bit version. And that's not bad. So, if you're going to be doing a cluster and you're using MLX distributed right now, it's better to get the same

exact machines. And I'm hopeful that somebody much smarter than me will figure out how to distribute this stuff among machines that are not the same.

So, what setup should you use? Cuz there's a lot of different options here. Should you go for something like the Mac Studio with 512 GB? Maybe. It's actually nice to have that much memory available without having to do all kinds of

clustering and heating up your office. And if you're going to be running a bunch of these just to match the amount of memory that that 512 machine gives you, I don't know why I'm making the Robert Dairo face, but you're going to

need a lot of these and it's going to be really expensive. Yeah, sure you can go with like 3060s or 3070s, a bunch of those. Those are going to be cheaper and they're going to run pretty nicely, but

you're dealing with a huge setup costs, which you can do cuz it's only once.

Operational costs though, those are ongoing, so keep an eye out for those. Now, if you do go with a clustering solution and you're using Apple Silicon, then there's a couple of really great options that I demonstrated here. Exo is

one of them. It's great if you want something simple and automatic. And it worked really well for me the last video. This video, I couldn't get it quite working, but I think they're working on something, an update. I've

been talking to Alex, the guy that created it, and they definitely have stuff in the works. Stay tuned on the channel. I'll be updating you if something new comes out. MLX distributed. Another great option just like I showed in this video. Ethernet

versus Thunderbolt. Thunderbolt is faster when you're doing data transfer back and forth, but Ethernet is going to be easier to set up and more flexible and more stable. Now, of course, clustering only makes sense when you're dealing with models that are too large

to fit on a single machine. If you're dealing with smaller models like 30 billion, 70 billion, those can all fit pretty comfortably at a nice quantization. Four bits to 8 bits on a 128 GB single machine. And anything

smaller than that, of course, as well. But for larger models, you're going to need to cluster or get the big boy here.

Exo. Adding the sixth machine to the mix. Yes, I'm in the green. Five Mac Studios and one MacBook Pro did it. I guess we can call it a day. Thanks for watching. If you want to catch my Mac

Mini video, you can watch it right over here. I'll see you next time. [Music]