Email or username:

Password:

Forgot your password?
Simon's posts Post Back to profile
Top-level
Jeff Triplett

@simon the 90B (55GB) might confuse people.

You do need ~88GB of RAM, not counting your context window, just to run the 90B model size. So 128 GB of RAM, or else you are going to get 1 token per 30 to 45 seconds or more of output while everything swaps around.

That small model is going to run very, very well on any M-series Mac with enough RAM.

13 November at 3:28 | Wall-to-wall | Open on mastodon.social
2 comments
Simon Willison

@webology thanks, updated that to say "Or the larger 90B model (55GB download, likely needs ~88GB of RAM) like this:"

13 November at 3:32 | Open on fedi.simonwillison.net
Jeff Triplett

@simon I hate that the memory size is so complicated. (that's my best rough guess btw)

Check out Matt's work youtube.com/@technovangelist

Check out this video youtube.com/watch?v=QfFRNF5AhM and one of Matt's about context windows. They are the best I have seen for explaining how much RAM one needs just to run one.

What annoys me is models are measured in Billions and context sizes in Kilobytes, and both are deceptive. i.e. 128K context size = 64 GB of RAM. NOT OBVIOUS at all.

@simon I hate that the memory size is so complicated. (that's my best rough guess btw)

Check out Matt's work youtube.com/@technovangelist

Check out this video youtube.com/watch?v=QfFRNF5AhM and one of Matt's about context windows. They are the best I have seen for explaining how much RAM one needs just to run one.

13 November at 3:40 | Open on mastodon.social
Go Up