The LPDDR5X decision was a puzzling one at this price point - that really hobbles the GPU. The architecture is similar to that of game consoles (e.g., the base PS5 with 16GB GDDR6 with an AMD semi-custom CPU), so the unified memory helps with the latency and zero/minimal-copy part. But the memory clock speeds, if it is not some beefed up version of LPDDR5X, are going to be less-than-ideal. I guess they were going for battery-life here.
Agree - I dont at all get that decision. Its not like you expect inference workloads to be bursty - and even then time to first token does matter. Or maybe they are like - well devs will put up with this since nothing else exists in the same form factor but for anything real they will have to invoke a cloud instance still.
The LPDDR5X decision was a puzzling one at this price point - that really hobbles the GPU. The architecture is similar to that of game consoles (e.g., the base PS5 with 16GB GDDR6 with an AMD semi-custom CPU), so the unified memory helps with the latency and zero/minimal-copy part. But the memory clock speeds, if it is not some beefed up version of LPDDR5X, are going to be less-than-ideal. I guess they were going for battery-life here.
My bad: it's a wired machine - so no battery-life considerations. Which makes this decision all the more puzzling.
Agree - I dont at all get that decision. Its not like you expect inference workloads to be bursty - and even then time to first token does matter. Or maybe they are like - well devs will put up with this since nothing else exists in the same form factor but for anything real they will have to invoke a cloud instance still.