On a…. What? A potato? A consumer gpu that doesn’t fit all of the 7B model so it is mem-moribund? Things with “patent pending” (nothing wrong with patents!) and permitting grad students to use it “for their degrees”. Just enough little vibe nudges that I feel confused and unmotivated to actually read the code/paper.
It runs 93x faster on than Zephyr 7B on a 4060 Ti (16GB of memory) that runs them both fully within VRAM. No memory bandwidth or capacity limitations for either model. It genuinely is a pretty fair comparison. I do get into it this in the paper. Unfortunately, I’m unable to edit my original post due to negative karma.
I do understand the vibes you’re talking about on the patent vibe side of things. It’s pretty damned presumptuous—why would grad students want to use this for their degrees? Unfortunately, putting it out there without that kind of disclaimer or specific clarity is also not really something I want to do, either. Researchers and academics often assume there’s an academic exception for IP. In this case, for now, there is not. However, I wanted to make it clear clear that this did not preclude individual, unfunded research like thesis papers for the like in an academic setting.
I apologize if I didn’t deliver it well. This is my first time trying to present anything of this nature, and I’ve tried to be be careful with my messaging, but this is a section where I have to admit I had a lot of trouble.
If this is a thing (and while I understand general skepticism towards extraordinary claims of this nature, I know that it is, because I’ve been kicking the tires on it for weeks), then it is something people are going to want to study. In that case, that early clarity matters. Since I am fairly certain that’s how this is going to shake out after further evaluation, I went ahead and specified upfront.
I appreciate you taking the time to write out what put you off, though, it’s helpful feedback.
Not a downvoter, but I am put off by things like:
| Runs 93x faster than Zephyr 7B
On a…. What? A potato? A consumer gpu that doesn’t fit all of the 7B model so it is mem-moribund? Things with “patent pending” (nothing wrong with patents!) and permitting grad students to use it “for their degrees”. Just enough little vibe nudges that I feel confused and unmotivated to actually read the code/paper.
Fair points!
It runs 93x faster on than Zephyr 7B on a 4060 Ti (16GB of memory) that runs them both fully within VRAM. No memory bandwidth or capacity limitations for either model. It genuinely is a pretty fair comparison. I do get into it this in the paper. Unfortunately, I’m unable to edit my original post due to negative karma.
I do understand the vibes you’re talking about on the patent vibe side of things. It’s pretty damned presumptuous—why would grad students want to use this for their degrees? Unfortunately, putting it out there without that kind of disclaimer or specific clarity is also not really something I want to do, either. Researchers and academics often assume there’s an academic exception for IP. In this case, for now, there is not. However, I wanted to make it clear clear that this did not preclude individual, unfunded research like thesis papers for the like in an academic setting.
I apologize if I didn’t deliver it well. This is my first time trying to present anything of this nature, and I’ve tried to be be careful with my messaging, but this is a section where I have to admit I had a lot of trouble.
If this is a thing (and while I understand general skepticism towards extraordinary claims of this nature, I know that it is, because I’ve been kicking the tires on it for weeks), then it is something people are going to want to study. In that case, that early clarity matters. Since I am fairly certain that’s how this is going to shake out after further evaluation, I went ahead and specified upfront.
I appreciate you taking the time to write out what put you off, though, it’s helpful feedback.