If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.
If you mean the transformer could literally output this as CoT.. that’s an interesting point. You’re right that “I should think about X” will let it think about X at an earlier layer again. This is still lossy, but maybe not as much as I was thinking.