Yeah, it’s just a temporary fix because we got hammered by an angry bot last night. I already lifted the relevant firewall (the robots.txt thing is made up, our robots.txt doesn’t block crawlers).
Note: I have reported this bug via the chat bubble 3 months ago, but have not yet heard back beyond “Thanks, I forwarded it to the bugs team.”. I feel a little bad for bring this issue up again.
The introducing comment.
I think removing this would add lots of results to Google. I am honestly unsure what’s the best there.
Often when I search site:slatestarcodex.com abc a result comes because some comment mentions something. Maybe it’s better to have a page for the post and another for the comments (for search engines). :shrug:
Comments are mostly indexed on Google via the post-page indexes, not the commentId permalinks. Indeed, the commentId permalinks confused a bunch of the crawlers because it resulted in lots of duplicate copies of the post text, and text from other comments showing up in the search results.
Are separate comment pages even desirable? Couldn’t the comment link be replaced with the anchor link which scrolls to the relevant comment (currently “see in context”)?
It’s a hotly debated topic! I think they are good because they provide a scroll-insensitive link to a piece of contend. Other people find them annoying. Maybe eventually we will find a UI solution that makes everyone happy.
Yeah, it’s just a temporary fix because we got hammered by an angry bot last night. I already lifted the relevant firewall (the robots.txt thing is made up, our robots.txt doesn’t block crawlers).
Relatedly, the robots.txt (in ForumMagnum) does block access to comment links via
So when pasting a link to a comment into a chat with an LLM, it won’t be able to read the comment. Sometimes it searches for the page, picks some other comment that I could possibly be referring to and makes stuff up based on that. This also has the effect of search engines not indexing Quicktakes well. e.g. googling
"I think are cool and put it in my room. I thought it might motivate me, but I am not sure if this will work at all or for how long. Feel free to steal. Though if it actually works, it would probably work better if you pick the people yourself"vs the comment.Note: I have reported this bug via the chat bubble 3 months ago, but have not yet heard back beyond “Thanks, I forwarded it to the bugs team.”. I feel a little bad for bring this issue up again.
The introducing comment. I think removing this would add lots of results to Google. I am honestly unsure what’s the best there. Often when I search
site:slatestarcodex.com abca result comes because some comment mentions something. Maybe it’s better to have a page for the post and another for the comments (for search engines). :shrug:Comments are mostly indexed on Google via the post-page indexes, not the
commentIdpermalinks. Indeed, the commentId permalinks confused a bunch of the crawlers because it resulted in lots of duplicate copies of the post text, and text from other comments showing up in the search results.Are separate comment pages even desirable? Couldn’t the comment link be replaced with the anchor link which scrolls to the relevant comment (currently “see in context”)?
It’s a hotly debated topic! I think they are good because they provide a scroll-insensitive link to a piece of contend. Other people find them annoying. Maybe eventually we will find a UI solution that makes everyone happy.