There are now enough answer engine crawlers that this is a policy rather than a toggle. The crawler table I work from lists thirteen of them, each with its vendor and whether the origin currently answers it, and the list keeps moving as vendors split one agent into several.
The part that makes the decision hard is that it is not reversible in any useful sense. Content already fetched and used does not come back when the directive changes. A block applied after the fact protects only what has not been read yet, which on an established site is the smaller half.
Three positions come up in practice. Allow everything and treat citations as distribution. Block the agents that collect for training and allow the ones that fetch to answer a live query. Or block everything and accept the loss. The middle position is the one most people describe, and it is also the one that needs re-checking every few months, because it depends on vendor documentation that changes without notice.
Two questions.
Has anyone measured referral behavior before and after a change, on a site with enough traffic for the comparison to mean anything? Most of what I have seen is a screenshot of one answer.
And for the sites you block, are you finding the content quoted anyway through a third party that allows the same crawler?
Swiss Knife SEO ModeratorAdministrator