Could a ideological mining pool use game theory to keep CP off the blockchain?
Suppose we want to keep CP out of the blockchain, and believe we can do that by capping OP_RETURN payload at 1000 bytes.
(Yes, there's certainly Byzantine and tortuous methods you can invent that would allow you to, in theory, transfer information into the blockchain that could be later unpacked into CP. But for purpose of this discussion, let's go with: "If it's easy for any node user to upload/download the raw data, users can and will, and this is bad.")
So the question is, what if a pool of miners, consisting of some fraction of the hashrate, demands this new rule?
This rule would be very much a classic textbook example of a soft fork, a new rule that restricts future transactions, so that all future transactions and blocks are still compatible with older blocks.
The not-so-efficient way to do this would be endless debates, pull requests, miner flagging, podcasts, rage-quits, etc. Instead, let's talk about what would happen if a pool decided they wanted to cause the soft fork by using a consensus hardball approach. How does the math and game theory behind that work?
Majority pool case
First, the easy case. If the pool has over 51% of the hashrate, they can absolutely do this. They could effect a not-full-consensus soft fork by declaring that, say
"After block 920000, any block containing an OP_RETURN with over 1000 bytes will be invalidated by our pool."
Because the pool has over 51% of the hashrate, they will eventually win any race, and so it would be irrational for any other miner to mine such a transaction. All miners would follow the incentives and reject such transactions, regardless of fee. It doesn't matter how high the fee is, if the block gets supplanted, you don't keep the fee. This is very simple game theory, provided that the threat to supplant the blocks is clear and credible. The soft fork would remain in effect indefinitely into the future, until a point when someone decided the pool had diminished in dominance and decided to challenge the pool, or the pool decided to abandon their policy. If Core believes it's best practice not to have a set of hard-coded rules which differ from the in-practice rules, they could always hard-code this into Core and it would become a more traditional form of soft fork.
Significant minority case
The more interesting case is when a pool has somewhat less than 50%. The math is very nonlinear, so we will look at a few cases.
Let's start with a situation when the ideological pool has 35% of the hashrate. This would start exactly as above, with the pool announcing a slightly weaker threat.
"After block 920000, any block containing an OP_RETURN with over 1000 bytes will be ignored. We will mine a competing chain until our chain falls two blocks behind the longest chain, at which point we will return to mine the longest chain."
We assume that this threat is clear and credible (we will analyze the costs of making this later.) This sets up a decision for other non-ideological miners: When presented with an offending transaction, they have a decision to include it or not. If the miner includes the transaction, an easy calculation says that there is over a 12% chance (35% times 35%) that the ideological pool will find the next two consecutive blocks (for some fun math getting a more precise payoff on the strategy see Chapter 6 of my book.) It follows that the expected revenue for mining such a block is reduced by 12%. If the block reward is (using recent numbers) around $350,000, roughly speaking the miner would be irrational to mine such a transaction unless the fee (or the sum of fees) exceeded $40,000.
So in simple terms; if 35% of miners agree to fight such blocks, they are essentially imposing a minimum $40,000 fee on such transactions.
Note this quick computation scales quadratically. If the ideological miners have 20% of the hashrate, there is at least a 4% chance the block is supplanted, which imposes a $14,000 fee.
Even a pool with 10% of the hashrate presents a 1% risk which suggest a $3500 minimum fee. Of course there is some level at which some non-ideological miners may choose to just ignore this risk.
This quick computation above underestimates the risk of including the transaction, however. There is 35% chance that the ideological miner creates an equal weight fork. At this instant, there is only the "first-seen" rule encouraging other miners to mine the first-seen chain. But if you build a block which extends a fork that will immediately be challenged, you are also facing the same $40,000 loss in expected revenue (but aren't getting any of the fee!) So if the other miners are rational, they would ignore the first-seen block and try to mine on the ideological chain. It follows that the risk can be significantly higher: If half of the unaffiliated miners are expected to go ahead and switch; the risk jumps from 12% to 25%.
Of course, there's a counter to this; if the pool that mined the original transaction is larger and interested in fighting back, unaffiliated miners face risk either way.
It makes sense that a minority pool of miners could effectively raise the prices to a prohibitive level. If these sort of transactions never gain any foothold, the ideological miners would have accomplished their purposes, this is a win for them.
Costs to the ideological pool
The pool is comprised of individual miners who are making their own decisions, even if the pool is committed to behaving ideologically. So we have to ask, what are they risking? Would a home miner who's spending $8 of electricity per day to make $10 in Bitcoin choose to participate in such a pool?
The answer largely depends on two related factors: 1) how frequently they expect to have to follow through, and 2) how many other miners commit to joining the pool.
Notice if that if the offending transactions are expected to be rare (say 1 per day) then over 99% of the time, the pool will operate as it usually does, and the revenue would be nearly the same; the expected revenue is only reduced in the case that the offending transaction is mined, then the pool wastes hash equalizing and then eventually losing the race. But this expected loss can be upperbounded by the probability that an offending block is mined.
Generally, if your pool with less than 50% of hashrate challenges a block, this is an expected loss. This expected loss is smaller when the pool hashrate is near 50%. It's almost a guaranteed loss with small hashrate (if you have 5%, the probability you win two blocks in a row is 0.25%, which is positive, barely.)
It's easier for the miner to think ideologically when the ideology only comes into play rarely: If you're mining rationally 99% of the time and ideologically 1% of the time, you will never suffer much of a hit.
Bottom line: If 35% of miners believe that such transactions would occur in less than 1% of blocks, and they are willing to stomach a 1% revenue risk to join this pool, this pool will be successful.
Notice we encounter another nonlinearity: The assumption that offending transactions occur only in 1% of the blocks would make them likely to occur almost never.
A counter to this is that it may be possible for offending transactions to slowly build up to over $40k worth in fees, at which point some miner would mine them. We can't rule this out.
If the transactions are to occur somewhat frequently it's much more costly to the pool. If a pool with 15% of the hashrate is constantly battling against blocks that occur once an hour, their revenue is going to drop by over 10%: this could discourage miners from joining the pool, making the pool less effective and encouraging more offending transactions.
Just as an aside: there's the interesting question of how the pool payout should be structured - should the pool payout more for hashes produced after the pool has equalized the chain? Lots of nonlinearities!
Bottom Line
If the small-OP_RETURN-payload team thinks that
1) Such transactions will be rare, say in less than 5% of blocks and
2) they can get 25-30% of miners to join them,
then they absolutely would be wise to form a pool which tries to supplant blocks containing such transactions. The result would be that such transactions would become prohibitively expensive, and would never get the critical mass necessary to become a threat.