Very good question. We just hired a new head of data science, so I wonder if he would answer differently than me.
If I were guessing from the outside I would guess churn rate. But unfortunately, our churn rate never changes. Our working theory there is that it primarily measures that we aren't charging very much.
The good news is that common sense and any measure we have of sentiment has already given us a strong signal that we are in the wrong place. So we don't have to A/B test our way anywhere. We can just jump to point B and refine from there.
On the subscriber side, sentiment via survey is probably the long term thing we'll watch. NPS but also specific things like are you healthier, feel more informed about world events, has this helped you in your career.
There are so many little quantitative measures though.
Effectively we are replacing a chunk of click bait with a chunk of high substance writing. But... that has a negative impact on CTRs.
I'm hoping that we see it in activity-type measures of retention, like what percentage of people read every day.
The author side also has a lot to watch. We started measuring hits as in how many articles pass 50k views. This platform used to manufacture a lot more hits and we think that hits have other side effects like boosting SEO, bringing in backlinks, viral tipping points.
Then does the mix of authors and author activity change? It's possible total volume of posts will go down. There's some number of them that are just here for money that subscribers aren't happy to be paying to them. So we have to be looking at slices of authors. Are authors that are boosted once more likely to keep writing? Do authors that joined our verified books program (presumably they produce a higher rate of boostable articles) produce more articles than they did before our recent changes? Does that group drive more external traffic (also an important part of how the Medium machine functions).
Basically, it's a big change that should hit everything we already measure but we might not fully parse out cause and effect.
