{'fraction_non_alphanumeric': 0.029350862777321282, 'fraction_numerical': 0.014511092851273624, 'mean_word_length': 4.839827255278311, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Digital Paywall Design: Implications for Content Demand & Subscriptions

Most online content publishers have moved to subscription-based business models regulated by digital paywalls. But, the managerial implications of such freemium content offerings are not well understood. We therefore utilized micro-level user activity data from the New York Times (NYT) to conduct a large scale study of the implications of digital paywall design for publishers. Specifically, we use a quasi-experiment that varied the (1) quantity (the number of free articles) and (2) exclusivity (the number of available sections) of free content available through the paywall to investigate the effects of paywall design on content demand, subscriptions, and total revenue. The paywall policy changes we studied suppressed total content demand by about 9.9%, reducing total advertising revenue. However, this decrease was more than offset by increased subscription revenue as the policy change led to a 31% increase in total subscriptions during our seven-month study, yielding net positive revenues of over $230,000. The results confirm an economically significant impact of the newspaper's paywall design on content demand, subscriptions, and net revenue. Our findings can help structure the scientific discussion about digital paywall design and help managers optimize digital paywalls to maximize readership, revenue, and profit.

Digital paywalls are essentially a price discrimination mechanism to sort readers according to their willingness-to-pay (Shapiro and Varian, 2013;Bhargava and Choudhary, 2001;Chellappa and Shivendu, 2005). Their basic goal is to create a separating equilibrium where those with high willingness-to-pay (WTP) are induced to subscribe while still allowing those with low WTP to be monetized through online advertising. 2 The goal of a digital paywall is to maximize revenue by regulating subscriptions and web traffic. In this way, they are similar to other canonical versioning mechanisms like those used by airlines which induce customers to self-select into Business and Economy classes according to their WTP to maximize producer surplus.
There are, however, key differences between the versioning mechanisms used by airlines and digital newspapers. Unlike airline seats, digital content is non-rival. Offering little 1 However, there are several exceptions to this. New media outlets such as http://vox.com, http: //politco.com, http://theringer.com, and http://buzzfeed.com have survived with mostly adbased monetization models owing to their exclusive content offerings such as podcasts which have helped maintain sustained engagement.
2 High WTP readers i.e. subscribers, also generate advertisement revenue as they also see some advertisements. However, it is dwarfed by the subscription revenue they generate. or no free content can increase short-run revenues as some fraction of marginal readers with sufficiently high WTP will subscribe. However, such conservative content policies would fail to bring new users to the platform as they reduce exposure. Furthermore, limiting freely available content makes it more difficult for new readers to determine how well the newspaper fits their content preferences, decreasing the chance that new readers will be persuaded to subscribe. 3 On the other hand, paywall policies that offer too much free content will bring many new readers to the platform, but will weaken the paywall as a separating mechanism since high WTP readers will have less incentive to subscribe. This tradeoff between short-run and long-run revenue generation is classically known as Arrow's Information Paradox (Arrow, 1962).
The optimal design of a digital paywall is a complex process as evidenced by newspapers' continuous tinkering with paywall designs. 4 It is important for newspapers and other content providers to understand the design-space of digital paywalls and the mechanisms by which the design parameters impact firm outcomes. Content producers have many important design choices to consider regarding 1) Quantity: The number of free articles that non-subscribers can access in each time-period. Publishers can, for example, implement an all-or-nothing paywall (e.g. The Financial Times and The Economist) which allows access to content for subscribers only, or a freemium paywall (e.g. the NYT or Boston Globe) which gives non-subscribers access to some free articles in each time-period (e.g. 5/month in the case of NYT and 2/month in the case of Boston Globe). 2) Exclusivity/Breadth: The exclusivity (or breadth) of content that non-subscribers can access. Non-subscribers can either have access to all content across all sections (low exclusivity) or a limited subset of content, like popular news or politics (high exclusivity), while subscribers have access to that as well as more niche content like in-person player interviews as is the case at ESPN. Throughout this paper we use the terms diversity of content or the breadth of content interchangeably. 3) Temporal Differentiation: The temporal inclusion or exclusion of content, such as full access for non-subscribers only on weekends, or only to monthly/quarterly special issues. This is common for some kinds of TV content where the free digital episode is delayed in time. 4) Porosity: Whether the paywall should allow free referrals to the newspaper's website from search engines, social media and news aggregators-sometimes referred to as a "porous" paywall.
In this paper we attempt to open the black box of digital paywall design. We study the impact of arguably the two most critical parameters of a digital paywall-the quantity of free articles and exclusivity or breadth of free content available to non-subscribers (i.e. whether there is some gated subscriber-only content). These parameters allow publishers to change the distribution of articles from which readers can sample and hence the perceived match between the content and the readers' preferences.
We exploit quasi-experimental variation in the quantity and exclusivity of free content offered by the NYT via their digital paywall to study the impact of these two paywall design parameters on content demand, subscriptions, and total revenue. A two-dimensional representation of the paywall design space considered in this paper is presented in Figure 1. Figure 1: The Paywall Design Space. Note: Media companies such as the New York Times (NYT) which have a porous paywall do not entirely fall into one of these 2× 2 buckets. NYT paywall offering (as of 2019) is "High" breadth and "Medium" quantity in terms of these two parameters.
Our work contributes to several areas of research. First, our study is related to the literature on product sampling and versioning of digital goods. Prior work in this literature has focused on the fact that digital goods are experience goods and consumers need time to derive value from them (Heiman and Muller, 1996;Heiman et al., 2001;Chellappa and Shivendu, 2005;Lehmann and Esteban-Bravo, 2006). Hence, firms can increase the propensity of a consumer to adopt their product or service by providing them free samples (Bawa and Shoemaker, 2004). Digital Paywalls and other freemium products, however, differ in an important way-their free offerings are perpetual, which can turn the free product into a close substitute for the premium product. Therefore, there is a significant risk of cannibalization of the premium product. There is a growing body of work that builds theoretical models of the economics of freemium services (Niculescu and Wu, 2014). There have also been empirical investigations into several aspects of freemiun products. Much of the empirical work, though, has focused on freemium products in a social or networked setting, for example, Oestreicher-Singer and Zalmanson (2013); Bapna et al. (2016) found increased social engagement and peer influence to be key drivers of subscriptions in freemium products respectively. Our work contributes to this burgeoning empirical literature by studying the screening procedure of a freemium product and in a non-networked setting. More precisely, we quantify how the design of the screening mechanism-in our case the quantity and exclusivity parameters of the digital paywallchanges readers' propensity to adopt the premium offering.
Our work also contributes to the literature on digital paywalls. This literature has mostly focused on the impact of the quantity parameter of a digital paywall on content demand. For instance, Chiou and Tucker (2013) find digital paywalls suppress demand; Lambrecht and Misra (2015) argue content providers can adjust the amount of free and premium content counter-cyclically in response to demand conditions. There has also been work studying the spillover effects of digital paywalls on print newspaper sales (Pattabhiramaiah et al., 2017) and social media sharing (Oh et al., 2016). Our work, however, is the first to study the impact of two digital paywall design parameters-quantity and exclusivity. All of the previous work has focused on a monolithic version of the digital paywall with only a variable quantity parameter. Hence, we are able to construct a more nuanced picture of the trade-offs at the heart of a digital paywall. It is unclear how paywalls impact subscriptions since premium versions are a close substitute for the free product. To the best of our knowledge, ours is the first study to quantify the impact of paywalls on readers' propensity to subscribe. Our analysis also goes a step further by using detailed data to explore the heterogeneity of the effect of the paywall on subscriptions.
Finally, we also study key decisions related to the digital paywall design in a multichannel setting, where readers can consume content via either the mobile app or the browser. While there is empirical work on consumption dynamics between different channels (Ansari et al., 2008;Avery et al., 2012;Athey et al., 2014;Deleersnyder et al., 2002), whether these channels have a synergistic or substitutive impact on content demand has been shown to be highly context dependent. Our work studies these multi-channel consumption dynamics in a novel news readership context. Our study has actionable implications for publishers' digital content strategies and makes several contributions to our understanding of digital disruption in online content industries. First, we econometrically identify the impact of paywall design on content demand. Content demand is an important metric directly linked to monetization. Less viewership leads to fewer ad impressions and therefore less advertising revenue. Second, we estimate the impact of paywall policy changes on the NYT's subscriber base. Until now there has only been anecdotal evidence that digital paywalls affect subscriptions (Kumar et al., 2013). 5 But, there exists no rigorous econometric quantification of such effects. Finally, we construct a detailed picture of the NYT's entire revenue stream in the presence of a digital paywall. In particular, how does the design of the paywall impact total revenue, comprised of ad revenue and subscription revenue. Is it the case that in spite of the low WTP for online news (Chyi, 2005), digital paywalls provide a sustainable digital business model for newspapers?
Our results suggest an economically significant relationship between paywall design, con-5 https://digiday.com/media/new-york-media-paywall-subscriptions-flexible/ tent demand, and subscriptions. The paywall policy changes, in quantity and exclusivity of free content, in our study decreased content consumption by about 9.9%. But, this decrease was more than offset by increased subscription revenue generated by a 31% increase in total subscriptions during our seven-month study. Taking these results together, the paywall change led to an economically significant net-positive impact of around $230,000 on the NYT's bottom-line. Our findings also suggest that paywall policy changes that let readers choose free content broadly from a variety of topical areas, rather than restricting the variety of free content content available, are more effective at increasing subscriptions, demand, and revenue.

Empirical Setting and Data
We use user-level data from the New York Times (NYT) to study digital paywall design. The NYT is the 17 th largest newspaper in the world by circulation and has won more Pulitzer Prizes than any other newspaper. 6 The scale and the heterogeneity of the NYT's global reader-base makes it a good context in which to study digital paywall design, and also allows us to generalize our findings to other similar newspapers such as The Washington Post, The Wall Street Journal (WSJ), LA Times, and USA Today, which together comprise a large portion of the total market for news consumption in USA.
Our data consists of user-level activity on the NYT's various online platforms for the seven month period from April to October 2013. The data track the browsing behavior of around 177 million unique visitors who accumulated over 777 million page views during this period. For the analysis in this paper, we construct a panel of 29,705,796 users who consumed NYT content in at least two different time-periods. 7 The users in our panel were either anonymous (identified by cookies) or registered/subscribed (which gives them the ability to comment on articles, save articles for future reading and get personalized content recommendations).

Quasi-experiment: A policy change in the digital paywall
The NYT launched their digital paywall in 2011. Since then they have implemented a porous paywall, through which unsubscribed users can read a fixed number of articles every month (currently five) 8 . Readers can access extra articles each month if they are referred to those articles through social media websites or search engines.
NYT distributes its digital content through three channels: (1) the main website (www. Figure 2: Details of the paywall setting change. Note: 1) "High Quantity"= Total Content Published, "High Diversity"= All Sections, "Low Quantity"= 3 articles per day, "Low Diversity"= Top News and Video sections. 2) As discussed later, since different users updated the app at different times, the one-week kick-off happened at different calendar times for different users.
nytimes.com) accessible from desktop computers, (2) the mobile website (www.mobile. nytimes.com) accessible via browsers on mobile devices (smartphones and tablets), and (3) the mobile app which can be installed on smartphones and tablets of all varieties. During our observation period the NYT paywall allowed ten free articles per month via channels (1) and (2). Visitors could however read an unlimited number of articles through the mobile app, but only from the Top News and Video sections.
However, on June 27, 2013, the NYT started metering their mobile apps such that unsubscribed users could only read 3 articles per day. At the same time, those articles could now be accessed from any section and not just from the Top News and Video sections. If, after hitting their quota, a user tries to access more articles, they see a pop-up in the mobile app urging them to become a subscriber. 9 To kick-off the update, users had a 1 week trial period from the time they updated the app during which they could freely read any number of articles from any sections. This change in the paywall's settings did not impact the readership on the browser channel and serves as a supply shock for the content consumed on the mobile app. Figure 2 displays the details of the quasi-experiment, which we describe in more detail below. We use this quasi-experimental variation in the quantity and exclusivity of the content available via the digital paywall to identify the impact of these two key elements of paywall design on content demand and subscriptions.
We tease apart the impact of the change in quantity (from unlimited access to 3 articles per day) and the change in diversity (from access to Top News and Video to access to all the sections), by decomposing the change into two phases-one in which only the quantity of content changed and the other in which only the breadth/exclusivity of available content changed; P1 and P2 in Figure 2 respectively. The paywall change in the mobile app was rolled out as part of an update available to download on day 88 of our observation period. However, not every user in our sample downloaded the updated version of the mobile app on the same day it became available as shown in Figure 3. This heterogeneity in the timing of updates provides user-level exogenous shocks to consumption that vary with time. As we show in our robustness checks, this differential updating is not correlated with any observable differences between users.
from making forward-looking adjustments to their reading behavior to a large extent, though it does not totally preclude it.

Variable Construction
Readership Variables: We construct the readership variables as the number of articles read by user i on day t on the mobile app N umArticles App it , on the browser N umArticles Browser it , and in total across both the mobile app and the browser as N umArticles T otal it . The choice of the unit of time as a day is primarily due to the perishable nature of news content and the strong diurnal patterns of content consumption. For instance, if user #10 read seven articles via the mobile app and three articles via the desktop and mobile browsers on day 76, then, the variables will be coded as N umArticles T otal 10,76 = 10, N umArticles App 10,76 = 7, and N umArticles Browser 10,76 = 3.
Subscription Variable: The subscription variable Subscribed it indicates the subscription status of user i on day t. Readers can subscribe to one of the four available bundles (1) all digital, (2) all digital and home delivery, (3) web and smartphone, (4) web and tablet. For simplicity and the ease of interpretability we pooled all the subscription bundles and coded the subscription status as a binary variable equal to 0 if the visitor was a non-subscriber (either an anonymous visitor or a registered non-subscriber) or 1 if the visitor was a subscriber.
Policy Variable: The entire policy change i.e. the combination of both the quantity and diversity/exclusivity change is operationalized as a binary indicator variable P aywallP olicy it . The P aywallP olicy it variable flips to one at different times for different users based on the time they updated the mobile app as shown in Figure 3. The earliest it flips to 1 is at the beginning of period P2 in Figure 2.

Summary Statistics
We created a panel of anonymous, registered, and subscribed users at the user-day level. There were a total of 29,705,796 users in our panel out of which 28,897,011 users were anonymous (identified via cookies) and the rest 808,785 were either registered or subscribed users. 78% of the visits made by our user panel were from U.S., 4% from Canada and the remaining 18% from the rest of the world (195 different countries). 67% of users had only one mobile device (iPhone, iPad, Android or iPodTouch), 31% had two devices and the remaining 2% was split between users with three or four devices. The devices used were split 10%, 44%, 45% and 1% between Android, iPhone, iPad and iPodTouch respectively. We have information on the genders of registered and subscribed users. 54% of such users did not declare their gender. Of the 46% of users who identified their gender, 61% were men and 39% were women. Table 1 displays summary statistics on the content consumption behavior of our user panel. As can be seen, the variance of the readership variables is greater than their mean, suggesting a long-tail.

Model Specifications
The quasi-experiment lends itself to a difference-in-difference (DiD) estimation strategy, which is our main model specification throughout the paper.

Impact on Content Demand
We split our analysis into three parts based on our research context. First, we assess the impact of the NYT paywall policy change on content demand in the mobile app channel, which is where the policy change was implemented. Next, we quantify the impact of the policy change on content demand in the browser channel, and finally, we estimate the impact on total content demand across both the mobile app and the browser channels. Our difference-in-difference (DiD) estimation considers subscribed readers as the control group as they were unaffected by the paywall change. Although subscribers certainly differ from non-subscribers in their level of consumption, the key identifying assumption of DiD is parallel trends. We qualitatively and quantitatively verify this assumption later in the paper. Our estimation is complicated by the fact that users can change their subscription status during our observation period. As can be seen from Figure 7, there is significant variation in the number of subscribers each day due to churn and an inflow of new subscribers. So, the composition of our treatment and control groups shift during the observation period. To address this issue, we focus only on users who did not change their subscription status during our observation period. Only a very small fraction (≈ 0.36% of the total 29,705,796) of the users changed their subscription status during our observation period. So bias from this selection is negligible.
Our exact model specifications are given in Equations 1, 2, 3.

N umArticles
These specifications differ only in their choice of dependent variables: mobile app readership N umArticles App it for (1), browser readership N umArticles Browser it (2), and finally, the total readership N umArticles T otal it (3). P aywallP olicy it captures the aggregate impact of the entire policy change (i.e. the change of both quantity and exclusivity) on readers. Our main independent variable is the interaction between the policy variable and an indicator for non-subscribers P aywallP olicy it × N otSubscribed i . The coefficient on this term captures the average treatment effect on the treated (ATT) of the paywall change on non-subscribers. Lastly, we incorporate a set of user and time fixed effects, denoted by γ i and δ t 10 respectively. 11 Since the dependent variables in the specifications above are skewed count variables we estimate them via a Poisson Regression (Athey and Imbens, 2006;Puhani, 2012;Shang et al., 2018). Another commonly employed alternative for estimating such models with over-dispersed dependent variables is to first log-transform the count variable(s) and then estimate the resulting model via a standard OLS regression. These "log-linearized" models are however known to provide biased estimates under heterokedasticity and when there are lots of zero counts in the data (Silva and Tenreyro, 2006). Hence, we use Poisson Regression as our main specification and show the robustness of our parameter estimates to log-linearized models. We follow the steps proposed by Shang et al. (2018) to compute partial elasticities for the Poisson difference-in-difference regressions. The impact of the paywall policy change in our case is given by the coefficient of the interaction term in the specifications 1-3 above, which can be interpreted easily in terms of differences-in-semielasticities (DIS) as exp (β 1 + β 2 ) − exp (β 1 ) where β 1 is the coefficient of P aywallP olicy it and β 2 is the coefficient of the interaction term.

Impact on Subscriptions
Next, we quantify the impact of the policy change on readers' subscription status-the number of non-subscribers induced to subscribe due to the policy change. Here, we do not have a natural control group for a DiD estimation strategy, so we use the heterogeneity in the readers' exposure to the paywall change to define a set of user groups that were differentially impacted by the policy change. We accomplish this in a couple of ways. Our first specification given in Equation 4 compares the varying subscription propensities of sub-populations of readers that hit the paywall i.e. they either tried to read more than the allotted quota of 3 free articles per day (and were shown a pop-up message to subscribe) or they consumed a wider variety of content once it was available. The readers who never exceeded the allotted quota of free articles or only consumed content from Top News and Video sections throughout never actually received the "treatment" of a paywall change and hence constitute our control group in this specification. This strategy allows us, to some degree, to decompose the differential impacts of the two components of the policy change. Although these groups may differ on meaningful dimensions, our identifying assumption relies on comparing parallel time trends in these groups before and after the policy change.
Our main independent variables of interest in this specification are the three interaction terms which permit us to assess the impact of paywall design on subscriptions. The coefficient on the two-way interaction term P aywallP olicy it × I(Exceed − Quantity) i captures the impact of paywall on the sub-population of readers who exceeded the quota of free articles but who were not interested in consuming content from the blocked sections. Similarly, the coefficient on the term P aywallP olicy it × I(Consume−Diverse) i quantifies the impact of paywall on the readers who consumed content from the blocked sections but who did not exceed their quantity quota of 3 free articles per day. Finally, the coefficient on the three-way-interaction term P aywallP olicy it ×I(Exceed−Quantity) i × I(Consume− Diverse) i captures the impact of paywall on readers who both exceeded their quota of free articles and also consumed more diverse content.
In our next specification we assess the impact of paywall on subscriptions by stratifying our reader-base differently. Our specification in Equation 5 harnesses the intensity of the treatment as it compares the differential propensity to subscribe based on the number of articles read prior to the paywall change came into effect. For instance, one should expect the readers who consumed (say) 20 articles on average per day to be impacted more by the paywall change to 3 articles per day compared to the ones who consumed only (say) 2 articles per day before the change.
Subscribed it = P aywallP olicy it + P aywallP olicy it × N umArticles P riorAvg i + γ i + δ t + it (5) The term N umArticles P riorAvg i in the above specification codes the intensity of the treatment as the average number of articles read by the user i prior to the paywall policy change.
Dynamic Effect of the Paywall change on subscriptions: Our specifications above capture the contemporaneous impact of the paywall policy change on subscriptions. However, there is also a sustained impact of the paywall change on the readers' propensity to subscribe. It is important to estimate this dynamic long-term impact in order to broadly understand the design of digital paywalls and the key temporal tradeoffs between the quantity and exclusivity parameters.
The specification for estimating the dynamic impact of the paywall is given in Equation 6. It is essentially the same as the specification in Equation 4 with the key difference that instead of interacting the treatment group indicator with the policy change variable P aywallP olicy it contemporaneously, we also add interactions of the treatment group indicator with the time dummy for each week until the end of our observation period. This entails adding all the interaction terms starting in week 13 when the paywall policy change was rolled-out until the end of our observation period in week 31.
∆ w in the above specification is the dummy variable for week w and δ t are the time fixedeffects. As earlier specifications, the time fixed-effects δ t are operationalized via day-level dummies ν day .
All our specifications also control for any time-invariant individual idiosyncrasies via the fixed-effects γ i . It is worth noting that our specifications above have a binary dependent variable Subscribed it and the preferred model for such a case is either a logit or probit. However, estimating such models with millions of individual and time fixed effects is extremely challenging for any software and takes extremely long to converge to the correct solution. Furthermore, it is also cumbersome to interpret the interaction terms in logit or probit models (Ai and Norton, 2003). So, following the lead of several researchers (Angrist and Pischke, 2008; Goldfarb and Tucker, 2011;Agrawal et al., 2015;Chatla and Shmueli, 2017;Taylor et al., 2019), we use standard linear probability models (LPM) for estimating our specifications. LPMs are typically a desirable modeling choice if a large fraction of the predicted probabilities lie inside the [0, 1] interval. In our case, all of the predicted probabilities lie between 0 and 1, and therefore LPM with robust standard errors will yield unbiased and consistent estimates (Horrace and Oaxaca, 2006;Chatla and Shmueli, 2017).

Results
In this section we present the empirical results quantifying the causal impact of the NYT paywall policy change. The results are split into three subsections. First, we present some model-free evidence highlighting the pre-and post-policy values of our outcome variables. Second, we present DiD estimates of the impact of the paywall change on content demand, and then finally we show the same estimates for subscriptions. We conclude this section by presenting analyses grounding the robustness of our findings.

Model-Free Evidence
Figures 4, 5, and 6 summarize the key economic variables-N umArticles App it , N umArticles Browser it , and N umArticles T otal it for the various user groups of interest. As can be seen, before the paywall change the consumption patterns of non-subscribers and subscribers are similar. This pattern of consumption persists for subscribers after the paywall change as they were not impacted by the paywall change. We observe a significant decline in readership after the paywall change came into effect. 12 It is not surprising that demand for content in the mobile app fell (Figure 4) as that was the channel where the paywall change was implemented, however, it is surprising to see the readership fall in the browser channel and hence the overall content demanded also falling sharply ( Figure 5, 6). A priori, as a result of this change, one could have expected that some readers might compensate for the decrease in supply of content in the mobile app channel by increasing their consumption in the browser and hence sustaining or even increasing their overall NYT content consumption. However, it seems that most of the readers either decreased their consumption or kept it the same as their pre-paywall levels, leading to an overall decrease in total readership. Figure 7 shows the number of subscribed users during our entire observation period. 13 Clearly, we see an economically significant increase in the number of subscribed users after the paywall change. We can not, as yet, quantify how much of this increased subscriber base is attributable to the paywall change.
Since our empirical analyses are based on DiD estimates, they rely on the parallel trends assumption. The parallel trends assumption in this case posits that, prior to the paywall shift, the key economic variables N umArticles App it , N umArticles Browser it , N umArticles T otal it , and Subscribed it should have similar parallel trends. So, in addition to the visual proof of the parallel trends assumption provided by Figures 4, 5, and 6, we perform a formal empirical test to verify this assumption for all our main specifications. 14 In particular, we show that the interactions of the treatment group indicator N otSubscribed i with the pre-treatment time dummies are jointly statistically insignificant. The p-values of the corresponding F-tests being 0.32, 0.38, 0.18, and 0.41 for the specifications with N umArticles App it , N umArticles Browser it , N umArticles T otal it , and Subscribed it dependent variables respectively. 15 12 Though, note that in the small one-week phase where the non-subscribers were allowed unfettered access to content there was an increase in total content consumption.
13 For privacy reasons, we have scaled the number of subscribers (y-axis) by a constant. 14 a) Details are provided in the robustness checks section. b) It is hard to show the parallel trends assumption visually for our specifications which have the subscription status as the dependent variable since there isn't a clear control group in that case, so we just provide an empirical proof for those specifications in the robustness checks section. 15 The null hypothesis was that the fits of the full and restricted models were the same.

Impact on Content Demand
We estimate the specifications in Equations 1, 2, and 3 to estimate the aggregate impact of the NYT policy change P aywallP olicy it on readership. The results in Table 2, specifically the coefficients of the term P aywalP olicy it × N otSubscribed it , show that the aggregate paywall change resulted in a 9.9% reduction in readership in total across both the mobile app and the browser. 16 This decrease is comprised of a 4.6% reduction in readership in the mobile app alone and a 3.5% decrease in readership in the browser alone relative to the control group. 17 The intent-to-treat (ITT) estimates for these specifications-which assumes everyone updated the mobile app at the same time-are similar to these estimates in both magnitude and directionality and are provided in the Appendix for completeness.  192,293,146 192,293,146 192,293,146 Table 2: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Note: 1). Robust standard errors are clustered at the level of users. 2). The variable N otSubscribed i codes the non-subscribed users (anonymous and registered)-our treatment group-as 1, and the subscribed users as 0. It is the complement of the subscription status variable Subscribed i (= 1 − N otSubscribed i ).
Next, we explore the impact of the paywall change on sub-populations of users that are likely to be differentially impacted by this policy change. Study of these sub-groups posits interesting managerial implications for NYT.

Impact on Registered Users:
As noted earlier, non-subscribers are composed of two distinct subgroups: anonymous users and registered users. In this section, we focus on the effect of the paywall change on registered users. This sub-population is of particular interest as registration itself is a signal for some type of intent. 18 We re-estimate Equations 1, 2, and 3, excluding anonymous users from the data. These results are reported in Table 3. Overall, we can see that the policy change had a greater impact on registered users as it resulted in a 7.6% decrease 19 in readership on the mobile app as opposed to 4.6% for the average user (cf. Table 2). However, the drop in browser readership is similar for the two groups. This could be explained by the fact that registered users are typically more engaged heavy consumers who often hit the paywall quantity limit. The inability to consume additional content either led them to consume NYT content via the print offering or it led them to abandon the NYT platform altogether. 20 Another possibility that we also can not rule out is the registered users deleting their browser cookies and consuming content as new users at a higher rate than an average user. On the other hand, anonymous users on average consumed fewer articles and were less likely to hit the free article limit. As a result, the content consumption of an average user did not drop as steeply as a registered user.   20 Several studies have demonstrated the increased tendency of readers to switch and "multi-home" among different online news outlets (Athey et al., 2014;Gentzkow et al., 2011). 21 Recall that our full dataset has many times more anonymous users than subscribed or registered users.

Impact on Users that Hit the Paywall:
Online content consumption has a long tail as most readers typically read only 1 or 2 articles. Hence, there is a large fraction of users that never actually hit the paywall. So, next we compute heterogeneous treatment effects for the users who actually "hit the paywall" and hence were impacted by the policy change. In our setup, this can happen in two scenarios-a) if they attempt to consume more articles than the limit of 3/day after the policy change and were shown a pop-up urging them to subscribe, or b) they consumed more diverse content i.e. read articles that were not part of the Top News and Video sections.
The results for the user groups that attempted to read more than 3 articles/day after the change in policy and those that consumed content outside the Top News and Video sections are shown in Tables 4 and 5 respectively. We can see an accentuated impact of the policy change on both these subsets of users; the impact on total content consumption is similar for both these groups of users-decreases of 18.7% and 19.2% respectively (Column 1 & Row 2 of Tables 4 and 5).  Table 4: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Same as Table 2 but the treatment group is only the sub-population of readers that tried to read more than 3 articles/day. Note: Robust standard errors are clustered at the level of users.
All these results point towards a story of the browser and the mobile app channels as having synergistic effects for news readership. Previous work (Forman et al., 2009;Avery et al., 2012) has shown that this could happen if for instance a channel provides additional utility to the consumers or offers different comparative advantages compared to the other channels. In our case this comparative advantage could be due to the user-friendly interface of the mobile app which can harness the device/platform specific characteristics of the operating system that a browser can not. Hence, a mobile app can provide additional utility to the consumers compared to the browser by lowering the search cost of content.  Table 5: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Same as Table 2 but the treatment group is only the sub-population of readers that consumed more diverse content after the paywall change. Note: Robust standard errors are clustered at the level of users.

Impact on Subscriptions
Our results from estimating Equations 4 and 5 are found in Table 6 below. For Equation 4 (column 1), our results suggest that for the NYT, both elements of the policy are increasing the propensity of users to subscribe. Specifically, since the coefficient on P aywallP olicy it × I(Exceed − Quantity) i is positive and statistically significant, the decrease in quantity is increasing the propensity of a particular subgroup of users to subscribe, which is in line with our theoretical expectations. As the P aywallP olicy it × I(M ore− Diverse) i coefficient is also positive and statistically significant, it follows that the increase in diversity is also driving a similarly increasing subscription propensity, albeit for a different subgroup. This result runs somewhat contrary to our initial expectations since having exclusive or gated content is generally used to induce subscriptions. 22 However, a key difference is that unlike the strictly multi-channel gated offerings of ESPN or YouTube, NYT had gated content only on the mobile app. So, a reader could have still consumed diverse content on the browser. Lastly, since the three-way interaction P aywallP olicy it ×I(Exceed−Quantity) i ×I(M ore−Diverse) i is positive and statistically significant, there seems to be complementarity between paywall design choices. 23 Our results of estimating Equation 5 (column 2 in Table 6) show that readership intensity prior to the paywall change mediates the impact of the policy change. Our point estimate suggests that for each additional marginal article read per day (on average), the paywall policy change increased subscription propensity by approximately 0.02. This supports our idea that the policy is more effective at inducing subscription for readers that are heavier consumers. Another explanation for these results is that prior readership intensity itself moderates the treatment effect. This would imply that if some "exogenous" shock boosted the readership of a particular individual, then the paywall change would be more likely to induce that individual to subscribe.
The intent-to-treat (ITT) estimates for these specifications-assuming everyone updated the mobile app at the same time-are similar to these estimates in both magnitude and directionality and are provided in the Appendix for completeness.  Table 6: Difference-in-difference estimates of the impact of the paywall policy change on subscriptions **p < 0.05,***p < 0.01. Results are computed for a panel of (users) n=29,705,796 and (days) t=212. Note: 1). Standard errors (shown in parenthesis) are clustered at the level of users.

Impact on Registered Users:
Here, we focus on the sub-population of registered users. Our results confirm the increased propensity of registered users to subscribe in response to the policy change compared to the control group. The paywall change increased the subscription probability of registered users that tried to access more free articles than their quota by 0.08 and subscription probability of the readers that read more diverse content by 0.05. Also, the marginal impact of an extra article read per day prior to the change was higher for registered users at 0.03 compared to the average non-subscriber in the control group.  Table 7: Difference-in-difference estimates of the impact of the paywall policy change on subscriptions **p < 0.05,***p < 0.01. Same as Table 6 but the treatment group is only the registered readers. Note: 1). Standard errors (shown in parenthesis) are clustered at the level of users.

Dynamic
Effect of the Paywall change on Subscriptions: Figure 8 shows the dynamic impact of the paywall change on subscriptions. As can be seen, the quantity restriction has the largest sustained impact on subscription propensity and its effect increases over time. The effect of increased variety, on the other hand, increases gradually over our observation period after an initial dip. The magnitudes of these two effects might not be directly comparable owing to the specifics of the paywall change. However, we conjecture that the seemingly lower effect-size of diversity of content could be due to the increased search costs of finding and reading content tailored to one's preferences.

Robustness Checks
We check the robustness of our findings in several ways. First, we present detailed empirical results verifying the presence of the parallel trends assumption required by several of our difference-in-difference specifications. Second, we show that the delay in updating the mobile app is not correlated with the readership on the mobile app and the total readership across all channels prior to the paywall shift. Third, we consider alternative ways of measuring readership other than the number of articles read. Fourth, we consider alternate functional forms of our specification, in particular log-linearized OLS count models and logit regression models. As a final robustness check, we perform our analyses at week-level granularity.

Empirically checking the parallel trends assumption for Difference-indifference (DiD) specifications:
First, we empirically verify the parallel trends assumption. In order to do that we generate interactions of week dummies ∆ w for weeks prior to the date that the policy change was implemented i.e. weeks 1 through 12, with the respective treatment indicators corresponding to different specifications. 24 If indeed there were parallel trends between the treated and control groups then all these pre-treatment interaction terms should be jointly statistically insignificant.
The resulting specifications are given in Equations 7-10. Table 8 shows the estimates for the readership specifications and Figure 9 demonstrates them visually for subscriptions. As can be seen, the parameter coefficients for pre-treatment interactions of the time dummies with treatment indicators are statistically insignificant individually. They are also insignificant collectively. 25 These results collectively suggest the presence of the 24 The time fixed-effects δ t are operationalized via day-level dummies as earlier. γ i represents the person fixed-effects.
25 A F-test of the two nested models, that is, one with all the parameters (including interactions parallel trends assumption.
4.4.2 Self-selection concerns due to the differential updating of the Mobile App: Differential updating of the mobile app provides useful individual-level variation in exposure to the paywall policy change. However, one potential concern regarding using this variation in our empirical specification is that a user's update timing may be endogenous. For example, early updaters might be frequent users of the app or they could be heavy NYT content consumers in general. Such non-random self-selection into the treatment could be problematic for the validity of our empirical analyses. So, in Figure 10 we show that there is no systematic pattern in the timing in which users update their app and readership on the mobile app N umArticles App it or total readership N umArticles T otal it . In order to quantify the dependence between the readership variables and time delay in updating the mobile app, we regressed the readership variables on time delay N umArticles App it , N umArticles T otal it . The F-statistics and the corresponding p-values for the two regressions involving mobile app readership and total readership were (F-stat: 1.01, p-val: 0.32) and (F-stat: .446, p-val: 0.51) respectively which suggest that we can not reject the null hypotheses that time delay is not a predictor of either mobile app readership or total readership. Hence, selection bias should not be a significant concern. However, we can of the treatment with pre-treatment time dummies) and one which is restricted to just the parameters corresponding to the true observed timing of the treatment in week 13 had p-values of 0.32, 0.38, 0.18, 0.41 for the specifications with N umArticles App it , N umArticles Browser it , N umArticles T otal it , and Subscribed it dependent variables respectively. This suggests that we can not reject the null hypothesis of the fits of the full and restricted models being the same.  192,293,146 192,293,146 192,293,146 Table 8: Robustness check showing parallel trends in the difference-in-difference specifications. *p < 0.10, **p < 0.05, ***p < 0.01.Note: 1). Robust standard errors are clustered at the level of users., 2). T i is the treatment indicator. 3).ν day : Average of the day-level dummies. As can be seen from the results in Table 9, the estimates of all the variables are comparable in magnitude and sign and hence in economic significance.  192,293,146 192,293,146 192,293,146 Table 9: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Note: Robust standard errors are clustered at the level of users.

Log-linearized OLS & Logit Model Specifications:
We employed Poisson regression models for the specifications when the dependent variable had a skewed (long-tailed) distribution such as N umArticles T otal it , N umArticles App it , or N umArticles Browser it . However, another popular alternative specification for such cases is log-linearized models, that is, we log-transform the skewed variables v it as log (v it + 1) and then use Ordinary Least Squares (OLS) to estimate the resulting specifications (Angrist and Pischke, 2008). Though, Silva and Tenreyro (2006) showed that such estimators are known to provide biased estimates of the true treatment effect, we still test the robustness of our Poisson Regression estimates from Table 2 to using log-linearized models owing to their high prevelance in previous literature. Estimation results are shown in Table 10, and it's easy to see that the impacts of the various variables are qualitatitvely and directionally similar as in Table 2. The policy change decreased total readership by 9.9% using the Poisson Regression specification and approximately 7% using the log-linearized specification. 26 Next, we used a simple linear probability model (LPM) to estimate the impact of the paywall policy change on subscriptions. LPM was our specification of choice in this case as opposed to a logit model owing to its simplicity and the ease of interpretability (Ai and Norton, 2003). Moreover, since logit models do not have a closed-form estimation procedure, they require iterative methods for optimization which can be slow for a big-data setup as ours which has millions of user and time fixed-effects. However, in spite of these difficulties, we were able to estimate the logit model specification on a randomly chosen subsample of 20000 users from our user-base leading to a total of 112,194 person-day observations. The results are shown in Table 11 and as can be seen they bear directional resemblance to the LPM results from Table 6.

Different (week-level) Temporal Granularity of Analysis:
Our main analyses in the paper are done at the granularity of a single day as the paywall policy change that we studied in this paper manifested as a daily change for the readers. Second, day-level analysis also makes sense since online content consumption patterns typically exhibit a strong diurnal nature as it is the granularity at which newspapers are published. So, here we check the robustness of our findings to an alternate temporal aggregation of data, in particular, data aggregated at week-level. Essentially, we re-estimate the models in Tables 2, 6 with week-level data. The results are shown in Tables 12, 13 and as can be seen, they are similar in magnitude and sign as the original results.      We used micro-level user activity data from one of the world's largest newspapers to study the digital paywall design. In particular, we use a NYT paywall policy change to establish the causal impact of the two most important paywall design parameters-the quantity and exclusivity of free content offered-on demand, subscriptions and revenue. We specifically examine the effects of these policy changes on individual-level consumption as well as on subscriptions.
The results suggest a statistically and economically significant impact of both the quantity and diversity parameters on subscriptions and demand. The paywall change not only depressed content demand in the mobile app-the channel in which these changes were implemented, but also decreased content consumption on the browser; reducing overall content consumption. The decrease in total readership was more pronounced for registered users. It is conceivable that since registered users are typically more engaged and loyal readers of the newspaper, the content constriction either led them to consume NYT content via the print offering or to delete their browser cookies, allowing them to consume content as a new user. Finally, it is also possible they abandoned the NYT and instead consumed news content from other sources. Several studies have documented this tendency of readers to switch among multiple online news platforms (Athey et al., 2014;Gentzkow et al., 2011). Unfortunately, we do not have access to the offline activity of the users or their broader internet consumption history to adjudicate among these potential explanations.
The paywall change also increased readers' likelihood of subscribing to the newspaper. It had an impact of engaging readers with more diverse/exclusive content to raise their willingness-to-pay as now they could consume content aligned with their preferences. But, at the same time the quantity constriction nudged them to become paid subscribers. So, cumulatively both these mechanisms helped convert registered users to subscribers. Just as with content consumption, these effects were stronger for registered users.

Managerial Implications
Our results have multiple managerial implications. First, they suggest news providers should consider freemium content offerings that let readers' choose the free content that they wish to consume. Second, while designing a digital paywall it is important to consider the interactions between the different paywall design parameters. The various design choices could have reinforcing or cannibalizing impacts on subscriptions. Third, online news providers should consider the multi-channel aspect of content consumption while splitting their marketing budget across different digital channels such as the mobile app and browser channels. Though the answer to this question is highly context dependent, we observed a synergy between the mobile app and the browser channel for news readership. This may suggest varying advertising intensity in these two channels since there is a risk of reaching the same user multiple times, wasting ad impressions and creating annoyance (Athey et al., 2014).

Revenue Impact:
The paywall changes we studied were successful for the NYT. As we saw in the results section (Table 2), they decreased the total number of article impressions across both the mobile app and the browser by approximately 9.9% compared to the levels before the policy change. Our calculations suggest that corresponds to a decrease of .043 articles per individual per day. 27 This amounts to around 149.4 million fewer impressions across both channels during our observation period. 28 Assuming one advertisement per page and an average CPM of $10.50 29 suggests a loss of around $1.57 million in digital ads revenue.
On the other hand, the paywall change positively impacted subscriptions, more than making up for the lost ad revenue. There are two main mechanisms through which the paywall change could impact subscriptions-via the inability to read news articles due to constriction in the number of free articles, or via the increased variety of news articles accessible after the change. We combine estimates of increased subscription odds from Table 6 and Figure 8 with the total number of individuals that hit the paywall and were part of our treatment groups. A rough estimate suggests that the policy change impacted about 12023 subscriptions. This is around 31% of the total 38490 new subscribers gained during our study. Conservatively assuming a customer lifetime value of one year and the average cost of a subscription bundle of around $150, this amounts to a net revenue impact of about $1.80 million from subscriptions. Subtracting the losses in ads revenue, we calculate the net profit from paywall design changes during our seven month study to be at least $230, 000. 30

Implications for Paywall Design:
Paywalls do increase subscription rates, but the effect is moderated by the different paywall design parameters. Decreasing the amount of free content (quantity) and the ability to choose content across all the sections (exclusivity), as opposed to just a few sections, increase subscription rates. In addition, we also find complementarity between these two choices. This suggests managers should strategize their paywall design based on the different parameters of the paywalls, as opposed to just focusing on quantity alone, as most 27 The average total readership (mobile app + browser) before the change was ≈.437 articles per individual per day, leading to a total decrease of approximately .043(=.437 × .099) articles per individual per day.
newspapers currently do.
Based on our results, we suggest that newspapers should not focus on the short-term ads revenue maximization, as they face severe competition from Google and Facebook to attract ad dollars. Rather, they should strive to convert online visitors to paid subscribers by offering differentiated content modulated via digital paywalls. As we saw in this paper, digital paywalls which match free content offerings to readers' preferences by letting them choose the content they want to consume could be effective at increasing newspapers' subscriptions and revenues. Broadly, our results reinforce the popular sentiment in the media industry that the historical newspaper business model of maximizing advertising revenue is no longer viable.
Our work also highlights the importance of the design of the screening mechanism for a freemium product and its impact on influencing the propensities of users to upgrade to the premium product. The quasi-experimental variation in our study allowed us to tease apart the impact of both the quantity and exclusivity parameters on users' subscription propensities. And, as we saw, both significantly increased the chances of subscription by themselves and they further complemented each other's impact. Without a structural model we cannot pin-down the optimal screening policy, but it is clear that any such policy should account for the various design parameters and consider the interactions among them. We encourage such structural modeling in future work.

Limitations and Future Research
Although our work improves our understanding of several underpinnings of a modern day newspaper's digital strategy, it is not without limitations. First, we only explored two of the design parameters of the digital paywall. In order to fully navigate the strategic landscape, it is important to understand the trade-offs associated with other paywall design choices e.g. social sharing, personalized content offerings, curation access. Second, as far as the quasi-experiment in this paper is concerned, there could be some residual issues of intertemporal substitution by forward-looking content consumers as well as there might be some framing effects of the introductory trial period. Third, our results are for a relatively small window of time (about 7 months), almost equally split before and after the paywall change. As part of future work, it will be interesting to quantify the longterm impacts of paywall design changes on readership and subscriptions. It will also be interesting to see if our finding of the positive impact of quantity and diversity/exclusivity of free content in driving subscriptions persists over time. Fourth, our analysis did not consider all the anonymous visitors to the website. Future work should consider the impact of the ones-and-dones also. A final shortcoming of this paper is that owing to the size and popularity of NYT our findings might not generalize well to a small-market newspaper. We hope our work will inspire future research to overcome these limitations in pushing the limits of our understanding of the relationship between digital paywall design, content demand, and revenue.
Below we report the intent-to-treat (ITT) estimates of the various results in the main body of the paper. Essentially, these estimates ignore the differential updating of the mobile app by the readers (shown in Figure 3) and assume that everyone updated their mobile app on the very first day that they were eligible to upgrade.  192,293,146 192,293,146 192,293,146 Table 2: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Note: 1). Robust standard errors are clustered at the level of users. 2). The variable N otSubscribed i codes the non-subscribed users (anonymous and registered)-our treatment group-as 1, and the subscribed users as 0. It is the complement of the subscription status variable Subscribed i (= 1 − N otSubscribed i ).   Table 3: Difference-in-difference estimates of the impact of the paywall policy change on the readership of registered users **p < 0.05,***p < 0.01. Note: Robust standard errors are clustered at the level of users.   Table 4: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Same as Table 2 but the treatment group is only the sub-population of readers that tried to read more than 3 articles/day. Note: Robust standard errors are clustered at the level of users.   Table 5: Difference-in-difference estimates of the impact of the paywall policy change on the readership variables **p < 0.05,***p < 0.01. Same as Table 2 but the treatment group is only the sub-population of readers that consumed more diverse content after the paywall change. Note: Robust standard errors are clustered at the level of users.