{'fraction_non_alphanumeric': 0.03398919275667203, 'fraction_numerical': 0.01942828857814627, 'mean_word_length': 4.820888355342137, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting Risk from Financial Reports with Regression

We address a text regression problem: given a piece of text, predict a real-world continuous quantity associated with the text's meaning. In this work, the text is an SEC-mandated financial report published annually by a publicly-traded company, and the quantity to be predicted is volatility of stock returns, an empirical measure of financial risk. We apply well-known regression techniques to a large corpus of freely available financial reports, constructing regression models of volatility for the period following a report. Our models rival past volatility (a strong baseline) in predicting the target variable, and a single model that uses both can significantly outperform past volatility. Interestingly, our approach is more accurate for reports after the passage of the Sarbanes-Oxley Act of 2002, giving some evidence for the success of that legislation in making financial reports more informative.


Introduction
We consider a text regression problem: given a piece of text, predict a R-valued quantity associated with that text. Specifically, we use a company's annual financial report to predict the financial risk of investment in that company, as measured empirically by a quantity known as stock return volatility.
Predicting financial risk is of clear interest to anyone who invests money in stocks and central to modern portfolio choice. Financial reports are a government-mandated artifact of the financial world that-one might hypothesize-contain a large amount of information about companies and their value. Indeed, it is an important question whether mandated disclosures are informative, since they are meant to protect investors but are costly to produce.
The intrinsic properties of the problem are attractive as a test-bed for NLP research. First, there is no controversy about the usefulness or existential reality of the output variable (volatility). Statistical NLP often deals in the prediction of variables ranging from text categories to linguistic structures to novel utterances. While many of these targets are uncontroversially useful, they often suffer from evaluation difficulties and disagreement among annotators. The output variable in this work is a statistic summarizing facts about the real world; it is not subject to any kind of human expertise, knowledge, or intuition. Hence this prediction task provides a new, objective test-bed for any kind of linguistic analysis.
Second, many NLP problems rely on costly annotated resources (e.g., treebanks or aligned bilingual corpora). Because the text and historical financial data used in this work are freely available (by law) and are generated as a by-product of the American economy, old and new data can be obtained by anyone with relatively little effort.
In this paper, we demonstrate that predicting financial volatility automatically from a financial report is a novel, challenging, and easily evaluated natural language understanding task. We show that a very simple representation of the text (essentially, bags of unigrams and bigrams) can rival and, in combination, improve over a strong baseline that does not use the text. Analysis of the learned models provides insights about what can make this problem more or less difficult, and suggests that disclosurerelated legislation led to more transparent reporting.

Stock Return Volatility
Volatility is often used in finance as a measure of risk. It is measured as the standard deviation of a stock's returns over a finite period of time. A stock will have high volatility when its price fluctuates widely and low volatility when its price remains more or less constant.
Let r t = Pt P t−1 − 1 be the return on a given stock between the close of trading day t − 1 and day t, where P t is the (dividend-adjusted) closing stock price at date t. The measured volatility over the time period from day t − τ to day t is equal to the sample s.d.: wherer is the sample mean of r t over the period. In this work, the above estimate will be treated as the true output variable on training and testing data. It is important to note that predicting volatility is not the same as predicting returns or value. Rather than trying to predict how well a stock will perform, we are trying to predict how stable its price will be over a future time period. It is, by now, received wisdom in the field of economics that predicting a stock's performance, based on easily accessible public information, is difficult. This is an attribute of well-functioning (or "efficient") markets and a cornerstone of the so-called "efficient market hypothesis" (Fama, 1970). By contrast, the idea that one can predict a stock's level of risk using public information is uncontroversial and a basic assumption made by many economically sound pricing mod-els. A large body of research in finance suggests that the two types of quantities are very different: while predictability of returns could be easily traded away by the virtue of buying/selling stocks that are under-or over-valued (Fama, 1970), similar trades are much more costly to implement with respect to predictability of volatility (Dumas et al., 2007). By focusing on volatility prediction, we avoid taking a stance on whether or not the United States stock market is informationally efficient.

Problem Formulation
Given a text document d, we seek to predict the value of a continuous variable v. We do this via a parameterized function f : where w ∈ R d are the parameters or weights. Our approach is to learn a human-interpretable w from a collection of Support vector regression (Drucker et al., 1997) is a well-known method for training a regression model. SVR is trained by solving the following optimization problem: (3) where C is a regularization constant and controls the training error. 1 The training algorithm finds weights w that define a function f minimizing the (regularized) empirical risk.
Let h be a function from documents into some vector-space representation ⊆ R d . In SVR, the function f takes the form: where Equation 4 re-parameterizes f in terms of a kernel function K with "dual" weights α i . be seen as a similarity function between two documents. At test time, a new example is compared to a subset of the training examples (those with α i = 0); typically with SVR this set is sparse. With the linear kernel, the primal and dual weights relate linearly: The full details of SVR and its implementation are beyond the scope of this paper; interested readers are referred to Schölkopf and Smola (2002). SVM light (Joachims, 1999) is a freely available implementation of SVR training that we used in our experiments. 2

Dataset
In the United States, the Securities Exchange Commission mandates that all publicly-traded corporations produce annual reports known as "Form 10-K." The report typically includes information about the history and organization of the company, equity and subsidiaries, as well as financial information. These reports are available to the public and published on the SEC's web site. 3 The structure of the 10-K is specified in detail in the legislation. We have collected 54,379 reports published over the period 1996-2006 from 10,492 different companies. Each report comes with a date of publication, which is important for tying the text back to the financial variables we seek to predict.
From the perspective of predicting future events, one section of the 10-K is of special interest: Section 7, known as "management's discussion and analysis of financial conditions and results of operations" (MD&A), and in particular Subsection 7A, "quantitative and qualitative disclosures about market risk." Because Section 7 is where the most important forward-looking content is most likely to be found, we filter other sections from the reports. The filtering is done automatically using a short, hand-written Perl script that seeks strings loosely matching the Section 7, 7A, and 8 headers, finds the longest reasonable "Section 7" match (in words) of more than 1,000 whitespace-delineated tokens.
Section 7 typically begins with an introduction like this (from ABC's 1998 Form 10-K, before tokenization for readability; boldface added): The following discussion and analysis of ABC's consolidated financial condition and consolidated results of operation should be read in conjunction with ABC's Consolidated Financial Statements and Notes thereto included elsewhere herein.
This discussion contains certain forward-looking statements which involve risks and uncertainties. ABC's actual results could differ materially from the results expressed in, or implied by, such statements. See "Regarding Forward-Looking Statements." Not all of the documents downloaded pass the filter at all, and for the present work we have only used documents that do pass the filter. (One reason for the failure of the filter is that many 10-K reports include Section 7 "by reference," so the text is not directly included in the document.) In addition to the reports, we used the Center for Research in Security Prices (CRSP) US Stocks Database to obtain the price return series along with other firm characteristics. 4 We proceeded to calculate two volatilities for each firm/report observation: the twelve months prior to the report (v (−12) ) and the twelve months after the report (v (+12) ).
Tokenization was applied to the text, including punctuation removal, downcasing, collapsing all digit sequences, 5 and heuristic removal of remnant markup. Table 1 gives statistics on the corpora used in this research; this is a subset of the corpus for which there is no missing volatility information. The drastic increase in length during the 2002-2003 period might be explained by the passage by the US Congress of the Sarbanes-Oxley Act of 2002 (and related SEC and exchange rules), which imposed revised standards on reporting practices of publicly-traded companies in the US.

Baselines and Evaluation Method
Volatility displays an effect known as autoregressive conditional heteroscedasticity (Engle, 1982). This means that the variance in a stock's return tends to change gradually. Large changes in price are presaged by other changes, and periods of stability tend to continue. Volatility is, generally speaking, not constant, yet prior volatility (e.g., v (−12) ) is a very good predictor of future volatility (e.g., v (+12) ). At the granularity of a year, which we consider here because the 10-K reports are annual, there are no existing models of volatility that are widely agreed to be significantly more accurate than our historical volatility baseline. We tested a state-of-theart model known as GARCH(1, 1) (Engle, 1982;Bollerslev, 1986) and found that it was no stronger than our historical volatility baseline on this sample.
Throughout this paper, we will report performance using the mean squared error between the predicted and true log-volatilities: 6 where N is the size of the test set, given in Table 1.

Experiments
In our experiments, we vary h (the function that maps inputs to a vector space) and the subset of the 5 While numerical information is surely informative about risk, recall that our goal is to find indicators of risk expressed in the text; automatic predictors of risk from numerical data would use financial data streams directly, not text reports. 6 We work in the log domain because it is standard in finance, due to the dynamic range of actual volatilities; the distribution over log v across companies tends to have a bell shape. data used for training. We will always report performance over test sets consisting of one year's worth of data (the subcorpora described in Table 1). In this work, we focus on predicting the volatility over the year following the report (v (+12) ). In all experiments, = 0.1 and C is set using the default choice of SVM light , which is the inverse of the average of h(d) h(d) over the training data. 7

Feature Representation
We first consider how to represent the 10-K reports. We adopt various document representations, all using word features. Let M be the vocabulary size derived from the training data. 8 Let freq(x j ; d) denote the number of occurrences of the jth word in the vocabulary in document d.
where N is the number of documents in the training set. This is the classic "TFIDF" score.
• LOG1P: h j (d) = log(1 + freq(x j ; d)). Rather than normalizing word frequencies as for TF, this score dampens them with a logarithm. We also include a variant of LOG1P where terms are the union of unigrams and bigrams.
Note that each of these preserves sparsity; when freq(x j ; d) = 0, h j (d) = 0 in all cases. For interpretability of results, we use a linear kernel. The usual bias weight b is included. We found it convenient to work in the logarithmic domain for the predicted variable, predicting log v instead of v, since volatility is always nonnegative. In this setting, the predicted volatility takes the form: Because the goal of this work is to explore how text might be used to predict volatility, we also wish to see whether text adds information beyond what can be predicted using historical volatility alone (the baseline, v (−12) ). We therefore consider models augmented with an additional feature, defined as h M +1 = log v (−12) . Since this is historical information, it is always available when the 10-K report is published. These models are denoted TF+, TFIDF+, and LOG1P+.
The performance of these models, compared to the baseline from Section 5, is shown in Table 2. We used as training examples all reports from the five-year period preceding the test year (so six experiments on six different training and test sets are shown in the figure). We also trained SVR models on the single feature v (−12) , with and without bias weights (b in Eq. 7); these are usually worse and never signficantly better than the baseline.
Strikingly, the models that use only the text to predict volatility come very close to the historical baseline in some years. That a text-only method (LOG1P with bigrams) for predicting future risk comes within 5% of the error of a strong baseline (2003-6) shows promise for the overall approach. A combined model improves substantially over the baseline in four out of six years (2003-6), and this difference is usually robust to the representation used. Table 3 shows the most strongly weighted terms in each of the text-only LOG1P models (including bigrams). These weights are recovered using the relationship expressed in Eq. 5.

Training Data Effects
It is well known that more training data tend to improve the performance of a statistical method; how-ever, the standard assumption is that the training data are drawn from the same distribution as the test data. In this work, where we seek to predict the future based on data from past, that assumption is obviously violated. It is therefore an open question whether more data (i.e., looking farther into the past) is helpful for predicting volatility, or whether it is better to use only the most recent data. Table 4 shows how performance varies when one, two, or five years of historical training data are used, averaged across test years. In most cases, using more training data (from a longer historical period) is helpful, but not always. One interesting trend, not shown in the aggregate statistics of Table 4, is that recency of the training set affected performance much more strongly in earlier train/test splits (2001-3) than later ones . This experiment leads us to conclude that temporal changes in financial reporting make training data selection nontrivial. Changes in the macro economy and specific businesses make older reports less relevant for prediction. For example, regulatory changes like Sarbanes-Oxley, variations in the business cycle, and technological innovation like the Internet influence both the volatility and the 10-K text.

Effects of Sarbanes-Oxley
We noted earlier that the passage of the Sarbanes-Oxley Act of 2002, which sought to reform financial reporting, had a clear effect on the lengths of the 10-K reports in our collection. But are the reports more informative? This question is important, because producing reports is costly; we present an empirical argument based on our models that the legis- 1996-2000 1997-2001 1998-2002 1999-2003 2000-2004 2001-2005 net loss  Table 3: Most strongly-weighted terms in models learned from various time periods (LOG1P model with unigrams and bigrams). "#" denotes any digit sequence.  Table 4: MSE of volatility predictions using reports from varying historical windows (1, 2, and 5 years), microaveraged across six train/test scenarios. Boldface marks best in a row. The historical baseline achieves 0.1576 MSE (see Table 2). lation has actually been beneficial.
Our experimental results in Section 6.1, in which volatility in the years 2004-2006 was more accurately predicted from the text than in 2001-2002, suggest that the Sarbanes-Oxley Act led to more informative reports. We compared the learned weights (LOG1P+, unigrams) between the six overlapping five-year windows ending in 2000-2005; measured in L 1 distance, these were, in consecutive order, 52.2, 59.9, 60.7, 55. The same pattern held when only words appearing in all five models were considered. Variation in the recency/training set size tradeoff ( §6.2), particularly during 2002-3, also suggests that there were substantial changes in the reports during that time.

Qualitative Evaluation
One of the advantages of a linear model is that we can explore what each model discovers about different unigram and bigram terms. Some manually selected examples of terms whose learned weights (w) show interesting variation patterns over time are shown in Figure 1, alongside term frequency patterns, for the text-only LOG1P model (with bigrams). These examples were suggested by experts in finance from terms with weights that were both large and variable (across training sets).
A particularly interesting case, in light of Sarbanes-Oxley, is the term accounting policies.  cates lower volatility. The frequency of the term also increases drastically over the same period, suggesting that the earlier weights may have been inflated. A more striking example is estimates, which averages one occurrence per document even in the 1996-2000 period, experiences the same term frequency explosion, and goes through a similar weight change, from strongly indicating high volatility to strongly indicating low volatility. As a second example, consider the terms mortgages and reit (Real Estate Investment Trust, a tax designation for businesses that invest in real estate). Given the importance of the housing and mortgage market over the past few years, it is interesting to note that the weight on both of these terms increases over the period from a strong low volatility term to a weak indicator of high volatility. It will be interesting to see how the dramatic decline in housing prices in late 2007, and the fallout created in credit markets in 2008, is reflected in future models.
Finally, notice that high margin and low margin, whose frequency patterns are fairly flat "switch places," over the sample: first indicating high and low volatility, respectively, then low and high. There is no a priori reason to expect high or low margins would be associated with high or low stock volatility. However, this is an interesting example where bigrams are helpful (the word margin by itself is uninformative) and indicates that predicting risk is highly time-dependent.

Delisting
An interesting but relatively infrequent phenomenon is the delisting of a company, i.e., when it ceases to be traded on a particular exchange due to dissolution after bankruptcy, a merger, or violation of exchange rules. The relationship between volatility and delisting has been studied by Merton (1974), among others. Our dataset includes a small number of cases where the volatility figures for the period following the publication of a 10-K report are unavailable because the company was delisted. Learning to predict delisting is extremely difficult because fewer than 4% of the 2001-6 10-K reports precede delisting.
Using the LOG1P representation, we built a linear SVM classifier for each year in 2001-6 (trained on the five preceding years' data) to predict whether a company will be delisted following its 10-K report. Performance for various precision measures is shown in Table 5 (%)  35  42  44  36  31  16 6 bulletin, creditors, dip, otc 5 court 4 chapter, debtors, filing, prepetition 3 bankruptcy 2 concern, confirmation, going, liquidation 1 debtorinpossession, delisted, nasdaq, petition Table 5: Left: precision of delisting predictions. The "oracle F 1 " row shows the maximal F 1 score obtained for any n. Right: Words most strongly predicting delisting of a company. The number is how many of the six years (2001-6) the word is among the ten most strongly weighted. There were no clear patterns across years for words predicting that a company would not be delisted. The word otc refers to "over-the-counter" trading, a high-risk market.
above 75% precision at 10% recall. Our best (oracle) F 1 scores occur in 2002 and 2003, suggesting again a difference in reports around Sarbanes-Oxley. Table 5 shows words associated with delisting.

Related Work
In NLP, regression is not widely used, since most natural language-related data are discrete. Regression methods were pioneered by Yang and Chute (1992) and Yang and Chute (1993) for information retrieval purposes, but the predicted continuous variable was not an end in itself in that work. Blei and McAuliffe (2007) used latent "topic" variables to predict movie reviews and popularity from text. Lavrenko et al. (2000b) and Lavrenko et al. (2000a) modeled influences between text and time series financial data (stock prices) using language models. Farther afield, Albrecht and Hwa (2007) used SVR to train machine translation evaluation metrics to match human evaluation scores and compared techniques using correlation. Regression has also been used to order sentences in extractive summarization (Biadsy et al., 2008). While much of the information relevant for investors is communicated through text (rather than numbers), only recently is this link explored. Some papers relate news articles to earning forecasts, stock returns, volatility, and volume (Koppel and Shtrimberg, 2004;Tetlock, 2007;Tetlock et al., 2008;Gaa, 2007;Engelberg, 2007). Das and Chen (2001) and Antweiler and Frank (2004) ask whether messages posted on message boards can help explain stock performance, while Li (2005) measures the association between frequency of words associated with risk and subsequent stock returns. Weiss-Hanley and Hoberg (2008) study initial public offering disclosures using word statistics. Many researchers have focused the related problem of predicting sentiment and opinion in text (Pang et al., 2002;Wiebe and Riloff, 2005), sometimes connected to extrinsic values like prediction markets (Lerman et al., 2008).
In contrast to text regression, text classification comprises a widely studied set of problems involving the prediction of categorial variables related to text. Applications have included the categorization of documents by topic (Joachims, 1998), language (Cavnar and Trenkle, 1994), genre (Karlgren and Cutting, 1994), author (Bosch and Smith, 1998), sentiment (Pang et al., 2002), and desirability (Sahami et al., 1998). Text categorization has served as a test application for nearly every machine learning technique for discrete classification.

Conclusion
We have introduced and motivated a new kind of task for NLP: text regression, in which text is used to make predictions about measurable phenomena in the real world. We applied the technique to predicting financial volatility from companies' 10-K reports, and found text regression model predictions to correlate with true volatility nearly as well as historical volatility, and a combined model to perform even better. Further, improvements in accuracy and changes in models after the passage of the Sarbanes-Oxley Act suggest that financial reporting reform has had interesting and measurable effects.