Friday, February 29, 2008

Project idea: structured financial data

As they are, things are pretty good for everyday investors on Main Street. The United States Securities and Exchange Commission makes the filings of public companies readily available on its website. If someone were interested in researching a company thoroughly, pretty much all the information is there: how much debt a company is carrying around, what kind of revenues it's pulling in, what the company's assets are worth, and so on.

So, things are pretty good, but they could be a lot better.

Now, I say "investors" and not "traders" because traders will typically need more than just information from the SEC. It's also a clear distinction that will limit the scope of the project I'm about to propose so that it's very clear what will be needed (and, more importantly, what will not).

The current situation

All this data is in HTML. It's great for human readability. Right now, that's exactly what everyone's doing: reading them one by one and poring over them. This is probably because there are a lot of loose, interesting facts that are specific to each company's way of reporting things. But among all that loose, unstructured data are structured tables containing information like a company's balance sheet, its cash flow, and its earnings.

For a stock market related project my team and I are doing, such data in a structured format would go a long way. Currently, there are data providers who would provide this kind of thing for a monstrously large monthly fee, perhaps because they bundle it up with other things. But this is public data. Nobody should have to pay a fee for it.

I personally tried to write a parser to automatically pull this data. I was reduced to entering some data by hand, but with around 8,000 companies to deal with, that would take me a couple of years, full-time. There is no way that this is feasible. Entering data by hand was pretty easy, but really tedious. What a human being can deduce by a quick glance, an automatically programmed parser would have trouble with.

Such a task would be, in my opinion, well-suited for being built and maintained by an open community. It's a big job — not particularly hard, just a lot of work. It doesn't make sense for any one company to feed these in; the ones that have this data are charging high rates for it. Everyone should be able to benefit from it, so contributors should include people from various organizations.

A proposed solution

The best solution I can think of would be to take the lead on the project initially. There would be a project website running on a simple web application. Since the number of companies is known, there would be a frequently updated status report saying what percentage of the task is done. For example, the front page could say "3000 out of 7500 companies with complete data" or "400 companies with incomplete data." To ensure the project's open nature, there would be a daily or weekly dump of the database. That way, no one party will control the data completely. Additionally, the community would have a reasonably complete backup.

It is essentially a massive data entry project. Not so fun, but for the participants there is potential fame and fortune as well as a strong mission. Once the data is available, programmers and financial analysts can graph data that would be buried in tables and easily compare data for multiple companies in a timespan on the order of a few seconds, rather than the current process, which could take minutes, hours, or even days.

The fields that would be stored would be standard items on a company's balance sheet, earnings statement, and cash flow statement. Eventually, if further patterns can be found, more fields would be added. There is a much more ambitious initiative which attempts to address the problem of all the fields, which I will discuss below.


Right now, there's a major push underway to address this exact problem: XBRL, which stands for eXtensible Business Reporting Language. I looked into it several times, and have found that it is characterized by slow progress and a quiet reluctance from a lot of the parties involved. Companies in the XBRL test program are few and far between. A lot of big names in financial research are entrenched, and they depend on keeping this structured data as revenue sources. It should be noted that if XBRL somehow succeeds, this open community project I'm proposing would not be needed at all.

The eventual XBRL specification that would be used for public companies' SEC filings would have structured fields to address all the data reported, but this could be a potentially huge problem because of the learning curve for new software tools that report preparers will have to get used to.

If there are any interested readers out there, please leave a comment and let me know what you think. This idea has been in my head for a couple of weeks now and I'm certain I left some gaping holes in my discussion of what I have in mind for this project.

Thursday, February 21, 2008

The gospel for stock market junkies

One of the recurring struggles among people who are part of the evangelical Christian tradition is finding the balance between faith and good works — knowing where each of them fits in and what they're for. Since I've been trying to read three chapters of the Bible every morning to get my bearings before jumping into what are typically full days, I read over something that reminded me of this.

Now, the "full days" have been dedicated to learning about stock market investing and exploring various angles for the individual investor to take when selecting stocks.

The price of a company's stock takes thousands of factors into account. One way to split up these factors is to consider 1) the book value or intrinsic worth of the company, and 2) the other things like projected growth, macro-economic concerns, takeover speculation, and a host of other factors that are not directly tied in to the numbers that the company is posting.

The stocks of most healthy companies will trade at prices above their book value. But when they're hammered by a bad economy or growth slows down, they won't go down to zero if they're solid companies, because people called value investors will swoop in and "support" the stock at a price at or somewhere near its book value. It's like a safety net, and these value investors keep stock prices at certain levels so they don't go down too far.

In good times and when a company's prospects look good, its stock will trade above the price set just by the raw numbers, whether the numbers are already in the bag or just strongly projected. At every point though, it's supported by its raw value, and every other gain in price is tacked on as extra.

I see us like stocks, and our spiritual well-being like the price of a stock. What God does for us is He gives us that solid book value. To get our balance sheets and cash flow in decent working order, so to speak, we must have faith and accept the Lord's gift of forgiveness. Then we're fixed, and our intrinsic worth is affirmed; there's no taking that away. We may be battered by the storms of life but we don't sink down past a certain level, because God sees value in us.

That faith can be enough to get by, but just as investors want to see a stock take off, I think it's in our interest also to grow, and not just rest on our laurels like some stocks languish at seemingly constant lows.

Faith gives us something like the value investor-supported base and ensures we don't sink past a certain low, but doing good deeds gives us growth and dynamism. On top of that, our good deeds often feed back to strengthen our faith, just like a company making constant process improvements and aggressively pursuing business will see the results feed back into the bottom line and strengthen the base of its value.

Tuesday, February 19, 2008

Freedom is shabby and inconvenient

As much as we value liberty and tout the virtues of living in a free society, I humbly submit that eternal vigilance is not the only price we pay for our freedom.

Anywhere there's freedom, you will find disorder, inconvenience, and shabbiness. We allow our freedoms to be restricted because we want some sort of orderliness, convenience, or good aesthetics. These are good things, but conceding to rigidity often means losing substance or meaning.

Take homeowners' associations, for example. With pretty much any HOA, you cannot get away with painting your house bright pink, letting the weeds grow to four feet in your front yard, and then parking your car among the weeds. It may be how you prefer to live your life and run your household, but by joining the HOA you are giving up freedoms that you would otherwise have, in exchange for the assurance that your neighbor across the street will not paint his house a loud pastel purple, because pink is okay but purple is not. You may be fine with how shabby your house looks, but you'd rather your neighbor not make his house an eyesore for you. So what you end up with is people being restricted; everyone's content, but chances are that nobody's house really looks any different from anyone else's.

This year, I'm living in Irvine, California. Having lived in more freewheeling places in California, like Davis, or San Bernardino, or Monterey Park, I feel the contrast here. Everything is much more convenient and well-planned, but it's hard to find shops that aren't part of a large national chain. In place of character and charm, I've got well-run homogeneity — which I actually like, but which plenty of others detest. (Even San Bernardino, with neither charm, character, nor well-run homogeneity, had good Mexican food.)

It's kind of like this with people, too. People may dress sharp, but I've found that many people who have taken the time to develop character on a deeper level, or hone their talents, are not the most fashionable or up to date when it comes to the latest trends. Of course, if you were really smart and considerate, you wouldn't be a complete slob and smell utterly foul due to all the time you spent "finding yourself" with little thought of more trivial things. All I'm saying is you just have to look good enough.

Or take a look at any Linux distribution, and compare it to Apple's Mac OS X or even to Microsoft Windows. Running Linux is the ultimate celebration of freedom, if you are using a computer. But you have to go through a lot of trouble to get wireless networking going, or even printing. The font display on any Linux distribution is mediocre at best, but you have access to change all of it, if you are so inclined. The trouble is, it's not fun at all to make these changes, and a lot of these difficulties arise due to inconsistencies and disagreements among all the "free" people about how to do things. With a proprietary, tightly controlled platform like Apple's or Microsoft's, there is much less freedom to tinker, and you have to trust that they make things "just work" but that's precisely why one would give up one's freedom. Most people just want to make things work.

Sometimes, we give up absolute freedom because it's more convenient to do so. Those who restrict us should give us something good in return, but I hope we don't lock ourselves in so tightly that there's no going back. Liberty has a price, and on top of that, it's not always pretty.