title: How to manage a huge Database (MySQL)
body: <p>I wrote a statistics/tracking platform to monitor my websites activity. For each user there is an entry created in a table with some relevant info: IP, Ref, GeoLocate countrycode etc.</p>
  
  <p>The database contains well over 7MM results and has a size of about 4GB.</p>
  
  <p>Now, I have a script which has to do SUM queries to get the numbers of impressions, clicks etc. for a defined time interval ($from_date &amp; $to_date).</p>
  
  <pre><code>$query = "SELECT COUNT(ip) as tot, sum(clicked=0) as a0, sum(clicked=1) as a1, sum(converted=0) as a2, sum(converted=1) as a3 FROM tf_data WHERE cid='".$cid."' and stamp between ".$from_date." and ".$to_date." "; 
  </code></pre>
  
  <p>Alone this query takes FOREVER to load, even tho I'm on a decent VPS Machine. I have some other queries like this in my summary page, which causes the script to timeout frequently (REQUEST TIMEOUT: This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.)</p>
  
  <p>What should I do? How can I handle that huge amout of data? Should I create a separate table and run a cron job to insert/update the impressions number, clicks number...?</p>
  
  <p>How is such a thing normally done?</p>
  
  <p>Thanks as always!</p>
  
  <p>EDIT: TABLE STRUCTURE:</p>
  
  <pre><code>CREATE TABLE `tf_data` (
  `click_id` int(11) NOT NULL AUTO_INCREMENT,
  `ip` varchar(225) NOT NULL,
  `agent` text NOT NULL,
  `referer` text NOT NULL,
  `stamp` text NOT NULL,
  `country` varchar(30) NOT NULL,
  `src` text NOT NULL,
  `adspot` varchar(250) NOT NULL,
  `cid` text NOT NULL,
  `adid` text NOT NULL,
  `lp` varchar(250) NOT NULL,
  `offer` int(11) NOT NULL,
  `clicked` int(11) NOT NULL,
  `converted` int(11) NOT NULL,
  `converted2` int(11) NOT NULL,
  `price` varchar(255) NOT NULL,
  PRIMARY KEY (`click_id`),
  UNIQUE KEY `ip` (`ip`)
  ) ENGINE=MyISAM AUTO_INCREMENT=9599999 DEFAULT CHARSET=latin1
  </code></pre>