title: Datomic query performance improvements
body: <p>I have a schema that looks similar to this in a Datomic database:</p>
  
  <pre><code>; --- tenant
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :tenant/guid
   :db/unique             :db.unique/identity
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :tenant/name
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :tenant/taks
   :db/valueType          :db.type/ref
   :db/cardinality        :db.cardinality/many
   :db.install/_attribute :db.part/db}
  
  ; --- task
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :task/guid
   :db/unique             :db.unique/identity
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :task/createdAt
   :db/valueType          :db.type/instant
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :task/name
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :task/subtasks
   :db/valueType          :db.type/ref
   :db/cardinality        :db.cardinality/many
   :db.install/_attribute :db.part/db}
  
  ; --- subtask
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :subtask/guid
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db/unique             :db.unique/identity
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :subtask/type
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :subtask/startedAt
   :db/valueType          :db.type/instant
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :subtask/completedAt
   :db/valueType          :db.type/instant
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :subtask/participants
   :db/valueType          :db.type/ref
   :db/cardinality        :db.cardinality/many
   :db.install/_attribute :db.part/db}
  
   ; --- participant
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :participant/guid
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db/unique             :db.unique/identity
   :db.install/_attribute :db.part/db}
  {:db/id                 #db/id[:db.part/db]
   :db/ident              :participant/name
   :db/valueType          :db.type/string
   :db/cardinality        :db.cardinality/one
   :db.install/_attribute :db.part/db}     
  </code></pre>
  
  <p>The tasks are pretty static over time but subtasks are added and removed on average about once per 5 minutes per task. I would say that each task on average has about 40 subtasks at any given time containing (almost always but there are a few exceptions) one participant. My sole purpose of using Datomic is to be able to see how tasks have evolved over time, i.e. I'd like to see the what a task looked like at a given time. To achieve I'm currently doing something similar to this:</p>
  
  <pre><code>(defn find-tasks-by-tenant-at-time 
      [conn tenant-guid ^long time-epoch]
      (let [db-conn (-&gt; conn d/db (d/as-of (Date. time-epoch)))
            task-ids (-&gt;&gt; (d/q '[:find ?taskIds
                                :in $ ?tenantGuid
                                :where
                                [?tenantId :tenant/guid ?tenantGuid]
                                [?tenantId :tenant/tasks ?taskIds]]
                              db-conn tenant-guid)
                         vec flatten)
            task-entities (map #(d/entity db-conn %) task-ids)
            dtos (map (fn [task]
                  (letfn [(participant-dto [participant]
                            {:id   (:participant/guid participant)
                             :name (:participant/name participant)})
                          (subtask-dto [subtask]
                            {:id           (:subtask/guid subtask)
                             :type         (:subtask/type subtask)
                             :participants (map participant-dto (:subtask/participants subtask))})]
                    {:id       (:task/guid task)
                     :name     (:task/name task)
                     :subtasks (map subtask-dto (:task/subtasks task))})) task-entities)]
            dtos))
  </code></pre>
  
  <p>Unfortunately this is extremely slow. It can take almost 60 seconds to return from this function if there are many tasks for a tenant (say 20) each containing roughly 40 subtasks. Am I doing something obviously wrong here? Is it possible to speed this up?</p>
  
  <p>Update:
  The entire dataset is roughly 2 Gb and the peer has 3.5Gb of memory (but it doesn't seem to make any difference if I decrease it to say 1.5 Gb) and the transactor has 1 Gb of memory. I'm using Datomic Free.</p>