{"action":"created","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1119","repository_url":"https://api.github.com/repos/markovmodel/PyEMMA","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1119/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1119/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1119/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/1119","id":235624260,"number":1119,"title":"A database (?) error on pyemma.coordinates.source()","user":{"login":"rafwiewiora","id":12088757,"avatar_url":"https://avatars1.githubusercontent.com/u/12088757?v=3","gravatar_id":"","url":"https://api.github.com/users/rafwiewiora","html_url":"https://github.com/rafwiewiora","followers_url":"https://api.github.com/users/rafwiewiora/followers","following_url":"https://api.github.com/users/rafwiewiora/following{/other_user}","gists_url":"https://api.github.com/users/rafwiewiora/gists{/gist_id}","starred_url":"https://api.github.com/users/rafwiewiora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafwiewiora/subscriptions","organizations_url":"https://api.github.com/users/rafwiewiora/orgs","repos_url":"https://api.github.com/users/rafwiewiora/repos","events_url":"https://api.github.com/users/rafwiewiora/events{/privacy}","received_events_url":"https://api.github.com/users/rafwiewiora/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-06-13T17:09:43Z","updated_at":"2017-06-13T20:32:04Z","closed_at":null,"body":"Hit a problem while trying to do k-means on my TICA-projected trajs, `pyemma.coordinates.source(glob.glob('*.npy'))` errors, while it worked perfectly ok on the featurized trajs (arrays are even same dimensions, keeping all TICs). \r\n\r\nHere's the error:\r\n```python\r\nIn [108]: source = pyemma.coordinates.source(glob.glob('*.npy'))\r\nObtaining file info:   1% (  76/5020) [#                                                                             ] eta 03:02 |13-06-17 12:58:31 pyemma.coordinates.data.util.traj_info_cache INFO     Cleaning database because it has too much entries or is too large.\r\nEntries: 50084. Size: 24.15MB. Configured max_entires: 50000. Max_size: 500MB\r\n---------------------------------------------------------------------------\r\nKeyError                                  Traceback (most recent call last)\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_cache.py in __getitem__(self, filename_reader_tuple)\r\n    186         try:\r\n--> 187             info = self._database.get(key)\r\n    188             if not isinstance(info, TrajInfo):\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_backends.py in get(self, key)\r\n    222         if not row:\r\n--> 223             raise KeyError()\r\n    224         info = self._create_traj_info(row)\r\n\r\nKeyError: \r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nAssertionError                            Traceback (most recent call last)\r\n<ipython-input-108-9d3c8ffd8f2f> in <module>()\r\n----> 1 source = pyemma.coordinates.source(glob.glob('*.npy'))\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/api.py in source(inp, features, top, chunk_size, **kw)\r\n    346             isinstance(inp, (list, tuple))\r\n    347             and (any(isinstance(item, (list, tuple, _string_types)) for item in inp) or len(inp) is 0)):\r\n--> 348         reader = create_file_reader(inp, top, features, chunk_size=chunk_size if chunk_size is not None else 100, **kw)\r\n    349 \r\n    350     elif isinstance(inp, _np.ndarray) or (isinstance(inp, (list, tuple))\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/reader_utils.py in create_file_reader(input_files, topology, featurizer, chunk_size, **kw)\r\n    103                 else:\r\n    104                     if suffix in ['.npy', '.npz']:\r\n--> 105                         reader = NumPyFileReader(input_list, chunksize=chunk_size)\r\n    106                     # otherwise we assume that given files are ascii tabulated data\r\n    107                     else:\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/numpy_filereader.py in __init__(self, filenames, chunksize, mmap_mode)\r\n     62 \r\n     63         self.mmap_mode = mmap_mode\r\n---> 64         self.filenames = filenames\r\n     65 \r\n     66     def _create_iterator(self, skip=0, chunk=0, stride=1, return_trajindex=False, cols=None):\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/datasource.py in filenames(self, filename_list)\r\n    114             for filename in filename_list:\r\n    115                 if config.use_trajectory_lengths_cache:\r\n--> 116                     info = TrajectoryInfoCache.instance()[filename, self]\r\n    117                 else:\r\n    118                     info = self._get_traj_info(filename)\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_cache.py in __getitem__(self, filename_reader_tuple)\r\n    204             info.abs_path = abs_path\r\n    205             # store info in db\r\n--> 206             self.__setitem__(info)\r\n    207 \r\n    208             # save forcefully now\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_cache.py in __setitem__(self, traj_info)\r\n    245 \r\n    246     def __setitem__(self, traj_info):\r\n--> 247         self._database.set(traj_info)\r\n    248 \r\n    249     def clear(self):\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_backends.py in set(self, traj_info)\r\n    215                             % (self.num_entries, (current_size*1.0 / 1024**2),\r\n    216                                config.traj_info_max_entries, config.traj_info_max_size))\r\n--> 217                 self._clean(n=self.clean_n_entries)\r\n    218 \r\n    219     def get(self, key):\r\n\r\n/cbio/jclab/home/rafal.wiewiora/anaconda3/lib/python3.6/site-packages/pyemma/coordinates/data/util/traj_info_backends.py in _clean(self, n)\r\n    349         cur = self._database.execute(stmnt)\r\n    350         self._database.commit()\r\n--> 351         assert cur.rowcount == len(ids), \"deleted not as many rows(%s) as desired(%s)\" %(cur.rowcount, len(ids))\r\n    352 \r\n    353         # iterate over all LRU databases and delete those ids, we've just deleted from the main db.\r\n\r\nAssertionError: deleted not as many rows(1) as desired(15025)\r\n```\r\n\r\nInterestingly, this hack makes it work:\r\n\r\n```python\r\nfnames = glob.glob('*.npy')\r\n\r\nfor i in np.arange(1, len(fnames)):                   \r\n    try:\r\n        source = pyemma.coordinates.source(fnames[0:i])\r\n    except:\r\n        source = pyemma.coordinates.source(fnames[0:i])\r\n```\r\n\r\n(i.e keep making the list of files longer by one and repeat when it fails) - I tried this once, started failing at 43 files, and then on every single file addition if fails once, works another time and so on:\r\n\r\n```python\r\nObtaining file info:  97% (42/43) [################################################################################  ] eta 00:01 |13-06-17 12:56:33 pyemma.coordinates.data.util.traj_info_cache INFO     Cleaning database because it has too much entries or is too large.\r\nEntries: 50057. Size: 24.15MB. Configured max_entires: 50000. Max_size: 500MB\r\nObtaining file info:  97% (43/44) [################################################################################  ] eta 00:01 -13-06-17 12:56:35 pyemma.coordinates.data.util.traj_info_cache INFO     Cleaning database because it has too much entries or is too large.\r\n(...)\r\n```\r\n\r\nOnly got it up to 88 files now, and keeps doing this fail-work pattern. I think I'm going to try running with this hack on all 5020 trajectories and seeing if that works (EDIT: this of course turned out to take way too much time to be reasonable) and if I can get those k-means running asap. "},"comment":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/comments/308239571","html_url":"https://github.com/markovmodel/PyEMMA/issues/1119#issuecomment-308239571","issue_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1119","id":308239571,"user":{"login":"marscher","id":170287,"avatar_url":"https://avatars3.githubusercontent.com/u/170287?v=3","gravatar_id":"","url":"https://api.github.com/users/marscher","html_url":"https://github.com/marscher","followers_url":"https://api.github.com/users/marscher/followers","following_url":"https://api.github.com/users/marscher/following{/other_user}","gists_url":"https://api.github.com/users/marscher/gists{/gist_id}","starred_url":"https://api.github.com/users/marscher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marscher/subscriptions","organizations_url":"https://api.github.com/users/marscher/orgs","repos_url":"https://api.github.com/users/marscher/repos","events_url":"https://api.github.com/users/marscher/events{/privacy}","received_events_url":"https://api.github.com/users/marscher/received_events","type":"User","site_admin":false},"created_at":"2017-06-13T20:32:04Z","updated_at":"2017-06-13T20:32:04Z","body":"On 13.06.2017 22:12, Rafal Wiewiora wrote:\n> Ok, so |rm -rf ~/.pyemma| resolved the 50000 trajectories problem - how \n> do I increase the database size though?\nhttp://www.emma-project.org/latest/Configuration.html#pyemma.util._config.Config.traj_info_max_entries\n\npyemma.config.save()\n> \n> I thought the problem with some jobs failing if many run at the same \n> time was that they were starting at the exact same time and at the same \n> point accessing the database - but from the failures I saw again with \n> this approach: (all failing with |sqlite3.OperationalError: database is \n> locked|)\nYes, but the database will be locked even for a longer time, if the \ncapacity is reached.\n> \n> |-bash-4.1$ qstat -u rafal.wiewiora hal-sched1.local: Req'd Req'd Elap \n> Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time \n> ----------------------- ----------- -------- ---------------- ------ \n> ----- ------ ------ --------- - --------- 8627576.hal-sched1.loc \n> rafal.wiewi batch dist_cross_tica 57772 1 1 20gb 144:00:00 R 97:20:00 \n> 8634457.hal-sched1.loc rafal.wiewi batch dih_kmeans 85293 1 16 20gb \n> 144:00:00 C -- 8634458.hal-sched1.loc rafal.wiewi batch dih_kmeans 23037 \n> 1 16 20gb 144:00:00 C -- 8634459.hal-sched1.loc rafal.wiewi batch \n> dih_kmeans 85382 1 16 20gb 144:00:00 R 00:25:47 8634460.hal-sched1.loc \n> rafal.wiewi batch dih_kmeans 23115 1 16 20gb 144:00:00 C -- \n> 8634461.hal-sched1.loc rafal.wiewi batch dih_kmeans 85694 1 16 20gb \n> 144:00:00 C -- 8634462.hal-sched1.loc rafal.wiewi batch dih_kmeans 24550 \n> 1 16 20gb 144:00:00 R 00:21:19 8634463.hal-sched1.loc rafal.wiewi batch \n> dih_kmeans 11864 1 16 20gb 144:00:00 C -- 8634464.hal-sched1.loc \n> rafal.wiewi batch dih_kmeans 31914 1 16 20gb 144:00:00 C -- \n> 8634465.hal-sched1.loc rafal.wiewi batch dih_kmeans 18271 1 16 20gb \n> 144:00:00 C -- 8634466.hal-sched1.loc rafal.wiewi batch dih_kmeans 2758 \n> 1 16 20gb 144:00:00 R 00:19:10 8634467.hal-sched1.loc rafal.wiewi batch \n> dih_kmeans 5000 1 16 20gb 144:00:00 C -- 8634468.hal-sched1.loc \n> rafal.wiewi batch dih_kmeans 8317 1 16 20gb 144:00:00 C -- \n> 8634469.hal-sched1.loc rafal.wiewi batch dih_kmeans 18393 1 16 20gb \n> 144:00:00 C -- 8634470.hal-sched1.loc rafal.wiewi batch dih_kmeans 20834 \n> 1 16 20gb 144:00:00 R 00:16:51 8634471.hal-sched1.loc rafal.wiewi batch \n> dih_kmeans 19457 1 16 20gb 144:00:00 R 00:16:20 8634472.hal-sched1.loc \n> rafal.wiewi batch dih_kmeans 21294 1 16 20gb 144:00:00 C -- \n> 8634473.hal-sched1.loc rafal.wiewi batch dih_kmeans 32242 1 16 20gb \n> 144:00:00 C -- 8634474.hal-sched1.loc rafal.wiewi batch dih_kmeans 5502 \n> 1 16 20gb 144:00:00 R 00:14:30 8634476.hal-sched1.loc rafal.wiewi batch \n> dih_kmeans 12612 1 16 20gb 144:00:00 C -- |\n> \n> Before this I see a very long ETA for the |pyemma.coordinates.source()|:\n> \n> |-bash-4.1$ tail -f dih_kmeans.o8634458 Obtaining file info: 86% \n> (4356/5020) [######################## ] eta 29:35 |^C -bash-4.1$ tail -f \n> dih_kmeans.o8634459 Obtaining file info: 81% (4074/5020) \n> [###################### ] eta 39:09 -^C -bash-4.1$ tail -f \n> dih_kmeans.o8634476 Obtaining file info: 0% ( 25/5020) [ ] eta 2:56:22 /^C |\n> \n> so it seems like what I really need to do is to wait until a previous \n> job has finished the |pyemma.coordinates.source()| before starting a \n> next one?\nNo actually not, one job reads/writes to the database, then frees the \nlock immediately. I've only tested this with < 20 jobs.\n\nYou can also disable this caching feature completely (see the config \ndocumentation).\n\nIn principal there is no good workaround other than what I've already \npointed out.\n\n"}}
{"actor":{"display_login":"marscher"}}