{"action":"created","issue":{"url":"https://api.github.com/repos/mudrod/mudrod/issues/163","repository_url":"https://api.github.com/repos/mudrod/mudrod","labels_url":"https://api.github.com/repos/mudrod/mudrod/issues/163/labels{/name}","comments_url":"https://api.github.com/repos/mudrod/mudrod/issues/163/comments","events_url":"https://api.github.com/repos/mudrod/mudrod/issues/163/events","html_url":"https://github.com/mudrod/mudrod/issues/163","id":227446303,"number":163,"title":"Full Ingest - Missing metadata_session_coocurrence_matrix.csv","user":{"login":"fgreg","id":1487133,"avatar_url":"https://avatars1.githubusercontent.com/u/1487133?v=3","gravatar_id":"","url":"https://api.github.com/users/fgreg","html_url":"https://github.com/fgreg","followers_url":"https://api.github.com/users/fgreg/followers","following_url":"https://api.github.com/users/fgreg/following{/other_user}","gists_url":"https://api.github.com/users/fgreg/gists{/gist_id}","starred_url":"https://api.github.com/users/fgreg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fgreg/subscriptions","organizations_url":"https://api.github.com/users/fgreg/orgs","repos_url":"https://api.github.com/users/fgreg/repos","events_url":"https://api.github.com/users/fgreg/events{/privacy}","received_events_url":"https://api.github.com/users/fgreg/received_events","type":"User","site_admin":false},"labels":[{"id":384457378,"url":"https://api.github.com/repos/mudrod/mudrod/labels/bug","name":"bug","color":"ee0701","default":true},{"id":422137874,"url":"https://api.github.com/repos/mudrod/mudrod/labels/core","name":"core","color":"0e8a16","default":false},{"id":384457381,"url":"https://api.github.com/repos/mudrod/mudrod/labels/help%20wanted","name":"help wanted","color":"128A0C","default":true}],"state":"open","locked":false,"assignee":{"login":"quintinali","id":8673393,"avatar_url":"https://avatars1.githubusercontent.com/u/8673393?v=3","gravatar_id":"","url":"https://api.github.com/users/quintinali","html_url":"https://github.com/quintinali","followers_url":"https://api.github.com/users/quintinali/followers","following_url":"https://api.github.com/users/quintinali/following{/other_user}","gists_url":"https://api.github.com/users/quintinali/gists{/gist_id}","starred_url":"https://api.github.com/users/quintinali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quintinali/subscriptions","organizations_url":"https://api.github.com/users/quintinali/orgs","repos_url":"https://api.github.com/users/quintinali/repos","events_url":"https://api.github.com/users/quintinali/events{/privacy}","received_events_url":"https://api.github.com/users/quintinali/received_events","type":"User","site_admin":false},"assignees":[{"login":"quintinali","id":8673393,"avatar_url":"https://avatars1.githubusercontent.com/u/8673393?v=3","gravatar_id":"","url":"https://api.github.com/users/quintinali","html_url":"https://github.com/quintinali","followers_url":"https://api.github.com/users/quintinali/followers","following_url":"https://api.github.com/users/quintinali/following{/other_user}","gists_url":"https://api.github.com/users/quintinali/gists{/gist_id}","starred_url":"https://api.github.com/users/quintinali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quintinali/subscriptions","organizations_url":"https://api.github.com/users/quintinali/orgs","repos_url":"https://api.github.com/users/quintinali/repos","events_url":"https://api.github.com/users/quintinali/events{/privacy}","received_events_url":"https://api.github.com/users/quintinali/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2017-05-09T17:50:20Z","updated_at":"2017-05-09T17:59:06Z","closed_at":null,"body":"Running full log ingestion results in the following error:\r\n\r\n```\r\n2017-05-09 09:48:48,582 INFO  server.ServerConnector (AbstractConnector.java:doStart(266)) - Started ServerConnector@47c7a9e5{HTTP/1.1}{0.0.0.0:4041}\r\n2017-05-09 09:48:48,582 INFO  server.Server (Server.java:doStart(379)) - Started @4290ms\r\n2017-05-09 09:48:48,865 INFO  handler.ContextHandler (ContextHandler.java:doStart(744)) - Started o.s.j.s.ServletContextHandler@64763e49{/metrics/json,null,AVAILABLE}\r\n2017-05-09 09:48:48,986 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:<init>(51)) - Started Mudrod Weblog Discovery Engine.\r\n2017-05-09 09:48:48,986 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:preprocess(98)) - Starting Web log preprocessing.\r\n2017-05-09 09:48:48,987 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:preprocess(106)) - Processing logs dated 201611.gz\r\n2017-05-09 09:48:49,036 INFO  pre.ImportLogFile (ImportLogFile.java:execute(74)) - Starting Log Import 201611.gz\r\n2017-05-09 09:48:50,010 INFO  util.Version (Version.java:logVersion(108)) - Elasticsearch Hadoop v5.2.0 [d85a257f9f]\r\n2017-05-09 09:48:50,355 INFO  rdd.EsRDDWriter (RestService.java:createWriter(562)) - Writing to [podaaclog201611.gz/rawhttp]\r\n2017-05-09 09:56:59,924 INFO  rdd.EsRDDWriter (RestService.java:createWriter(562)) - Writing to [podaaclog201611.gz/rawftp]\r\n2017-05-09 10:00:10,210 INFO  pre.ImportLogFile (ImportLogFile.java:execute(78)) - Log Import complete. Time elapsed 681 seconds\r\n2017-05-09 10:00:10,218 INFO  pre.CrawlerDetection (CrawlerDetection.java:execute(93)) - Starting Crawler detection rawhttp.\r\n2017-05-09 10:00:12,695 INFO  pre.CrawlerDetection (CrawlerDetection.java:checkByRateInParallel(159)) - Original User count: 6365\r\n[Stage 7:>                                                         (0 + 4) / 96]2017-05-09 10:04:22,330 INFO  pre.CrawlerDetection (CrawlerDetection.java:checkByRateInParallel(177)) - User count: 6317\r\n2017-05-09 10:04:22,335 INFO  pre.CrawlerDetection (CrawlerDetection.java:execute(102)) - Crawler detection complete. Time elapsed 252 seconds\r\n2017-05-09 10:04:22,352 INFO  pre.SessionGenerator (SessionGenerator.java:execute(66)) - Starting Session Generation.\r\n2017-05-09 10:16:47,998 INFO  pre.SessionGenerator (SessionGenerator.java:genSessionByRefererInParallel(224)) - Initial Session count: 6084392\r\n2017-05-09 10:36:13,317 INFO  pre.SessionGenerator (SessionGenerator.java:execute(71)) - Session generating complete. Time elapsed 1910 seconds.\r\n2017-05-09 10:36:13,415 INFO  pre.SessionStatistic (SessionStatistic.java:execute(66)) - Starting Session Summarization.\r\n[Stage 10:====================================================>   (90 + 4) / 96]2017-05-09 10:36:57,595 INFO  pre.SessionStatistic (SessionStatistic.java:processSessionInParallel(172)) - Final Session count: 893\r\n                                                                                2017-05-09 10:36:57,596 INFO  pre.SessionStatistic (SessionStatistic.java:execute(79)) - Session Summarization complete. Time elapsed 44 seconds.\r\n2017-05-09 10:36:57,599 INFO  pre.RemoveRawLog (RemoveRawLog.java:execute(41)) - Starting raw log removal.\r\n2017-05-09 10:45:14,315 INFO  pre.RemoveRawLog (RemoveRawLog.java:execute(47)) - Raw log removal complete. Time elapsed 496 seconds.\r\n2017-05-09 10:45:14,318 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:preprocess(125)) - Web log preprocessing for logs dated 201611.gz complete. Time elapsed 3385 seconds.\r\n2017-05-09 10:45:14,370 INFO  pre.HistoryGenerator (HistoryGenerator.java:execute(47)) - Starting HistoryGenerator...\r\n/Users/greguska/data/mudrod/201611/UserHistoryMatrix.csv\r\npodaaclog\r\n2017-05-09 10:45:14,525 INFO  pre.HistoryGenerator (HistoryGenerator.java:execute(53)) - HistoryGenerator complete. Time elapsed 0 seconds\r\n2017-05-09 10:45:14,528 INFO  pre.ClickStreamGenerator (ClickStreamGenerator.java:execute(48)) - Starting ClickStreamGenerator...\r\npodaaclog\r\n[podaaclog201611.gz]click stream number:1296\r\n2017-05-09 10:45:19,730 INFO  pre.ClickStreamGenerator (ClickStreamGenerator.java:execute(65)) - ClickStreamGenerator complete. Time elapsed 5 seconds.\r\n2017-05-09 10:45:19,730 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:preprocess(134)) - Web log preprocessing (user history and clickstream) complete.\r\n2017-05-09 10:45:19,730 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:process(182)) - Starting Web log processing.\r\n2017-05-09 10:45:19,731 INFO  process.ClickStreamAnalyzer (ClickStreamAnalyzer.java:execute(49)) - Starting ClickStreamAnalyzer...\r\n2017-05-09 10:45:20,444 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data is not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:45:20,490 INFO  jni.JniLoader (JniLoader.java:liberalLoad(90)) - successfully loaded /var/folders/rz/qtbsq0bs49z341cfvmpnnlzd3yz9g6/T/jniloader97316236944444993netlib-native_system-osx-x86_64.jnilib\r\n2017-05-09 10:45:22,199 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data was not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:45:23,684 INFO  process.ClickStreamAnalyzer (ClickStreamAnalyzer.java:execute(66)) - ClickStreamAnalyzer complete. Time elapsed: 3s\r\n2017-05-09 10:45:23,685 INFO  process.UserHistoryAnalyzer (UserHistoryAnalyzer.java:execute(47)) - Starting UserHistoryAnalyzer...\r\n2017-05-09 10:45:23,948 INFO  process.UserHistoryAnalyzer (UserHistoryAnalyzer.java:execute(56)) - UserHistoryAnalyzer complete. Time elapsed: 0s\r\n2017-05-09 10:45:23,948 INFO  discoveryengine.WeblogDiscoveryEngine (WeblogDiscoveryEngine.java:process(192)) - Web log processing complete. Time elaspsed 4 seconds.\r\n2017-05-09 10:45:23,951 INFO  discoveryengine.MetadataDiscoveryEngine (MetadataDiscoveryEngine.java:preprocess(46)) - *****************Metadata preprocessing starts******************\r\n2017-05-09 10:45:23,951 INFO  pre.ApiHarvester (ApiHarvester.java:execute(54)) - Starting Metadata harvesting.\r\n2017-05-09 10:45:23,953 INFO  pre.ApiHarvester (ApiHarvester.java:harvestMetadatafromWeb(119)) - *****************Metadata downloading starts******************\r\n2017-05-09 10:45:24,520 INFO  pre.ApiHarvester (ApiHarvester.java:harvestMetadatafromWeb(137)) - Directory is created!\r\n2017-05-09 10:45:39,586 INFO  pre.ApiHarvester (ApiHarvester.java:harvestMetadatafromWeb(166)) - *****************Metadata downloading finishes******************\r\n2017-05-09 10:45:41,267 INFO  pre.ApiHarvester (ApiHarvester.java:execute(66)) - Metadata harvesting completed. Time elapsed: 17\r\n2017-05-09 10:45:41,268 INFO  discoveryengine.MetadataDiscoveryEngine (MetadataDiscoveryEngine.java:preprocess(53)) - *****************Metadata preprocessing ends******************Took 17s\r\n2017-05-09 10:45:41,268 INFO  discoveryengine.MetadataDiscoveryEngine (MetadataDiscoveryEngine.java:process(60)) - *****************Metadata processing starts******************\r\n2017-05-09 10:45:41,269 INFO  pre.MatrixGenerator (MatrixGenerator.java:execute(61)) - *****************Metadata matrix starts******************\r\n2017-05-09 10:45:43,350 INFO  pre.MatrixGenerator (MatrixGenerator.java:execute(76)) - *****************Metadata matrix ends******************Took 2s\r\n2017-05-09 10:45:43,351 INFO  process.MetadataAnalyzer (MetadataAnalyzer.java:execute(66)) - *****************Metadata Analyzer starts******************\r\n2017-05-09 10:45:43,423 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data is not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:45:43,443 INFO  jni.JniLoader (JniLoader.java:load(62)) - already loaded netlib-native_system-osx-x86_64.jnilib\r\n2017-05-09 10:45:46,418 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data was not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:45:51,974 INFO  process.MetadataAnalyzer (MetadataAnalyzer.java:execute(85)) - *****************Metadata Analyzer ends******************Took 8s\r\n2017-05-09 10:45:51,975 INFO  discoveryengine.MetadataDiscoveryEngine (MetadataDiscoveryEngine.java:process(70)) - *****************Metadata processing ends******************Took 10s\r\n2017-05-09 10:45:51,985 INFO  discoveryengine.RecommendEngine (RecommendEngine.java:<init>(24)) - Started Mudrod Recommend Engine.\r\n2017-05-09 10:45:51,985 INFO  discoveryengine.RecommendEngine (RecommendEngine.java:preprocess(29)) - *****************Recommendation preprocessing starts******************\r\n2017-05-09 10:45:51,986 INFO  pre.ApiHarvester (ImportMetadata.java:execute(48)) - *****************Metadata harvesting starts******************\r\n2017-05-09 10:45:54,851 INFO  pre.ApiHarvester (ImportMetadata.java:execute(54)) - *****************Metadata harvesting ends******************Took 2s\r\n2017-05-09 10:45:54,854 INFO  pre.MetadataTFIDFGenerator (MetadataTFIDFGenerator.java:execute(47)) - *****************Dataset TF_IDF Matrix Generator starts******************\r\n2017-05-09 10:45:58,159 INFO  pre.MetadataTFIDFGenerator (MetadataTFIDFGenerator.java:execute(58)) - *****************Dataset TF_IDF Matrix Generator ends******************Took 3s\r\n2017-05-09 10:45:58,162 INFO  pre.SessionCooccurence (SessionCooccurence.java:execute(53)) - *****************Dataset session_based similarity Generator starts******************\r\n2017-05-09 10:45:58,334 INFO  pre.SessionCooccurence (SessionCooccurence.java:execute(70)) - *****************Dataset session_based  similarity Generator ends******************Took 0s\r\n2017-05-09 10:45:58,335 INFO  pre.NormalizeVariables (NormalizeVariables.java:execute(46)) - *****************processing metadata variables starts******************\r\n2017-05-09 10:45:59,566 INFO  pre.NormalizeVariables (NormalizeVariables.java:execute(52)) - *****************processing metadata variables ends******************Took 1s\r\n2017-05-09 10:45:59,566 INFO  discoveryengine.RecommendEngine (RecommendEngine.java:preprocess(47)) - *****************Recommendation preprocessing  ends******************Took 7s {}\r\n2017-05-09 10:45:59,566 INFO  discoveryengine.RecommendEngine (RecommendEngine.java:process(53)) - *****************Recommendation processing starts******************\r\n2017-05-09 10:45:59,567 INFO  process.AbstractBasedSimilarity (AbstractBasedSimilarity.java:execute(43)) - *****************abstract similarity calculation starts******************\r\n2017-05-09 10:45:59,661 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data is not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:46:07,916 WARN  distributed.RowMatrix (Logging.scala:logWarning(66)) - The input data was not directly cached, which may hurt performance if its parent RDDs are also uncached.\r\n2017-05-09 10:46:20,406 INFO  process.AbstractBasedSimilarity (AbstractBasedSimilarity.java:execute(65)) - *****************abstract similarity calculation ends******************Took 20s\r\n2017-05-09 10:46:20,410 INFO  process.VariableBasedSimilarity (VariableBasedSimilarity.java:execute(66)) - *****************calculating metadata variables based similarity starts******************\r\n2017-05-09 10:47:10,277 INFO  process.VariableBasedSimilarity (VariableBasedSimilarity.java:execute(76)) - *****************calculating metadata variables based similarity ends******************Took 49s\r\n2017-05-09 10:47:10,278 INFO  process.sessionBasedCF (sessionBasedCF.java:execute(43)) - *****************Session based metadata similarity starts******************\r\norg.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/greguska/data/mudrod/201611/metadata_session_coocurrence_matrix.csv\r\n\tat org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251)\r\n\tat org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)\r\n\tat org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)\r\n\tat scala.Option.getOrElse(Option.scala:121)\r\n\tat org.apache.spark.rdd.RDD.partitions(RDD.scala:250)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)\r\n\tat scala.Option.getOrElse(Option.scala:121)\r\n\tat org.apache.spark.rdd.RDD.partitions(RDD.scala:250)\r\n\tat org.apache.spark.rdd.ZippedWithIndexRDD.<init>(ZippedWithIndexRDD.scala:44)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$zipWithIndex$1.apply(RDD.scala:1293)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$zipWithIndex$1.apply(RDD.scala:1293)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\r\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:362)\r\n\tat org.apache.spark.rdd.RDD.zipWithIndex(RDD.scala:1292)\r\n\tat org.apache.spark.api.java.JavaRDDLike$class.zipWithIndex(JavaRDDLike.scala:342)\r\n\tat org.apache.spark.api.java.AbstractJavaRDDLike.zipWithIndex(JavaRDDLike.scala:45)\r\n\tat gov.nasa.jpl.mudrod.utils.MatrixUtil.loadVectorFromCSV(MatrixUtil.java:352)\r\n\tat gov.nasa.jpl.mudrod.semantics.SemanticAnalyzer.calTermSimfromMatrix(SemanticAnalyzer.java:73)\r\n\tat gov.nasa.jpl.mudrod.recommendation.process.sessionBasedCF.execute(sessionBasedCF.java:49)\r\n\tat gov.nasa.jpl.mudrod.discoveryengine.RecommendEngine.process(RecommendEngine.java:64)\r\n\tat gov.nasa.jpl.mudrod.main.MudrodEngine.startFullIngest(MudrodEngine.java:250)\r\n\tat gov.nasa.jpl.mudrod.main.MudrodEngine.main(MudrodEngine.java:388)\r\n2017-05-09 10:47:10,295 INFO  process.sessionBasedCF (sessionBasedCF.java:execute(57)) - *****************Session based metadata similarity ends******************Took 0s\r\n2017-05-09 10:47:10,295 INFO  discoveryengine.RecommendEngine (RecommendEngine.java:process(68)) - *****************Recommendation processing ends******************Took 70s {}\r\n2017-05-09 10:47:10,295 INFO  main.MudrodEngine (MudrodEngine.java:startFullIngest(251)) - *****************full ingest has been finished successfully*****************\r\n2017-05-09 10:47:10,340 INFO  server.ServerConnector (AbstractConnector.java:doStop(306)) - Stopped ServerConnector@47c7a9e5{HTTP/1.1}{0.0.0.0:4041}\r\n```"},"comment":{"url":"https://api.github.com/repos/mudrod/mudrod/issues/comments/300250558","html_url":"https://github.com/mudrod/mudrod/issues/163#issuecomment-300250558","issue_url":"https://api.github.com/repos/mudrod/mudrod/issues/163","id":300250558,"user":{"login":"fgreg","id":1487133,"avatar_url":"https://avatars1.githubusercontent.com/u/1487133?v=3","gravatar_id":"","url":"https://api.github.com/users/fgreg","html_url":"https://github.com/fgreg","followers_url":"https://api.github.com/users/fgreg/followers","following_url":"https://api.github.com/users/fgreg/following{/other_user}","gists_url":"https://api.github.com/users/fgreg/gists{/gist_id}","starred_url":"https://api.github.com/users/fgreg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fgreg/subscriptions","organizations_url":"https://api.github.com/users/fgreg/orgs","repos_url":"https://api.github.com/users/fgreg/repos","events_url":"https://api.github.com/users/fgreg/events{/privacy}","received_events_url":"https://api.github.com/users/fgreg/received_events","type":"User","site_admin":false},"created_at":"2017-05-09T17:59:06Z","updated_at":"2017-05-09T17:59:06Z","body":"@quintinali Ok. If that feature is unavailable we should also disable the `process.sessionBasedCF (sessionBasedCF.java:execute(43)) - *****************Session based metadata similarity starts******************` step so we don't see the exception."}}
{"actor":{"display_login":"fgreg"}}