{"push_id":1773317669,"size":13,"distinct_size":0,"ref":"refs/heads/shs-ng/M6","head":"4a7e2677615dedbcc981b8b9cf139f339781556d","before":"4470c45a18cc2d9693048c7575eb3307b00aaaa7","commits":[{"sha":"cc6662b54f25cab3274a3536c9bb722371278b88","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M1: feedback.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/cc6662b54f25cab3274a3536c9bb722371278b88"},{"sha":"f9678852c0496653bcd6e7fe57abcedbd2044492","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M1.2: In-memory KVStore implementation.\n\nThis change adds an in-memory implementation of KVStore that can be\nused by the live UI.\n\nThe implementation is definitely not optimized, neither for speed nor\nspace, but should be fast enough for using in the listener bus.\n\nThe change slightly changes the semantics of KVStore views; now, by\ndefault, they don't guarantee ordering, which makes it faster to\niterate over data in the in-memory store when trimming the number\nof elements (a feature that will be useful later on). The disk store\nby default still returns elements in sorted order.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/f9678852c0496653bcd6e7fe57abcedbd2044492"},{"sha":"aee60dcfc3ab7ed8e3897d4ef9df91e183000e80","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M2: Store FsHistoryProvider listing data in a KVStore.\n\nThe application listing is still generated from event logs, but is now stored\nin a KVStore instance. By default an in-memory store is used, but a new config\nallows setting a local disk path to store the data, in which case a LevelDB\nstore will be created.\n\nThe provider stores things internally using the public REST API types; I believe\nthis is better going forward since it will make it easier to get rid of the\ninternal history server API which is mostly redundant at this point.\n\nI also added a finalizer to LevelDBIterator, to make sure that resources are\neventually released. This helps when code iterates but does not exhaust the\niterator, thus not triggering the auto-close code.\n\nHistoryServerSuite was modified to not re-start the history server unnecessarily;\nthis makes the json validation tests run more quickly.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/aee60dcfc3ab7ed8e3897d4ef9df91e183000e80"},{"sha":"a22d8a22cf94486c33b849943c6588b9e16eaeaf","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M3: Add initial listener implementation to collect app state.\n\nThe initial listener is based on the existing JobProgressListener (and others),\nand tries to mimin their behavior as much as possible. The change also includes\nsome minor code movement so that some types and methods from the initial history\nserver code code can be reused.\n\nNote the code here is not 100% correct. This is meant as a building ground for\nthe UI integration in the next milestones. As different parts of the UI are\nported, fixes will be made to the different parts of this code to account\nfor the needed behavior.\n\nI also added annotations to API types so that Jackson is able to correctly\ndeserialize options, sequences and maps that store primitive types.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/a22d8a22cf94486c33b849943c6588b9e16eaeaf"},{"sha":"dcd003b0f5c2ff2cb180b467d9f5c0fad63c5de6","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.0: Initial UI hook up.\n\nThis change adds some building blocks for hooking up the new data store\nto the UI. This is achieved by returning a new SparkUI implementation when\nusing the new KVStoreProvider; this new UI does not currently contain any\ndata for the old UI / API endpoints; that will be implemented in M4.\n\nThe interaction between the UI and the underlying store was isolated\nin a new AppStateStore class. Code in later patches will call into this\nclass to retrieve data to populate the UI and API.\n\nSome new indexed fields had to be added to the stored types so that the\ncode could efficiently process the API requests.\n\nOn the history server side, some changes were made in how the UI is used.\nBecause there's state kept on disk, the code needs to be more careful about\nclosing those resources when the UIs are unloaded; and because of that some\nlocking needs to exist to make sure it's OK to move files around. The app\ncache was also simplified a bit; it just checks a flag in the UI instance\nto check whether it should be used, and tries to re-load it when the FS\nlisting code invalidates a loaded UI.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/dcd003b0f5c2ff2cb180b467d9f5c0fad63c5de6"},{"sha":"4d00c4906ced3b5ca00a6eba4f17f802a465ba4a","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.1: Port environment page to new backend.\n\nThis change moves the environment page to the new UI module,\nmaking modifications so that its data is stored in and retrieved\nfrom the new store.\n\nSome temporary internal API changes were made to maintain the current\ncode working when using the new UI implementation. These may be cleaned\nup when the old UI code is cleaned up and only the new implementation\nremains.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/4d00c4906ced3b5ca00a6eba4f17f802a465ba4a"},{"sha":"63a6ceb16e1e170d4f6cc9b67d51433e40f97bc2","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.2: Port executors page to new backend.\n\nThe executors page is built on top of the REST API, so the page itself\nwas easy to hook up to the new code.\n\nSome other pages depend on the `ExecutorListener` class that is being\nremoved, though, so they needed to be modified to use data from the\nnew store. Fortunately, all they seemed to need is the map of executor\nlogs, so that was somewhat easy too.\n\nThe executor timeline graph required some extra code to save the\nexecutor-related events in the UI store. This just implements the\nexisting functionality, without making any changes related to efficiency\nor scalability of that graph.\n\nI had to change some of the test golden files because the old code would\nreturn executors in \"random\" order (since it used a mutable Map instead\nof something that returns a sorted list), and the new code returns executors\nin id order.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/63a6ceb16e1e170d4f6cc9b67d51433e40f97bc2"},{"sha":"940e95f74c4a32c593021277418c22074fd62712","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.3: Port StorageTab to the new backend.\n\nThis required adding information about StreamBlockId to the store,\nwhich is not available yet via the API. So an internal type was added\nuntil there's a need to expose that information in the API.\n\nThe UI only lists RDDs that have cached partitions, and that information\nwasn't being correctly captured in the listener, so that's also fixed,\nalong with some minor (internal) API adjustments so that the UI can\nget the correct data.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/940e95f74c4a32c593021277418c22074fd62712"},{"sha":"66327c934bbde162dbcd8e1b2ad4727671c7278b","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.4: Port JobsTab and StageTab to the new backend.\n\nThis change is a little larger because there's a whole lot of logic\nbehind these pages, all really tied to internal types and listeners.\nThere's also a lot of code that was moved to the new module.\n\n- Added missing StageData and ExecutorStageSummary fields which are\n  used by the UI. Some json golden files needed to be updated to account\n  for new fields.\n\n- Save RDD graph data in the store. This tries to re-use existing types as\n  much as possible, so that the code doesn't need to be re-written. So it's\n  probably not very optimal.\n\n- Some old classes (e.g. JobProgressListener) still remain, since they're used\n  in other parts of the code; they're not used by the UI anymore, though, and\n  will be cleaned up in a separate change.\n\n- Save information about active pools in the disk store; this could potentially\n  be avoided, since it's most probably not much data, but it makes it easier\n  later to add this kind of information to the API and to history if wanted.\n\n- Because the new store sorts things slightly differently from the previous\n  code, some json golden files had some elements within them shuffled around.\n\n- The retention unit test in UISeleniumSuite was disabled because the code\n  to throw away old stages / tasks hasn't been added yet.\n\n- The job description field in the API tries to follow the old behavior, which\n  makes it be empty most of the time, even though there's information to fill it\n  in. For stages, a new field was added to hold the description (which is basically\n  the job description), so that the UI can be rendered in the old way.\n\n- A new stage status (\"SKIPPED\") was added to account for the fact that the API\n  couldn't represent that state before. Because of the way the new code tracks\n  stages, they would end up showing up as \"PENDING\" in the UI.\n\n- The API used to expose \"executorRunTime\" as the value of the task's duration,\n  which wasn't really correct (also because that value was easily available\n  from the metrics object); this change fixes that by storing the correct runtime,\n  which also means a few expectation files needed to be updated to account for\n  the new durations (and also sorting differences due to the changed values).","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/66327c934bbde162dbcd8e1b2ad4727671c7278b"},{"sha":"b6fbf3a5975799892d7885860fc8aa320ef38c56","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.5: Simplify API resource structure.\n\nWith the new UI store, the API resource classes have a lot less code,\nsince there's no need for complicated translations between the UI\ntypes and the API types. So the code ended up with a bunch of files\nwith a single method declared in them.\n\nThis change re-structures the API code so that it uses less classes;\nmainly, most sub-resources were removed, and the code to deal with\nsingle-attempt and multi-attempt apps was simplified.\n\nThe only change was the addition of a method to return a single\nattempt's information; that was missing in the old API, so trying\nto retrieve \"/v1/applications/appId/attemptId\" would result in a\n404 even if the attempt existed (and URIs under that one would\nreturn valid data).\n\nThe streaming API resources also overtook the same treatment; the\nstreaming backend is still not hooked up to the store, but once it\nis, the code in the remaining classes will be simplified even\nfurther.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/b6fbf3a5975799892d7885860fc8aa320ef38c56"},{"sha":"fcaf5c5a881a509dd61b8e5a9733b3d896469aa0","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.6: Remove JobProgressListener.\n\nThe only remaining use of this class was the SparkStatusTracker, which\nwas modified to use the new state store. The test code to wait for\nexecutors was moved to TestUtils and now uses the SparkStatusTracker API.\n\nIndirectly, ConsoleProgressBar also uses this data. Because it has\nsome lower latency requirements, a shortcut to efficiently get the\nactive stages from the active listener was added to the AppStateStore.\n\nAs part of this change I also modified the streaming UI to read the needed\ndata from the store, which was missed in the previous patch that made\nJobProgressListener redundant.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/fcaf5c5a881a509dd61b8e5a9733b3d896469aa0"},{"sha":"0f53cad609640a2a1cebd6c86cbe90c6b891aae3","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M5: Store SQL UI data in the app state store.\n\nThis change replaces the SQLListener with a new implementation that\nsaves the data to the same store used by the SparkContext's state\nstore. For that, the types used by the old SQLListener had to be\nupdated a bit so that they're more serialization-friendly.\n\nThe interface for getting data from the store was abstracted into\na new class, SQLAppStateStore (following the convention used in\ncore). The session's shared state now contains a pointer to this\nstore instead of the listener itself.\n\nAnother change is the way that the SQL UI hooks up into the core\nUI or the SHS. The old \"SparkHistoryListenerFactory\" was replaced\nwith a new \"AppStatePlugin\" that more explicitly differentiates\nbetween the two use cases: parsing event logs and showing the UI.\nThe SHS hooks into that new API to talk to the SQL module; the live\nUI still needs to be installed explicitly, but the SQL code now\nuses the same code to do it (in the SQLAppStatePlugin implementation).\n\nThe metrics gathering code was re-worked a bit so that the types\nused are less memory hungry and more serialization-friendly. This\nshould help with the caching layer used during writes; while the\ndata is cached in memory it should use less of it, and when it needs\nto be loaded from disk, it shouldn't be terribly expensive.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/0f53cad609640a2a1cebd6c86cbe90c6b891aae3"},{"sha":"4a7e2677615dedbcc981b8b9cf139f339781556d","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M6: Add cleaning of old elements from the store.\n\nThis change restores the functionality that keeps a limited number of\ndifferent types (jobs, stages, etc) depending on configuration, to avoid\nthe store growing indefinitely over time.\n\nThe feature is implemented by creating a new type (ElementTrackingStore)\nthat wraps a KVStore and allows triggers to be set up for when elements\nof a certain type meet a certain threshold. Triggers don't need to\nnecessarily only delete elements, but the current API is set up in a way\nthat makes that use case easier.\n\nThe new store also has a trigger for the \"close\" call, which makes it\neasier for listeners to register code that should be run before the\nstore is closed.\n\nThe old configurations for cleaning up the stored elements from the core\nand SQL UIs are now active again, and the old unit tests are re-enabled.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/4a7e2677615dedbcc981b8b9cf139f339781556d"}]}
{"actor":{"display_login":"vanzin"}}