{"push_id":1789563240,"size":5,"distinct_size":1,"ref":"refs/heads/shs-ng/M9","head":"f730e9e235928fdc2f694a650da59b9e579b916a","before":"55eea134e600c44fcd8a1a783523b20d24129ecc","commits":[{"sha":"fb06444d40ab83cee628da6b5f382d8ef48f42d0","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M5: Store SQL UI data in the app state store.\n\nThis change replaces the SQLListener with a new implementation that\nsaves the data to the same store used by the SparkContext's state\nstore. For that, the types used by the old SQLListener had to be\nupdated a bit so that they're more serialization-friendly.\n\nThe interface for getting data from the store was abstracted into\na new class, SQLAppStateStore (following the convention used in\ncore).\n\nAnother change is the way that the SQL UI hooks up into the core\nUI or the SHS. The old \"SparkHistoryListenerFactory\" was replaced\nwith a new \"AppStatePlugin\" that more explicitly differentiates\nbetween the two use cases: processing events, and showing the UI.\nBoth live apps and the SHS use this new API (previously, it was\nrestricted to the SHS).\n\nNote on the above: this causes a slight change of behavior for\nlive apps; the SQL tab will only show up after the first execution\nis started.\n\nThe metrics gathering code was re-worked a bit so that the types\nused are less memory hungry and more serialization-friendly. This\nshould help with the caching layer used during writes; while the\ndata is cached in memory it should use less of it, and when it needs\nto be loaded from disk, it shouldn't be terribly expensive.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/fb06444d40ab83cee628da6b5f382d8ef48f42d0"},{"sha":"f52dd0f2e25b253264eea1b5a0edb6bf9dd534c6","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M6: Add cleaning of old elements from the store.\n\nThis change restores the functionality that keeps a limited number of\ndifferent types (jobs, stages, etc) depending on configuration, to avoid\nthe store growing indefinitely over time.\n\nThe feature is implemented by creating a new type (ElementTrackingStore)\nthat wraps a KVStore and allows triggers to be set up for when elements\nof a certain type meet a certain threshold. Triggers don't need to\nnecessarily only delete elements, but the current API is set up in a way\nthat makes that use case easier.\n\nThe new store also has a trigger for the \"close\" call, which makes it\neasier for listeners to register code that should be run before the\nstore is closed.\n\nThe old configurations for cleaning up the stored elements from the core\nand SQL UIs are now active again, and the old unit tests are re-enabled.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/f52dd0f2e25b253264eea1b5a0edb6bf9dd534c6"},{"sha":"91894b6840df7ba55bf2b2d0fb02e741f7cac02e","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M7: Limit disk usage of the history server.\n\nThis change adds a new configuration option and support code that limits\nhow much disk space the SHS will use. The default value is pretty generous\nso that applications will, hopefully, only rarely need to be replayed.\n\nThis works by keeping track of how much data each application is using;\nbecause it's not possible to know, before replaying, how much space will\nbe needed, it's possible that usage will exceed the configured limit\ntemporarily. The code uses the concept of a \"lease\" to try to limit how\nmuch the SHS will exceed the limit.\n\nActive UIs are also tracked, so they're never deleted. This works in\ntandem with the existing option of how many active UIs are loaded; because\nunused UIs will be unloaded, their disk stores will also become candidates\nfor deletion. If the data is not deleted, though, re-loading the UI is\npretty quick.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/91894b6840df7ba55bf2b2d0fb02e741f7cac02e"},{"sha":"5dae520a9ac4ac776e10cf72ee226895d8a91b3e","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M8: Delete stale application data from SHS.\n\nDetect the deletion of event log files from storage, and remove\ndata about the related application attempt in the SHS.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/5dae520a9ac4ac776e10cf72ee226895d8a91b3e"},{"sha":"f730e9e235928fdc2f694a650da59b9e579b916a","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M9: Stage page speed up.\n\nThere are two main changes to speed up rendering of the tasks list\nwhen rendering the stage page.\n\nThe first one makes the code only load the tasks being shown in the\ncurrent page of the tasks table, and information related to only\nthose tasks. One side-effect of this change is that the graph that\nshows task-related events now only shows events for the tasks in\nthe current page, instead of the previously hardcoded limit of \"events\nfor the first 1000 tasks\". That ends up helping with readability,\nthough.\n\nTo make sorting efficient when using a disk store, the task wrapper\nwas extended to include many new indices, one for each of the sortable\ncolumns in the UI, and metrics for which quantiles are calculated.\n\nThe second changes the way metric quantiles are calculated for stages.\nInstead of using the \"Distribution\" class to process data for all task\nmetrics, which requires scanning all tasks of a stage, the code now\nuses the KVStore \"skip()\" functionality to only read tasks that contain\ninteresting information for the quantiles that are desired.\n\nThis is still not cheap; because there are many metrics that the UI\nand API track, the code needs to scan the index for each metric to\ngather the information. Savings come mainly from skipping deserialization\nwhen using the disk store, but the in-memory code also seems to be\nfaster than before (most probably because of other changes in this\npatch).\n\nWith the above changes, a lot of code in the UI layer could be simplified.","distinct":true,"url":"https://api.github.com/repos/vanzin/spark/commits/f730e9e235928fdc2f694a650da59b9e579b916a"}]}
{"actor":{"display_login":"vanzin"}}