{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16648","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16648/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16648/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16648/events","html_url":"https://github.com/apache/spark/pull/16648","id":201903575,"number":16648,"title":"[SPARK-18016][SQL][CATALYST] Code Generation: Constant Pool Limit","user":{"login":"bdrillard","id":1728813,"avatar_url":"https://avatars3.githubusercontent.com/u/1728813?v=4","gravatar_id":"","url":"https://api.github.com/users/bdrillard","html_url":"https://github.com/bdrillard","followers_url":"https://api.github.com/users/bdrillard/followers","following_url":"https://api.github.com/users/bdrillard/following{/other_user}","gists_url":"https://api.github.com/users/bdrillard/gists{/gist_id}","starred_url":"https://api.github.com/users/bdrillard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdrillard/subscriptions","organizations_url":"https://api.github.com/users/bdrillard/orgs","repos_url":"https://api.github.com/users/bdrillard/repos","events_url":"https://api.github.com/users/bdrillard/events{/privacy}","received_events_url":"https://api.github.com/users/bdrillard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":43,"created_at":"2017-01-19T16:11:57Z","updated_at":"2017-10-17T18:24:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16648","html_url":"https://github.com/apache/spark/pull/16648","diff_url":"https://github.com/apache/spark/pull/16648.diff","patch_url":"https://github.com/apache/spark/pull/16648.patch"},"body":"[class_splitting] Code Generation: Constant Pool Limit\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nSupports code generation for large structures that would previously trigger a Constant Pool limit exception, as noted in [SPARK-18016](SPARK-18016). In this fix, when the volume of generated code for the class would exceed 1600k bytes, a new private nested class is declared, and any new functions that would have been inlined to the outer class with an `addNewFunction` call are inlined to the new nested class instead. `addNewFunction` also would now return the name of the function registered (class-qualified, if it would be inlined to a nested class), so that the caller of the function can call it even if inlined to a different class. Additional nested classes are generated if the threshold is met subsequent times. These nested classes are instantiated and declared at the bottom of the generated outer class. \r\n\r\nBecause private nested classes have access to the outer class's global state, but their functions and local state do not count towards the outer class's Constant Pool, and that they can be instantiated in the same outer class without the need to declare additional classes and handle the dependency injection, they seem to be a good candidate to solve this particular issue.\r\n\r\nOne key quality of this patch is that the common path for code generation remains unaffected. The 1600k threshold necessary to split of a nested class should only be exceeded in scenarios where the schema is extremely large. Generated code for most use cases will still be inlined entirely to the single outer class.\r\n\r\nOne other feature of this patch is the compaction of primitive declarations and simply-assigned object declarations (objects initialized to null, or not initialized at all) into arrays. Code generation can sometimes produce an excess of global state, so we reduce the number of declarations that would count towards the Constant Limit by creating arrays such that for a given variable of common type and declaration, it can be mapped to a specific index of an array. Using arrays of mutable state, the number of variables in global state is less constrained by the constant pool limit, but by the maximum size of an array.\r\n\r\nThis patch splits code (only code registered through the `addNewFunction` call) among the outer class and nested classes, and creates arrays for compactable mutable state like below:\r\n\r\n```\r\n     /* 6 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection {\r\n     /* 7 */    // Global \"mutable\" state\r\n     /* 8 */    private Object[] references;\r\n     /* 9 */    private UnsafeRow result;\r\n    /* 10 */    private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder holder;\r\n    /* 11 */    private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter;\r\n    /* 12 */    UTF8String[] mutableStateArray1 = new UTF8String[24003]; // compacted states\r\n    /* 13 */    java.lang.STring[] mutableStateArray = new java.lang.String[8001];\r\n    /* 14 */    int[] mutableStateArray3 = new int[8000];\r\n    /* 15 */    boolean[] mutableStateArray2 = new boolean[32004];\r\n    /* 16 */\r\n    /* 17 */    public SpecificUnsafeProjection(Object[] references) {\r\n    /* 18 */        this.references = references;\r\n    /* 19 */        result = new UnsafeRow(8001);\r\n    /* 20 */        holder = new org.apache.spark.sql.catalyst.expressions.codegen.BuffereHolder(result, 256032);\r\n    /* 21 */        rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 8001);\r\n    /* 22 */        // array initializations for compacted states\r\n    /* 23 */        for (int i = 0; i < mutableStateArray1.length; i++) {\r\n    /* 24 */            mutableStateArray1[i] = null;\r\n    /* 25 */        }\r\n    /* 26 */\r\n    /* 27 */        for (int i = 0; i < mutableStateArray2.length; i++) {\r\n    /* 28 */            mutableStateArray2[i] = false;\r\n    /* 29 */        }\r\n    /* 30 */     }\r\n    /* 31 */     // method inlined to outer class\r\n    /* 32 */     private void evalIfCondExpr50(InternalRow i) {\r\n    /* 33 */         boolean isNull703 = i.isNullAt(0);\r\n    /* 34 */         org.apache.spark.sql.Row value703 = isNull703 ? null : \r\n    /* 35 */             ((org.apache.spark.sql.Row)i.get(0, null));\r\n    /* 36 */\r\n    /* 37 */         if (isNull703) {\r\n    /* 38 */             throw new NullPointerException(((java.lang.String) references[200]));\r\n    /* 39 */         }\r\n    /* 40 */         boolean isNull701 = true;\r\n    /* 41 */         boolean value701 = false;\r\n    /* 42 */         if (!false) {\r\n    /* 43 */\r\n    /* 44 */             mutableStateArray3[49] = 50;\r\n    /* 45 */\r\n    /* 46 */             isNull701 = false;\r\n    /* 47 */             if (!isNull701) {\r\n    /* 48 */                 value701 = value703.isNullAt(mutableStateArray3[49]);\r\n    /* 49 */             }\r\n    /* 50 */         }  // examples of mutableStateArray aliases for primitive global state\r\n    /* 51 */         mutableStateArray2[200] = isNull701;\r\n    /* 52 */         mutableStateArray2[201] = value701;\r\n    /* 53 */     }\r\n     ...              \r\n /* 70344 */     public UnsafeRow apply(InternalRow i) {\r\n /* 70345 */         nestedClassInstance5.apply589_0(i); // referencing a nested class method\r\n     ...\r\n /* 70398 */         return result;\r\n /* 70399 */     }\r\n /* 70400 */     // Instantiation of nested classes\r\n /* 70401 */     private NestedClass5 nestedClassInstance5 = new NestedClass();\r\n /* 70402 */     private NestedClass4 nestedClassInstance4 = new NestedClass();\r\n     ...\r\n /* 70406 */     private NestedClass nestedClassInstance = new NestedClass();\r\n /* 70407 */     // Declaration of a nested class\r\n /* 70408 */     private NestedClass5 {\r\n /* 70409 */         // Code inlined to a nested class\r\n /* 70410 */         private void apply519_0(InternalRow fieldName1105) {\r\n     ...\r\n/* 340829 */     } // end of last nested class\r\n/* 348030 */ } // end of the outer class\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nAdded a new test to the DataframeComplexTypeSuite that tests converting a large structure to a dataset. Ran full regression tests across every module.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/337323460","html_url":"https://github.com/apache/spark/pull/16648#issuecomment-337323460","issue_url":"https://api.github.com/repos/apache/spark/issues/16648","id":337323460,"user":{"login":"bdrillard","id":1728813,"avatar_url":"https://avatars3.githubusercontent.com/u/1728813?v=4","gravatar_id":"","url":"https://api.github.com/users/bdrillard","html_url":"https://github.com/bdrillard","followers_url":"https://api.github.com/users/bdrillard/followers","following_url":"https://api.github.com/users/bdrillard/following{/other_user}","gists_url":"https://api.github.com/users/bdrillard/gists{/gist_id}","starred_url":"https://api.github.com/users/bdrillard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdrillard/subscriptions","organizations_url":"https://api.github.com/users/bdrillard/orgs","repos_url":"https://api.github.com/users/bdrillard/repos","events_url":"https://api.github.com/users/bdrillard/events{/privacy}","received_events_url":"https://api.github.com/users/bdrillard/received_events","type":"User","site_admin":false},"created_at":"2017-10-17T18:24:50Z","updated_at":"2017-10-17T18:24:50Z","author_association":"CONTRIBUTOR","body":"@kiszk please see #19518 for part 2 of this original PR, and thanks!"}}
{"actor":{"display_login":"bdrillard"}}