{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/353","repository_url":"https://api.github.com/repos/databricks/spark-csv","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/353/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/353/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/353/events","html_url":"https://github.com/databricks/spark-csv/issues/353","id":164477254,"number":353,"title":"Access underlying parser settings","user":{"login":"XapaJIaMnu","id":2027221,"avatar_url":"https://avatars0.githubusercontent.com/u/2027221?v=3","gravatar_id":"","url":"https://api.github.com/users/XapaJIaMnu","html_url":"https://github.com/XapaJIaMnu","followers_url":"https://api.github.com/users/XapaJIaMnu/followers","following_url":"https://api.github.com/users/XapaJIaMnu/following{/other_user}","gists_url":"https://api.github.com/users/XapaJIaMnu/gists{/gist_id}","starred_url":"https://api.github.com/users/XapaJIaMnu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/XapaJIaMnu/subscriptions","organizations_url":"https://api.github.com/users/XapaJIaMnu/orgs","repos_url":"https://api.github.com/users/XapaJIaMnu/repos","events_url":"https://api.github.com/users/XapaJIaMnu/events{/privacy}","received_events_url":"https://api.github.com/users/XapaJIaMnu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-07-08T07:51:19Z","updated_at":"2017-05-08T06:42:35Z","closed_at":null,"body":"Hey,\n\nI have a CSV which contains fields with very long lines. and I get the following exception:\n`\nException in task 0.0 in stage 1.0 (TID 1)\ncom.univocity.parsers.common.TextParsingException: Length of parsed input (100001) exceeds the maximum number of characters defined in your parser settings (100000). \nHint: Number of characters processed may have exceeded limit of 100000 characters per column. Use settings.setMaxCharsPerColumn(int) to define the maximum number of characters a column can have\nEnsure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse\n`\nThe setting in question is found at: [https://github.com/databricks/spark-csv/blob/master/src/main/scala/com/databricks/spark/csv/readers/readers.scala#L62](url). As far as I can tell there isn't a way to set it since the parser is not exposed (correct me if I am wrong, I am a beginner in scala).  Is there a way to setting it other than editing the spark-csv code?\n\nCheers,\n\nNick\n"},"comment":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/comments/299785796","html_url":"https://github.com/databricks/spark-csv/issues/353#issuecomment-299785796","issue_url":"https://api.github.com/repos/databricks/spark-csv/issues/353","id":299785796,"user":{"login":"SindhujaM","id":16665962,"avatar_url":"https://avatars0.githubusercontent.com/u/16665962?v=3","gravatar_id":"","url":"https://api.github.com/users/SindhujaM","html_url":"https://github.com/SindhujaM","followers_url":"https://api.github.com/users/SindhujaM/followers","following_url":"https://api.github.com/users/SindhujaM/following{/other_user}","gists_url":"https://api.github.com/users/SindhujaM/gists{/gist_id}","starred_url":"https://api.github.com/users/SindhujaM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SindhujaM/subscriptions","organizations_url":"https://api.github.com/users/SindhujaM/orgs","repos_url":"https://api.github.com/users/SindhujaM/repos","events_url":"https://api.github.com/users/SindhujaM/events{/privacy}","received_events_url":"https://api.github.com/users/SindhujaM/received_events","type":"User","site_admin":false},"created_at":"2017-05-08T06:42:35Z","updated_at":"2017-05-08T06:42:35Z","body":"How to resolve ???\r\n\r\nERROR: org.apache.spark.executor.Executor - Exception in task 0.0 in stage 1.0 (TID 1)\r\ncom.univocity.parsers.common.TextParsingException: Length of parsed input (100001) exceeds the maximum number of characters defined in your parser settings (100000). \r\nIdentified line separator characters in the parsed content. This may be the cause of the error. The line separator in your parser settings is set to '\\n'. Parsed content: [url=http://forums.hardwarezone.com.sg/member.php?u=422183]HaoGuShiXiongDi[/url] has reported a post.[\\n]\r\n\t[\\n]\r\n\tReason:[quote]Spamming a wall of text.[\\n]....\r\n......................\r\n..........................\r\n\r\nI have loaded 1.5 GB of my CSV file in DataFrame . But getting Lenght exceed exception. Kindly help me to resolve this..\r\n\r\nMY Sample\r\n\r\nDataFrame df = sqlContext.read().format(\"com.databricks.spark.csv\")\r\n\t\t\t\t.option(\"header\", \"true\")\r\n\t\t\t\t.option(\"inferSchema\", \"true\")\r\n\t\t\t\t.option(\"delimiter\", \",\")\r\n\t\t\t\t.option(\"escape\", \";\")\r\n\t\t\t\t.option(\"parserLib\", \"UNIVOCITY\")\r\n\t\t\t\t.load(input).toDF();\r\n\r\ndf.write().format(\"com.databricks.spark.csv\").option(\"header\", \"false\")\r\n\t\t.save(\"output.csv\");\r\n\t\t"}}
{"actor":{"display_login":"SindhujaM"}}