title: Pyspark error instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':"
body: <p>I was trying this code i also found on stackoverflow</p>
  
  <pre><code>from pyspark.mllib.linalg.distributed import RowMatrix
  rows = sc.parallelize([(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)])
  
  # Convert to RowMatrix
  mat = RowMatrix(rows)
  
  # Calculate exact and approximate similarities
  exact = mat.columnSimilarities()
  approx = mat.columnSimilarities(0.05)
  
  # Output
  exact.entries.collect()
  [MatrixEntry(0, 2, 0.991935352214),
   MatrixEntry(1, 2, 0.998441152599),
   MatrixEntry(0, 1, 0.997463284056)]
  </code></pre>
  
  <p>then when I ran exaxt.entries I got this error </p>
  
  <pre><code>---------------------------------------------------------------------------
  IllegalArgumentException                  Traceback (most recent call last)
  &lt;ipython-input-3-912f64c8ec62&gt; in &lt;module&gt;()
  ----&gt; 1 print exact.entries
  
  D:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\pyspark\mllib\linalg\distributed.pyc in entries(self)
      824         # DataFrame on the Scala/Java side. Then we map each Row in
      825         # the DataFrame back to a MatrixEntry on this side.
  --&gt; 826         entries_df = callMLlibFunc("getMatrixEntries", self._java_matrix_wrapper._java_model)
      827         entries = entries_df.rdd.map(lambda row: MatrixEntry(row[0], row[1], row[2]))
      828         return entries
  
  D:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\pyspark\mllib\common.pyc in callMLlibFunc(name, *args)
      128     sc = SparkContext.getOrCreate()
      129     api = getattr(sc._jvm.PythonMLLibAPI(), name)
  --&gt; 130     return callJavaFunc(sc, api, *args)
      131 
      132 
  
  D:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\pyspark\mllib\common.pyc in callJavaFunc(sc, func, *args)
      121     """ Call Java Function """
      122     args = [_py2java(sc, a) for a in args]
  --&gt; 123     return _java2py(sc, func(*args))
      124 
      125 
  
  D:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py in __call__(self, *args)
     1131         answer = self.gateway_client.send_command(command)
     1132         return_value = get_return_value(
  -&gt; 1133             answer, self.gateway_client, self.target_id, self.name)
     1134 
     1135         for temp_arg in temp_args:
  
  D:\opt\spark\spark-2.2.0-bin-hadoop2.7\python\pyspark\sql\utils.pyc in deco(*a, **kw)
       77                 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
       78             if s.startswith('java.lang.IllegalArgumentException: '):
  ---&gt; 79                 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
       80             raise
       81     return deco
  
  IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':"
  </code></pre>
  
  <p>I installed Spark on windows10 using this site <a href="https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c" rel="nofollow noreferrer">https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c</a></p>
  
  <p>any fix on this? thanks</p>
  
  <p>*update: additional error from cmd 
  I dont understand the error. there actually exist a metastore_db in my folder</p>
  
  <pre><code>18/02/04 16:48:22 ERROR Schema: Failed initialising database.
  Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
  java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@700d488a, see the next exception for details.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
          at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection.&lt;init&gt;(Unknown Source)
          at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
          at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
          at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
          at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
          at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at java.sql.DriverManager.getConnection(Unknown Source)
          at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
          at com.jolbox.bonecp.BoneCP.&lt;init&gt;(BoneCP.java:416)
          at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
          at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)
          at org.datanucleus.store.rdbms.RDBMSStoreManager.&lt;init&gt;(RDBMSStoreManager.java:298)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
          at java.lang.reflect.Constructor.newInstance(Unknown Source)
          at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
          at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
          at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
          at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
          at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
          at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
          at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
          at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
          at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
          at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
          at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
          at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
          at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
          at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
          at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
          at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
          at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:57)
          at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:66)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
          at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
          at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:199)
          at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.&lt;init&gt;(SessionHiveMetaStoreClient.java:74)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
          at java.lang.reflect.Constructor.newInstance(Unknown Source)
          at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:86)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
          at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
          at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
          at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
          at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
          at org.apache.hadoop.hive.ql.metadata.Hive.&lt;clinit&gt;(Hive.java:166)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
          at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:191)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
          at java.lang.reflect.Constructor.newInstance(Unknown Source)
          at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
          at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362)
          at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266)
          at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
          at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:194)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
          at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
          at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
          at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:193)
          at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105)
          at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
          at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35)
          at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
          at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
          at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
          at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
          at scala.Option.getOrElse(Option.scala:121)
          at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:129)
          at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:126)
          at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65)
          at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:283)
          at org.apache.spark.mllib.api.python.PythonMLLibAPI.getMatrixEntries(PythonMLLibAPI.scala:1207)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
          at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
          at py4j.Gateway.invoke(Gateway.java:280)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:214)
          at java.lang.Thread.run(Unknown Source)
  Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@700d488a, see the next exception for details.
          at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
          ... 113 more
  Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database C:\Users\USER\Desktop\ColumnSimilarities\metastore_db.
          at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
          at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source)
          at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
          at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.store.raw.RawStore$6.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.store.raw.RawStore.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
          at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.store.access.RAMAccessManager$5.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.store.access.RAMAccessManager.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
          at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.db.BasicDatabase$5.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.db.BasicDatabase.bootServiceModule(Unknown Source)
          at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
          at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
          at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
          at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
          at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
          at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(Unknown Source)
          ... 110 more
  </code></pre>
  
  <p>*Update</p>
  
  <p>I used this portion of code I found in the documentations it was working.</p>
  
  <pre><code>from pyspark.mllib.linalg.distributed import  RowMatrix, IndexedRowMatrix,CoordinateMatrix, MatrixEntry
  
  #Create an RDD of coordinate entries.
  #   - This can be done explicitly with the MatrixEntry class:
  # entries = sc.parallelize([MatrixEntry(0, 0, 1.2), MatrixEntry(1, 0, 2.1), MatrixEntry(6, 1, 3.7)])
  #   - or using (long, long, float) tuples:
  entries = sc.parallelize([(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)])
  #([(0, 0, 1.2), (1, 0, 2.1), (2, 1, 3.7)])
  mat1 = RowMatrix(entries)
  
  exact = mat1.columnSimilarities()
  exact.entries.collect()
  </code></pre>