Then error as 18/04/26 21:03:44 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, .th, executor 1): : Task failed while writing rows.Īt .$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285)Īt .$$anonfun$write$1.apply(FileFormatWriter.scala:197)Īt .$$anonfun$write$1.apply(FileFormatWriter.scala:196)Īt .nTask(ResultTask.scala:87)Īt .n(Task.scala:109)Īt .Executor$n(Executor.scala:345)Īt .runWorker(ThreadPoolExecutor.java:1149)Īt $n(ThreadPoolExecutor.java:624)Ĭaused by: : .maxCompressedLength(I)IĪt .maxCompressedLength(Native Method)Īt .maxCompressedLength(Snappy.java:316)Īt 圜press(Snapp圜ompressor.java:67)Īt .(CompressorStream.java:81)Īt .(CompressorStream.java:92)Īt $press(CodecFactory.java:112)Īt $ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:89)Īt 1.writePage(ColumnWriterV1.java:153)Īt 1.flush(ColumnWriterV1.java:241)Īt 1.flush(ColumnWriteStoreV1.java:126)Īt (InternalParquetRecordWriter.java:159)Īt (InternalParquetRecordWriter.java:111)Īt (ParquetRecordWriter.java:112)Īt .close(ParquetOutputWriter.scala:42)Īt .$SingleDirectoryWriteTask.releaseResources(FileFormatWriter.scala:405)Īt .$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:396)Īt .$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:269)Īt .$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:267)Īt .Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)Īt .$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:272)ġ8/04/26 21:03:44 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times aborting jobġ8/04/26 21:03:44 ERROR datasources.FileFormatWriter: Aborting job null. Sql("INSERT INTO parquet_table_name VALUES(1, 'test')") Sql("CREATE TABLE parquet_table_name (x INT, y STRING) STORED AS PARQUET") So the only thing you can set is the compression codec, using dataframe.write().format("orc").option("compression","snappy").I'm trying to create Hive table with snappy compression via Spark2. Note that the default compression codec has changed with Spark 2 before that it was zlib This can be one of the known case-insensitive shorten
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |