"IllegalStateException: Recursive update" when converting Hudi table to Delta #466
Open
3 of 4 tasks
Labels
bug
Something isn't working
Comments
lucasmo
changed the title
|
Thanks for reporting the issue @lucasmo. Can you share the contents of .hoodie folder i.e For getting unblocked you can delete the metadata table inside .hoodie it should have the below file path and try the sync again. |
|
@vinishjail97 Can you confirm that deleting the metadata prefix in S3 will not cause any issues (as long as I do it when something else is not writing)? Presumably that means it will just recreate it. I can provide a zip file; is there a way to get it to you personally? |
|
Yes that's correct. For the existing corrupted metadata, you can drop the file in this issue or share it on my email |
|
@vinishjail97 Sent to your email. I have deleted the .hoodie/metadata/ prefix in S3, but the save now causes this error (not xtable, the hudi save). Can you advise? I have rolled back the changes by putting the metadata folder back in place. |
|
Your hudi writer needs to have |
|
So this gets me one step closer, but still not working. The initial load works (I'll add the log at the end). The followup incremental load fails like this: The first load succeeds like this: Note this is the writer's configuration: full_cow_hudi_options = {
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.partitionpath.field": "op_date",
"hoodie.datasource.write.precombine.field": "event_millis",
"hoodie.datasource.write.recordkey.field": "hidden_unique_id",
"hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
"hoodie.payload.ordering.field": "event_millis",
"hoodie.insert.shuffle.parallelism": 2,
"hoodie.upsert.shuffle.parallelism": 2,
"hoodie.metadata.enable": "false",
}
df.write.format("hudi").options(**full_cow_hudi_options).mode("append").save("s3://hidden-s3-bucket/hidden-prefix")Since it's changed from the initial run, I am running xtable df51515. I'm happy to meet and/or help debug this but I'm not sure how I'd start. |
|
@lucasmo Can you share the dump for this clean commit ? It should be something like 20240702171813028.clean in .hoodie folder.
|
|
That clean commit no longer exists, but I ran again (with the same error).
I found Output of {
"startCleanTime": "20240708201227546",
"timeTakenInMillis": 517,
"totalFilesDeleted": 7,
"earliestCommitToRetain": "20240708111705091",
"lastCompletedCommitTimestamp": "20240708201123175",
"partitionMetadata": {
"op_date=2024-07-03": {
"partitionPath": "op_date=2024-07-03",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"a8214808-28b4-4c0f-8e77-193013379fd6-0_0-22-137_20240707181719248.parquet"
],
"successDeleteFiles": [
"a8214808-28b4-4c0f-8e77-193013379fd6-0_0-22-137_20240707181719248.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-07-04": {
"partitionPath": "op_date=2024-07-04",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"bb63b6ef-1a83-4a74-a736-c8418d811254-0_0-22-132_20240708081624436.parquet"
],
"successDeleteFiles": [
"bb63b6ef-1a83-4a74-a736-c8418d811254-0_0-22-132_20240708081624436.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-07-05": {
"partitionPath": "op_date=2024-07-05",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"525f1b45-7b9a-4c9e-8d83-a7cc157bb8fd-0_0-22-130_20240708091702825.parquet"
],
"successDeleteFiles": [
"525f1b45-7b9a-4c9e-8d83-a7cc157bb8fd-0_0-22-130_20240708091702825.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-07-06": {
"partitionPath": "op_date=2024-07-06",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"19976be6-94e0-4e15-a7a6-629b04c8f415-0_2-22-132_20240708091702825.parquet"
],
"successDeleteFiles": [
"19976be6-94e0-4e15-a7a6-629b04c8f415-0_2-22-132_20240708091702825.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-07-07": {
"partitionPath": "op_date=2024-07-07",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"620b8bd3-aaea-44e0-8ad7-0cfe055b58ee-0_3-22-133_20240708091702825.parquet"
],
"successDeleteFiles": [
"620b8bd3-aaea-44e0-8ad7-0cfe055b58ee-0_3-22-133_20240708091702825.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-07-08": {
"partitionPath": "op_date=2024-07-08",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"22f5d34a-48e5-44aa-bdd7-6172fc53ed82-0_4-22-134_20240708091702825.parquet"
],
"successDeleteFiles": [
"22f5d34a-48e5-44aa-bdd7-6172fc53ed82-0_4-22-134_20240708091702825.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
},
"op_date=2024-06-26": {
"partitionPath": "op_date=2024-06-26",
"policy": "KEEP_LATEST_COMMITS",
"deletePathPatterns": [
"d777fbc1-63ef-4fde-8940-1fca04abef1e-0_5-22-147_20240707201632859.parquet"
],
"successDeleteFiles": [
"d777fbc1-63ef-4fde-8940-1fca04abef1e-0_5-22-147_20240707201632859.parquet"
],
"failedDeleteFiles": [],
"isPartitionDeleted": {
"boolean": false
}
}
},
"version": {
"int": 2
},
"bootstrapPartitionMetadata": {
"map": {}
}
} |
|
@vinishjail97 This is really puzzling. I was able to boil it down to a very simple test case that blows up with the recursive error, which has nothing at all to do with XTable. Using the following java dependencies, versions taken from the XTable pom
The following code triggers the recursive exception: import org.apache.avro.Schema;
import org.apache.avro.specific.SpecificData;
Schema schema = new Schema.Parser().parse("{\"type\":\"record\",\"name\":\"HoodieCleanPartitionMetadata\",\"namespace\":\"org.apache.hudi.avro.model\",\"fields\":[{\"name\":\"partitionPath\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"policy\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"deletePathPatterns\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"successDeleteFiles\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"failedDeleteFiles\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"isPartitionDeleted\",\"type\":[\"null\",\"boolean\"],\"default\":null}]}");
SpecificData.getForSchema(schema);I got the schema by running Bumping the hudi-common dependency to 0.14.1 seems to have addressed the issue in the test case. |
|
I tried re-producing the issue by running the same test as part of xtable project itself but didn't see any error, my test and schema can be found here. Another thing you can try in your local is whether downgrading the avro version to 1.8.2 and try parsing or running the sync again ? This is the version being used in hudi. |
|
@vinishjail97 I copied and pasted your test into an xtable-utilities test and I see the failure condition there: #485 The same test passes in xtable-core |
|
Okay, I was able to figure out why this isn't causing an error in xtable-core but it is causing one in xtable-utilities. Both have Running this sample code: Schema schema = new Schema.Parser().parse("{\"type\":\"record\",\"name\":\"HoodieCleanPartitionMetadata\",\"namespace\":\"org.apache.hudi.avro.model\",\"fields\":[{\"name\":\"partitionPath\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"policy\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"deletePathPatterns\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"successDeleteFiles\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"failedDeleteFiles\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}},{\"name\":\"isPartitionDeleted\",\"type\":[\"null\",\"boolean\"],\"default\":null}]}");
System.out.println("Class for schema: " + SpecificData.get().getClass(schema));
The issue is that both |
|
@vinishjail97 can you comment on the linked issue apache/hudi#11602? They are saying you shouldn't use this jar file, but this is out of my scope to negotiate. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Search before asking
Please describe the bug 🐞
I’m trying to use XTable to convert a hudi source to a delta target and I am receiving the following exception. The table is active and frequently updated. It is being actively queried as a hudi table.
Is there any other debug information I can provide to make this more useful?
My git head is 4a96627
OS is Linux/Ubuntu
Java 11
Modified log4j2.xml to set level=trace for org.apache.hudi, o.a.xtable
Run with stacktrace:
config.yaml:
hoodie.properties from the table:
I submitted this to the dev@ mailing list and received no response, so filing as an issue.
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: