DorisExporter marshal all data to single json array lead to stream load request too small

[flashmouse]

Component(s)

exporter/doris

Describe the issue you're reporting

DorisExporter use stream load to insert data into doris. stream load have a default load limitation streaming_load_max_mb (default 10GB), but currently DorisExporter marshal all data to a single json array so request size restricted by streaming_load_json_max_mb (default only 100MB) so the maximum request body size each stream load request DorisExporter send is only 100MB. As a result, otel collector have to send too many small requests to doris, it's very ineffective.

I hope DorisExporter could reduce write requests frequency to doris. I think implementation below maybe feasible:

1. each dTrace parse to one single json line, and set strip_outer_array=fasle
2. parse []*dTrace to csv format to bypass the limitation of streaming_load_json_max_mb

[github-actions]

Pinging code owners:

• exporter/doris: @atoulme @joker-star-l

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[joker-star-l]

Thanks for your advice, I will change it with the first method you mentioned.

[joker-star-l]

By the way, another reason for too many requests might be that you don't have a batch processor in the config file.

[flashmouse]

I had added batch processor and caught error said my data size exceed streaming_load_json_max_mb so I decrease the parameters of batch procerror to fit it.