eotransform.streamed_process.streamed_process

streamed_process(source: Iterable[StreamedInT], process: Transformer[StreamedInT, StreamedOutT], dst_sink: Sink[StreamedOutT], executor: ThreadPoolExecutor, loading_timeout: float = 120, storing_timeout: float = 120) → None[source]

Interleaves the data loading from source with the data processing, and finally putting it into the sink. This can be used to hide IO processed i.e.: thread_0: IN->IN->IN thread_1: ->PR->PR->PR thread_2: ->SK->SK->SK

Parameters:

  • source – Input data fed to the process transformation

  • process – Transformation changing the input data to the output data

  • dst_sink – Sink receiving the output data

  • executor – Thread pool executor used to spawn async threads for IO hiding

  • loading_timeout – Timeout for the loading process in seconds (default: 120)

  • storing_timeout – Timout for the storing process in seconds (default: 120)