C# Client Library
A C# Client Library for the AnalyzeRe REST API
Loading...
Searching...
No Matches
AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer Class Reference

Converts buffered bytes to strings. More...

Public Types

enum  CompressionStrategy { CompressEachChunk , CompressEntireFile }
 Indicates whether the resulting chunks of compressed data should be pieces of a zip file, or a self-contained zip file each. More...
 

Public Member Functions

 GZipBufferedBytesProducer (IProducerConsumerBuffer< BufferedBytes > source, int? min_buffer_length=null, int? max_buffer_length=null, CompressionStrategy compressionStrategy=CompressionStrategy.CompressEachChunk, CompressionLevel compressionLevel=CompressionLevel.Fastest)
 Construct a producer of strings from buffered bytes.
 
virtual void Cancel ()
 Stop the producer from reading any more bytes. Warning: This will leave the stream in its current position and any bytes that haven't been inserted into the buffer are lost.
 
void Dispose ()
 Cancel the producer thread and dispose of the queue.
 
void Start (CancellationToken? cancellationToken=null)
 Start the asynchronous producer.
 
TProduced TryTake (out bool success, CancellationToken? cancellation)
 Attempts to remove an item from the produced list. May block if there is not yet an item ready, or return success:false.
 

Protected Member Functions

virtual void Dispose (bool disposing)
 Cancel the producer thread and dispose of the queue.
 
override bool IsProducerFinished ()
 Indicates whether it can produce more items.
 
override void OnDispose ()
 Dispose actions.
 
override void OnStart ()
 Invoked before the producer thread is started.
 
override bool TryProduceNext (out BufferedBytes buffer)
 Attempts to read from the stream.
 
override bool TryProduceNext (out BufferedBytes next, int buffer_size)
 Compress the next BufferedBytes.
 
bool TryProduceNext (out TProduced next)
 The method which asynchronously produces items.
 

Properties

CancellationTokenSource Cancellation [get]
 Can be used to cancel the buffering process.
 
bool CanTake [get]
 Indicates whether a call to Take will ever return another item.
 
bool IsConsumerWaiting [get]
 Determines whether the consumer of this buffer is currently waiting for it to produce some data.
 
bool IsRunning [get]
 Indicates whether the producer is currently running.
 
int MaximumBufferLength = 33554432 [get, set]
 The max length of the each byte array to generate. (Default 32 MiB)
 
int MinimumBufferLength = 16384 [get, set]
 The minimum length of the byte array to generate when the consumer of this buffer is currently waiting for more data. (Default 16 KiB) Set this lower if the consumer frequently spends too much time waiting for the producer to send the prepare the next byte array. As soon as the consumer requests more data, we will return the current buffer, if we've filled more than this many bytes. Note that smaller buffers may be returned if the end of the source stream is reached.
 
BlockingCollection< TProduced > Produced [get]
 The collection of objects produced.
 
int TotalBytesIn [get]
 Returns the total number of input bytes processed by this compressor.
 
int TotalBytesOut [get]
 The total number of output bytes produced after compression.
 
long TotalBytesRead [get]
 The total number of bytes that have been buffered since start.
 

Detailed Description

Converts buffered bytes to strings.

Definition at line 9 of file GZipBufferedBytesProducer.cs.

Inheritance diagram for AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer:
AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< BufferedBytes >

Member Enumeration Documentation

◆ CompressionStrategy

Indicates whether the resulting chunks of compressed data should be pieces of a zip file, or a self-contained zip file each.

Enumerator
CompressEachChunk 

Each output chunk will be its own zip file. Required by protocols that decompress each chunk as you upload.

CompressEntireFile 

All output chunks will be part of the same zip file. Required by protocols that only decompress the entire file once it is uploaded and recombined.

Definition at line 40 of file GZipBufferedBytesProducer.cs.

Constructor & Destructor Documentation

◆ GZipBufferedBytesProducer()

AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.GZipBufferedBytesProducer ( IProducerConsumerBuffer< BufferedBytes source,
int?  min_buffer_length = null,
int?  max_buffer_length = null,
CompressionStrategy  compressionStrategy = CompressionStrategy::CompressEachChunk,
CompressionLevel  compressionLevel = CompressionLevel::Fastest 
)
inline

Construct a producer of strings from buffered bytes.

Parameters
sourceThe source collection from which buffered bytes can be retrieved.
min_buffer_length(optional) The minimum size a buffered byte array will have. The first object returned will have this length to ensure the consumer becomes unblocked as soon as possible. See AdaptiveBufferedBytesProducer.MinimumBufferLength for details and defaults.
max_buffer_length(optional) The maximum size the buffered byte arrays can reach. See AdaptiveBufferedBytesProducer.MaximumBufferLength for details and defaults.
compressionStrategy(Optional - Default: CompressionStrategy.CompressEachChunk) Whether the resulting chunks should be pieces of a zip file or a self-contained zip file each.
compressionLevel(Optional - Default: Fastest) The GZip compression algorithm to use. Better compression requires a more powerful CPU to not become the bottleneck, but if the CPU can keep up, increasing the compression level will reduce the number of bytes that need to be transmitted and thus improve upload speed.

Definition at line 79 of file GZipBufferedBytesProducer.cs.

Member Function Documentation

◆ Cancel()

virtual void AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Cancel ( )
inlinevirtualinherited

Stop the producer from reading any more bytes. Warning: This will leave the stream in its current position and any bytes that haven't been inserted into the buffer are lost.

Definition at line 172 of file ProducerConsumerBuffer.cs.

◆ Dispose() [1/2]

void AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Dispose ( )
inlineinherited

Cancel the producer thread and dispose of the queue.

Definition at line 76 of file ProducerConsumerBuffer.cs.

◆ Dispose() [2/2]

virtual void AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Dispose ( bool  disposing)
inlineprotectedvirtualinherited

Cancel the producer thread and dispose of the queue.

Parameters
disposingWhether dispose is being called by the dispose method or the finalizer

Definition at line 91 of file ProducerConsumerBuffer.cs.

◆ IsProducerFinished()

override bool AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.IsProducerFinished ( )
protected

Indicates whether it can produce more items.

Returns
True if it is finished producing items.

◆ OnDispose()

override void AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.OnDispose ( )
inlineprotectedvirtual

Dispose actions.

Reimplemented from AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< BufferedBytes >.

Definition at line 105 of file GZipBufferedBytesProducer.cs.

◆ OnStart()

override void AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.OnStart ( )
inlineprotectedvirtual

Invoked before the producer thread is started.

Reimplemented from AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer.

Definition at line 114 of file GZipBufferedBytesProducer.cs.

◆ Start()

void AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Start ( CancellationToken?  cancellationToken = null)
inlineinherited

Start the asynchronous producer.

Parameters
cancellationTokenCancellation token that will notify this producer if it is cancelled and halt the producer thread.
Exceptions
InvalidOperationExceptionProducer is already running.

Definition at line 110 of file ProducerConsumerBuffer.cs.

◆ TryProduceNext() [1/3]

override bool AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer.TryProduceNext ( out BufferedBytes  buffer)
inlinesealedprotectedinherited

Attempts to read from the stream.

Parameters
bufferThe next buffer to add to the queue.
Returns
True if some bytes were successfully buffered.

Definition at line 78 of file AdaptiveBufferedBytesProducer.cs.

◆ TryProduceNext() [2/3]

override bool AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.TryProduceNext ( out BufferedBytes  next,
int  buffer_size 
)
inlineprotected

Compress the next BufferedBytes.

Parameters
nextThe next BufferedBytes to add to the queue.
buffer_sizeThe desired size of output BufferedBytes to create.
Returns
True if an item was successfully produced.

Definition at line 130 of file GZipBufferedBytesProducer.cs.

◆ TryProduceNext() [3/3]

bool AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.TryProduceNext ( out TProduced  next)
abstractprotectedinherited

The method which asynchronously produces items.

Parameters
nextThe next item to add to the blocking collection.
Returns
True if an item was successfully produced.

◆ TryTake()

TProduced AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.TryTake ( out bool  success,
CancellationToken?  cancellation 
)
inlineinherited

Attempts to remove an item from the produced list. May block if there is not yet an item ready, or return success:false.

Parameters
successReturns true if a valid item was returned.
cancellationA cancellation token which can be used to cancel this Take operation.
Returns
The item removed from the list.
Exceptions
BufferProducerExceptionIf the producer thread raised an exception.
InvalidOperationExceptionThe producer thread is not currently running, either because the Start method has not been called, or because the Cancel method has been called.

Definition at line 138 of file ProducerConsumerBuffer.cs.

Property Documentation

◆ Cancellation

CancellationTokenSource AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Cancellation
getprotectedinherited

Can be used to cancel the buffering process.

Definition at line 53 of file ProducerConsumerBuffer.cs.

◆ CanTake

bool AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.CanTake
getinherited

Indicates whether a call to Take will ever return another item.

Exceptions
BufferProducerExceptionIf the producer thread raised an exception.

HACK: Need some way of surfacing asynchronous errors without consumers assuming we're done, so raise any asynchronous exception here. TODO: Revisit Implementation such that BufferProducerException is discovered elsewhere CanTake can simply be made to return false if an exception occurred.

Definition at line 36 of file ProducerConsumerBuffer.cs.

◆ IsConsumerWaiting

bool AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.IsConsumerWaiting
getprotectedinherited

Determines whether the consumer of this buffer is currently waiting for it to produce some data.

Definition at line 23 of file ProducerConsumerBuffer.cs.

◆ IsRunning

bool AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.IsRunning
getinherited

Indicates whether the producer is currently running.

Definition at line 26 of file ProducerConsumerBuffer.cs.

◆ MaximumBufferLength

int AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer.MaximumBufferLength = 33554432
getsetinherited

The max length of the each byte array to generate. (Default 32 MiB)

Definition at line 21 of file AdaptiveBufferedBytesProducer.cs.

◆ MinimumBufferLength

int AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer.MinimumBufferLength = 16384
getsetinherited

The minimum length of the byte array to generate when the consumer of this buffer is currently waiting for more data. (Default 16 KiB) Set this lower if the consumer frequently spends too much time waiting for the producer to send the prepare the next byte array. As soon as the consumer requests more data, we will return the current buffer, if we've filled more than this many bytes. Note that smaller buffers may be returned if the end of the source stream is reached.

Definition at line 30 of file AdaptiveBufferedBytesProducer.cs.

◆ Produced

BlockingCollection<TProduced> AnalyzeRe.LargeDataUpload.ProducerConsumerBuffer< TProduced >.Produced
getprotectedinherited

The collection of objects produced.

Definition at line 19 of file ProducerConsumerBuffer.cs.

◆ TotalBytesIn

int AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.TotalBytesIn
get

Returns the total number of input bytes processed by this compressor.

Definition at line 33 of file GZipBufferedBytesProducer.cs.

◆ TotalBytesOut

int AnalyzeRe.LargeDataUpload.GZipBufferedBytesProducer.TotalBytesOut
get

The total number of output bytes produced after compression.

Definition at line 36 of file GZipBufferedBytesProducer.cs.

◆ TotalBytesRead

long AnalyzeRe.LargeDataUpload.AdaptiveBufferedBytesProducer.TotalBytesRead
getinherited

The total number of bytes that have been buffered since start.

Definition at line 33 of file AdaptiveBufferedBytesProducer.cs.


The documentation for this class was generated from the following file: