Class KNetSourceTask<TTask>

Namespace

MASES.KNet.Connect

Assembly

MASES.KNet.dll

An implementation of KNetTask<TTask> for source task

public abstract class KNetSourceTask<TTask> : KNetTask<TTask>, IKNetTask, ITask, IKNetConnectLogging where TTask : KNetSourceTask<TTask>

Type Parameters

TTask

The class which extends KNetSourceTask<TTask>

Inheritance

object

KNetTask

KNetTask<TTask>

KNetSourceTask<TTask>

Implements

IKNetTask

ITask

IKNetConnectLogging

Derived

KNetConnectSourceTask

Inherited Members

KNetTask<TTask>.Version()

KNetTask.ExecuteOnTask(string, params object[])

KNetTask.ExecuteOnTask<T>(string, params object[])

KNetTask.DataToExchange<T>()

KNetTask.DataToExchange(object)

KNetTask.Context<T>()

KNetTask.Properties

KNetTask.Connector

KNetTask.TaskId

KNetTask.StartInternal()

KNetTask.Start(Map<String, String>)

KNetTask.Start(IReadOnlyDictionary<string, string>)

KNetTask.StopInternal()

KNetTask.Stop()

KNetTask.VersionInternal()

KNetTask.IsTraceEnabled

KNetTask.IsDebugEnabled

KNetTask.IsInfoEnabled

KNetTask.IsWarnEnabled

KNetTask.IsErrorEnabled

KNetTask.LogTrace(string)

KNetTask.LogTrace(string, JVMBridgeException)

KNetTask.LogDebug(string)

KNetTask.LogDebug(string, JVMBridgeException)

KNetTask.LogInfo(string)

KNetTask.LogInfo(string, JVMBridgeException)

KNetTask.LogWarn(string)

KNetTask.LogWarn(string, JVMBridgeException)

KNetTask.LogError(string)

KNetTask.LogError(string, JVMBridgeException)

object.ToString()

object.Equals(object)

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

Properties

Context

The SourceTaskContext

public SourceTaskContext Context { get; }

Property Value

SourceTaskContext

ReflectedTaskClassName

Set the ReflectedTaskClassName of the connector to a fixed value

public override string ReflectedTaskClassName { get; }

Property Value

string

Methods

CreateRecord<TValue>(string, Schema, TValue, DateTime)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<object, TValue> CreateRecord<TValue>(string topic, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

topic string

The name of the topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<object, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TValue>(string, int?, Schema, TValue, DateTime)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<object, TValue> CreateRecord<TValue>(string topic, int? partition, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<object, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, Schema, TKey, Schema, TValue)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, Schema keySchema, TKey key, Schema valueSchema, TValue value)

Parameters

topic string

The name of the topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, Schema, TValue)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, Schema valueSchema, TValue value)

Parameters

topic string

The name of the topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, int?, Schema, TKey, Schema, TValue)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value)

Parameters

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, int?, Schema, TKey, Schema, TValue, DateTime)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, int?, Schema, TKey, Schema, TValue, DateTime, Headers)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value, DateTime timestamp, Headers headers)

Parameters

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

headers Headers

The Headerss; may be null or empty

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKey, TValue>(string, int?, Schema, TValue)

Creates a new SourceRecord<TKey, TValue>

public SourceRecord<TKey, TValue> CreateRecord<TKey, TValue>(string topic, int? partition, Schema valueSchema, TValue value)

Parameters

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKey, TValue>

A newvly allocated SourceRecord<TKey, TValue>

Type Parameters

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values).

CreateRecord<TKeySource, TOffset, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, Schema, TValue, DateTime)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, object, TValue> CreateRecord<TKeySource, TOffset, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<TKeySource, TOffset, object, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, int?, Schema, TValue, DateTime)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, object, TValue> CreateRecord<TKeySource, TOffset, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, int? partition, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<TKeySource, TOffset, object, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, Schema, TKey, Schema, TValue)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, Schema keySchema, TKey key, Schema valueSchema, TValue value)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type within Map<K, V> of sourcePartition

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, Schema, TValue)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, Schema valueSchema, TValue value)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type within Map<K, V> of sourcePartition

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, int?, Schema, TKey, Schema, TValue)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type within Map<K, V> of sourcePartition

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, int?, Schema, TKey, Schema, TValue, DateTime)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value, DateTime timestamp)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type within Map<K, V> of sourcePartition

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, int?, Schema, TKey, Schema, TValue, DateTime, Headers)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, int? partition, Schema keySchema, TKey key, Schema valueSchema, TValue value, DateTime timestamp, Headers headers)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

keySchema Schema

The schema for the key; may be null

key TKey

The key; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

timestamp DateTime

The timestamp; may be null

headers Headers

The Headerss; may be null or empty

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type within Map<K, V> of sourcePartition

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource>, Map<String, TOffset>, string, int?, Schema, TValue)

Creates a new SourceRecord<TKeySource, TOffset, TKey, TValue>

public SourceRecord<TKeySource, TOffset, TKey, TValue> CreateRecord<TKeySource, TOffset, TKey, TValue>(Map<String, TKeySource> sourcePartition, Map<String, TOffset> sourceOffset, string topic, int? partition, Schema valueSchema, TValue value)

Parameters

sourcePartition Map<String, TKeySource>

The parameter represents a single input sourcePartition that the record came from (e.g. a filename, table name, or topic-partition).

sourceOffset Map<String, TOffset>

The parameter represents a position in that sourcePartition which can be used to resume consumption of data.

topic string

The name of the topic; may be null

partition int?

The partition number for the Kafka topic; may be null

valueSchema Schema

The schema for the value; may be null

value TValue

The value; may be null

Returns

SourceRecord<TKeySource, TOffset, TKey, TValue>

A newvly allocated SourceRecord<TKeySource, TOffset, TKey, TValue>

Type Parameters

TKeySource

The type within Map<K, V> of sourcePartition

TOffset

The type within Map<K, V> of sourceOffset

TKey

The type of the key to be inserted in Kafka

TValue

The type of value to be inserted in Kafka

Remarks

These values can have arbitrary structure and should be represented using Org.Apache.Kafka.Connect.Data.* objects (or primitive values). For example, a database connector might specify the sourcePartition as a record containing { "db": "database_name", "table": "table_name"} and the sourceOffset as a long containing the timestamp of the row.

OffsetAt<TKeySource, TOffset>(String, TKeySource)

Get the offset for the specified partition. If the data isn't already available locally, this gets it from the backing store, which may require some network round trips.

protected Map<String, TOffset> OffsetAt<TKeySource, TOffset>(String keyName, TKeySource keyValue)

Parameters

keyName String

The identifier used when was called OffsetForKey<T>(String, T)

keyValue TKeySource

The value used when was called OffsetForKey<T>(String, T)

Returns

Map<String, TOffset>

Return the Map<K, V> associated to the element identified from keyName and keyValue which is an object uniquely identifying the offset in the partition of data

Type Parameters

TKeySource

The type of the key set when was called OffsetForKey<T>(String, T) to generated first parameter of SourceRecord

TOffset

The type of the offset set when was called OffsetForKey<T>(String, T) to generated second parameter of SourceRecord

OffsetForKey<T>(String, T)

Generates a Map<K, V> to be used in SourceRecord

protected Map<String, T> OffsetForKey<T>(String identifier, T value)

Parameters

identifier String

The identifier to be associated in first, or second, parameter of a SourceRecord

value T

The value to be inserted and associated to the identifier

Returns

Map<String, T>

A Map<K, V>

Type Parameters

T

The identifier type

Poll()

Implement the method to execute the Poll action

public abstract IList<SourceRecord> Poll()

Returns

IList<SourceRecord>

The list of SourceRecord to return to Apache Kafka Connect framework

PollInternal()

Public method used from Java to trigger Poll()

public void PollInternal()