Installation

This connector can be used inside a Docker stack or installed as a general Kafka Connect plugin.

Developer guide

This MongoDB-Sink-Connector works based on RecordConverters to convert a SinkRecord to a Document. The default RecordConverter is GenericRecordConverter, which converts a record-key as _id and adds a field for every field-name from record-value. The GenericRecordConverter supports conversion of most of the primitive types and collections.

For Avro records with complex schemas, or for custom collection format it is recommended to write your own RecordConverter and register it to an extended RecordConverterFactory. Writing a custom RecordConverter is relatively straight forward. The interface requires two methods to be implemented.

/**
 * Converts Kafka records to MongoDB documents.
 */
public interface RecordConverter {
    /**
     * <p>The schema names used are the fully qualified (including namespace) and case-sensitive
     * names. If the converter requires records with both a key and a value schema, the  returned
     * format is "KeySchemaName-ValueSchemaName". If the key is not required, only "ValueSchemaName"
     * may be returned. KeySchemaName and ValueSchemaName may be substituted by the Object class
     * that it supports. If the converter supports all types of data, return null.
     */
    Collection<String> supportedSchemaNames();

    /**
     * Convert a Kafka record to a BSON document.
     *
     * @param record record to convert
     * @return BSON document
     * @throws DataException if the record cannot be converted by the current converter.
     */
    Document convert(SinkRecord record) throws DataException;
}

Info

For more details on configuring & installing MongoDb Sink Connector, refer to the readme on our Github repository.