I/O API

Inputs

Every module can have zero or more inputs, as well as outputs. Inputs take data from other modules, outputs push data to subsequent modules. Inputs and outputs preferrably have one of the predefined types, but can in general take on any custom flatbuffer defined type.

The predefined types are

Event
Frame
IMU
Trigger
BoundingBox

Input definition

Inputs can only be defined in the initInputs static function. The number of inputs as well as their types and names must be fixed for every module, and can not be changed at runtime. However, to take on a variable number of input signals, inputs can me marked as optional. In that case, the code has to check at runtime if an input is connected.

Inputs get defined in the static void initInputs(dv::InputDefinitionList &in) function. The argument is a modifiabable container that offers functions to add inputs of different types:

in.addEventInput(const std::string &name, bool optional = false)
in.addFrameInput(const std::string &name, bool optional = false)
in.addIMUInput(const std::string &name, bool optional = false)
in.addTriggerInput(const std::string &name, bool optional = false)
in.addBoundingBoxInput(const std::string &name, bool optional = false)

To add an input for a custom flatbuffer type, use the following function. The type identifier is the four character type identifier string.

in.addInput(const std::string &name, const std::string &typeIdentifier, bool optional = false)

Inputs at runtime

Inputs can be accessed in constructor, run and configUpdate functions. Inheriting from dv::ModuleBase provides an object called inputs.

To retrieve an input with a known name and type, call the respective function

const auto input = inputs.getEventInput(const std::string &name);
const auto input = inputs.getFrameInput(const std::string &name);
const auto input = inputs.getIMUInput(const std::string &name);
const auto input = inputs.getTriggerInput(const std::string &name);
const auto input = inputs.getBoundingBoxInput(const std::string &name);

To retrieve an input for a custom flatbuffer type <T>, call

const auto input = inputs.getInput<T>(const std::string &name);

Input APIs

Not all input types share the same information. Therefore, the APIs for the different input types differ slightly.

Data

const auto inEvents = input.events();

Returns an (random access) iterable data structure of the latest events to arrive at the input. One can convenently loop over the arrays with for (const auto &event : input.events()) {}.

The returned value can also be implicitly converted to a const dv::EventStore. It can directly be passed to any function that takes a const dv::EventStore&

Meta data

int width = input.sizeX();
int height = input.sizeY();
cv::Size size = input.size();

Return the width / height of the input data. The maximum x-coordinate of an event at this input is input.sizeX() - 1. This value is depends on what module / camera is attached to the input at runtime. E.g. In case of a low-resolution DVS128 it would be 128, in case of a high-resolution DVS346 it would be 346. The same holds for the y-coordinate.

Frame Data

const auto frame = input.frame()

Individual Frame Meta Data

The following methods for accessing the frame contents are supported:

const auto &pixels = frame.pixels(); A reference to a vector type for raw pixel access
cv::Mat *matPtr = frame.getMatPointer(); A pointer to an OpenCV Mat representing this frame

Each frame has additional metadata, that can differ for each frame

long timestamp = frame.timestamp(); The timestamp of the frame
long timestamp = frame.timestampStartOfFrame(); The timestamp of the start of frame
long timestamp = frame.timestampEndOfFrame(); The timestamp of the end of frame
long timestamp = frame.timestampStartOfExposure(); The timestamp ofthe start of the exposure
long timestamp = frame.timestampEndOfExposure(); The timestamp ofthe end of the exposure
dv::FrameFormat format = frame.format(); The format of the frame. Options are GRAY, BGR, BGRA
int width = frame.sizeX(); The width of this specific frame. The width can be smaller or equal to the input width, but never larger.
int height = frame.sizeY(); The height of this specific frame. The height can be smaller or equal to the input height, but never larger.

Input Meta Data

int width = input.sizeX();
int height = input.sizeY();
cv::Size size = input.size();

Returns the maximum dimensions of the input. Individual frames on the input can be smaller than this, but never bigger.

Data

input.data()

Retrieved the newest data on the this input. The returned data type is (random access) iterable. A convient way to iterate through the newest data is to call for (const auto &sample : input.data()) {}

Data

input.data()

Retrieved the newest data on the this input. The returned data type is (random access) iterable. A convient way to iterate through the newest data is to call for (const auto &sample : input.data()) {}

Data

input.data()

Retrieved the newest data on the this input. The returned data type is (random access) iterable. A convient way to iterate through the newest data is to call for (const auto &boundingBox : input.data()) {}

Input Meta Data

int width = input.sizeX();
int height = input.sizeY();
cv::Size size = input.size();

Returns the maximum dimensions of the input. A bounding box on this input is always in respect to these dimensions.

Common for all inputs

The following functions are available for all input types, including custom flatbuffer types.

input.isConnected()

For optional inputs, this function returns true iff the input is connected to an output in the current runtime.

input.getOriginDescription()

Returns a string describing the origin of the data. In most cases, this string will give some information about the original creator of the data, like serial number of the camera. However, this is not guaranteed, as every module the data passes through is allowed to alter the string it passes down in any way.

input.data()

Advanced use Retrieved the newest data on the this input. The returned data type is equivalent to a shared pointer type. Dereferencing the return value gives the flatbuffer type <T> of the latest data.

input.infoNode()

Advanced use Returns the underlying config trees info node about the output connected to this input. This info node provides information about dimensions etc about the input. Convenience functions such as the event inputs sizeX etc. are based on this information. An input of a custom flatbuffer type can have arbitrary meta information about the connection, which an be obtained from the info node. For example, to get an integer with key sizeX from the info node, call input.infoNode().getInt("sizeX")

Outputs

Output definition

Outputs can only be defined in the static initOutputs function. The number, names and types of outputs have to be constant during runtime.

Outputs are defined in the static void initOutputs(dv::OutputDefinitionList &out) function. The argument is a modifiabable container that offers functions to add outputs of different types:

out.addEventOutput(const std::string &name)
out.addFrameOutput(const std::string &name)
out.addIMUOutput(const std::string &name)
out.addTriggerOutput(const std::string &name)
out.addBoundingBoxOutput(const std::string &name)

To add an output of a custom flatbuffer type, use the generic function with your custom types four-character type identifier.

out.addOutput(const std::string &name, const std::string &typeIdentifier)

Output setup

Upon initialization of your module, outputs have to be set up in the constructor. Setting up outputs means assigning them required meta information such as dimensions and source identifiers. In most cases, one wants to set up an output in terms of an input. For example, a filter module takes events of a certain dimension and wants to emit events with the same dimension. This is why setup takes place at initialization time, rather than static.

Event, Frame and Bounding Box outputs

output.setup(int sizeX, int sizeY, const std::string &originDescription)

Sets up the output with width sizeX and height sizeY and the supplied origin description. It is advisable to copy the origin description from an input, as the purpose of this field is to keep track of the original creator of the data.

output.setup(const RuntimeInput &input)

Sets up the output with the same parameters as a compatible input. Event, Frame and Bounding Box inputs are compatible with each other. For example, setting up a frame output with the same parameters as an Event input would look like outputs.getFrameOutput("frames").setup(inputs.getEventInput("events"));

Other outputs

output.setup(const std::string &originDescription)

Sets up the output with the given origin description. It is advisable to copy the origin description from an input, as the purpose of this field is to keep track of the original creator of the data.

Advanced use This is sufficient for the provided IMU and Trigger types. For a custom flatbuffer type, you may require additional information to be present in the output info config node. To set up these custom fields, put them in the output info node obtained by output.infoNode()

Send data to outputs

Data can be sent to outputs at any time in the run function. One can send as many data packets as needed (or none). Any sending of data has to happen on the thread where the run function runs on.

Sending data is easy. Depending on the datatype, we expose different convenience functions to use.

Events, IMU, Trigger

To send data to an output, simply stream the data to the desired output. In case of events, IMU, and trigger data, the piped data elements are first appended to an out packet. They only get sent out when piping in dv::commit.

outputs.getEventOutput("events") << event1 << event2 << dv::commit;

Alternatively, one can specifically obtain the the output container, append to it and commit it.

auto outEvents = outputs.getEventOutput("events").events();
outEvents.push_back(event1);
outEvents.push_back(event2);
outEvents.commit();

After committing, the output container gets reassigned and can be used again immediately after.

For IMU and Trigger outputs, one uses the data() function instead of the events() function. (The data() function is also available for events and functually identical).

auto outIMU = outputs.getIMUOutput("imu").data();
outIMU.push_back(imu1);
outIMU.push_back(imu2);
outIMU.commit();

The output containers for events, IMU and trigger also have stream operator support, as

auto outTrigger = outputs.getTriggerOutput("trigger").data();
outTrigger << trigger1 << trigger2;
outTrigger.commit();

All methods are equivalent in performance.

Frames

OpenCV Frames

To send an OpenCV frame to an output, simply use the stream operator on the output.

outputs.getFrameOutput("frames") << myFrame << dv::commit;

OpenCV frames do not have a notion of timestamps. By default, all frames sent out this way would have a timestamp of 0. To make sure the frames on the output have timestamps, one can stream in the timestamp first.

outputs.getFrameOutput("frames") << timestamp << myFrame << dv::commit;

Alternatively, one can get the frame at the output, and use that to commit the OpenCV matrix manually. This is used, when one wants to set additional parameters such as exposure times.

auto outFrame = outputs.getFrameOutput("frames").frame();
outFrame.setTimestamp(timestamp);
outFrame.setTimestampStartOfExposure(timeStartOfExposure);
outFrame.setTimestampEndOfExposure(timeEndOfExposure);
outFrame.setMat(openCVFrame);
outFrame.commit()

Non-OpenCV Frames

// Get current output frame
auto outFrame = outputs.getFrameOutput("frames").frame();
// Setting the format and size makes the pixel buffer allocate the right amount of storage
outFrame.setFormat(dv::FrameFormat::BGR);
outFrame.setSize(640, 480);
// Set the timestamp
outFrame.setTimestamp(timestamp);
// .pixels gives a writable, vector compatible buffer with enough storage.
// You can write the image in there with any method
std::copy(myBuffer, outFrame.pixels());
// As soon as all the data is ready, calling commit sends the data out
outFrame.commit();

Non OpenCV frames are bit harder to send out. The idea is to generate an empty array on the output, by setting the format as well as the size. Then write the image data in the buffer obtained by pixels(). As soon as all the data is ready, call commit() to send the frame out.

Note Set the size and format before writing any data to the pixel array. After calling commit, one has to set the size and format again for the next frame, which gives a new array.