nicetoolbox.visual.media.components

Components module for defining various visual components.

Classes:

GazeIndividualComponent: Class for visualizing individual gaze data. BodyJointsComponent: Class for visualizing body joints data. HandJointsComponent: Class for visualizing hand joints data. FaceLandmarksComponent: Class for visualizing face landmarks data. GazeInteractionComponent: Class for visualizing gaze interaction data. ProximityComponent: Class for visualizing proximity data. KinematicsComponent: Class for visualizing kinematics data.

Classes

BodyJointsComponent	Class for visualizing body joint data.
BodyMeshComponent	Class for visualizing body mesh data.
Component	Abstract class for defining visual components.
EmotionIndividualComponent	Class for visualizing emotion individual data.
EyeClosedStateComponent	Class for reading eye closed/open state.
EyeClosureComponent	Class for visualizing eye closure (EAR) data.
FaceBoundingBoxComponent	Class for visualizing face bounding box data.
FaceLandmarksComponent	Class for visualizing face landmarks data.
GazeFusionComponent	Class for visualizing fused gaze.
GazeInteractionComponent	Class for visualizing gaze interaction data.
HandJointsComponent	Class for visualizing hand joints data.
HeadOrientationComponent	Class for visualizing head orientation data.
KinematicsComponent	Class for visualizing kinematics data.
ProximityComponent	Class for visualizing proximity data.

class nicetoolbox.visual.media.components.BodyJointsComponent(visualizer_config: Dict, io, logger, component_name: str)

Class for visualizing body joint data.

calculate_middle_eyes() → Tuple[ndarray, ndarray | None]

Calculate the middle of the eyes for the both dimensions. If has only one camera, then 3d returns as None

Returns:

The middle eyes 2d and 3d data.

Return type:

Tuple[np.ndarray, np.ndarray]

visualize(frame_idx: int) → None

Visualize the body joints component.

Combines the _log_data and _log_skeleton methods to visualize the body joints component in either 2D or 3D.

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.BodyMeshComponent(visualizer_config: Dict, io, logger, component_name: str, calib: Dict)

Class for visualizing body mesh data.

visualize(frame_idx: int) → None

Abstract method to visualize the component.

class nicetoolbox.visual.media.components.Component(visualizer_config, io, logger, component_name)

Abstract class for defining visual components.

visualizer_config

Configuration settings for the visualizer.

Type:

dict

component_name

The name of the component.

Type:

str

logger

The viewer object for logging the visualizations.

Type:

viewer.Viewer

algorithm_list

The list of algorithms used for the component.

Type:

list

component_prediction_folder

The path to the component prediction folder.

Type:

str

canvas_list

The list of canvases for the component.

Type:

list

algorithms_results

The list of algorithm results for the component.

Type:

list

canvas_data

The dictionary of canvas data for the component.

Type:

dict

abstract visualize()

Abstract method to visualize the component.

class nicetoolbox.visual.media.components.EmotionIndividualComponent(visualizer_config: Dict, io, logger, component_name: str, bbox_tuples: List[Tuple[ndarray, List[str]]] | None = None)

Class for visualizing emotion individual data.

Draws each subject’s face bounding box coloured by their strongest emotion and labelled with it. The boxes come from face_bounding_box (paired in via bbox_tuples) rather than from this component’s own NPZ, so the emotion overlay and the box overlay always agree on where the face is.

The emotion labels are read from the NPZ’s axis3, so whatever set the detector predicts is what gets drawn - the count is not assumed.

The valence_arousal array is additionally plotted as a scalar timeseries, one plot per tracked (subject, camera) pair with a line per dimension. Pairs the detector never tracked are all-NaN and are dropped rather than plotted empty.

camera_names

The camera names.

Type:

List[str]

subject_names

The subject names.

Type:

List[str]

bbox_per_alg

Per-algorithm (boxes, camera_names). Empty when no bounding box component is wired.

Type:

List[Tuple[np.ndarray, List[str]]]

valence_arousal_per_alg

Per-algorithm valence/arousal, or None when the canvas is empty or the detector does not emit it.

Type:

List[np.ndarray | None]

Initialize the EmotionIndividualComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

• bbox_tuples (List[Tuple[np.ndarray, List[str]]], optional) – Per-algorithm (bbox_2d, camera_names) from face_bounding_box, positionally aligned with this component’s algorithms. Defaults to None (nothing is drawn).

visualize(frame_idx: int) → None

Visualize the emotion individual component on each camera view.

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.EyeClosedStateComponent(visualizer_config: Dict, io, logger, component_name: str)

Class for reading eye closed/open state.

Carries no overlay of its own — like GazeInteractionComponent, it exists to hand its per-eye binary state to EyeClosureComponent, which recolors the eye contours with it.

camera_names

The camera names.

Type:

List[str]

subject_names

The subject names.

Type:

List[str]

Initialize the EyeClosedStateComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

get_closed_state_data() → List[Tuple[ndarray, List[str]]]

Get the closed-state data for every configured eye_closure_threshold instance, in list order.

Each entry corresponds positionally to [media.eye_closed_state].algorithms[i] and is intended to be paired with [media.eye_closure_score].algorithms[i] for coloring.

Returns:

List of (data, eye_labels) tuples, one per algorithm. Data is (subjects, cameras, frames, eyes) with 1 = closed, 0 = open, NaN = missing.

visualize(frame_idx: int) → None

Abstract method to visualize the component.

class nicetoolbox.visual.media.components.EyeClosureComponent(visualizer_config: Dict, io, logger, component_name: str, closed_state_tuples: List[Tuple[ndarray, List[str]]] | None = None, state_camera_names: List[str] | None = None)

Class for visualizing eye closure (EAR) data.

Draws a closed contour around each eye, mirroring the eye outlines the eye_closure_ear detector draws on its own mp4.

Everything is read from the component NPZ: eye_landmarks_2d carries the eye points already sliced out by the detector, labelled left_eye_* / right_eye_* on axis3, so splitting the points by label prefix is enough — no keypoint mapping lookup needed.

When an eye_closed_state instance is paired in (via closed_state_tuples), each contour is colored by that eye’s closed/open state instead of the algorithm’s static color.

The score array is additionally plotted as a scalar timeseries, one plot per tracked (subject, camera) pair with a line per eye. Pairs the detector never tracked are all-NaN and are dropped rather than plotted empty.

The contours are camera-view only; the detector produces no 3D eye data.

camera_names

The camera names.

Type:

List[str]

subject_names

The subject names.

Type:

List[str]

eye_indices_per_alg

Per algorithm, the (left, right) positions within the axis3 landmark labels.

Type:

List[Tuple[List[int], List[int]]]

closed_state_per_alg

Per-algorithm closed-state pair (data, eye_labels). Empty when no state component is wired.

Type:

List[Tuple[np.ndarray, List[str]]]

state_camera_names

The closed-state camera axis, which may be a different subset than this component’s own. Empty when no state component is wired.

Type:

List[str]

Initialize the EyeClosureComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

• closed_state_tuples (List[Tuple[np.ndarray, List[str]]], optional) – Per-algorithm (closed_state, eye_labels) from eye_closed_state, positionally aligned with this component’s algorithms. Defaults to None (static coloring).

• state_camera_names (List[str], optional) – The closed-state camera axis, used to index the state arrays by camera name. Defaults to None.

visualize(frame_idx: int) → None

Visualize the eye closure component on each camera view.

Each eye is drawn as its own entity so the two contours stay independently addressable in the viewer, and each tracked (subject, camera) EAR score is logged as its own scalar timeseries.

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.FaceBoundingBoxComponent(visualizer_config: Dict, io, logger, component_name: str)

Class for visualizing face bounding box data.

Draws one box per subject per camera view, labelled with the detection confidence.

The boxes also anchor other face-level overlays: EmotionIndividualComponent recolors them by the dominant emotion rather than drawing boxes of its own, so the two never disagree about where a face is.

camera_names

The camera names.

Type:

List[str]

subject_names

The subject names.

Type:

List[str]

Initialize the FaceBoundingBoxComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

get_bbox_data() → List[Tuple[ndarray, List[str]]]

Get the bounding boxes for every configured algorithm, in list order.

Each entry corresponds positionally to [media.face_bounding_box].algorithms[i] and is intended to be paired with a face-level component that draws on top of the boxes.

Returns:

List of (data, camera_names) tuples, one per algorithm. Data is (subjects, cameras, frames, x0/y0/x1/y1/conf), NaN where no face was assigned.

visualize(frame_idx: int) → None

Visualize the face bounding boxes on each camera view.

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.FaceLandmarksComponent(visualizer_config, io, logger, component_name)

Class for visualizing face landmarks data.

class nicetoolbox.visual.media.components.GazeFusionComponent(visualizer_config: Dict, io, logger, component_name: str, calib: Dict, eyes_middle_3d_data: ndarray | None = None, look_at_data_tuples: List[Tuple[ndarray, List[str]]] | None = None)

Class for visualizing fused gaze.

Each configured algorithm is a gaze_fusion instance (e.g. gaze_fusion_weighted, gaze_fusion_per_subject) — the component draws one overlay per instance so different fusion strategies can be compared side-by-side. All data (fused 3D direction, per-camera 2D reprojection, 2D face origin) is read from each fusion NPZ directly.

calib

The calibration parameters.

Type:

dict

camera_names

The camera names.

Type:

List[str]

subject_names

The subject names.

Type:

List[str]

eyes_middle_3d_data

The 3D eyes middle data (from body_joints).

Type:

np.ndarray

look_at_data

The look at data.

Type:

np.ndarray

look_at_labels

The look at labels.

Type:

List[str]

origins_per_alg

Per-camera 2D face origin per algorithm (S, C, F, 2).

Type:

List[np.ndarray]

fused_gaze_3d_per_alg

Fused 3D gaze per algorithm (S, 1, F, 4).

Type:

List[np.ndarray]

projected_gaze_2d_per_alg

Reprojected 2D gaze per algorithm (S, C, F, 3).

Type:

List[np.ndarray]

visualize(frame_idx: int) → None

Visualize each configured gaze_fusion instance as its own overlay.

Every algorithm in self.algorithm_list produces a full set of arrows (one per subject) drawn with that algorithm’s color. Look-at coloring, when enabled, overrides the static color only for alg_idx=0 (there is a single gaze_distance instance driving it).

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.GazeInteractionComponent(visualizer_config, io, logger, component_name)

Class for visualizing gaze interaction data.

Initialize the GazeInteractionComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

get_lookat_data() → List[Tuple[ndarray, List[str]]]

Get the look at data for every configured gaze_distance instance, in list order.

Each entry corresponds positionally to [media.gaze_interaction].algorithms[i] and is intended to be paired with [media.gaze_fusion].algorithms[i] for coloring.

Returns:

List of (data, labels) tuples, one per algorithm.

visualize()

Abstract method to visualize the component.

class nicetoolbox.visual.media.components.HandJointsComponent(visualizer_config, io, logger, component_name)

Class for visualizing hand joints data.

Initialize the HandJointsComponent by calling the BodyJointsComponent constructor.

Parameters:

• visualizer_config (dict) – The visualizer configuration settings.

• io – The input/output object.

• logger – The logger object.

• component_name (str) – The name of the component.

class nicetoolbox.visual.media.components.HeadOrientationComponent(visualizer_config: Dict, io, logger, component_name: str)

Class for visualizing head orientation data.

Initialize the HeadOrientationComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

visualize(frame_idx: int) → None

Visualize the head orientation component.

Combines the _log_data method to visualize the head orientation component in either 2D.

Parameters:

frame_idx (int) – The frame index.

class nicetoolbox.visual.media.components.KinematicsComponent(visualizer_config: Dict, io, logger, component_name: str)

Class for visualizing kinematics data.

Initialize the KinematicsComponent.

Reads the pre-aggregated velocity_bodypart_{dim} output produced by velocity_body (one detector instance per dim). Axes3 = bodypart labels, axis4 = [velocity, confidence].

visualize(frame_idx: int) → None

Visualize the kinematics component from the pre-aggregated per-bodypart velocity.

3D data lives on the single [“3d”] pseudo-camera slot (one metric per subject/bodypart); 2D data has one metric per (subject, camera, bodypart).

class nicetoolbox.visual.media.components.ProximityComponent(visualizer_config: Dict, io, logger, component_name: str, eyes_middle_3d_data: Tuple[ndarray, List[str]] | None = None, eyes_middle_2d_data: Tuple[ndarray, List[str]] | None = None)

Class for visualizing proximity data.

Initialize the ProximityComponent.

Parameters:

• visualizer_config (Dict) – The visualizer configuration settings.

• io – The input/output object.

• logger (viewer.Viewer) – The viewer rerun object.

• component_name (str) – The name of the component.

• eyes_middle_3d_data (Tuple[np.ndarray, List[str]], optional) – The 3D eyes middle data. Defaults to None.

• eyes_middle_2d_data (Tuple[np.ndarray, List[str]], optional) – The 2D eyes middle data. Defaults to None.

visualize(frame_idx: int) → None

Visualize the proximity component. 3D goes on the single [‘3d’] slot; 2D uses per-camera slots.