ObjectDetector Class Reference

This class implements a modular and easy to use object detector for a single camera. Given a camera name, this class will detect objects using the depth measure from a ZED camera and/or inferenced objects from a custom trained model. This class and it's detections are ran in a different thread. More...

#include <ObjectDetector.h>

Inheritance diagram for ObjectDetector:

Collaboration diagram for ObjectDetector:

Public Member Functions
	ObjectDetector (BasicCamera *pBasicCam, const int nNumDetectedObjectsRetrievalThreads=5, const bool bUsingGpuMats=false)
	Construct a new ObjectDetector object.
	ObjectDetector (ZEDCamera *pZEDCam, const int nNumDetectedObjectsRetrievalThreads=5, const bool bUsingGpuMats=false)
	Construct a new ObjectDetector object.
std::future< bool >	RequestDepthDetectionOverlayFrame (cv::Mat &cvFrame)
	Request a copy of a frame containing the object detection overlays from the depth library.
std::future< bool >	RequestTensorflowDetectionOverlayFrame (cv::Mat &cvFrame)
	Request a copy of a frame containing the object detection overlays from the tensorflow model.
std::future< bool >	RequestDetectedDepthObjects (std::vector< depthobject::DepthObject > &vDepthObjects)
	Request the most up to date vector of detected objects from OpenCV's Depth algorithm.
std::future< bool >	RequestDetectedTensorflowObjects (std::vector< tensorflowobject::TensorflowObject > &vTensorflowObjects)
	Request the most up to date vector of detected objects from our custom tensorflow model.
IPS &	GetIPS ()
	Accessor for the Frame I P S private member.
Public Member Functions inherited from AutonomyThread< void >
	AutonomyThread ()
	Construct a new Autonomy Thread object.
virtual	~AutonomyThread ()
	Destroy the Autonomy Thread object. If the parent object or main thread is destroyed or exited while this thread is still running, a race condition will occur. Stopping and joining the thread here insures that the main program can't exit if the user forgot to stop and join the thread.
void	Start ()
	When this method is called, it starts a new thread that runs the code within the ThreadedContinuousCode method. This is the users main code that will run the important and continuous code for the class.
void	RequestStop ()
	Signals threads to stop executing user code, terminate. DOES NOT JOIN. This method will not force the thread to exit, if the user code is not written properly and contains WHILE statement or any other long-executing or blocking code, then the thread will not exit until the next iteration.
void	Join ()
	Waits for thread to finish executing and then closes thread. This method will block the calling code until thread is finished.
bool	Joinable () const
	Check if the code within the thread and all pools created by it are finished executing and the thread is ready to be closed.
AutonomyThreadState	GetThreadState () const
	Accessor for the Threads State private member.
IPS &	GetIPS ()
	Accessor for the Frame I P S private member.

Private Member Functions
void	ThreadedContinuousCode () override
	This code will run continuously in a separate thread. New frames from the given camera are grabbed and the objects for the camera image are detected, filtered, and stored. Then any requests for the current objects are fulfilled.
void	PooledLinearCode () override
	This method holds the code that is ran in the thread pool started by the ThreadedLinearCode() method. It copies the data from the different data objects to references of the same type stored in a queue filled by the Request methods.
void	UpdateDetectedObjects (std::vector< depthobject::DepthObject > &vNewlyDetectedObjects)
void	UpdateDetectedObjects (std::vector< tensorflowobject::TensorflowObject > &vNewlyDetectedObjects)

Private Attributes
Camera< cv::Mat > *	m_pCamera
bool	m_bUsingZedCamera
bool	m_bUsingGpuMats
int	m_nNumDetectedObjectsRetrievalThreads
IPS	m_IPS
std::vector< depthobject::DepthObject >	m_vDetectedDepthObjects
std::vector< tensorflowobject::TensorflowObject >	m_vDetectedTensorObjects
cv::Mat	m_cvNormalFrame
cv::Mat	m_cvProcFrame
cv::Mat	m_cvDepthMeasure
cv::cuda::GpuMat	m_cvGPUNormalFrame
cv::cuda::GpuMat	m_cvGPUDepthMeasure
std::queue< containers::FrameFetchContainer< cv::Mat > >	m_qDetectedObjectDrawnOverlayFrames
std::queue< containers::DataFetchContainer< std::vector< depthobject::DepthObject > > >	m_qDetectedDepthObjectCopySchedule
std::queue< containers::DataFetchContainer< std::vector< tensorflowobject::TensorflowObject > > >	m_qDetectedTensorflowObjectCopySchedule
std::shared_mutex	m_muPoolScheduleMutex
std::mutex	m_muFrameCopyMutex
std::mutex	m_muDepthDataCopyMutex
std::mutex	m_muTensorflowDataCopyMutex

Additional Inherited Members
Public Types inherited from AutonomyThread< void >
enum	AutonomyThreadState
Protected Member Functions inherited from AutonomyThread< void >
void	RunPool (const unsigned int nNumTasksToQueue, const unsigned int nNumThreads=2, const bool bForceStopCurrentThreads=false)
	When this method is called, it starts/adds tasks to a thread pool that runs nNumTasksToQueue copies of the code within the PooledLinearCode() method using nNumThreads number of threads. This is meant to be used as an internal utility of the child class to further improve parallelization. Default value for nNumThreads is 2.
void	RunDetachedPool (const unsigned int nNumTasksToQueue, const unsigned int nNumThreads=2, const bool bForceStopCurrentThreads=false)
	When this method is called, it starts a thread pool full of threads that don't return std::futures (like a placeholder for the thread return type). This means the thread will not have a return type and there is no way to determine if the thread has finished other than calling the Join() method. Only use this if you want to 'set and forget'. It will be faster as it doesn't return futures. Runs PooledLinearCode() method code. This is meant to be used as an internal utility of the child class to further improve parallelization.
void	ParallelizeLoop (const int nNumThreads, const N tTotalIterations, F &&tLoopFunction)
	Given a ref-qualified looping function and an arbitrary number of iterations, this method will divide up the loop and run each section in a thread pool. This function must not return anything. This method will block until the loop has completed.
void	ClearPoolQueue ()
	Clears any tasks waiting to be ran in the queue, tasks currently running will remain running.
void	JoinPool ()
	Waits for pool to finish executing tasks. This method will block the calling code until thread is finished.
bool	PoolJoinable () const
	Check if the internal pool threads are done executing code and the queue is empty.
void	SetMainThreadIPSLimit (int nMaxIterationsPerSecond=0)
	Mutator for the Main Thread Max I P S private member.
int	GetPoolNumOfThreads ()
	Accessor for the Pool Num Of Threads private member.
int	GetPoolQueueLength ()
	Accessor for the Pool Queue Size private member.
std::vector< void >	GetPoolResults ()
	Accessor for the Pool Results private member. The action of getting results will destroy and remove them from this object. This method blocks if the thread is not finished, so no need to call JoinPool() before getting results.
int	GetMainThreadMaxIPS () const
	Accessor for the Main Thread Max I P S private member.
Protected Attributes inherited from AutonomyThread< void >
IPS	m_IPS

Detailed Description

This class implements a modular and easy to use object detector for a single camera. Given a camera name, this class will detect objects using the depth measure from a ZED camera and/or inferenced objects from a custom trained model. This class and it's detections are ran in a different thread.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-24

Constructor & Destructor Documentation

ObjectDetector() [1/2]

ObjectDetector::ObjectDetector	(	BasicCamera *	pBasicCam,
		const int	nNumDetectedObjectsRetrievalThreads = 5,
		const bool	bUsingGpuMats = false
	)

Construct a new ObjectDetector object.

Parameters

pBasicCam	- A pointer to the BasicCam camera to get frames from for detection.
nNumDetectedObjectsRetrievalThreads	- The number of threads to use when fulfilling requests for the detected depth objects. Default is 5.
bUsingGpuMats	- Whether or not the given camera name will be using GpuMats.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-10

{
    // Initialize member variables.
    m_pCamera                             = pBasicCam;
    m_bUsingZedCamera                     = false;    // Toggle ZED functions off.
    m_nNumDetectedObjectsRetrievalThreads = nNumDetectedObjectsRetrievalThreads;
    m_bUsingGpuMats                       = bUsingGpuMats;
    m_IPS                                 = IPS();
}

ObjectDetector() [2/2]

ObjectDetector::ObjectDetector	(	ZEDCamera *	pZEDCam,
		const int	nNumDetectedObjectsRetrievalThreads = 5,
		const bool	bUsingGpuMats = false
	)

Construct a new ObjectDetector object.

Parameters

pZEDCam	- A pointer to the ZEDCam camera to get frames from for detection. Override for ZED camera.
nNumDetectedObjectsRetrievalThreads	- The number of threads to use when fulfilling requests for the detected depth objects. Default is 5.
bUsingGpuMats	- Whether or not the given camera name will be using GpuMats.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-07

{
    // Initialize member variables.
    m_pCamera                             = pZEDCam;
    m_bUsingZedCamera                     = true;    // Toggle ZED functions off.
    m_nNumDetectedObjectsRetrievalThreads = nNumDetectedObjectsRetrievalThreads;
    m_bUsingGpuMats                       = bUsingGpuMats;
}

Member Function Documentation

RequestDepthDetectionOverlayFrame()

std::future< bool > ObjectDetector::RequestDepthDetectionOverlayFrame

(

cv::Mat &

cvFrame

)

Request a copy of a frame containing the object detection overlays from the depth library.

Parameters

cvFrame

- The frame to copy the detection overlay image to.

Returns

std::future<bool> - The future that should be waited on before using the passed in frame. Future will be true or false based on whether or not the frame was successfully retrieved.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-11

{
    // Assemble the DataFetchContainer.
    containers::FrameFetchContainer<cv::Mat> stContainer(cvFrame, PIXEL_FORMATS::eDepthDetection);
 
    // Acquire lock on pool copy queue.
    std::unique_lock<std::shared_mutex> lkScheduler(m_muPoolScheduleMutex);
    // Append frame fetch container to the schedule queue.
    m_qDetectedObjectDrawnOverlayFrames.push(stContainer);
    // Release lock on the frame schedule queue.
    lkScheduler.unlock();
 
    // Return the future from the promise stored in the container.
    return stContainer.pCopiedFrameStatus->get_future();
}

RequestTensorflowDetectionOverlayFrame()

std::future< bool > ObjectDetector::RequestTensorflowDetectionOverlayFrame

(

cv::Mat &

cvFrame

)

Request a copy of a frame containing the object detection overlays from the tensorflow model.

Parameters

cvFrame

- The frame to copy the detection overlay image to.

Returns

std::future<bool> - The future that should be waited on before using the passed in frame. Future will be true or false based on whether or not the frame was successfully retrieved.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-11

{
    // Assemble the DataFetchContainer.
    containers::FrameFetchContainer<cv::Mat> stContainer(cvFrame, PIXEL_FORMATS::eTensorflowDetection);
 
    // Acquire lock on pool copy queue.
    std::unique_lock<std::shared_mutex> lkScheduler(m_muPoolScheduleMutex);
    // Append frame fetch container to the schedule queue.
    m_qDetectedObjectDrawnOverlayFrames.push(stContainer);
    // Release lock on the frame schedule queue.
    lkScheduler.unlock();
 
    // Return the future from the promise stored in the container.
    return stContainer.pCopiedFrameStatus->get_future();
}

RequestDetectedDepthObjects()

std::future< bool > ObjectDetector::RequestDetectedDepthObjects

(

std::vector< depthobject::DepthObject > &

vDepthObjects

)

Request the most up to date vector of detected objects from OpenCV's Depth algorithm.

Parameters

vDepthObjects

- The vector the detected depth objects will be saved to.

Returns

std::future<bool> - The future that should be waited on before using the passed in object vector. Future will be true or false based on whether or not the objects were successfully retrieved.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-07

{
    // Assemble the DataFetchContainer.
    containers::DataFetchContainer<std::vector<depthobject::DepthObject>> stContainer(vDepthObjects);
 
    // Acquire lock on pool copy queue.
    std::unique_lock<std::shared_mutex> lkScheduler(m_muPoolScheduleMutex);
    // Append detected object fetch container to the schedule queue.
    m_qDetectedDepthObjectCopySchedule.push(stContainer);
    // Release lock on the frame schedule queue.
    lkScheduler.unlock();
 
    // Return the future from the promise stored in the container.
    return stContainer.pCopiedDataStatus->get_future();
}

RequestDetectedTensorflowObjects()

std::future< bool > ObjectDetector::RequestDetectedTensorflowObjects

(

std::vector< tensorflowobject::TensorflowObject > &

vTensorflowObjects

)

Request the most up to date vector of detected objects from our custom tensorflow model.

Parameters

vTensorflowObjects

- The vector the detected tensorflow objects will be saved to.

Returns

std::future<bool> - The future that should be waited on before using the passed in object vector. Future will be true or false based on whether or not the objects were successfully retrieved.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-07

{
    // Assemble the DataFetchContainer.
    containers::DataFetchContainer<std::vector<tensorflowobject::TensorflowObject>> stContainer(vTensorflowObjects);
 
    // Acquire lock on pool copy queue.
    std::unique_lock<std::shared_mutex> lkScheduler(m_muPoolScheduleMutex);
    // Append detected object fetch container to the schedule queue.
    m_qDetectedTensorflowObjectCopySchedule.push(stContainer);
    // Release lock on the frame schedule queue.
    lkScheduler.unlock();
 
    // Return the future from the promise stored in the container.
    return stContainer.pCopiedDataStatus->get_future();
}

GetIPS()

IPS & ObjectDetector::GetIPS

(

)

Accessor for the Frame I P S private member.

Returns

IPS& - The detector objects iteration per second counter.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-10

{
    // Return Iterations Per Second counter.
    return m_IPS;
}

ThreadedContinuousCode()

void ObjectDetector::ThreadedContinuousCode ( )

overrideprivatevirtual

This code will run continuously in a separate thread. New frames from the given camera are grabbed and the objects for the camera image are detected, filtered, and stored. Then any requests for the current objects are fulfilled.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-07

Implements AutonomyThread< void >.

{
    // Create future for indicating when the frame has been copied.
    std::future<bool> fuNormalFrame;
    std::future<bool> fuDepthMeasureCopyStatus;
 
    // Check if the camera is setup to use CPU or GPU mats.
    if (m_bUsingZedCamera)
    {
        // Check if the ZED camera is returning cv::cuda::GpuMat or cv:Mat.
        if (m_bUsingGpuMats)
        {
            // Grabs normal frame and depth measure from ZEDCam. Dynamic casts Camera to ZEDCamera* so we can use ZEDCam methods.
            fuNormalFrame            = dynamic_cast<ZEDCamera*>(m_pCamera)->RequestFrameCopy(m_cvGPUNormalFrame);
            fuDepthMeasureCopyStatus = dynamic_cast<ZEDCamera*>(m_pCamera)->RequestDepthCopy(m_cvGPUDepthMeasure);
 
            // Wait for requested frames to be retrieved.
            if (fuDepthMeasureCopyStatus.get() && fuNormalFrame.get())
            {
                // Download mat from GPU memory.
                m_cvGPUNormalFrame.download(m_cvNormalFrame);
                m_cvGPUDepthMeasure.download(m_cvDepthMeasure);
            }
            else
            {
                // Submit logger message.
                LOG_WARNING(logging::g_qSharedLogger, "ObjectDetector unable to get normal frame or depth measure from ZEDCam!");
            }
        }
        else
        {
            // Grabs normal frame and depth measure from ZEDCam. Dynamic casts Camera to ZEDCamera* so we can use ZEDCam methods.
            fuNormalFrame            = dynamic_cast<ZEDCamera*>(m_pCamera)->RequestFrameCopy(m_cvNormalFrame);
            fuDepthMeasureCopyStatus = dynamic_cast<ZEDCamera*>(m_pCamera)->RequestDepthCopy(m_cvDepthMeasure);
 
            // Wait for requested frames to be retrieved.
            if (!fuDepthMeasureCopyStatus.get() || !fuNormalFrame.get())
            {
                // Submit logger message.
                LOG_WARNING(logging::g_qSharedLogger, "ObjectDetector unable to get normal frame or depth measure from ZEDCam!");
            }
        }
    }
    else
    {
        // Grab frames from camera.
        fuNormalFrame = dynamic_cast<BasicCamera*>(m_pCamera)->RequestFrameCopy(m_cvNormalFrame);
 
        // Wait for requested frames to be retrieved.
        if (!fuNormalFrame.get())
        {
            // Submit logger message.
            LOG_WARNING(logging::g_qSharedLogger, "ObjectDetector unable to get requested frames from BasicCam!");
        }
    }
 
    // Call detection methods and inference.
 
    // TODO: Implement when ready, commented out to suppress warnings.
    // Merge the newly detected objects with the pre-existing detected objects
    // this->UpdateDetectedObjects(vNewlyDetectedObjects);
 
    // Call FPS tick.
    m_IPS.Tick();
 
    // Acquire a shared_lock on the detected objects copy queue.
    std::shared_lock<std::shared_mutex> lkSchedulers(m_muPoolScheduleMutex);
    // Check if the detected object copy queue is empty.
    if (!m_qDetectedDepthObjectCopySchedule.empty() || !m_qDetectedTensorflowObjectCopySchedule.empty() || !m_qDetectedObjectDrawnOverlayFrames.empty())
    {
        size_t siQueueLength =
            std::max({m_qDetectedDepthObjectCopySchedule.size(), m_qDetectedTensorflowObjectCopySchedule.size(), m_qDetectedObjectDrawnOverlayFrames.size()});
        // Start the thread pool to store multiple copies of the detected objects to the requesting threads
        this->RunDetachedPool(siQueueLength, m_nNumDetectedObjectsRetrievalThreads);
        // Wait for thread pool to finish.
        this->JoinPool();
        // Release lock on frame copy queue.
        lkSchedulers.unlock();
    }
}

Here is the call graph for this function:

PooledLinearCode()

void ObjectDetector::PooledLinearCode ( )

overrideprivatevirtual

This method holds the code that is ran in the thread pool started by the ThreadedLinearCode() method. It copies the data from the different data objects to references of the same type stored in a queue filled by the Request methods.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-08

Implements AutonomyThread< void >.

{
    //  Detection Overlay Frame queue.
    // Acquire sole writing access to the detectedObjectCopySchedule.
    std::unique_lock<std::mutex> lkObjectOverlayFrameQueue(m_muFrameCopyMutex);
    // Check if there are unfulfilled requests.
    if (!m_qDetectedObjectDrawnOverlayFrames.empty())
    {
        // Get frame container out of queue.
        containers::FrameFetchContainer<cv::Mat> stContainer = m_qDetectedObjectDrawnOverlayFrames.front();
        // Pop out of queue.
        m_qDetectedObjectDrawnOverlayFrames.pop();
        // Release lock.
        lkObjectOverlayFrameQueue.unlock();
 
        // Check which frame we should copy.
        switch (stContainer.eFrameType)
        {
            case PIXEL_FORMATS::eDepthDetection: *(stContainer.pFrame) = m_cvProcFrame; break;
            case PIXEL_FORMATS::eTensorflowDetection: *(stContainer.pFrame) = m_cvProcFrame; break;
            default: *(stContainer.pFrame) = m_cvProcFrame;
        }
 
        // Signal future that the frame has been successfully retrieved.
        stContainer.pCopiedFrameStatus->set_value(true);
    }
 
    //  DepthObject queue.
    // Acquire sole writing access to the detectedObjectCopySchedule.
    std::unique_lock<std::mutex> lkDepthObjectQueue(m_muDepthDataCopyMutex);
    // Check if there are unfulfilled requests.
    if (!m_qDetectedDepthObjectCopySchedule.empty())
    {
        // Get frame container out of queue.
        containers::DataFetchContainer<std::vector<depthobject::DepthObject>> stContainer = m_qDetectedDepthObjectCopySchedule.front();
        // Pop out of queue.
        m_qDetectedDepthObjectCopySchedule.pop();
        // Release lock.
        lkDepthObjectQueue.unlock();
 
        // Copy the detected objects to the target location
        *(stContainer.pData) = m_vDetectedDepthObjects;
 
        // Signal future that the frame has been successfully retrieved.
        stContainer.pCopiedDataStatus->set_value(true);
    }
 
    //  TensorflowObject queue.
    // Acquire sole writing access to the detectedObjectCopySchedule.
    std::unique_lock<std::mutex> lkTensorflowObjectQueue(m_muTensorflowDataCopyMutex);
    // Check if there are unfulfilled requests.
    if (!m_qDetectedTensorflowObjectCopySchedule.empty())
    {
        // Get frame container out of queue.
        containers::DataFetchContainer<std::vector<tensorflowobject::TensorflowObject>> stContainer = m_qDetectedTensorflowObjectCopySchedule.front();
        // Pop out of queue.
        m_qDetectedTensorflowObjectCopySchedule.pop();
        // Release lock.
        lkTensorflowObjectQueue.unlock();
 
        // Copy the detected objects to the target location
        *(stContainer.pData) = m_vDetectedTensorObjects;
 
        // Signal future that the frame has been successfully retrieved.
        stContainer.pCopiedDataStatus->set_value(true);
    }
}

---

The documentation for this class was generated from the following files:

• src/vision/objects/ObjectDetector.h
• src/vision/objects/ObjectDetector.cpp