TensorflowTPU< T, P > Class Template Reference

This class is designed to enable quick, easy, and robust handling of .tflite models for deployment and inference on the Coral EdgeTPU Accelerator. More...

#include <TensorflowTPU.hpp>

Public Types
enum class	DeviceType { eAuto , ePCIe , eUSB }
enum class	PerformanceModes { eLow , eMedium , eHigh , eMax }

Public Member Functions
	TensorflowTPU (std::string szModelPath, PerformanceModes ePowerMode=PerformanceModes::eHigh, unsigned int unMaxBulkInQueueLength=32, bool bUSBAlwaysDFU=false)
	Construct a new TensorflowTPU object.
	~TensorflowTPU ()
	Destroy the TensorflowTPU object.
void	CloseHardware ()
	Release all hardware and reset models and interpreters.
TfLiteStatus	OpenAndLoad (DeviceType eDeviceType=DeviceType::eAuto)
	Attempt to open the model at the given path and load it onto the EdgeTPU device.
bool	GetDeviceIsOpened () const
	Accessor for the Device Is Opened private member.

Static Public Member Functions
static std::vector< edgetpu::EdgeTpuManager::DeviceEnumerationRecord >	GetHardwareDevices ()
	Retrieve a list of EdgeTPU devices from the edge API.
static std::vector< std::shared_ptr< edgetpu::EdgeTpuContext > >	GetOpenedHardwareDevices ()
	Retrieve a list of already opened EdgeTPU devices from the edge API.

Protected Member Functions
edgetpu::EdgeTpuManager *	GetEdgeManager ()
	Retrieves a pointer to an EdgeTPUManager instance from the libedgetpu library.
std::string	DeviceTypeToString (edgetpu::DeviceType eDeviceType)
	to_string method for converting a device type to a readable string.

Protected Attributes
std::string	m_szModelPath
edgetpu::EdgeTpuManager::DeviceEnumerationRecord	m_tpuDevice
edgetpu::EdgeTpuManager::DeviceOptions	m_tpuDeviceOptions
std::unique_ptr< tflite::FlatBufferModel >	m_pTFLiteModel
std::shared_ptr< edgetpu::EdgeTpuContext >	m_pEdgeTPUContext
std::unique_ptr< tflite::Interpreter >	m_pInterpreter
bool	m_bDeviceOpened

Private Member Functions
virtual T	Inference (const P &tInput, const float fMinObjectConfidence, const float fNMSThreshold)=0

Detailed Description

template<typename T, typename P>
class TensorflowTPU< T, P >

This class is designed to enable quick, easy, and robust handling of .tflite models for deployment and inference on the Coral EdgeTPU Accelerator.

Template Parameters

T	- The type used for the return of the Inference() method.
P	- The type used for the argument of the Inference() method.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-24

Member Enumeration Documentation

DeviceType

template<typename T , typename P >

enum class TensorflowTPU::DeviceType

strong

        {
            eAuto,    // Any open device will be picked. Prioritizes PCIe device if not already in use.
            ePCIe,    // Attempt to use a PCIe device for this model.
            eUSB      // Attempt to use a USB device for this model.
        };

PerformanceModes

template<typename T , typename P >

enum class TensorflowTPU::PerformanceModes

strong

        {
            eLow,       // Power saver mode. Low power draw and little heat output, but not great performance.
            eMedium,    // Balanced. Medium power draw, medium performance.
            eHigh,      // Performance mode. High power draw and increased heat output, great performance.
            eMax        // Maximum clock speed. Max performance, but greatest power draw and heat output. Could damage device in hot environments.
        };

Constructor & Destructor Documentation

TensorflowTPU()

template<typename T , typename P >

TensorflowTPU< T, P >::TensorflowTPU ( std::string  szModelPath, PerformanceModes  ePowerMode = PerformanceModes::eHigh, unsigned int  unMaxBulkInQueueLength = 32, bool  bUSBAlwaysDFU = false  )

inline

Construct a new TensorflowTPU object.

Parameters

szModelPath	- The path to the model to open and inference on the EdgeTPU.
ePowerMode	- The desired power mode of the device.
unMaxBulkInQueueLength	- Input queue length for device. Larger queue may improve USB performance going from device to host.
bUSBAlwaysDFU	- Whether or not to always reload firmware into the device after when this object is created.

Note

The given model must be a tflite model custom compiled to map operations to the EdgeTPU refer to https://coral.ai/docs/edgetpu/models-intro/#compiling and https://coral.ai/docs/edgetpu/compiler/#system-requirements

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Initialize member variables.
            m_szModelPath                                  = szModelPath;
            m_tpuDeviceOptions["Usb.MaxBulkInQueueLength"] = std::to_string(unMaxBulkInQueueLength);
            m_bDeviceOpened                                = false;
 
            // Determine which power mode should be set.
            switch (ePowerMode)
            {
                case PerformanceModes::eLow: m_tpuDeviceOptions["Performance"] = "Low"; break;
                case PerformanceModes::eMedium: m_tpuDeviceOptions["Performance"] = "Medium"; break;
                case PerformanceModes::eHigh: m_tpuDeviceOptions["Performance"] = "High"; break;
                case PerformanceModes::eMax: m_tpuDeviceOptions["Performance"] = "Max"; break;
                default: m_tpuDeviceOptions["Performance"] = "High"; break;
            }
 
            // Determine if firmware should be loaded ever time code is started.
            if (bUSBAlwaysDFU)
            {
                // Always load firmware.
                m_tpuDeviceOptions["Usb.AlwaysDfu"] = "True";
            }
            else
            {
                // Only load firmware on first init of device.
                m_tpuDeviceOptions["Usb.AlwaysDfu"] = "False";
            }
        }

~TensorflowTPU()

template<typename T , typename P >

TensorflowTPU< T, P >::~TensorflowTPU ( )

inline

Destroy the TensorflowTPU object.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-10-24

        {
            // Check if device has been opened.
            if (m_bDeviceOpened)
            {
                // Close tflite interpreter.
                m_pInterpreter.reset();
                // Close edgetpu hardware.
                m_pEdgeTPUContext.reset();
                // Close model.
                m_pTFLiteModel.reset();
            }
        }

Member Function Documentation

CloseHardware()

template<typename T , typename P >

void TensorflowTPU< T, P >::CloseHardware ( )

inline

Release all hardware and reset models and interpreters.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2024-03-31

        {
            // Set opened toggle.
            m_bDeviceOpened = false;
            // Close tflite interpreter.
            m_pInterpreter.reset();
            // Close edgetpu hardware.
            m_pEdgeTPUContext.reset();
            // Close model.
            m_pTFLiteModel.reset();
        }

OpenAndLoad()

template<typename T , typename P >

TfLiteStatus TensorflowTPU< T, P >::OpenAndLoad ( DeviceType  eDeviceType = DeviceType::eAuto )

inline

Attempt to open the model at the given path and load it onto the EdgeTPU device.

Parameters

eDeviceType

- An enumerator specifying which device this model should run on. (PCIe, USB, or autoselect)

Returns

TfLiteStatus - The Tensorflow Lite status of the model interpreter. Status will be TfLiteOk if model was successfully opened and loaded onto the device.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Create instance variables.
            TfLiteStatus tfReturnStatus = TfLiteStatus::kTfLiteCancelled;
            std::vector<edgetpu::EdgeTpuManager::DeviceEnumerationRecord> vValidDevices;
 
            // Determine which device is going to be used for this model.
            switch (eDeviceType)
            {
                case DeviceType::eAuto: m_tpuDevice.type = edgetpu::DeviceType(-1); break;
                case DeviceType::ePCIe: m_tpuDevice.type = edgetpu::DeviceType::kApexPci; break;
                case DeviceType::eUSB: m_tpuDevice.type = edgetpu::DeviceType::kApexUsb; break;
                default: m_tpuDevice.type = edgetpu::DeviceType(-1); break;
            }
 
            // Load compiled Edge TPU model as a flatbuffer model.
            m_pTFLiteModel = tflite::FlatBufferModel::VerifyAndBuildFromFile(m_szModelPath.c_str());
            // Check if model was successfully opened.
            if (m_pTFLiteModel != nullptr)
            {
                // Get a list of available devices and already opened devices.
                std::vector<edgetpu::EdgeTpuManager::DeviceEnumerationRecord> vDevices      = this->GetHardwareDevices();
                std::vector<std::shared_ptr<edgetpu::EdgeTpuContext>> vAlreadyOpenedDevices = this->GetOpenedHardwareDevices();
 
                // Get list of valid, unopened devices.
                // Loop through available devices.
                for (unsigned int unIter = 0; unIter < vDevices.size(); ++unIter)
                {
                    // Create instance variables.
                    bool bValidDevice = true;
 
                    // Loop through all opened devices.
                    for (unsigned int nJter = 0; nJter < vAlreadyOpenedDevices.size(); ++nJter)
                    {
                        // Check if current available device has already been opened.
                        if (vAlreadyOpenedDevices[nJter]->GetDeviceEnumRecord().path == vDevices[unIter].path)
                        {
                            // Set device as not valid.
                            bValidDevice = false;
                        }
                        // Determine if we should check device type.
                        else if (eDeviceType != DeviceType::eAuto)
                        {
                            // Check if device type matches.
                            if (vDevices[unIter].type != m_tpuDevice.type)
                            {
                                // Set device as not valid.
                                bValidDevice = false;
                            }
                        }
                    }
 
                    // Check if still valid.
                    if (bValidDevice)
                    {
                        // Append to valid devices vector.
                        vValidDevices.emplace_back(vDevices[unIter]);
                    }
                }
 
                // Check if any valid devices were found.
                if (vValidDevices.size() > 0)
                {
                    // Loop through each device until one successfully opens.
                    for (unsigned int unIter = 0; unIter < vValidDevices.size() && !m_bDeviceOpened; ++unIter)
                    {
                        // Submit logger message.
                        LOG_INFO(logging::g_qSharedLogger,
                                 "Attempting to load {} onto {} device at {} ({})...",
                                 m_szModelPath,
                                 this->DeviceTypeToString(vValidDevices[unIter].type),
                                 vValidDevices[unIter].path,
                                 this->DeviceTypeToString(vValidDevices[unIter].type));
 
                        // Attempt to open device.
                        m_pEdgeTPUContext = this->GetEdgeManager()->OpenDevice(vValidDevices[unIter].type, vValidDevices[unIter].path, m_tpuDeviceOptions);
 
                        // Only proceed if device opened.
                        if (m_pEdgeTPUContext != nullptr && m_pEdgeTPUContext->IsReady())
                        {
                            // Create custom tflite operations for edge tpu.
                            tflite::ops::builtin::BuiltinOpResolverWithXNNPACK tfResolver;
                            tfResolver.AddCustom(edgetpu::kCustomOp, edgetpu::RegisterCustomOp());
                            // Create tflite interpreter with model and operations resolver.
                            if (tflite::InterpreterBuilder(*m_pTFLiteModel, tfResolver)(&m_pInterpreter) != kTfLiteOk)
                            {
                                // Submit logger message.
                                LOG_ERROR(logging::g_qSharedLogger,
                                          "Unable to build interpreter for model {} with device {} ({})",
                                          m_szModelPath,
                                          vValidDevices[unIter].path,
                                          this->DeviceTypeToString(vValidDevices[unIter].type));
 
                                // Release interpreter and context.
                                m_pInterpreter.reset();
                                m_pEdgeTPUContext.reset();
 
                                // Update return status.
                                tfReturnStatus = TfLiteStatus::kTfLiteUnresolvedOps;
                            }
                            else
                            {
                                // Bind the given context device with interpreter.
                                m_pInterpreter->SetExternalContext(kTfLiteEdgeTpuContext, m_pEdgeTPUContext.get());
                                // Attempt to allocate necessary tensors for model onto device.
                                if (m_pInterpreter->AllocateTensors() != kTfLiteOk)
                                {
                                    // Submit logger message.
                                    LOG_WARNING(logging::g_qSharedLogger,
                                                "Even though device was opened and interpreter was built, allocation of tensors failed for model {} with device {} ({})",
                                                m_szModelPath,
                                                vValidDevices[unIter].path,
                                                this->DeviceTypeToString(vValidDevices[unIter].type));
 
                                    // Release interpreter and context.
                                    m_pInterpreter.reset();
                                    m_pEdgeTPUContext.reset();
 
                                    // Update return status.
                                    tfReturnStatus = TfLiteStatus::kTfLiteDelegateDataWriteError;
                                }
                                else
                                {
                                    // Submit logger message.
                                    LOG_INFO(logging::g_qSharedLogger,
                                             "Successfully opened and loaded model {} with device {} ({})",
                                             m_szModelPath,
                                             vValidDevices[unIter].path,
                                             this->DeviceTypeToString(vValidDevices[unIter].type));
 
                                    // Set toggle that model is opened with device.
                                    m_bDeviceOpened = true;
 
                                    // Update return status.
                                    tfReturnStatus = TfLiteStatus::kTfLiteOk;
                                }
                            }
                        }
                        else
                        {
                            // Submit logger message.
                            LOG_ERROR(logging::g_qSharedLogger,
                                      "Unable to open device {} ({}) for model {}.",
                                      vValidDevices[unIter].path,
                                      this->DeviceTypeToString(vValidDevices[unIter].type),
                                      m_szModelPath);
                        }
                    }
                }
                else
                {
                    // Submit logger message.
                    LOG_ERROR(logging::g_qSharedLogger,
                              "No valid devices were found for model {}. Device type is {}",
                              m_szModelPath,
                              this->DeviceTypeToString(m_tpuDevice.type));
                }
            }
            else
            {
                // Submit logger message.
                LOG_ERROR(logging::g_qSharedLogger, "Unable to load model {}. Does it exist at this path? Is this actually compiled for the EdgeTPU?", m_szModelPath);
            }
 
            // Return status.
            return tfReturnStatus;
        }

Here is the call graph for this function:

Here is the caller graph for this function:

GetDeviceIsOpened()

template<typename T , typename P >

bool TensorflowTPU< T, P >::GetDeviceIsOpened ( ) const

inline

Accessor for the Device Is Opened private member.

Returns

true - Model has been successfully opened and loaded onto device.

false - Model has not yet been opened or loaded onto a device.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

{ return m_bDeviceOpened; }

GetHardwareDevices()

template<typename T , typename P >

static std::vector< edgetpu::EdgeTpuManager::DeviceEnumerationRecord > TensorflowTPU< T, P >::GetHardwareDevices ( )

inlinestatic

Retrieve a list of EdgeTPU devices from the edge API.

Returns

std::vector<edgetpu::EdgeTpuManager::DeviceEnumerationRecord> - A vector containing device records for currently connected devices. Each device record contains a type (usb/pcie) and path.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Create instance variables.
            edgetpu::EdgeTpuManager* tpuEdgeManagerInstance = edgetpu::EdgeTpuManager::GetSingleton();
 
            // Check if edgetpu singleton objects are supported.
            if (tpuEdgeManagerInstance != nullptr)
            {
                // Get a list of devices from the edgetpu api.
                return tpuEdgeManagerInstance->EnumerateEdgeTpu();
            }
            else
            {
                // Return empty vector.
                return std::vector<edgetpu::EdgeTpuManager::DeviceEnumerationRecord>();
            }
        }

Here is the caller graph for this function:

GetOpenedHardwareDevices()

template<typename T , typename P >

static std::vector< std::shared_ptr< edgetpu::EdgeTpuContext > > TensorflowTPU< T, P >::GetOpenedHardwareDevices ( )

inlinestatic

Retrieve a list of already opened EdgeTPU devices from the edge API.

Returns

std::vector<edgetpu::EdgeTpuManager::DeviceEnumerationRecord> - A vector containing device records for currently opened devices. Each device record contains a type (usb/pcie) and path.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Create instance variables.
            edgetpu::EdgeTpuManager* tpuEdgeManagerInstance = edgetpu::EdgeTpuManager::GetSingleton();
 
            // Check if edgetpu singleton objects are supported.
            if (tpuEdgeManagerInstance != nullptr)
            {
                // Get a list of devices from the edgetpu api.
                return tpuEdgeManagerInstance->GetOpenedDevices();
            }
            else
            {
                // Return empty vector.
                return std::vector<std::shared_ptr<edgetpu::EdgeTpuContext>>();
            }
        }

Here is the caller graph for this function:

GetEdgeManager()

template<typename T , typename P >

edgetpu::EdgeTpuManager * TensorflowTPU< T, P >::GetEdgeManager ( )

inlineprotected

Retrieves a pointer to an EdgeTPUManager instance from the libedgetpu library.

Returns

edgetpu::EdgeTpuManager* - A pointer to the manager. Will be nullptr if not supported on this operating system.

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Create instance variables.
            edgetpu::EdgeTpuManager* tpuEdgeManagerInstance = edgetpu::EdgeTpuManager::GetSingleton();
 
            // Check if edgetpu singleton objects are supported.
            if (tpuEdgeManagerInstance == nullptr)
            {
                // Submit logger message.
                LOG_CRITICAL(logging::g_qSharedLogger, "Unable to get EdgeTPU manager! This operating system does not support singletons.");
            }
 
            // Get a list of devices from the edgetpu api.
            return tpuEdgeManagerInstance;
        }

Here is the caller graph for this function:

DeviceTypeToString()

template<typename T , typename P >

std::string TensorflowTPU< T, P >::DeviceTypeToString ( edgetpu::DeviceType  eDeviceType )

inlineprotected

to_string method for converting a device type to a readable string.

Parameters

eDeviceType

- The edgetpu device type. (kApexUsb or kApexPci)

Returns

std::string - The equivalent string. (USB or PCIe)

Author

clayjay3 (clayt.nosp@m.onra.nosp@m.ycowe.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om)

Date

2023-11-11

        {
            // Determine which device type string should be returned.
            switch (eDeviceType)
            {
                case edgetpu::DeviceType::kApexUsb: return "USB";
                case edgetpu::DeviceType::kApexPci: return "PCIe";
                default: return "Not Found";
            }
        }

Here is the caller graph for this function:

Inference()

template<typename T , typename P >

virtual T TensorflowTPU< T, P >::Inference ( const P &  tInput, const float  fMinObjectConfidence, const float  fNMSThreshold  )

privatepure virtual

Implemented in yolomodel::tensorflow::TPUInterpreter.

---

The documentation for this class was generated from the following file:

• src/interfaces/TensorflowTPU.hpp