🍿
IoT Snacks
Part 7by Muhammad

TinyML vs Full ML: When to use which (and why it matters in IoT)

TinyML vs Full ML When to use which

TinyML vs Full ML: When to use which is a practical decision you make on almost every real IoT machine learning project. This comparison breaks down the tradeoffs across latency, cost, accuracy, power, connectivity, privacy and maintainability, with concrete selection rules and code you can run. It is written for intermediate embedded and IoT engineers who already know basic machine learning concepts and want deployment level clarity.

Table of Contents

TinyML vs Full ML: When to use which

This decision is less about which approach is “better” and more about where inference must happen and what constraints are non negotiable. In IoT, the biggest constraints usually come from one of these: power budget, connectivity reliability, real time latency, unit cost, privacy rules or the complexity of the task.

You can treat TinyML as “do enough ML on the device to meet product requirements” and Full ML as “use bigger models and stronger hardware to maximize capability” (often on an edge gateway or cloud). Many successful products combine both using a tiered architecture.

Definitions and where they run

What people mean by TinyML

TinyML usually means running machine learning inference on microcontrollers (MCUs) or similarly constrained embedded targets. Typical targets are Arm Cortex-M, ESP32 class SoCs or low power RISC-V MCUs. In practice, TinyML systems often use:

  • Quantized models (commonly int8) to reduce memory and compute
  • Small architectures (keyword spotting CNNs, anomaly detection with 1D CNNs, tiny vision models for low resolution images)
  • On-device feature extraction like MFCCs for audio or simple DSP pipelines
  • Frameworks like TensorFlow Lite for Microcontrollers (TFLM), CMSIS-NN, microTVM or vendor SDK runtimes

What people mean by Full ML

Full ML typically means you run larger models on a capable CPU, GPU, TPU or NPU, either:

  • In the cloud (managed inference endpoints, autoscaling, global availability)
  • On an edge server or gateway (x86, Arm Linux, Jetson, Intel NPU, Coral TPU)
  • On a high-end embedded Linux device (SoC with NPU, more RAM and storage)

Full ML is where you can deploy transformers, larger convolutional networks, ensembles, retrieval augmented generation (RAG) pipelines and more advanced monitoring and retraining loops.

Decision matrix (quick picks)

If you only read one section, use this matrix. The “winner” indicates what usually fits best.

Constraint / Goal TinyML (on MCU) Full ML (edge server or cloud)
Hard real-time response (< 50 ms) Best (no network round trip) Possible at edge, risky via cloud
Always offline / intermittent connectivity Best Only if you can run on an on-prem edge server
Very low power (coin cell, energy harvesting) Best (if model fits and duty cycle is low) Not applicable unless inference is rare and offloaded
Complex tasks (rich vision, language, multi-sensor fusion) Limited Best
Strict privacy (no raw data leaves device) Best (local inference) Possible with on-prem edge and strong controls
Fast iteration, A/B testing, frequent model updates Harder (firmware lifecycle) Best (CI/CD for models)
Lowest per-unit BOM cost at scale Often best (MCU is cheap) Can win if it lets you simplify sensors and device compute
Lowest cloud bill Best Can be expensive at high volume
Security attack surface Smaller network exposure but harder to patch globally Centralized patching but broader internet facing exposure

Latency, bandwidth and connectivity

Latency: what actually matters

When latency matters, measure end-to-end time from “sensor event happens” to “actuator response”. TinyML often wins because it avoids:

  • Wi-Fi association delays
  • Cellular uplink scheduling delays
  • Backhaul jitter
  • Cloud queueing and cold starts

Full ML can still meet low latency if inference runs on a local edge server on the same LAN and your device publishes only lightweight features. Cloud inference is usually the last choice for hard real-time control loops.

Bandwidth: raw sensor data is expensive

Streaming raw audio, high-rate vibration or images quickly becomes expensive and power hungry. A common threshold is: if you cannot afford to send raw data continuously, you either need TinyML for local decisions or a hybrid approach where the device sends only features or short event clips.

Practical examples:

  • Vibration monitoring: send features (RMS, kurtosis, band energies) and only upload raw windows when an anomaly triggers
  • Audio: do keyword spotting locally, upload audio only on trigger or for periodic audits
  • Vision: do motion detection or low-res prefiltering locally, then upload selected frames to an edge server

Power and battery life

Power is often the deciding factor for TinyML. A microcontroller running an int8 model for 10 to 50 ms and sleeping the rest of the time can stay within a tight energy budget. A radio that transmits frequently can dominate power consumption, so local inference can save energy by reducing transmissions.

Rules of thumb that hold up in many deployments:

  • Radio costs more than compute for small inference workloads, especially on cellular and Wi-Fi
  • Duty cycle is everything: even an efficient MCU model burns power if you run it continuously
  • Memory accesses matter: poorly optimized models waste energy on RAM and flash traffic

If you already have a powered gateway (industrial PC, router, vehicle ECU), Full ML at the edge can be power neutral for the sensor nodes because it lets them stay simple and low power.

Accuracy and model capability

Model size and the “capability ceiling”

Full ML generally gives you a higher ceiling: larger receptive fields, better robustness, richer representations and the ability to use modern architectures (transformers, larger CNN backbones, multi-modal fusion). TinyML models can be surprisingly good, but you will hit limits when you need:

  • High resolution vision (object detection on 640×480 and up)
  • Fine-grained classification with many classes
  • General language understanding or generation
  • Complex temporal modeling (long context) beyond small recurrent or 1D CNN approaches

Quantization effects

TinyML often relies on int8 quantization. Many models tolerate this well, but some are sensitive, especially if you also prune aggressively. If you need the last few points of accuracy for a safety critical classifier, Full ML on a more capable edge device may be the safer path.

Sensor quality can beat model size

A useful reminder in IoT: model improvements cannot always compensate for poor sensing. Sometimes switching from a cheap mic to a better mic array, adding an accelerometer axis or improving analog front-end filtering will make a TinyML approach viable. This is one of the biggest levers you have if you want to keep inference on device.

Privacy, security and compliance

TinyML is attractive when privacy requirements prohibit sending raw data off-device. If you can do on-device inference and only transmit metadata or aggregated scores, you reduce exposure and often simplify compliance for sensitive signals (audio, images, biometric adjacent sensors).

Full ML is not automatically “less private”. You can run Full ML in an on-prem edge environment, use confidential computing, encrypt data in transit and at rest and apply strict access controls. The difference is operational: more moving parts means more policy and monitoring work.

Security tradeoffs to consider:

  • TinyML devices: smaller remote attack surface if they rarely connect, but patching and fleet management can be harder
  • Full ML services: easier centralized patching and observability, but you must harden endpoints and manage credentials at scale

Total cost (BOM, cloud and ops)

BOM and hardware cost

TinyML can reduce bill of materials (BOM) by letting you use a low-cost MCU and avoid a Linux class SoC. That said, TinyML can also push you toward a slightly bigger MCU (more flash and RAM) than you would otherwise need.

Full ML may increase device BOM if you need an NPU capable gateway or a more powerful edge box, but it can also reduce sensor node cost by keeping them “dumb” and offloading compute.

Cloud cost and scaling

If you ship 50,000 devices streaming data for cloud inference, cloud costs can eclipse hardware costs. TinyML reduces cloud inference volume by filtering at the edge. If you still need cloud analytics, you can upload summaries, features or periodic samples.

Ops cost (often underestimated)

Full ML usually requires more MLOps (machine learning operations): model registry, deployment pipelines, monitoring drift and data labeling loops. TinyML requires strong embedded release processes, over-the-air (OTA) firmware updates and on-device telemetry. Both have ongoing costs, just in different places.

Development workflow and MLOps

TinyML workflow realities

  • Debugging: you need tooling to inspect intermediate tensors, timing, memory usage and quantization issues
  • Data collection: you often must collect data on the real device because sensor placement and noise matter
  • Updates: model updates often ship as firmware, which means tighter validation and staged rollouts

Full ML workflow realities

  • Rapid iteration: you can deploy models daily without touching device firmware
  • Observability: easier to log features and predictions, run shadow deployments and do canary releases
  • Dependency management: you must manage Python environments, CUDA versions or runtime compatibility

For “living” models that require frequent retraining, Full ML usually fits better. For stable tasks like keyword spotting or anomaly detection on a fixed machine, TinyML can run for years with minimal updates.

Hardware reality check (what fits on MCUs)

Before you commit to on-device inference, do a quick feasibility check. These are common ranges, not guarantees:

  • Flash: 256 KB to a few MB for firmware plus model weights
  • RAM: 64 KB to 512 KB is common, a few MB on higher-end MCUs
  • Compute: tens to a few hundred MHz, often with DSP instructions

Two memory buckets matter:

  • Weights: stored in flash (int8 helps a lot)
  • Activation arena: RAM used for intermediate tensors, often the limiting factor

If your model barely fits, you will fight stability issues. In that case, either simplify the model, reduce input size, redesign the pipeline (feature extraction) or move up to an edge gateway for Full ML.

Code example 1: TinyML audio classifier (microcontroller)

This example shows a real TinyML pattern: run an int8 TensorFlow Lite model with TensorFlow Lite for Microcontrollers (TFLM) and classify a short audio feature frame (for example MFCC features) on an MCU. It assumes you already converted your model to a C array (common in TFLM projects) and you have an input feature vector ready.

Prerequisites

  • A TFLM-enabled project (Arduino, PlatformIO or a vendor SDK)
  • An int8 TFLite model converted to a C array (for example via xxd -i)
  • Input features sized exactly as the model input tensor expects

Minimal TFLM inference code (C++)

// Runs int8 TensorFlow Lite Micro inference on an MCU using a compiled-in .tflite model.
#include <cstdint>
#include <cstring>

#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/micro/system_setup.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"

// 1) Provide your model bytes (generated from a .tflite file)
extern const unsigned char g_model_tflite[];
extern const unsigned int g_model_tflite_len;

// 2) Tune this based on your model, use interpreter->arena_used_bytes() to measure.
constexpr int kTensorArenaSize = 60 * 1024;
alignas(16) static uint8_t tensor_arena[kTensorArenaSize];

static tflite::MicroInterpreter* interpreter = nullptr;
static TfLiteTensor* input = nullptr;
static TfLiteTensor* output = nullptr;

bool tinyml_init() {
  tflite::InitializeTarget();

  const tflite::Model* model = tflite::GetModel(g_model_tflite);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    return false;
  }

  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
      model, resolver, tensor_arena, kTensorArenaSize);
  interpreter = &static_interpreter;

  if (interpreter->AllocateTensors() != kTfLiteOk) {
    return false;
  }

  input = interpreter->input(0);
  output = interpreter->output(0);

  // Expect int8 input for a quantized TinyML model.
  if (input->type != kTfLiteInt8 || output->type != kTfLiteInt8) {
    return false;
  }
  return true;
}

// feature_vector must match input tensor size in bytes.
// For example: MFCC frames flattened to [N] int8 values.
int tinyml_infer(const int8_t* feature_vector, size_t feature_bytes) {
  if (!interpreter || !input || !output) return -1;
  if (feature_bytes != static_cast<size_t>(input->bytes)) return -2;

  std::memcpy(input->data.int8, feature_vector, feature_bytes);

  if (interpreter->Invoke() != kTfLiteOk) return -3;

  // Argmax on int8 logits.
  int best_i = 0;
  int8_t best_v = output->data.int8[0];
  for (int i = 1; i < output->dims->data[output->dims->size - 1]; i++) {
    int8_t v = output->data.int8[i];
    if (v > best_v) {
      best_v = v;
      best_i = i;
    }
  }
  return best_i;
}

How to use it

  1. Call tinyml_init() once at boot.
  2. Generate features (MFCCs or other) and quantize them to int8 using the same scale and zero-point used during training.
  3. Call tinyml_infer() for each feature window.

Why this matters for TinyML vs Full ML: When to use which: this on-device approach gives you deterministic latency and reduces radio usage, but it also forces you to manage quantization, memory arena sizing and firmware rollout for model updates.

Code example 2: Full ML inference API (edge server)

This example runs a Full ML style deployment on an edge Linux box (or cloud VM): a FastAPI service that loads a scikit-learn model and serves predictions over HTTP. This pattern is common for gateways that aggregate multiple sensors and run heavier models than an MCU can handle.

Step 1: Train and save a model (Python)

# Trains a simple classifier and saves it as a file for an edge inference service.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
import joblib

X, y = make_classification(
    n_samples=5000,
    n_features=16,
    n_informative=10,
    n_redundant=2,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=200))
])

model.fit(X_train, y_train)
print("Test accuracy:", model.score(X_test, y_test))

joblib.dump(model, "edge_model.joblib")
print("Saved edge_model.joblib")

Step 2: Serve it with FastAPI (Python)

# Starts an HTTP inference server that loads a saved scikit-learn model.
from fastapi import FastAPI
from pydantic import BaseModel, conlist
import joblib
import numpy as np

app = FastAPI(title="Edge Full ML Inference")
model = joblib.load("edge_model.joblib")

class PredictRequest(BaseModel):
    # 16 features, adjust to your real feature vector
    x: conlist(float, min_length=16, max_length=16)

@app.post("/predict")
def predict(req: PredictRequest):
    X = np.array([req.x], dtype=np.float32)
    proba = model.predict_proba(X)[0].tolist()
    pred = int(np.argmax(proba))
    return {"pred": pred, "proba": proba}

Step 3: Run and test (bash)

# Installs dependencies, runs the API then sends a prediction request.
python -m venv .venv
. .venv/bin/activate
pip install -U pip
pip install fastapi uvicorn scikit-learn joblib numpy

uvicorn app:app --host 0.0.0.0 --port 8000

# In another terminal:
curl -s http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"x":[0.1,0.2,0.3,0.1,0.0,0.5,0.2,0.2,0.1,0.1,0.0,0.3,0.2,0.1,0.4,0.2]}'

Why this matters for TinyML vs Full ML: When to use which: this Full ML pattern makes updates easy (swap the model file or redeploy the container) and supports heavier feature engineering, but it adds network dependency and an endpoint you must secure and monitor.

Hybrid patterns (best of both)

Many real deployments do not pick one. They tier the system so the device does fast filtering and the backend does deeper reasoning.

Pattern 1: TinyML trigger, Full ML confirm

  • Device runs TinyML keyword spotting or anomaly detection continuously
  • On trigger, device uploads a short raw window or richer features
  • Edge or cloud runs a larger model to confirm and classify

This reduces bandwidth and power but still gives you high accuracy on events that matter.

Pattern 2: Full ML training, TinyML deployment

You train in the cloud with large pipelines, then distill or quantize for on-device inference. The cloud also handles data labeling and periodic retraining, then you ship a new model via OTA.

Pattern 3: TinyML per-sensor, Full ML for fleet analytics

Each node makes local decisions, while the backend aggregates outcomes and context for predictive maintenance dashboards, drift monitoring and root-cause analysis. You avoid sending raw data but still get fleet-level insight.

Common pitfalls and how to avoid them

Pitfall: picking TinyML without validating memory and latency

Fix: prototype early on your target MCU, measure:

  • Peak RAM (tensor arena used bytes)
  • Inference time worst case, not average
  • Power during inference and sleep

Pitfall: cloud inference for a task that needs deterministic response

Fix: push inference to device or a local gateway. If you must use cloud, add buffering and ensure the system fails safe when connectivity drops.

Pitfall: assuming Full ML is “free” because the cloud scales

Fix: estimate cost per inference and multiply by expected device count and sampling rate. Add egress costs if you stream data out of region.

Pitfall: model accuracy looks great in the lab, fails in the field

Fix: collect field data early, validate across temperature, mounting variation, aging sensors, different operators and ambient noise. Plan for drift monitoring in both TinyML and Full ML deployments.

Practical checklist (decide in 30 minutes)

  1. What is the maximum acceptable end-to-end latency? If it is sub-100 ms and needs to work offline, favor TinyML or edge Full ML on a gateway.
  2. Can you afford to transmit raw sensor data? If no, favor TinyML or feature-based uploads.
  3. What is your power budget? If you have a battery target measured in months or years, avoid frequent radio transmissions and lean toward TinyML triggers.
  4. How complex is the task? If it needs high resolution vision, language or long context, Full ML is the likely winner.
  5. How often will the model change? If you need frequent updates and experimentation, Full ML is simpler operationally.
  6. What are the privacy constraints? If raw audio or images cannot leave the device, prioritize TinyML or on-prem edge Full ML.
  7. What is your fleet update capability? If OTA is limited or costly, avoid TinyML models that require frequent firmware updates.

Run this checklist with stakeholders from embedded, cloud, security and product. It turns “TinyML vs Full ML: When to use which” into an engineering decision instead of a preference.

Conclusion

TinyML is the best fit when you need low latency, offline operation, low power and reduced bandwidth by running inference directly on constrained devices. Full ML is the best fit when you need higher accuracy ceilings, complex models, rapid iteration and centralized deployment and monitoring. In many IoT systems the winning architecture is hybrid: TinyML filters and triggers locally, then Full ML confirms, explains or aggregates at the edge or in the cloud.