Plant identification sounds like a solved problem until you try to ship it inside a real product.
At first, it looks straightforward: plants already have a scientific classification system. We have the Linnaean taxonomy — kingdom, family, genus, species. We have public datasets. We have modern vision models. So why not just build a hierarchy, let the model walk down the tree, and identify the plant?
That was not how it worked for PlantCare.
We were building an on-device plant identification model for an iPhone app focused on yard trees and shrubs. The goal was not to name every plant on earth. The goal was narrower and more practical: identify the plants people actually prune, then route them to the right pruning guidance.
That product constraint changed the machine learning problem.
A transformer-based vision-language model sounds attractive here because it can appear more flexible. It can reason over names, descriptions, and taxonomy. But for this use case, that is not the same as being reliable.
Plant ID in the real world is not a text reasoning task. It is a noisy visual classification task. Users do not upload perfect botanical specimens. They upload whatever they see in their yard: a branch, a few leaves, a winter silhouette, bark texture, a blurry canopy, or a plant without flowers or fruit. The most useful diagnostic attributes are often missing.
That is the core problem.
A botanist may distinguish two taxa using flower structure, fruit, bud arrangement, leaf underside, bark, seasonality, location, or growth habit. But a single phone photo may contain only one or two of those signals. Sometimes it contains none of the decisive ones.
This is also why taxonomy alone does not solve the problem.
The Linnaean system is a biological classification system, not a deterministic visual decision tree for casual photos. Two plants can be taxonomically close but visually different in the images users submit. Two plants can also be taxonomically different but look almost identical in a leaf-only photo. A hierarchy can describe biological relationships, but it does not guarantee that each split is visually observable from the input image.
That is where a deterministic hierarchical classifier breaks down.
In theory, you could classify first by family, then genus, then species. In practice, each step creates an error surface. If the model makes a mistake high in the hierarchy, everything below that point is wrong. Worse, many hierarchy nodes are not visually separable from the available image. The classifier is being asked to make a botanical distinction without the botanical evidence.
So instead of treating plant ID as a taxonomy traversal problem, we treated it as a product-shaped machine learning problem.
We needed a qualified dataset and a flat visual classifier trained directly on the supported taxa that matter for the app.
That distinction matters.
A flat classifier does not pretend the model can always infer the correct biological path. It learns the visual boundaries that actually appear in the training images. It can be evaluated directly against the supported plant set. It also lets us design practical fallback behavior: exact taxa where we have enough clean data, grouped labels where pruning guidance can be shared, and unsupported paths where the model should abstain.
Our early instinct was to start with broad public plant datasets. We tested PlantNet-300K, iNaturalist 2021, and GBIF. But “large plant dataset” and “good product dataset” are not the same thing.
PlantNet was useful, but not mainly as the positive source for the plants we cared about. iNaturalist helped in some areas, but adding it broadly was not consistently beneficial. GBIF ended up being the strongest source for positive examples aligned to our supported pruning taxa.
That became the first real lesson: dataset suitability mattered more than dataset size.
The second lesson was that generic model rankings did not decide the winner.
We compared multiple mobile-scale backbones on the same GBIF-only supported task, including MobileNetV4, EfficientNetV2-B0, and MobileNetV3-Large. On our cleaned task, MobileNetV4 performed best. The result was not about choosing the most fashionable architecture. It was about matching the model to the dataset, label space, and on-device constraint.
But the biggest improvement did not come from changing the backbone.
It came from cleaning the taxonomy.
Our original label space mixed exact species, horticultural groups, and broad genus-level buckets. On paper, that looked reasonable. In training, it created a messy softmax problem. Some classes were visually coherent. Others were broad “spp.” buckets that grouped together too much variation.
So we changed the task.
We created an exact-taxa GBIF-only ablation and removed the noisiest broad buckets. That produced a cleaner 26-class problem. On the cleaned test subset, the improvement held up:
- baseline GBIF-only MobileNetV4: top-1 0.7617
- cleaned exact-taxa MobileNetV4: top-1 0.7706
- top-3 improved from 0.8786 to 0.9120
- calibration improved meaningfully as well
The top-1 gain was modest, but the direction was important. It showed that our next improvement was not going to come from throwing a larger or more general model at the problem. It was going to come from making the training task match the real product.
That is the main point.
Plant identification is not solved by a transformer, a taxonomy tree, or a larger dataset by default. Real-world plant ID fails at the gap between biological classification and visible evidence. If the image does not show flowers, fruit, bark, buds, or other diagnostic traits, the model cannot magically recover them.
The useful path is more disciplined:
- build a qualified dataset
- clean the label space
- train on the taxa the product actually supports
- evaluate supported classes separately from unknowns
- use fallback groups where the product can tolerate ambiguity
- abstain when the model should not guess
- ship a model small enough to run reliably on-device
For PlantCare, that meant training a flat, mobile-scale visual classifier around the plants people actually prune, rather than pretending a deterministic hierarchy could solve the messy reality of user photos.
The biggest gains did not come from a more exotic model.
They came from making the machine learning problem honest.