I made a page where you walk through the same garden in first person, switching between the senses of a crow, pit viper, bat, fly, honeybee, dog, cat, mantis shrimp and more. Senses humans cannot perceive (ultraviolet, heat, echoes, polarization, near-infrared) are "translated" into colors we can see. It uses Three.js with my own shaders.
https://sishiya.github.io/manabi-lab/animal-senses/?lang=en
This post covers how one garden produces pictures for many different senses.
The views are estimates based on research and cannot be used to diagnose eyes. The app labels its explanations as "established", "estimate" or "effect".
Give objects "per-sense" properties
Besides the color the human eye sees, each vertex in the garden carries these properties (painted by paint()):
| Attribute | Contents |
|---|---|
aCol | color seen by the human eye (linear RGB) |
aP | UV reflectance, daytime temperature, night temperature, whether it glows |
aA | opacity to visible light, UV and heat |
aX | how well echoes return, near-infrared reflectance, whether it is a smooth polarizing surface |
For example, sunflower petals have a UV reflectance of 0.03 at the base and 0.58 at the tip. To us they are evenly yellow, but in UV the dark base forms a "target" pattern (values are estimates).
Opacity is split per kind of light because of garbage bags: showing that a bag hides its contents in visible light but not in UV requires different transparency for each kind of light.
Draw the same scene many times with different modes
The scene shader switches what it draws with uMode:
| uMode | What it draws |
|---|---|
| 0 | visible light |
| 1 | ultraviolet |
| 2 | heat (temperature / 60) |
| 3 | echoes (distance / 40, and how much returns based on surface angle) |
| 4 | near-infrared (sunlight + the camera's IR light, which falls off with distance squared) |
| 5 | polarization (Fresnel equations for smooth surfaces, Rayleigh scattering for the sky) |
Only the needed passes are rendered to render targets, and the compositing shader COMP_FS mixes them with a formula for each animal:
- Crow: UV overlaid in purple on visible light
- Pit viper: heat blurred over a 320-cell width and overlaid (the pit organ cannot see fine shapes)
- Dog, cat: dichromat color formula (Viénot 1999) and blur; the cat gets brighter and softer at night
- Mantis shrimp: grayscale, with color quantized to 12 hues only inside a horizontal band that sweeps up and down
You can also split the screen with a human view on the left.
Compound eyes: read a cube map on a hexagonal grid
Flies and bees see almost all the way around. So the page renders a 256 px cube map at the camera position, converts it to azimuth and elevation, and packs 336° across the screen.
That is divided into a hexagonal grid, and each hexagon reads just one direction, at its center. Mipmaps blur it by each ommatidium's acceptance angle. For the bee, a UV cube is rendered too and colors are shifted: "green → red, blue → green, UV → blue".
Bats: echoes come back at the speed of sound
The bat remembers up to six call times. For each pixel, when
age − 2d / 343 ≥ 0 (age = time since the call, d = distance to the object) the echo has returned, so the pixel lights up and fades over 0.12 seconds. Where the call's wavefront is right now (d ≈ 343·age) is drawn as a blue line.
When a moth is ahead, the call interval shortens from 0.1 to 0.05 to 0.012 seconds (the feeding buzz).
With "hear the echoes" on, the real 80→25 kHz sweep is played stretched 10× as 8→2.5 kHz, with delay and volume set by each object's distance, direction and size.
Nearsight, farsight and presbyopia from the "distance pass"
The distance pass made for echoes also works for the focus errors of human eyes. For each pixel:
needed power = 1/distance + hyperopia (myopia is negative) Below 0 is the myopic error; anything beyond the accommodation range (Hofstetter's minimum, 15 − 0.25×age) is the hyperopic or presbyopic error. The blur circle's diameter is "pupil diameter × error", with the pupil at 3 mm by day and 6.5 mm at night. You hold a paperback 35 cm away, so with presbyopia you can see just the book go blurry.
Color vision differences use the matrices of Machado et al. (2009).
Careful with flicker
Flies sense fast time, so they should see the flicker of a fluorescent lamp (100 brightness changes per second). On screen, though, the flicker becomes 2–25 times per second, inside the range most likely to trigger photosensitive seizures (3–30).
Measuring the first version, about 9% of the screen changed relative luminance by 0.1 or more. So now the flicker appears only when you turn on "see the lamp flicker", and the brightness swing is reduced to 85–100% (a real lamp drops to about 50%). With prefers-reduced-motion, it cannot be turned on. It is now 0%.
Small gotchas
- Drawing a translucent film double-sided without depth writes put the back face over the front, so the bag's contents showed through in UV where they should be hidden → translucent objects are front-face only
- Having a GLSL function and a local variable share a name (
cat) is risky, so they becamecatVanddogV - Merging static objects into one mesh (
bakeStatic()) cut draw calls from hundreds to dozens
Main sources
- F. Viénot, H. Brettel, J. D. Mollon (1999) (dichromat color formula)
- G. M. Machado, M. M. Oliveira, L. A. F. Fernandes (2009) (color vision matrices)
- T. H. Bullock, F. P. J. Diecke (1956) (sensitivity of the pit viper's pit organ)
The other sources (about 35) are in the app.
This page is part of Manabi Lab, a series of small learning apps that run in the browser (English and Japanese). The code and this post were written together with AI (Claude).
