Small sources come out better, and faster

The estimate raises an obvious question — why is the small folder the quick one? — and the answer turns out to be the same answer to a question about quality that nobody asks until afterwards.

The time half, which is the easy half

Tiling is over the source. A 500-pixel square at ESPCN’s 208-pixel stride is nine passes; a 4,000-pixel square is 400 passes, forty-four times as many, on the same network at the same milliseconds each. The factor you asked for changes nothing about that count, because neither network takes a factor.

Passes for one file, ESPCN, by source size
Source	Passes	Seconds at 230 ms
500 × 500	9	2.1
1,000 × 1,000	25	5.8
2,000 × 2,000	100	23
4,000 × 4,000	400	92

So the folder of thumbnails is not quick because it is small in megabytes. It is quick because there is less of it for the network to look at, and the relationship is quadratic: doubling the long edge quadruples the wait.

The quality half, which is the interesting one

What the network was trained to do

Both graphs here were trained by taking sharp photographs, shrinking them by a fixed amount with a known filter, and learning to put back what the shrinking took away. That is the entire task. The training pairs were made by a downscale, so what the network is good at is undoing a downscale.

A 500-pixel supplier thumbnail is very often literally that: a large studio photograph that somebody resized. The network is being asked to do the exact thing it was trained on, and it does it well.

What a large original is instead

A 4,000-pixel photograph straight off a sensor has not been downscaled. What limits it is lens softness, sensor noise, motion at the moment of exposure and the camera’s own sharpening — none of which resembles a clean downscale, and none of which the network has a learned inverse for. It will enlarge the picture competently and it will enlarge the noise along with it, because at that scale the noise is the detail as far as the network can tell.

The practical rule

The good case is a small, sharp original that used to be larger. The poor case is a large, soft original that never was. That is why this site opens at 2× on small inputs and refuses to start a folder of twelve-megapixel photographs: the second job is worse on both counts at once, which is rare enough to be worth saying plainly.

What this does not mean

It does not mean shrinking a photograph first and then enlarging it will help. Throwing detail away to give a network something familiar to undo loses more than it returns, and this queue cannot shrink anything in any case.
It does not mean a small blurry original will come out sharp. Softness in the source is not what either network learned to remove, and it will be enlarged faithfully.
It does not mean the result contains information that was not there. It contains a plausible guess at what a larger photograph of the same thing would have looked like, which is useful and is not the same claim.