The delay is different for LC, HE and HEv2. The amount for each profile is fixed in samples, so it also differs in time based on samplerate. The delays should be less then 1 AAC frame, so there is not much that can be done on the AAC encoder side.
What samplerates were the input wav files you used?
Sample rate 48K for the WAVs.
Also a correction (!!) : the 54ms files were encoded with a different bitrate than the 94ms file. So rpp3po's previous observation that, given the same encoding parameters and kind of input, the delay is always the same is apparently correct. My bad, very sorry. It's been just that much of a file juggling here, things got a little messed up.
Example info for a short clip
$ wavinfo 1.wav
File:
Name: 1.wav
File Size: 11730272
Format:
Type: Microsoft PCM
Channels: 6
Sample Rate: 48000 Hz
Avg bytes/sec: 576000
Block Align: 12 bytes
Bit Width: 16
Channel Mask: 0x03F
Data:
Start: 44
Data Size: 11730228
Samples: 977519
Playing Time: 20.36 sec
which - at 224k bitrate - gets encoded to
$ MP4Box -info 1.m4a
* Movie Info *
Timescale 90000 - Duration 00:00:20.462
Fragmented File no - 1 track(s)
File Brand mp42 - version 0
Created: GMT Wed Feb 25 16:23:35 2009
File has no MPEG4 IOD/OD
Chapters:
Chapter #1 - 00:00:00.097 - ""
iTunes Info:
Encoder Software: Nero AAC codec / Aug 6 2007
Track # 1 Info - TrackID 1 - TimeScale 48000 - Duration 00:00:20.462
Media Info: Language "Undetermined" - Type "soun:mp4a" - 480 samples
MPEG-4 Config: Audio Stream - ObjectTypeIndication 0x40
MPEG-4 Audio AAC LC - 6 Channel(s) - SampleRate 24000 - SBR SampleRate 48000
Self-synchronized
So to conclude this, what would those fixed padding sample amounts then be for each profile? And how many samples a single AAC frame?