RMS Size refers to a time frame in which the average signal is calculated.
RMS is Root-Mean-Square or the square-root of the arithmetic mean of the square of the signal. RMS is averaging the area displaced by the signal, the area between the waveform and the linear zero line (not 0dB, but the axis). As the waveform swings both above (+) and below (-) the centreline, the polarity of the swings has to be disregarded. Luckily, in maths, anything multiplied by itself (squaring) ends up positive. The signal can then be averaged (arithmetic mean over the timeline/window ED mentions or its integration time) as the positive and negative halves won't now cancel each other out -and finally the inverse to squaring is executed -square root. See also: http://en.wikipedia.org/wiki/Root_mean_square
RMS-size or RMS-window is the average RMS-value over a certain time-frame
A compressor working in RMS mode reacts to signal above a set threshold but it measures on the average level of the signal (over a period of time).
RMS-size close to 0ms is referred to as peak-mode, and is used in limiters.
A larger RMS-size can give smoother operation, and is preferred for leveling the signal.
Generally, setting this value low gives more compression, and less compression when set higher.
You can set the RMS window-size for the master VU-meter. Default is 500ms.
See: Master VU Settings