Canspeed -
To state the obvious, rowing and the ergometer are slightly different beasts. The erg is simpler, is less sensitive to technique & requires no coordination with others. Though most of the very fastest times on the erg have been recorded by OTW rowers, that likely reflects the fact that there are few, if any, non-Olympic athletes with the same level of conditioning.
There are certainly talents & physical characteristics that are typical of elite heavyweight rowers - tall (6'2"+); big but not huge (195-240lbs), freakishly high VO2 Max,...
Speed, in the sense that differentiates 100m sprinters from 1500m guys in track, is pretty much a non-factor. Rowing just doesn't require that much quickness.
I am convinced that the single most important differentiator is mental toughness and control. While this is probably true for many, if not most, sports at the elite level, I think it is more the case for rowing.
A few interesting comments & links in this UK forum thread on elite oarsmen.
http://www.concept2.co.uk/forum/viewtop ... &sk=t&sd=a
None of this sounds like it would hold you back from performing very well & possibly at the elite level on the erg. Proof, as always, is in the last 500m of the race.
Cheers. Patrick.