...for the brush tool it refers to the brushes palette as to what style of brush you are using...
...if you want a dotted line brush you will need to create a new artbrush or pattern brush, or use the 'borders lines' available in the flyout menu of the brushes palette under 'open brush library'. There is one in there, probably not one you would want though...
..to create your own, draw a straight line, apply the dashed settings to it from the stroke palette, drag this line to the brushes palette, select new art brush...
...the pencil tool, in my version (CS2) does honor the stroke palettes settings...
Andrew