Talking you about VDC, yes? For me known maximal vertical resolution reached is 800 pixels interlaced. Mode used allows maximally 600x800 for memory limit. Please PM me - when I will be free somewhere on disk I'll find this mode.
Commodore 64 was great, Commodore 128 is bigger, better, faster and more powerful... Commodore 65 was almost here, but C256 is coming and it will be earthquake...
I have done modes with vertical resolutions of 400 pixels (which would be 800 interlaced). Technically there is no real limit for the vertical height, you can define char height as 32 and set the vertical register to 255 which would be about 8192 lines. It all depends on the display device if you are able to get a stable picture. As I noted in VDC Mode Mania, the vertical refresh rate of the 640x400 non-interlace mode is about 37 Hz. If you want even more lines you need devices able to display even lower Hz. The 1901 barely does 37 Hz, on some you get a rolling picture. Not sure how low multisync-monitors go, but basically your user base shrinks to zero at that point.
Horizontal sync is a lot worse in that matter, anything to far off PAL or NTSC-standard will lead to non-display on most monitors.
Not sure what you mean by that? Horizontal total (r0) should always be either 126 (NTSC) or 127 (PAL). Horizontal displayed (r1) is usually 80 so you have borders to the right and left depending on where the horizontal sync position (r2) is. So if you only display 50 columns instead of 80 you should have those centered.
Vertical is similar. You set vertical total (r4) and vertical displayed (r6) and the vertical sync position (r7). At PAL-resolution this means you only display 200 lines out of 312 which means borders at the top and bottom. With NTSC borders are not as bad with 200 out of 262 lines.
Because of those unused display areas I created two text-modes in VDC Mode Mania that make better use of the display area. For PAL this is a 9x11 char-mode and for NTSC 9x9.
I've gotten the 1084 up to 42 rows, after that it tends to loose sync and since the 1084 has no vertical hold control like the 1901 the picture will start rolling at that point. The 1901 can barely display 50 rows. I remember trying this back in 1992 or so and could not turn the knob far enough so the picture would not roll. Some 20 years later it suddenly worked. Either way: my current thinking is you should aim for PAL- and NTSC-compatible-modes that most devices can display (back then and now).
Ah, now I get what you meant. NTSC is usually 525 (half)-lines and PAL 625. Home-computers usually only use the first field, so 262 lines (NTSC) or 312 lines (PAL). Within this area you need to deduct some lines for VBLANK which usually gets you 240 lines in NTSC and 288 in PAL. This would mean 30 text-lines in NTSC and 36 in PAL. You would just need to adjust register 6 accordingly (and register 7 to center the picture). Either way at this resolution you will most likely be in the overscan-area on most display devices. It's a good idea to let the user choose the best display-position (reg 7) at those resolutions.