iPaint / BASIC 8 format

Well, if I'm just making stuff up, I would make like AVI / IFF / WAV files. After the load address, which should be irrelevent:
.text "IMGH" ;packed id
.word +7 ; packet size
.byte $08 ; cell x
.byte $02 ; cell y
.word +640 ; width
.word +200 ; height
.byte flags ; compression, interlace, double-width
.byte fg*16+bg ; only for monochrome (adjust packet size if included)

followed in any order by packets like

.text "IPD0" ;image pixel data 0 (even field)
.word 8+80*200 ;packet size
.word startx ; 0 [could be more for progressive image]
.word starty ; 0
.word dx ; 640 [could be less for progressive image]
.word dy ; 200 [could be less for progressive image]
.byte ... ;80x200

.text "ICD0" ;image color data 0 (even field, not in monochrome image)
.word 4+80*100 ;packet size
.byte startx ; 0
.byte starty ; 0
.byte dx ; 80 [could be less for progressive image]
.byte dy ; 100 [could be less for progressive image]
.byte ... ;80x100

and then the following would only be in interlaced images...

.text "IPD1" ;image pixel data 1 (odd field)
.word 8+640*200 ;packet size
.word startx ; 0
.word starty ; 0
.word dx ; 640 [could be less for progressive image]
.word dy ; 200 [could be less for progressive image]
.byte ... ;640x200

.text "ICD1" ;image color data 1 (odd field, not in monochrome image)
.word 4+80*100 ;packet size
.byte startx ; 0
.byte starty ; 0
.byte dx ; 80 [could be less for progressive image]
.byte dy ; 100 [could be less for progressive image]
.byte ... ;80x100

That is pretty similar to the format proposed/tested on former Alive! forum, but it did not allow multiple packets in the file: there could only be a fixed number of pixel and color segements (packets without any id/length); a WORD pointer was used in the header that gave an offset into the file were each could be found; the program would (might?) have to caclulate the size of each data segment

This method allows for multiple packets in a file. So you could have a progressive image; where it loads 2 rasters of pixels (one packet) and then 80 bytes of color (next packet) and then keeps repeating/alternating like that for full image. Useful if an image is too big to fit in VDC RAM, or when decrompression takes a long time... also useful to make file conversions easier; GEOS and JPEG for example use a similar method. If you don't want progressive image you just need 2 packets (or one for monochrome)... or 4 for interlace. You can also have extra packets that a simple program could ignore; like comments or audio or animation or compression parameters

How about we can that format, VDC : Video Display Codex

It has been a while seen I looked at a GeoPaint file, but as I recall, each VLIR record is a horizontal span of variable length. So one VLIR record would be 8 pixels tall and X cards wide. And a card is 8 pixels wide. This made for great "compression" because not all 80 cards (640 pixels) need to be in each span... only the ones that are actually used. So a VLIR record might only have 2 cards, or might be 80 or anywhere in between. I think (but could easily be wrong) that a VLIR record was simply omitted (has 0,0 track/sector pointer) if a span has no used pixels (all background). The nice thing of course is that the file format has no real limit in height and the width is (could be) potentially much longer... Of course GeoPaint limits these values.

+7 means decimal 7, as opposed to $7 hexadecimal 7. Yeah, silly me! Obviously I've spent to much time using MONITOR

In the "off-the-top-of-my-head" format, all values in the header and in all in the "pixel" packets are in terms of pixels. Just in case somebody needs to import/convert a file that is not a multiple of the cell size (or a decompression scheme is not aligned with cell size). Actually it seems this would be most useful for a paint program, since you don't normally select an area to cut/copy/paste to be exactly aligned with cell boundraries (and using "lasso select" never ends up on cell boundraries)... strange that BASIC 8 Paint / iPaint do not use this! Of course I understand neither of them allow "lasso select". Sorry, going off-topic again...

In my not-well-thought-out proposal, the only thing that is in terms of cells (i.e., pixels / cell_width or pixels / cell_height) is the "color" packets. This because the hardware requires color to be addressed in terms of cells, and never by pixels. Well, unless you wanted to set the VDC to 1x1 mode Even if you make the cells smaller, the VDC limits the number of cells to 8 bits both horizontally and vertically. So you should notice that the "color" packets use .BYTE while everything else uses .WORD

"Video Display Codex" sounds like as good a name as any, although that implies (by the acronym VDC) it is for the 80-column VDC chip.  So far, everything would work just as well with the VIC-II or VIC-I, assuming the correctly cell-size was used.  If we're going to invent a format, might as well make it generic.

The idea I proposed above is a bit verbose, but should be practical for an image that is not in "progressive" format (like GeoPaint / JPEG).  So a simple non-progressive file would have about 64 bytes of meta-data, which is about 1/4 of a block, which is almost nothing compared to a typical VDC image which will be 64 blocks after compression (128 blocks or more without compression).  But if you used the headers I proposed in a progressive image, the meta-data (overhead) would be a lot more; so we might want to think of something more efficient.

So, as a comparison, here is the format proposed by OtherGuy (as best I can reconstruct it)

.text "OGUY" ;whatever, magic ID
.word size ;all pixel and color data (bytes)
.byte cell_width
.byte cell_height
.byte num_cells_x
.byte num_cells_y ;the actual number would be 2x if interlace
.byte compression ;and other flags, like interlace/monochrome
.byte fg          ;foreground, only used in monochrome
.byte bg          ;background, only used in monochrome
.byte bd          ;border, only used in non-monochrome (same register as foreground)
.word 0           ;padding
.word offset_pixel_0 ;unsigned; relative to address of "OGUY"
.word offset_color_0 ;dito (zero if monochrome)
.word offset_pixel_1 ;dito; zero if not interlaced
.word offset_color_1 ;dito; zero if not interlaced

That proposal only uses 24 bytes and allows each data segment to be anywhere in the file, although the number of segments is strictly fixed.  In particular, he said at first he was using pixels first, followed by colors, but later decided it looked better (while loading the image) to put the colors first and the pixels after.  Because he used pointers in the header, he did not have to change the format or his viewer; just the way the file was recorded.

Can we come up with a compromise between my idea and his?  Something that allows large/progressive images (multiple segments of arbitrary size), but it is more compact than my method... ?  One idea I had was in each "pixel" packet, to make coordinates in terms of cells relative to the top-left cell.  This would allow the use of BYTEs instead of WORDs, but the viewer would need to be smart enough to special-case the first and last cells in a raster, and the first and last rasters in an image (but only in the case when the image is not an exact multiple of cell-size).  The only other problem I can think of is that may cause some compression routines to not be possible... but in such cases, I imagine you could use a different packet ID...

Along the same lines, I was thinking the size of my method could be reduced if each packet ID was 2 bytes instead of four.  But there should be some method to clearly destinguish a "FourCC" from a "TwoCC"... like set the high bit in the second character ?

Along the same lines, AVI/IFF/WAV files normally use a 32-bit size with each packet.  This would normally be unneeded on a C128, but in the case of an image too large for VDC RAM, or maybe one containing a lot of meta-data (comments, animation, audio), the file size may exceed 64K... so at least 24 bit would be needed sometimes.  So, I was thinking only a packet with a "FourCC" ID could possibly have a DWORD size, but normally the size would be 16-bit WORD, and this could be indicated by setting the high-bit in the last character.  To be compatible (not that is needed) with AVI/IFF/WAV files, the size would be DWORD unless the high-bit of the last char is set...

Continuing this idea, a "TwoCC" packet would, by default, have a WORD size (never can be DWORD), or could have a BYTE size if the first char of ID had its high-bit set.  Now in case you are confused, here is summary:

;huge packet, like an over-sized VDC image
.text "V", "D", "C", " "    ;no high bits
.dword total_size           ;like AVI/IFF/WAV

;typical packet (FourCC)
.text "I", "P", "D", "0" | $80 ;high bit in 4th char
.word packet_size           ;like 640 for pixels in a span of 8x8 cells

;shortend packet (TwoCC)
.text "D", "0" | $80        ;high bit in 2nd char
.word packet_size           ;like 640 for pixels in a span of 8x8 cells

;tiny packet (TwoCC)
.text "D" | $80, "0" | $80  ;high bit in 1st char (8-bit size), high bit in 2nd char (TwoCC)
.byte packet_size           ;like 160 for pixels in a span of 8x2 cells

;tiny packet (TwoCC) example 2
.text "C" | $80, "0" | $80  ;high bit in 1st char (8-bit size), high bit in 2nd char (TwoCC)
.byte packet_size           ;like 80 for colors in a span of 8x? cells

Of yeah, AVI and WAV files use little-endian byte order, and so should are format.  IFF uses big-endian because of the Motorola CPU in Amigas.


Also, I was thinking the format could be used for any CBM image (assuming correct cell-size).  In order to allow that, we should have a byte in the header that indicates the palette being used in the source file.  This would allow a viewer to display most images no matter the hardware with color translation.  Of course the translation may not look pretty

Well, it turns out we were both right and both wrong!  I was thinking geoPaint did a variable number of cards per single row.  If I understand your comment, it does it in multiples of 2 cards... well I found interesting article in C=Hacking #3 that discusses geoPaint format.  (P.S. sorry for going off-topic)

Well according to the article (it has been forever since I looked at hex dump of geoPaint file), it always writes 80 cards of pixel data (640 bytes), followed by another 80 cards of pixel data (640 bytes), followed by 8 bytes of junk, followed by 80 cards of color data (80 bytes), followed by 80 cards of color data (80 bytes).

So, I guess you were closer than me, in that it (conceptually) does cards vertically by 2 (although in reality it stores 160 cards horizontally).

Also, a 0,0 block pointer in the VLIR block indicates end-of-file; a 0,255 pointer indicates "no data" for a span (640x16 pixels).

Edit

I'll summarize the compression scheme based on reading the article,
 %00nnnnnn = n Raw data bytes
 %01nnnnnn = n Card Copy (described below)
 %1nnnnnnn = n RLE bytes

There are no special values of 'n' except for Raw (type %00), where n=0 is end of file.  In particular, an 'n' of zero for RLE or Card Copy is an error.

Card Copy reads the next 8 bytes of raw data, and then writes those 'n' times to the bitmap.  This works well for geoPaint images which is VIC-II encoding of pixel data, and more importantly the fact that geoPaint fill patterns use 8 bytes.

In my experience with real images (not something created with geoPaint, or some other paint program), copy of 8 bytes is only occassionaly useful... usually when this appears in a real image, it is a simple background (in which case RLE works fine), or it looks like "fake" / computer generated / over-compressed (especially if n > 2).  *Blah* can't really explain it in words...