Tired of ads? Upgrade to paid account and never see ads again!
Deunan
They see me rollin'... 
15th-Mar-2014 05:23 pm
Decision has been made, there will be a short production run of GDEMU soon - as in, a few weeks. After that, when I can tell just how many people are actually going to pay for it, I will decide if I will continue or not. More details will follow once I get everything sorted out.

In other news, my testers notified me of some recently done Dreamcast dumps that appear to be broken on GDEMU, so I investigated.
Seems like there is a group, or groups, that started doing GD-ROM dumps using PC drives with a so-called "swap method". I did that too once, other than having slightly different "audio offsets" there is nothing preventing such dumps from working - if done correctly. And that's the key word here. Well, the group(s) in question decided they know better and introduced changes into tracks' positions and lengths - to "fix" things I guess. The worst part is they've just shoved all their changes into .gdi files and typical user has no way of knowing if it's a proper dump or not.

GD-ROM images made using GD drive in Dreamcast are not perfect either, but frankly no CD image format is, or will ever be, unless we store raw EFM stream (assuming we can even read it glitch-free, which is debatable). But small flaws can be corrected if the method is good in principle and the changes are always consistent. For GDIs it means you need to adjust for missing pregaps (not really needed) on data tracks and starting address on audio tracks. My guess is the new dumps were supposed to overcome that, but instead are plain broken.

As an example let's take a look at Chu-Chu Rocket, the PAL version. The new .gdi file says this for tracks 3 and 4:

3 45000 4 2352 "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es) (Track 03).bin" 0
4 355602 0 2352 "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es) (Track 04).bin" 0

Track 3 file is 730535904 bytes long, that is 310602 raw sectors. Since it's missing the pregap I mentioned, the actual length of the track is 150+310602=310752.
Let's add this length to it's starting address, to see where it ends: 45000+310752=355752. See the problem? Track 4 starts at 355602, which is less than 355752. This creates an impossible situation where tracks overlap.

But wait, what if we treated this new image just like the old dumps and actually did apply the fix for audio tracks as usual? Then it'd be 355602 plus 150 fix which equals 355752 and it checks out. Yay! Maybe this is the correct way after all, let's investigate further:

18 505295 0 2352 "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es) (Track 18).bin" 0
19 505819 4 2352 "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es) (Track 19).bin" 0

Track 18 is audio, 1232448 bytes long, which makes it 524 "sectors". If we add the fix, then it starts at 505295+150=505445, and ends at 505445+524=505969. Oops. Track 19 started already, at 505819... Now track 18 end overlaps with track 19 pregap.

The funny thing is, the situation with track 18 overlapping track 19 will kinda work still, since no code ever tries to acces the pregap, so the collision will never be discovered. But this tells you a lot about quality of this dump. Can it be fixed? Maybe. Question is, why break and fix it, rather than just use a known good dump.

I rest my case.
Comments 
15th-Mar-2014 06:38 pm (UTC) - Put me down for one!
I'll buy one, or even more depending on the price!
17th-Mar-2014 11:21 am (UTC) - Re: Put me down for one!
Yes, put me down for one too. Also following the other project on Assemblergames too...
15th-Mar-2014 06:40 pm (UTC) - that group
Ooh, I know that group. They are big-mouthed pompous buffoons who have no clue on how to manage software standards (as in, their own ripping standards). And they don't even have a decent coder, they just mix and match a dozen different tools for ripping.

Which wouldn't be a problem if they weren't so incredibly obnoxious about how they are better than everyone. Even worse, many people believe that too, on viral levels.

If you ask me, you should just include a note that "rips from these groups are defective, GDEMU only works with backups you make yourself on your Dreamcast", and not bother fixing support for their broken rips. If we are lucky, they may just consider it a wake-up call.
15th-Mar-2014 11:55 pm (UTC) - Re: that group
"rips from these groups are defective, GDEMU only works with backups you make yourself on your Dreamcast" -- it's like 'proper encrypted NDS roms are defective, since they don't work on flashcards, decrypted ones are the only proper dumps possible'.
16th-Mar-2014 01:31 am (UTC) - Re: that group
So dumping roms with a rom dumper -> correct, but dumping gdroms with a gdrom drive -> not correct?

Why not just write a Dreamcast app that reads everything fine in the first place? That would make dumping easier, faster, more efficient, would allow way more people to make "proper" dumps, and you wouldn't have to suffer from the deficiencies of using TOC faking.
16th-Mar-2014 01:46 am (UTC) - Re: that group
Because even on a PC you can't reach certain areas without swapping tricks and special firmware commands on certain drives. You won't be able to read the entire disc contents on a DC.
15th-Mar-2014 06:50 pm (UTC)
The track itself (index 01 in CD terms) starts at 45000, if you add the pregap, it should start at 45000-150=44850, so other tracks are unaffected, there is no overlapping and the .gdi is correct.
15th-Mar-2014 07:33 pm (UTC)
So many things wrong with that statement... No, track 3 starts at 45000 with index 0, which makes it a 2s pause, thus a pre-gap. The actual data start at 45150 and you can verify that by inspecting sector header in raw dump. Better yet, go read ECMA-130 and stop spouting this nonsense...
15th-Mar-2014 08:01 pm (UTC)
No, track 3 starts at 44850 with index 0. ECMA-130 has nothing about GD-ROMs. Find the copy of "GD-ROM Format Basic Specifications Ver. 2.14" and open page 8. It's the official Sega document and it says the first track of the HD zone always starts at AMSF 10:00:00 - Index 00/10:02:00 - Index 01. LBA 0 is AMSF 00:02:00, LBA -150 is 00:00:00, AMF 10:00:00 is LBA 44850, that's where the 1st HD zone track starts. And, I repeat, it's the official Sega document. Also, you can always check the data and subs by yourself, the game data always starts at 45000 and 150 sectors before that is always pregap. Cheers.

Edited at 2014-03-15 08:02 pm (UTC)
15th-Mar-2014 08:22 pm (UTC)
1) GD-ROM is a CD-ROM in all aspects, except it's partitioned into low and high density area. All ECMA-130 rules on track layout have been followed.
2) Track 3, first one in HD area, starts at 10:00:00 as you yourself just said. This part is already track 3 index 0, there is no "pregap" of any kind before it. Page 8 has a nice figure, on it, pay attention to TNO and Index values next time you read it.
3) LBA 0 is not AMSF 00:02:00, the same document, the same page, has a value called "FAD". That's LBA in SEGA's naming scheme, they use it in all documents related to GD-ROM. Go read them as well.
4) Do not mix subcode values into this, it's because of people like you who don't understand what subcode is, and how it's encoded and used, that we have this mess with shifted track postions in the first place. Go read my blog entry on PCE games while you're at it.

You are mixing logical layout with physical layout and tell me ECMA-130 does not apply here? What does, then? Do you even know how a drive finds the seek position it's supposed to read? I bet in your mind it's all magic and "just happens".
15th-Mar-2014 08:46 pm (UTC)
There is a huge misunderstanding here. That 'group' you've mentioned uses the PC-standard logical layout. Dump any CD with CloneCD and check the AMSF for the first sector in the dump - it's 00:02:00. And it equals to LBA 0. Sectors 00:00:00 to 00:02:00 (LBA -150 to LBA 0) contain the pregap to the first track, it is readable only by drives with overreading into lead-in capability and only partially (starting from LBA -140 or -130 in the best possible case, it is always omitted by all the dumping tools).

So if we talk about FADs, not LBAs, all the images of 'that group' has the first track starting with FAD 150 (equals to LBA 0) and third track (first track of the HD Zone) starting with FAD 45150 (LBA 45000). And the question really is: whether gdi format expects LBA values or FAD values? That 'group' uses LBA values and you treat them as FAD values, which are 150 sectors off. The dump itself is correct and, I repeat, has no overlapping.
15th-Mar-2014 09:20 pm (UTC)
I will concede that using "LBA" is not the best way to avoid further confusion, but I absolutely refuse to use the term "FAD", since it stand for "frame address". And these things we are addressing are not frames, but 1/75 fractions of a second. Or sections, as the audio part of the spec would call it. Also, SEGA introduces a term "LSN" which stands for "logical sector number" and that is indeed shifted -2s compared to FAD. But they never use it.

Point here is: What we are trying to do here is create a dump at a semi-logical level, but with audio included. The only way to get any meaningful results is to stick to a few simple rules:
- since we don't dump lead-in, we can use physical address assigned to user area on the disc
- that address, FAD, can be also called LBA because it partitions the whole disc into addressable blocks
- it happens that data tracks use these values for physical sector addressing, so it maps 1:1
- we can partition audio tracks as well and pretend there are "sectors" in there which also can be addressed this way
- all GD-ROMs are so well mastered that we can use these rules to create a dump without any overlaps

This is already cheating a bit since in reality audio tracks are not addressed this way, subcode Q is used exclusively, and it can have an offset to FAD - that is allowed as per spec.
But, as long as we are using sectors as our point of reference, we stick to one addressing system and not mix logical based Mode 1 header addressing with sub Q one! GDI were always sector based so we stick to that and not introduce another dumping system that completly breaks things.

If you want you can create another format, call it something else than GDI, and then you can use Q or mixed addresses as you like, though hopefully with some sane system behing it. But please don't break already established format that works, even if it's not perfect. Yours is even less.

And BTW I don't care what software does what. CloneCD is as flawed as any other program that rips CDs into sector-based data. You can never fix the issues that way, only conceal them, or just plain fake the dump by pretending data/audio can never overlap. Again, read my blog on PCE discs.

So, I'm not arguing that "my definition" of LBA is not shifted -2s and yours is. That dump is broken because it mixed logical and physical addresses. I can tell you this: no matter what you (or those dumpers) think LBA 0 is, the code on Dreamcast will request firs sector of track 3 by specifying address 45150 in the command to the drive. And that is the value in Mode 1 header the drive will look for. So I belive the previous dumps were much more "correct" in choosing addressing method than this new idea.
15th-Mar-2014 10:05 pm (UTC)
First of all, LBA is an official term. And it's not a CloneCD invention - it's just how the reading commands work. Check the "CD-ROM SCSI-2 Command Set Reference Manual", there is a "Read CD" command, which uses LBA as argument and "Read CD MSF", which uses AMSF. Check the page 157 (page 170 of the document, at least, in my version), it says to count "from the first block of the first track" and, in fact, when you read the sector 0, it returns the one with AMSF 00:02:00. And according to "GD-ROM Protocol SPI (Sega Packet Interface) Specifications" (pages 54 and 56), Dreamcast uses FAD in reading commands instead of LBA/LSN, so, yes, there's a clear contradiction.
But looking at TOSEC/Dumpcast images - the first track also always starts from 0 and the third track from 45000 in .gdi with no pregaps, so they also use LBAs. The differ is that their .gdi points to the INDEX 01 (in CD terms) and "that group's" to the INDEX 00 for the tracks with pregaps and to the INDEX 01 for the tracks without pregaps and that's definetely not correct: if you treat .gdi as TOC, where all the first indexes should be shown, it's the broken addressing, yes.
"1 0 4 2352 track01.bin 0" -- what is the meaning of the last 0? Maybe it should be used to show the true beginning of the track? 0 for tracks with pregaps stripped out and 150 for tracks with 2-sec pregaps inside? Can't find the proper format description.

Edited at 2014-03-15 10:15 pm (UTC)
15th-Mar-2014 11:07 pm (UTC)
Again, don't care. It's not SCSI-2 system, it's not like SCSI documents define anything but it's own communication protocol, and it was only meant to address data tracks and not mixed mode discs. The term LBA is also used on other block devices but with the same idea behind it. IT ADDRESSES BLOCKS OF DATA.

All previous dumps always had data tracks start at index 0 but actual files at index 1 - AFAIR the GD drive will not dump pregaps, and we don't need them anyway. Your dump artificially added empty space and "pregap" to track 19 of Chu Chu - except it's not a real pregap because header says index 1. If not for that hack the track data would be shifted relative to it's new, "fixed" postition. Somehow we didn't have to resort to such methods with previous dumps, gee, I wonder why.

So, now this new dump is bigger, contains unused space fillers, and has mixed addressing and created overlaps.

Also, take a look at track 3 first sector data, there is actually pseudo-TOC structure in there at offset 0x100 (+16 in raw images). See the values? 5E B0 00 41? Index 1, data track, address 45150? I suppose SEGA guys were also stupid and didn't know about the track starting at 44850.

Even IF your explanation about pregap made any sense whatsoever, it would still create an overlap of tracks 18 and 19. In fact it would be even worse now, if data track really start - as you say - at exactly the address indicated in the GDI file then it would overlap not pregap but actual data. But, I suppose there is some other convoluted explanation for this problem as well?

To sum this up: You've changed a somewhat flawed but working format to create even worse one, bigger for no good reason, that can't be easily distinguished from previous dumps unless some sanity checks and a lot of value patching is added to the code. You apparently have zero knowledge of how things actually work and just stick to a trendy ripping software and consider that the authority. Your lack of understanding the subject made you apply some silly notion of "logical" sector addressing to the whole area of the disc. It only kinda works because you didn't touch track 3 so games without audio will work, and in case of split data the last track is padded with empty space. Bravo.

Please educate yourself before you come back here. If not I will put you on spam list, because at this point it's how much values your replies bring.

Edited at 2014-03-15 11:10 pm (UTC)
15th-Mar-2014 11:52 pm (UTC)
"except it's not a real pregap because header says index 1" -- since, as you've already noticed, the data is dumped using the swapping, there's no need to fake/generate anything - all the data IS there, including pregaps. Data pregaps are also dumpable (with certain drives), but they aren't in the dump, since emus (and burning tools, if we talk about CDs) expect no pregap there and won't work, if there is.
Also, what header? Indexes are only in subchannels. "00 FF FF FF FF FF FF FF FF FF FF 00 10 01 74 01" -- this one? That's the track mode, not index, pregaps should be in the same mode as the track itself, according to your beloved ECMA-130 (page 18 of the document, page 28 of the pdf), so it must end with 01 (since the track is in Mode 1). For Mode 2 (CD-i, PSX) it ends with 02. And the subchannels are correct there, showing INDEX 00.

"GD drive will not dump pregaps, and we don't need them anyway" - it's like early decrypted Neo Geo roms - they contain the data not directly read from the roms, but the data the system receives after reading. So the dump received from the DC is, in fact, the same temporary truncated thing, that contains the playable game, yes, but can't be called a proper dump at all.

There's no overlapping, since the dump contains all the data from 45000 (45150 FAD) to leadout dumped in a single piece, then splitted by tracks according to subs - nothing is generated and no overlapping in the dump itself (unless you mean .gdi file is screwing the emulator which reads the wrong bytes, then yes, such 'overlapping' is possible). The only problem is that .gdi format itself is too limited and in this case, yes, gdis aren't correct from the emulator's point of view, since they expect no pregaps there, but it's a .gdi format's problem.

And sorry, but it's you not educated enough, you've proven it already in PCE CD discussion earlier, now you prove it again. Also, you're a pretty rude person. Cheers.

Edited at 2014-03-16 12:52 am (UTC)
16th-Mar-2014 01:14 am (UTC)
Not conviced. Pre-gap before last data track should contain mode 0 sectors before mode 1 (sorry, I wrote index where I meant mode). While it is possible it was mastered that way, it's more likely your drive is just not reading mode 0 properly and masking that with zeroes, since the Q sub for that part will say it's audio data. Again, it's all explained in EMCA-130 you disregarded. So whether it's really a proper pregap dump is debatable since only EFM or frame dump could confirm that.

Also, I don't think you quite understand the possible implications of using a swap disk. Namely, drive only reads TOC once, on the swap and not the actual dumped GD-ROM. I won't bother explaining why it's important, you won't get it anyway.

Obviously you know more about Dreamcast and GD-ROMSs then I do. I tip my hat to you sir. One of these days you'll explain, I suppose, why you dump "pregap" on last data track but not track 1 and 3. No encrypted Neo Geo ROMs there, I assure you. It's nice you've finally admitted you broke the format by messing with it though. Maybe you'll realize that the dump itself is flawed as well one day.

Also, I might be rude but you've never _even once_ addressed the pretty obvious contradictions I pointed out in sector header and pseudo-TOC data. No, you know better, facts be damned, specs be damned, CloneCD rules. Keep dumping "perfect" images at logical sector level and post-processing the files to fit your arbitrary rules of Q sub matching. And welcome to my ignore list. You should've ended there after the PCE discussion fiasco, that we both agree on.
16th-Mar-2014 01:42 am (UTC)
From ECMA-130 (since you don't want to look there):
-
Pre-gap : A first part of a Digital Data Track not containing user data and encoded as a Pause. It is divided
into two intervals:
- first interval: at least 75 Sections (at least 1 s) coded as the preceding track, i.e. the Control field
(see 22.3.1) of the q-channel (see 22.3) and, in case of a preceding Digital Data Track, the setting
of the Sector Mode byte are identical with those of the previous Information Track;
- second interval: at least 150 Sections (at least 2 s) in which the Control field of the q-channel and
the setting of the Sector Mode byte are identical with those of the part of the track where user
data is recorded. In this interval of the Pre-gap the data is structured in Sectors.
-

The pregap of Chu Chu Rocket's Track 19 is exactly 75 empty sections (audio pregap format, due to the track 18 is audio) and 150 sections of Mode 1 data sectors, due to the track itself is data.

"pretty obvious contradictions I pointed out in sector header and pseudo-TOC data" -- I've said already, it's a .gdi format limitation not letting to describe the pregaps properly, so it results with wrong addressing in .gdi.

"One of these days you'll explain, I suppose, why you dump "pregap" on last data track but not track 1 and 3." - already explained above, they are dumped, but not included in the dumps themselves, since they won't load in any emulator and even cuesheet format doesn't let you to determine the pregap before the first track to mount such image properly. If you're interested, you may look for Rawdump set, it was a special proof-of-concept project with proper complete dumps, with dumped (not generated) subchannels from the lead-in area with TOCs and with first pregaps. These are the real dumps for MAME-graded preservation purposes, opposite to stripped gdi ones for playing.
16th-Mar-2014 01:27 am (UTC)
And this is exactly why I mentioned that you guys have no clue on software standards - less so on creating a new one than understanding another.

If you want to make a proper standard, then create your own, instead of trying to bend everything in ways they were not meant to be (ultimately only creating more problems for everyone, just because of a fuss over 4kbyte worth of zeros).
16th-Mar-2014 02:21 am (UTC) - awesome man just awesome
good work man and such an amazing feat!!! congrats to you on a job well done cant wait to see some videos of it in action if there is i oughta check it out

cant believe some of these comments you been getting the gull of some people just ridiculous
16th-Mar-2014 04:14 am (UTC)
Can we play TOSEC dumps?
16th-Mar-2014 08:47 am (UTC)
TOSEC and Dumpcast. Or your own dumps. Or dumps done on PC drive as long as it's not the redump's / trurip's braindead hackery.
This also includes CD images in CDI format for homebrew, and ISO as well though obviously that's not self-bootable. Can be used as data carrier though for homebrew apps.
16th-Mar-2014 11:41 pm (UTC) - Fantastic News
Congrats on your work buddy, seems I picked tan interesting time to start reading into the DC scene again!

Would love to buy at least one if I can! I'll keep a close eye and on your posts in case I'm lucky enough :)
18th-Mar-2014 11:37 pm (UTC) - Buying a GDEMU
Outsider checking in, I will definitely buy one!
22nd-Mar-2014 08:49 pm (UTC) - beautiful!
i'll take 2!!!! SRSLY 2.
25th-Mar-2014 11:05 am (UTC)
Hello,

Very very good job
I too am interested to buy a device.

goodbye
25th-Mar-2014 11:57 pm (UTC) - GD Emu
I'm down for one, when I tell me friends I recon they will be too
30th-Mar-2014 12:56 am (UTC) - Count me in...
... Depending on price and ease of installation, anyway.
31st-Mar-2014 07:31 pm (UTC) - Manufacturing in China
Evening mate, been following your project with interest for a while. This used to be the holy grail back in my scene days and its such a buzz seeing an old dream come true.

Just dropping you a line to say that if you need any help getting larger quantities manufactured in China then I'd be more than happy to help you out by throwing some resources your way. I'm also willing to help you source the financial backing but given this day and age you'd honestly be better off crowd sourcing it.

You can drop me a line via here if interested.

Either way, kudos on the graft - a job well done :)
31st-Mar-2014 08:06 pm (UTC) - Slowpoke here
I'm in. I'd buy one, or even a few. :)
3rd-Apr-2014 11:23 am (UTC) - I want one!
That nice, I want one to continue my Shenmue translation project... !
4th-Apr-2014 12:34 am (UTC) - Awesome! I'll totally buy one!
I saw on your page that pre-orders are already closed, but as soon as they are available, I will get one.

Any idea if/when preorders will reopen?

Edited at 2014-04-04 12:43 am (UTC)
19th-Aug-2014 03:26 pm (UTC) - GDROM Explorer: version 1.6.2 out
User japanese_cake referenced to your post from GDROM Explorer: version 1.6.2 out saying: [...] errors that need to be fixed. For those who have read Deunan's post regarding bad GDI dumps [...]
18th-Sep-2014 10:04 am (UTC)
Here is a quote from a NFO, which should throw some water to Deunan's position, which I fully support.

Title....: Eldorado Gate 4 (J)

Supplier.: Hykan - Date.....: 25/12/2010
Selfboot.: Hykan - Genre....: RPG
Platform.: SEGA DC - Files....: 22x20MB
Origin...: JAP/NTSC - Filename.: Hyk-EG4.partxx.rar
Type.....: .nrg 80min - Ripped...: Nothing

[...]

Capcom have throw in a ton of protections into this game:

1. LBA checks
Easily fixed by using LBA 45000 scheme.

2. Country Check
Check if IP.BIN (in memory) is "J " not "JUE".

These can be easily fixed by setting IP.BIN to "J " but then it won't
boot in USA and Europe consoles. Using CDX to boot the "J" game in
foreign consoles will render enemies to have no AI too.

I have found the "U" check routines but not the "J " routines. So I
wrote a small piece of code to run at the beginning of 1st_read.bin
to patch the in memory "JUE" string to "J " as they are not needed
after the game booted.

3. GD Tracks Checks
Check that the LBA of Track 3 is 45000.
since I used 45000 D/D format, there is no track 3, so I redirected it to
check track 2. Note that Echelon's 11700 scheme uses only 2 tracks also.

4. GD Size Checks
Check that that Size of the disk is 0x0861B4
This page was loaded Mar 27th 2015, 11:54 am GMT.