Deunan
KEEP off my lawn 
13th-Feb-2012 02:57 pm
Funny how sometimes things happen, seemingly completely irrelevant to each other, and at some point you realize it's all connected.

I've been asked to look at PC-Engine and new methods of dumping CD-ROM based games for this system. Turns out there are some issues with current dumps, and most importantly some games are still unplayable. We're not sure if it's due to bugs in emulators or not but surely having a correct dump would help a lot.
The big word here is "correct". Dumping unprotected CDs compatible with Yellow Book is easy, however, when you get data sectors inside an audio track it's another story. There's been a lot of deliberation on the subject of CD-ROM dumps, especially when it comes to audio tracks and so-called "audio offset" - something that depends on particular drive/firmware and causes the actual data to be slightly shifted. This is usually way less than 1/75th of a second but many dumpers find it unacceptable that the same disc produces two different results, which cannot be directly compared.

The way I see it audio offset can't be fully accounted for when you dump the tracks on a PC drive. Sure, you can stick to one particular drive model to be able to compare the dumps but this cannot be really called "correct approach". Another problem is inability to directly read and dump special areas on the CD, like lead-in, lead-out, and some track pregaps (which might contain data - as part of a protection scheme or just due to shoddy mastering). I know that some people insist on dumping subchannel data for analysis and then proceed to read the disc according to that - but such methods are flawed as well. First of all, the subs are not protected by correction codes and contain bit glitches. Many are easy to spot and correct, either "by hand" or with a partial redump, but one can never be sure that the Q subchannel was read and reported as-is, or was it perhaps recreated by drive firmware when CRC checksum mismatched. Second, Q subs are not required for each and every section of the track. The drive must be able to deal with missing or damaged subs and this is also why data sectors carry their own address fields. Audio can only be addressed with Q sub, hence the seek function is only required to get as close as possible to requested audio chunk but it might not be a perfect hit. And this is why early drives had so many problems with stuttering audio rips - something that is mostly fixed on current models. Mostly :) Lastly though, many people don't realize that Q sub packets are stored for CIRC-encoded data, that in turn is a shuffled version of the actual raw data. So, in other words, when you remove CIRC and fix any C1/C2 errors, the order of data changes back and it's hard to tell what Q address it really belongs to. Again, not a problem for data sectors that have their own header and address field but it makes audio positioning rather inaccurate - hence the offset.

One method of dumping CDs that is not plagued with these issues is to try and grab raw frames, after EFM processing and before these enter CIRC decoder. We can have 1:1 mapping of data and subchannels, except we now have to process everything software-side in the emulator. Not only that, we need to actually get dumps like that first, and that is not possible with unmodified consumer equipment. Even when we do, we might end up with bit glitches (perfectly fixable with CIRC but still present in the dump) that will require a redump of perhaps several discs to get rid of. There are also weak EFM patterns in some protection schemes but these we can't do anything about, since even capturing analog stream coming directly from the optical pickup would not let us read those properly. So, no point in bothering. This new idea certainly has it's own problems but these can be overcome, so it's at least worth proper research.

So, what's this connection I mentioned? Well, I got a link to a document from KEEP. It stands for "KEEPING EMULATION ENVIRONMENTS PORTABLE" and it's EU-funded project aimed at preserving old games (well, multimedia in general). In theory it's great news that there are organizations like that and at least some politicians and/or civil servant could be convinced to fund them. In reality though the situation looks pretty grimm still.

Here's a link: http://www.keep-project.eu/ezpub2/index.php?/eng/About-KEEP/News/Newsletter-4

I wrote about this several times already but I feel like I need to keep reminding you, and myself, every now and then - so that you don't take things for granted. I urge you people to read this document, it's not all that long. After I did, I came to realise three things:

1) These guys have money.
Apparently got more than 3 million euro so far. You can tour the world with it and have open, free meetings for people. Make no mistake, I'm not saying it's bad, far from it - we need people to understand what's at stake and what the problems are. I just hope this is not the only thing the money were used for.

2) All they did is code UI in JAVA.
Yes, the greatest achievement so far is some silly-looking UI for a couple of open-source emulators. I mean, seriously, most of the document praises this... thing... as if it was the ultimate solution to game preservation. I've seen high-school projects more ambitious than this. Are the authors of WinUAE or VICE aware that KEEP is using their work? I'm not sure so I might be somewhat out of line here, but it really angers me when open-source projects are used like this, just because these are free. Free as in free beer in this case.

3) They are fully aware it's illegal.
At least they worked that out. Even went as far as to specifically point out legal status of bypassing copy protection in Germany and France, and the fact that there are no exceptions, whatsoever, for individuals doing research. I'm willing to bet there were games for Amiga and C64 that were copy-protected, and that authors of WinUAE and VICE had to study those in order to properly emulate. Now, if you can play a game in emulator that was stored on a floppy or a tape in a way that prevented copying, then surely this system has now been bypassed. So, it's illegal. And yet... no-one cares if it's KEEP? I mean, if I was to throw a conference like this and show Makaron working, then I'd most likely have lawyers (and possibly police) knocking on my door the next morning.



Please spread this little rant of mine. I'd like to have these KEEP people made aware of my own existence, and all the other people that were doing emulation and preservation long before they even had the idea. And we are all individuals working for free in our spare time, to beat the clock. Because some of the systems are becoming dust as we speak and soon there won't be anything left to preserve. Whether it's dumping or emulation, we often have to bypass original protection systems to get results, which is illegal. As is storing software you're not licensed to have, even if it's the last copy on Earth. We have the skills and knowledge to do these things and must keep a low-profile, whereas organizations like KEEP do very little to actually change the law but have no problems boasting success when using existing emulators.

If you think I'm overreacting then just answer yourself this little question: What would KEEP be able to show during these meetings if they did not use free emulators? That's right, nothing. Not even that UI since there would be no point to it without actual emulation. And systems and games that were researched and dumped long before they started. I really love the idea of raising awareness on horrible laws we have and all, but to me it feels like KEEP is trying to grab the money while reinventing the wheel. Rather than try and help/fund/coordinate existing projects, they take what is free and claim to have done all the dirty work. As if.
Comments 
14th-Feb-2012 11:04 pm (UTC) - redump.org
Anonymous
Given your apparent expertise, what do you think of projects like redump.org? Dumps are done reading as much of the sub code as possible, reading lead in and out etc.

:)
15th-Feb-2012 06:17 pm (UTC) - Re: redump.org
As I said, dumping using subchannel data as reference works for some systems, but it's not universal solution.

Q sub address and data sector address are not the same thing. In fact some protection systems are based on this. Also, there is no proper way of dumping things like data sectors hidden in audio tracks, so we still need to agree on how to do that and a storage format for keeping such dumps.
Or, we could dump the disc as raw frames and not have to worry about it - though, as mentioned, this approach has it's own problems. So I'll see if it's worth the trouble :)
14th-Feb-2012 11:15 pm (UTC) - Please spread this little rant of mine.
15th-Feb-2012 09:47 pm (UTC) - dumb question
Anonymous
only one question urges me.
WHO THE HELL GAVE THEM ALL THOSE MONEY?

can anyone confirm that's not a "project" sponsored (and paid) by UE?
16th-Feb-2012 07:36 am (UTC)
Anonymous
Not sure how the audio track things related to KEEP. Did I miss something?

For weak EFM, correct me if I am wrong, it seems that the CD reader can read, but can't burn the weak EFM unless it is artificially modified to keep DSV low.
17th-Feb-2012 02:28 pm (UTC)
Apparently you did. Read my post again please.

EFM is handled by the drive's DSP on the fly. There are some rules on how bytes are scrambled and then encoded, and the protection is based on this. There are patterns that make the drive pick suboptimal encodings that eventually cause signal level problems down the line. Since these patterns could, in theory, happen by chance (and not due to protection), the newer drives were made to handle things better. But you can still manufacture a pressed disc with EFM patterns that are simply invalid - these will produce specific read errors that cannot be duplicated.
16th-Feb-2012 06:16 pm (UTC)
Anonymous
Any update on your gd rom drive emulator?
17th-Feb-2012 02:32 pm (UTC)
Busy with other things for the moment. Job, life, that stuff.
17th-Feb-2012 11:14 pm (UTC)
Anonymous
I've been asked to look at PC-Engine and new methods of dumping CD-ROM based games for this system. Turns out there are some issues with current dumps, and most importantly some games are still unplayable. We're not sure if it's due to bugs in emulators or not but surely having a correct dump would help a lot.

If I was you - I would first check out on real hardware.
20th-Feb-2012 05:15 am (UTC) - Dumping
Anonymous
Some CD/DVD Drives (plextor) have a special command to read sectors raw in scrambled form. Detecting sub-channel offset in relation to the sectors they belong to is also not that big of a deal. Detecting the factory write offset of yellow book discs with CDDA is also not that big of a deal with these drives, and using the information to get the same data dumped from ANY drive which can overread is both possible and proven. However I think this is a side track of what this rant should be about...

Of course VICE and WinUAE authors know about copy protections and how to emulate the hardware well enough so that the protections pass the checks. However researching each protection is not necessary if you truly know how the hardware functions and how to emulate it properly. Such as it is with VICE, of course there are people outside of the project who pass along any incompatibilities with any particular piece of software and it can be deduced usually what it causing the issue through independent analysis...

Using these open source emulators without consent of course is the real issue which should be focused on, especially for monetary gain without compensating the authors, and violating the terms of the licenses. That is the real issue IMO and maybe you should stick to that :P
20th-Feb-2012 12:19 pm (UTC) - Re: Dumping
Any data returned as sectors is already cooked by the drive. You just bypass one layer of processing, is all. You do realize that audio tracks have no sectors, right? The stream is partitioned into smaller chunks so it can be read by PC, but where exactly these chunks start/end is down to the drive hardware/firmware. And this is what causes the offset to be different on each CD-ROM drive.

Also, thanks to how CIRC encoder works the offset between Q sub and data sector address might not be constant for the whole disc. But yes, this is not the main issue here, I just wanted to point out that any dump format based on tracks/sectors can never be free of these problems, not to mention copy protection systems.

EU laws state that any device capable of bypassing a functional copy protection mechanism is illegal, both creation and possession. And by "device" it also means software soulutions. It doesn't really matter if the system in question was made with bypassing protection in mind, it's the end result that counts. It's even possible to block scientific papers from being published or even discussed - that has happened.
In US the recent rulings have allowed some more freedom but that's pretty much just for cell phones and personal use. It's not very hard to argue that publicly available emulator is not really "personal" anymore.

As for the EULA, the GPL allows you to even sell the software, there is no violation here. It's not nice of KEEP to use the software like that but they are not breaking any laws or terms. However, for an organization that claims to bring one, true uber-emulator for all possible gaming systems it's really silly to use existing emulators and at the same time claim these must be replaced. All they have shown is that emulation is the key to preserving old games - and we kinda knew that for years now. If I knew I could get 3M for saying obvious things I'd sure sign up.
1st-Mar-2012 01:00 am (UTC) - Raspberry Pi
Anonymous
Seems the Raspberry Pi launched today :)
Looks good, I'm Going to try to get one shortly.

Deunan, do you think makaron would run on it?

Specs:
Broadcom BCM2835 700MHz ARM1176JZFS processor with FPU and Videocore 4 GPU
GPU provides Open GL ES 2.0, hardware-accelerated OpenVG, and 1080p30 H.264 high-profile decode
GPU is capable of 1Gpixel/s, 1.5Gtexel/s or 24GFLOPs with texture filtering and DMA infrastructure
256MB RAM
Boots from SD card, running the Fedora version of Linux
10/100 BaseT Ethernet socket
HDMI socket
USB 2.0 socket
RCA video socket
SD card socket
Powered from microUSB socket
3.5mm audio out jack
Header footprint for camera connection
Size: 85.6 x 53.98 x 17mm
1st-Mar-2012 09:13 pm (UTC) - Re: Raspberry Pi
Sorry, no:
- 700MHz ARM is nowhere near fast enough to properly emulate Dreamcast. While, with enough work, some games could perhaps be made playable, this would be just a gimmick.
- Broadcom hasn't made the DSP available and without FPU of some sort the ARM is not really worth much. It's not yet clear if they will unlock it or not in future.
- It's not running Windows and doesn't have DirectX interface I havily use :)

I'm really interested in this project and will probably get a board for myself, but it's not very good base for emulation. Seriously, while it's amazing how little power these things draw, you have to understand that there are just a couple of CPUs out there that can match x86 performance, clock for clock. And only because these are modern designs that did not have to be backwards compatible with 30+ years worth of code.
1st-Mar-2012 11:51 pm (UTC) - Re: Raspberry Pi
Anonymous
That's a shame, still going to pick up a few, even if it's just for the 1080p playback. I could throw together a box to put it in and tuck it behind my tv.

But it should be ok for say, any emu pre psx/n64 right?
2nd-Mar-2012 12:39 pm (UTC) - Re: Raspberry Pi
Not sure if that ARM is fast enough but PSX can be emulated fully in software. That probably needs a functional FPU though.

See, the main problem is without FPU of some sort you can't make good use of GPU. I mean, you need to feed GPU with vertex data per-frame, and for that you have to compute it first. And, to add insult to injury, the 2D acceleration is not being used now due to some problems. The GUI probably feels a bit heavy and unresponsive because of that.

2D acceleration is usually not needed if you have decent 3D and FPU (but can be better suited for some tasks, like font scaling and rendering). If I had to pick just one, I'd choose FPU. Then 3D, then 2D acceleration.

H264 should not be affected by lack of either since the chip offers hardware decoding (and I bet rendering assitance as well) so for this task CPU only needs to supply the data stream. And this is, AFAIK, already enabled and there's even a player included in the software pack.
2nd-Mar-2012 07:34 pm (UTC) - Re: Raspberry Pi
Anonymous
Thanks for the explanation Deunan. I heard something about the drivers not being complete (that's why the word processor in one of the vids they showed was slow) But I'd imagine it would take too long to sort that out once the device has been out for while.

Thanks Deunan.
This page was loaded Jul 24th 2014, 2:32 am GMT.