Deunan

Condition red

2013-04-10T15:07:05Z

Spot differences between two pictures:

The first one was taken on system with Catalyst 13.1 WHQL and the second one on 13.2 beta. Day wasted on fixing this particular "bug". And before someone points out "This is what you get for using beta drivers" - there is a reason I need to be on 13.2 or newer.

A little help

2013-01-28T23:47:42Z

Dave Murphy from devkitPro.org is trying to raise some money for his project. Please go visit this web page and see if you can help.

devkitPro is a great dev platform for all Nintendo DS homebrew, and also other consoles that I'm not really that familiar with: GBA, GP32, PSP, GC, Wii. It's a very nice set of free tools and libraries for Windows OS, and we should support it because there aren't really all that many.

In fact I'm using it myself - for ARM-related stuff, including GDEMU project. The VMS for NDS was built using devkitPro for example. There is some work being done to have SH4 family support in there as well, so that should also be interesting to Dreamcast homebrew community.

GPU computing

2013-01-26T17:36:32Z

Thanks to an article on SemiAccurate I learned about new AMD gadget called, wait for it, Gizmo. You can see it here.
Gizmo was most likely inspired by the success of Raspberry Pi as dev boards like this existed before but were never this cheap, or even available to general public. Let's compare it to RPi and Intel products then :P

- It's a PC (even runs Windows 7 since that's how AMD guys measured the silicon temperatures under stress).

First, it needs 3V "button" lithium battery, which is mandatory but apparently not part of the kit. In fact it has to be a battery with wires and a small plug, like in laptops, so forget about buying it in TESCO. Tsk.
Then you'll need a SATA hard drive, or SSD, so again forget about cheap SD cards. I suppose a CF card with IDE-to-SATA interface might do the trick if you don't need performance.
Lastly it will obviously need more power than a USB phone charger can provide, much more. The good news is it will accept anything from 9 to 24 volts, so it can be run on 12V lead-acid (car) battery for example.

So, compared to RPi it's not really that great for small projects. It's on par with some of the Intel N-series Atom ITX boards, like D945GSEJT or DN2800MT. Its form factor places it somewhere between RPi and ITX.

- It needs cooling (unless used in a lab environment).

While the board is all passive-cooled it's clearly stated in the docs that this is just enough for 25C ambient temperature and only without a case. If you want to put it in a case or use at higher temperatures you'll need to add a fan to the CPU radiator. There is a fan connector on the PCB for that purpose, though I would've like bigger heatsinks.
The CPU itself is rated at 6.4W but there's also the companion chip (the "south bridge") to consider. The VRM section is also going to generate some heat but I assume it can deal with it in most situations.

Again, a win for RPi and possibly also for the two Atom boards I mentioned since these will work in a case if there is enough convection present. I've seen fanless cases for these Atoms boards so it can be done. Obviously though it depends a lot on where that case will be put :) It might work in an air-conditioned room but not otherwise in summer heat. It's not black and white here.

- It's an APU.

And now we're talking. It's not that much smaller than ITX board and possibly runs hotter so does it have any good sides to it? Yup, the computing power available.
It's a dual-core fully out-of-order AMD64 architecture CPU clocked at 1GHz. That might not look very impressive compared to 1.86GHz N2800 Atom, which is also 64-bit capable and dual-core, with Hyper-Threading to boot, but Atoms are in-order architecture. Turns out it's difficult to make code that would not choke in-order CPUs so much. The compilers are to blame although some code (semi-random branching for example) is just not predictable enough to properly optimize.
The APU is not just CPU though, it's also the GPU next to it. Radeon HD 6250 in this particular case, with 80 shaders clocked at 280MHz.

So why exactly is a measly mobile GPU, the lowest of all AMD has to offer, that much of a win? Because its 80 shaders equal to 1 compute unit (CU), and you can do other stuff with it than just drive VGA output.

To make a point here I've run some tests. My code was trying to brute-force crack M4-type encryption key from dumped NAOMI data. These keys are only 32 bit long and the encryption algorithm is not even that complicated once you see it - again, thanks to Andreas Naive for making "obvious" things actually obvious to us, mere mortals :)
I wrote a cracker in C that, given a key, will decode 8 bytes of data and compare it with known pattern to check for match. To scan entire key space you need to run this code 4294967296 times. A typical, simple approach would be to create a cracking procedure that takes a key value as an argument and then make a loop that will call this procedure 2^32 times, checking the result. Here's how long it takes:

* Intel Core2 Duo E6600
- 1 core @ 2400MHz (2nd core not used)
- full out-of-order architecture
- Windows 7 Professional 64-bit
- 64-bit code (MinGW64 4.5.3 -O2)
+ 415s

* AMD Athlon XP processor 1700+
- 1 core @ 1466.909MHz
- full out-of-order architecture
- Debian Linux, 2.6.32 kernel
- 32-bit code (gcc 4.4.5 -O2)
+ 937s

* Intel Atom N270
- 1 core @ 1596.095MHz (HT not used)
- in-order architecture
- Debian Linux, 2.6.32 kernel
- 32-bit code (gcc 4.4.5 -O2)
+ 2799s

* Raspberry Pi ARM11
- 1 core @ 900MHz (O/C, core @ 450MHz, SDRAM @ 450MHz)
- ARMv6 architecture
- Raspbian Linux, 3.2.27 kernel
- 32-bit code (gcc 4.6.3 -O2)
+ 3378s

As you can see it takes some time, and the in-order Atom and RPi ARM are especially bad at it. And my RPi is running overclocked, the typical values are 700MHz for CPU, 250MHz for core and 400MHz for SDRAM so in reality it's even worse. Obviously you don't want to run crackers on your small dev board but what if this was face/shape recognition based on images from small camera on a robot? That does seem like a plausible use case.

Now there's this stuff called OpenCL which lets you distribute your computation-heavy tasks over multiple CPU cores, and also GPU compute units. I used the same cracker, except the main loop was thrown out and replaced by OCL framework. Here's how it went:

* Intel Core2 Duo E6600
- 2 cores @ 2400MHz
- full out-of-order architecture
- Windows 7 Professional 64-bit
- OpenCl code (AMD APP 2.6)
+ 105s

* AMD/ATI Radeon HD 5770
- 10 compute units @ 850MHz
- GPU architecture
- Windows 7 Professional 64-bit
- OpenCl code (AMD APP 2.6)
+ 6s

Yeah, that's whole 6 seconds. Not all code gets that much of a boost on GPU, this one was integer based with some logic operations but didn't have many branches in it. Even the CPU version got twice as fast as simple C code, most likely due to aggresive compiler optimizations - most loops had just 4 passes so it's a great place to unroll and use SSE2 vectorization.

Now, my 5770 has 10CUs clocked at 850MHz so in total 8500PU - "power units". It run for 6 seconds so it needed 51000PUs to complete the task. The 6250 has only 1CU at 280MHz so 280PUs total. 51000/280=182 seconds. In reality probably a bit more due to slower data transfers. Compare that to Atom results and you'll see why having that GPU is important :)
With dual-core CPU you can easily run a lot of data processing and offload the really heavy stuff to GPU, so it appears to be a great dev board for more advanced projects.

Now why did I bother with this long-winded explanation? Well, it looks like AMD has got all three next-gen consoles in the bag. We've had a lot of "insider leaks" lately, most of it is wishful thinking taken for gospel, especially when it comes to fanboys. Silly people. It's not about raw power anymore. Consoles will not be able to beat PCs with the numbers, not unless you want them to draw 1kW of power and cost the same as rack full of servers. It's about being smart with what limited resources you have. One can argue that's always been the case but this generation will show it even more. A typical PC that can run games in 1080p in 3D at 60fps would need some 300-400 Watts of power. Next gen consoles are promising the same level of fidelity (well, we shall see about that I guess) at half that power. This is what I find most interesting. I couldn't care less if the CPUs are 1.8 or 3.2GHz and how may gigabytes of RAM there are inside.

BTW, I've made some additonal calculations. My RPi runs on 5V and draws 0.5A so it used up 5V * 0,5A * 3378s = 8445Ws to get the calculations done. My Radeon 5770 has 108W TDP so let's assume I actually hit that, and that the rest of my PC drew 150W, which is VERY safe assumption as CPU was idle and so were the HDDs. (108W + 150W) * 6s = 1548Ws. So not only it was faster but also used less power :) Nice things, these compute units. With 16 thousands 128-bit wide registers it's no wonder each takes so much silicon space.

End of the world

2012-12-18T14:18:43Z

I might not belive that the world will end this December but two of my PCs decided not to wait and commited suicide.

My netbook died first, about a month ago, one day simply didn't turn on and that was it. No amount of messing with its internals would help. It was an old hand-me-down with Atom N270 that I got for free because of failed HDD. I replaced it, reinstalled OS and kept using it for a year or so. It had Win XP, 1024x600 matte screen, 1G of RAM and the battery would hold for about 2 hours - which was good enough for my needs. Hell, it flew with me around Europe a few times. I wasn't using it much at home so I don't need to replace it right away but I sure miss it.

Yesterday another N270 gave up the ghost, this time it was my Linux system that I keep running 24/7 for various purposes: router for my private LAN, WiFi AP, FTP/NFS server, and most importantly my dev machine for Dreamcast and NAOMI since I keep my cross-compiler tools there. I liked this board too, it was all-passive cooled and required only 12V input from a brick-type PSU so there were no fans at all. I think the BGA balls cracked because I would get random reboots lately and last week the system would not boot up until it has cooled down to room temperature. Eventually even that stopped working and now it will reboot randomly within 10 seconds of powering up, cold or not. So, right now half of my flat has no Internet and I need to fix that ASAP.

I ordered a new board, it's another Atom (N2800 this time) since I really want to keep the energy usage down to bare minimum and I don't need a lot of CPU power. Even N270 could easily deal with 100Mb/s traffic on both NICs while streaming from HDD, and it was 2.5W rated. Yeah, I know, it doesn't include the north bridge which was doing most of the job connecting all system components together :) So N2800 might be 6.5W but I expect NM10 to have improved over the old 945 (and GPU is now part of the CPU as well). I was also interested in AMD Brazos family but those chips are much more powerful and require active cooling, and I don't need Radeon HD in a headless PC. The good news is the new board will also be powered by single 12V so at least I get to keep the PSU - hopefully. I already had to buy a new memory stick (DDR3 now instead of DDR2), a new low-profile NIC (no PCI slot, just one PCI-E x1), and a new N-capable WiFi card (miniPCI-E). Well, at least my netbook HDD is going to be reused :P

There is one more old PC that I have, and obviously my main one that is not very old but it has its years. I swear, one of them dies in the next few weeks and I'm buying a replacement and calling it Apollo 13. In the meantime I started doing more frequent backups.

Anyway, so what's up with the GDEMU project. Well, there is progress but I've hit some problems - as usual. I came up with new logic for the FPGA and it works perfectly (so far) between MCU and FPGA but fails on the GD bus. And I have no idea why, I've tried pretty much everything by now, except adding some pull-ups to control lines but I don't expect this to help much. Doesn't look like an electrical problem.
The prototype works when FPGA is clocked within a very specific frequency range, but not really otherwise. BIOS loads the game, I get to see the first screen or so and then it dies because DMA goes completly out of sync - I still have tons of data in the buffer but the console expects to see end-of-DMA interrupt already. So obviously I'm missing a lot of read requests but I don't know why. Must be another race condition that I can't figure out. So, why not let it run at the frequency it works? Because the problem is still there, just not as obvious. It's not stable either way and you wouldn't want your game to freeze 3 hours in and who knows how long since last save, right?

To combat that I've finally gave in and bought USB based JTAG programmer for Altera FPGAs. Those things are costly but I found a cheap clone that should work nice. I expect it to arrive in a few days. With live JTAG uplink I will be able to transfer new settings directly rather than have to swap SD cards as I do now, and more importantly I'll be able to run a logic analyzer to see what is going on.

The world will most likely not end but thanks to all those troubles (and GoG discounts :) my bank account balance just might.

EDIT: Looks like it could be electrical issue after all. Well, I'm going to rip the Dreamcast apart now and solder some proper wires for ground return path. Lets see what that does.

Oh, and here's a photo:

As you can see I got the JTAG unit today and I'm fresh out of USB ports on the hub :)

EDIT 2: Apparently one year was not enough to add proper idle support for Cedar Trail Atoms to Linux kernel. Not even the bleeding edge 3.7.1. If you run dmesg |grep intel_idle you'll see this:

intel_idle: does not run on family 6 model 54

So, if you're in this situation as well and you don't mind compiling kernel from sources, try this hack:

1) Locate drivers/idle/intel_idle.c in the source tree
2) Make a backup copy just in case :)
3) Edit the file, find "intel_idle_ids" table
4) Add "ICPU(0x36, idle_cpu_atom)," line to it, but keep it sorted by model code

Compile and install the modules and kernel. Reboot. Enjoy.

Now, I'm not saying this is the proper way of doing it but 20mA less current draw from PSU (at 12.2V) says it's working. I haven't seen any nasty side effects yet.

Genesis contd.

2012-09-03T15:35:40Z

And behold, it was very good...

Okay folks, since there are so many questions about the GD-EMU project and noone can be bothered to read the answers from the time I showed you my first iteration of the idea, here it is all again:

1) Ready when?
No idea. I would not be making a custom PCB and ordering new parts and working on it if I didn't belive it can be done, but at the same time I cannot (and will not) make any promises about delivery dates. Obviously though if I can't make it work as I'd like in the next few months it's going to be shelved again.

2) How much?
Again, no idea. In fact it's not even decided I will be selling those. If it doesn't seem like I can turn a profit without investing all my free time into it, I'll just stop at prototype phase. While I understand that it would upset many of you, I'm not a charity worker. It's one thing to code a free application and share it with the world and quite another manufacturing a hardware device for sale.

All I can say right now is the prototype is pretty expensive (compared to a price of a working, pre-owned Dreamcast). But that is true for all prototypes. Things get considerably cheaper when mass-produced. Then again it's quite possible the first batches will still be priced higher because of low volume of sales - I'm sure as hell not going to invest my own money into this.

3) Kickstarter? Preorders?
While Kickstarter seems like a good option, it's a no-no because I'm not a US resident. End of story right there. I will also not take any kind of preorders (or other money offers) until I'm certain the device will work and can be manufactured in suitable quantities. Things get serious when money are involved and I'm a rather cautious person.

4) Features?
It will be a 100% compatible replacement for GD-ROM drive, except using SD cards. It might offer better loading times but otherwise will function in the same way. It's meant to provide a backup solution for the laser and other mechanical parts of the drive which are no longer in production and fail after so many years of use. While many of you will interpret this last sentence as "it will play game rips" I'd like to point out that I never condoned software piracy. I think I made my point clear when I refused to fix any bugs in Makaron that were related to CDI rips of the games (as opposed to proper GDI images). Many of these "bugs" were actually how the rips worked on a real console, although these could be somewhat helped if I wanted to. But I didn't. So, if you are/were a Dreamcast user then you should be familiar with region locks, video cable restrictions, bootable (or not) homebrew, etc. Using GD-EMU will not remove/help with any of these. You might try image patching, sure, but I will not give any support for these modifications if there are any problems.

As for user interface - I like simple things that work as expected. I've seen too many projects that looked nice but didn't deliver what was promised in the first place. My goals are perfect compatibility and stability. Anything else is extra. I think 2 buttons is enough to select which game on the card should be "inserted".
If that's not enough for you, code a good Dreamcast app that will select games from the card - it can be put as the first image on it, which will boot by default. Then we can talk about how to make the hardware do what the app/user wants.

5) USB link to PC?
That's in plans, but no work has been done yet. I'm not even sure the USB port on the prototype works properly :) So, eventually yes, but probably not from the start. USB host support (as in USB HDDs and FLASH drives) is probably not going to happen. Did I mention I like simple solutions?

6) Other features?
Well, if it ever happens that I make tons of profit on these things, which I doubt, I might reconsider my stance on UI, USB host, and other things. But that would have to be a considerable amount of money to motivate me :)

7) Open source?
Highly unlikely. If only because some people could just take all my work and start selling their own devices. While I'm not stopping anyone from creating a different/better project, they better be prepared to spend as much time on it as I have. I've already helped many people by sharing important bits and pieces of info, and even programs made by me. There is goodwill and there is stupidity - and I have to say that more often than not I've came to regret my decisions. Once burned...

8) Pics or it didn't happen.
There are photos of my all-FPGA approach on this blog, and even some short movies on YT of it working (with minor issues) if you know where to look. I will post pictures of the V2 prototype connected once it actually does work. I'm redoing much of my FPGA code and this might take some time as I want to try another approach.

Genesis

2012-08-30T12:44:47Z

Let there be light:

And there was light:

More pictures to follow soon :)

Status so far:
Voltage regulators - check
MCU starts - check
Bootloader operational using 3V3 UART - check
MCU JTAG - check
C runtime stub + simple exception handling - check
Status LEDs - check
UART 115200 8N1 console - check
Interrupts - check (~~need to investigate if registers are really properly saved though~~)
External RAM - check (problem found, ~~should be~~ fixed now)
High speed SD interface - in progress

Random fun fact: Many SD/SDHC cards exhibit various little quirks in SPI mode so the code needs to be aware of those to work properly in every case. One would think the native SD protocol is so tightly standardized that there should be no such surprises. Well, I just found a bunch of 2GB Kingston SDs that respond to ACMD41 with bad CRC7...

EDIT: Turns out the R3 answer is the only one not protected by CRC7, that space is marked as reserved and just filled with all-ones. I'm still not getting the busy bit within reasonable times on these Kingstons but I suppose reading the docs few more times might teach me something new again.

Anyway, here's the actual thing:

Now it's a proper prototype, with all these wires and blinking LEDs. A few things are still missing on the PCB but right now I need to get SD protocol working so I can fetch FPGA configuration image and test it.

EDIT:

High speed SD interface - check
DMA on SD i/f - check
Basic FAT support - check
FPGA - in progress

I'm using my own FAT library, which has no write support but it was designed to be fast while consuming as little RAM as possible. In fact current SD cards are so fast it makes sector buffering impractical, since the lookups and LRU queues kill any gains with additional overhead. I suppose it'd be different if the CPU was clocked above some 400MHz and had some fast L1 cache.
Right now I get average of ~10MB/s in test that seeks to random part of 1.2GB file and reads 1-3500 consecutive 2352-byte long chunks. This is to simulate RAW image reads for GD-ROM. So pretty well I'd say, a nice boost compared to 2.5MB/s I got over SPI.

The native SD interface required a pretty much complete rewrite of some code, so I'm not 100% sure it's stable and all, but seems to work for hours without problems so far.

With a piece of wire

2012-06-23T00:56:44Z

SDR in 5 easy steps:

1) Design your radio

2) Build it

3) Get control software

4) Get decoder software

5) Amaze your friends

Considering I got this far with a 2m piece of wire hanged across my window, I'd say it's a success :)

My name is Bond

2012-06-04T11:06:41Z

Disclaimer: This entry is not emulator related. If you're here only for Makaron news, please ignore it.

I found a very interesting paper a few days ago, here's a link: Breakthrough silicon scanning discovers backdoor in military chip

This is just a brief summary of a side attack on Actel/Microsemi ProASIC3 chips. The power analysis idea alone is mighty interesting - for such a big silicon structure as FPGA it seems impossible at first, but then it's not the whole chip that has been put under scrutiny. It's just the part responsible for JTAG and some of the internal addressing structrure, but even then measuring the subtle current fluctuations is not exactly trivial. While simple on paper, this task requires a number of fast and well-calibrated ADCs and most importantly a software capable of processing the collected data. I could probably come up with a good enough electrical component of such testbed but that software part is like magic to me - and I'm no stranger to programming :) Ah well, I suppose it's better for me to accept that certain math problems are way above my head, less stress that way (I guess ignorance IS a bliss after all).
Anyway, ProASIC3 series have been well designed, with countermeasures in place to thwart this type of attacks. The paper mentions fuzzy clock sources and very low leakage transistors, as well as carefull structure design - all these factors contribute greatly to the chip's security. It is, after all, touted the most secure FPGA available on the market today. But there's a twist: apparently the chip has some undocumented features that allow one to read it's configuration and internal memories even after it has been secured with secret key.

You see, when it comes to JTAG it's pretty much normal that most (if not all) of the protocol commands intended for factory testing are kept secret. This is to protect the manufacturer secrets, nothing strange to it. However, it's one thing when the design has, say, additional structures to improve yields and some of them are disabled even if fully operational in order to keep all chips with the same specs. It's a dirty little trick but in the end acceptable. It's quite another story with intentional backdoors placed into the design.

Hidden features like these are handy for the people who make the product, in case there is a problem that needs to be fixed after the production has been completed. It doesn't have to be a hardware thing - remember the first X360 DVD drives? Those could be reprogrammed if you just knew how to do it, sometimes even via the SATA link so there wasn't even any need to open the cover. This was a firmware backdoor and once discovered it was ultimately used enable the drives to accept recordable media as original games. I seriously doubt Microsoft had any knowledge of this prior to discovery by hackers, this was most likely something the drive manufacturer added on it's own to be able to reuse returned/repaired drives. This 'cost saving' in company A caused company B much grief and money loss.

So, the big question: Was it Actel that put this 'feature' in the design, or was this added by the Chinese factory that actually made the chips? Either way this will have far reaching consequences as ProASIC3 series is often used in military equipment. So I wonder if it's just a blunder or a some sort of espionage attempt. Actel does claim that their products are secure and the contents of the chip cannot be easily recovered as there is no way of doing so. Clearly, that is a lie. Obviously the company now says that there isn't any backdoor but you can't prove that something doesn't exist. On the other hand, the researchers will need to verify their claim by showing a succesful attack attempt to the public. So I guess we wait and see what happens. If it turns out to be true Actel's whole CPLD/FPGA branch could be finished.

In other, somewhat related news: I've taken some interest in software defined radios. I have to say that just by researching the subject my understanding of modern signal processing technology has gone up quite a bit. The best SDRs out there use FPGAs for signal processing - ADC interface, sampling rates, decimation, digital filtering, etc. A proper design can cover a big chunk of frequency spectrum while sampling at many Ms/s. The best way to go about it is to use quadrature demodulators to produce I/Q signals. These can be then freely processed PC-side (as long as you have CPU power to do so) to obtain anything from AM radio audio to satellite imagery. Problem is, these toys are costly :) I'm not about to throw 3-4k$ at my 'new hobby' so the only other options are:

1) Self-made QAM mixer/demodulator that uses audio frequency output. This can be plugged into modern PC soundcard and sampling rates of 96kHz or even 192kHz are avaiable, at a very nice 16-bit resolution. Some cards can even go as high as 24 bits, though in reality the SNR is probably not going to exceed some 20-22 bits even in best of conditions. I've actually designed a PCB based on YU1LM DR2A design, which is pretty simple and anyone can make it cheaply since there aren't any hard to come by parts. It needs external VFO at 4x the local oscillator frequency but I can make that too. Worst case scenario I'll buy a cheap DDS kit based on AD9850 or something like that.

Here's what the PCB will look like:

2) For some 20$ you can buy a USB DVB-T dongle with Realtek RTL2832 chip inside. That chip model is important, apparently it can be put into a 'dumb' mode where it samples and passes the I/Q data unmodified to the PC. With a proper tuner attached it will cover 80-1700MHz range, with some holes though, and the SNR depends on the frequency. The best tuner seems to be Elonics E4000 but other ones (FC0012/FC0013 for example) will also work, though with more holes so less frequency coverage. This mode was meant for software processing of FM and DAB/DAB+ radio signals, and it seems to be undocumented feature not present in other designs. RTL2832 can only provide 8-bit data but it makes up for that in the bandwidth of 1Ms/s up to 3.2Ms/s. The best setting is an integer multiple of the on-board resonator, usually 28.8MHz, so that there are no sample drops. So, while far from perfect it's a great bargain for 20 bucks.

And the best thing, even though it's called a radio, is that such setup is pretty much a PC-based spectrum analyzer. These things are quite expensive as standalone hardware so it'll be interesting to see if a self-made project like this can be useful for something other than listening to police radio traffic :)

Still cranking

2012-03-12T13:41:20Z

I didn't get into finer details of the low-level CD dump in my previous post because it would get really long and boring - some people however took that as a lack of proper understanding of the complexity of such project on my part. Rest assured I did give it a lot of thought :)

I'm fully aware that a raw bitstream from the optical pickup contains many bit glitches, as EFM is far from perfect. After all this is what we have CIRC for. If the drive logic can deal with those glitches, so can we. The only difference is the drive will correct (where possible) the bad symbols in the stream and produce a correct output, whereas we want to come up with a clean input stream that would not even need correcting. This is more problematic but still doable.

What is a bit glitch? It's exactly as it sounds, a single bit having wrong value (1 instead of 0, or the opposite) somewhere in the stream. It's usually just one bit per byte, and doesn't happen very often, but more serious read errors happen as well from time to time. How often is "not very" you might wonder. Well, let's analyze Ranma dump I've made.

To keep things a bit less messy I will only use tracks 3-55 of the dump here, all of which are audio. 257060 audio sections to be precise, for a somewhat round number. Since each section is accompanied by exactly one Q sub packet, we have exactly 257060 of those as well. Knowing that this particular subchannel has a CRC we can easily check the integrity of all of them. And it turns out 151 are glitched.

In case you're not clear on why we stick to subchannels and not actual data, it's because subs are not protected by CIRC so a dump will not be repaired on the fly. Or at least should not be, some drives can actually do that for P & Q subs. It's not all that difficult (might be a bit slow though) - knowing that it's usually one bit that goes wrong we just keep flipping all the 96 bits in the Q packet and recalculate the CRC. Eventually we find the right one and have repaired the stream. Now, I do realize that we could end up with a bad repair and CRC collision but that is very unlikely for a single bit change. Also, keep in mind that it might be the CRC area that has the glitched bit :)

Sure enough, out of those 151 bad packets 150 were fixed by flipping one bit. One case was two bits going wrong next to each other, so it could be a fluke, or a case of a "walking bit" problem. These are also easily fixed but require another pass over the data and more complicated algorithm. In the end all packets were repaired though. CIRC is more complicated in principle but also designed for fixing the errors, not just detecting them. This is why we could have a dump that has many glitches but can be used since it auto-repairs itself when processed properly. But, obviously, we want to have clean one - and the question is how do we obtain one.

We could just repair the image after the dump is done, that will probably fix most of the glitches. Some might be tricky enough to warrant a partial redump to make sure, but that could be automated, we could even do two passes in all cases to have more material to work with. Unless the disc is damaged and keeps returning errors for one particular area, the glitches are otherwise pretty random, caused mostly by various harmonic frequencies in the rotation speed introduced by the spindle motor. Pickup positioning and focus is also not perfect and subject to both vibrations and thermal noise in the coil drivers.

Ranma, by the way, is also not exactly an example of good CD mastering. The data LBA to Q sub offset is maybe not as big as on Lady Phantom but it's there. Also, there are mode-2 Q sub packets present on the disc, even if empty, so it makes one wonder why didn't they delete them altogether if Catalog Number was not recorded. I'm mentioning that because mode-2 Q does not carry addressing information except the fraction field.

And, just to put things into perspective here, since you migth think that 151 glitches per (pretty much) a whole disc is not a big deal: This was just the Q subchannel data. There are 2352 bytes in the sector (when dumped raw or as audio section) and another 96 in the subchannels (98 actually but 2 bytes are used internally as a sync field, and not reported). A raw sector plus sub packet, when rejoined together, is 2448 bytes total. The Q subchannel alone is just 12 bytes. Since we got 151 errors by analyzing just 12 bytes out of 2448, then it means the total number of glitches would be about 30 thousands for the whole dump. And that is what I call a problem, and why we don't have a low-level dumper yet.

Cranking old engines

2012-03-02T22:31:46Z

Lookit what goodies came in the mail today:

There is a reason why I'm showing you Lady Phantom - this is one of the somewhat problematic CD-ROMs for PCE. First, the TOC:

-> Track 01  (Audio, 00:47:65, LBA: 0 / 00:02:00)
-> Track 02  (Mode 1, LBA: 3590 / 00:49:65)
-> Track 03  (Audio, 02:34:63, LBA: 5157 / 01:10:57)
-> Track 04  (Audio, 02:23:59, LBA: 16770 / 03:45:45)
-> Track 05  (Audio, 02:48:09, LBA: 27554 / 06:09:29)
-> Track 06  (Audio, 02:30:08, LBA: 40163 / 08:57:38)
-> Track 07  (Audio, 02:48:02, LBA: 51421 / 11:27:46)
-> Track 08  (Audio, 02:27:37, LBA: 64023 / 14:15:48)
-> Track 09  (Audio, 00:33:26, LBA: 75085 / 16:43:10)
-> Track 10  (Audio, 00:09:50, LBA: 77586 / 17:16:36)
-> Track 11  (Audio, 00:33:54, LBA: 78311 / 17:26:11)
-> Track 12  (Audio, 02:13:59, LBA: 80840 / 17:59:65)
-> Track 13  (Mode 1, LBA: 90874 / 20:13:49)
-> Track 14  (Audio, 02:14:25, LBA: 91550 / 20:22:50)
-> Track 15  (Mode 1, LBA: 101625 / 22:37:00)
-> Track 16  (Audio, 02:55:64, LBA: 102325 / 22:46:25)
-> Track 17  (Mode 1, LBA: 115514 / 25:42:14)
-> Track 18  (Audio, 02:05:73, LBA: 116214 / 25:51:39)
-> Track 19  (Mode 1, LBA: 125662 / 27:57:37)
-> Track 20  (Audio, 01:54:28, LBA: 126384 / 28:07:09)
-> Track 21  (Mode 1, LBA: 134962 / 30:01:37)
-> Track 22  (Audio, 01:43:38, LBA: 135678 / 30:11:03)
-> Track 23  (Mode 1, LBA: 143441 / 31:54:41)
-> Track 24  (Audio, 02:30:03, LBA: 144149 / 32:03:74)
-> Track 25  (Mode 1, LBA: 155402 / 34:34:02)
-> Track 26  (Audio, 01:52:62, LBA: 156088 / 34:43:13)
-> Track 27  (Mode 1, LBA: 164550 / 36:36:00)
-> Track 28  (Audio, 02:09:10, LBA: 165266 / 36:45:41)
-> Track 29  (Mode 1, LBA: 174951 / 38:54:51)
-> Track 30  (Audio, 00:37:56, LBA: 175575 / 39:03:00)
-> Track 31  (Mode 1, LBA: 178406 / 39:40:56)
-> Track 32  (Audio, 01:21:72, LBA: 179070 / 39:49:45)
-> Track 33  (Mode 1, LBA: 185217 / 41:11:42)
-> Track 34  (Audio, 01:43:24, LBA: 185889 / 41:20:39)
-> Track 35  (Mode 1, LBA: 193638 / 43:03:63)
-> Track 36  (Audio, 01:11:69, LBA: 194338 / 43:13:13)
-> Track 37  (Mode 1, LBA: 199732 / 44:25:07)
-> Track 38  (Audio, 01:31:39, LBA: 200368 / 44:33:43)
-> Track 39  (Mode 1, LBA: 207232 / 46:05:07)
-> Track 40  (Audio, 03:42:69, LBA: 208632 / 46:23:57)
-> Track 41  (Audio, 01:01:60, LBA: 225351 / 50:06:51)
-> Track 42  (Mode 1, LBA: 229986 / 51:08:36)
-> Track 43  (Audio, 04:11:57, LBA: 230654 / 51:17:29)
-> Track 44  (Mode 1, LBA: 249536 / 55:29:11)
-> LeadOut  (LBA: 250814 / 55:46:14)

It's very rare for any disc to have so many data and audio tracks mixed together. PCE doesn't use ISO9660 so the devs just split the game data as they saw fit.
If this was a well-mastered Yellow Book compliant CD-ROM then we'd just rip the tracks, ignore pre/postgaps and it'd be a complete, working dump. It's not so easy in this case. Let's analyze the transition from track 14 to track 15.

-> Track 14  (Audio, 02:14:25, LBA: 91550 / 20:22:50)
-> Track 15  (Mode 1, LBA: 101625 / 22:37:00)

The last sections of track 14 are silent, this is normal except for cases where you don't want gaps in audio. Since the next track is data it's a good idea to do this properly. Except, you know, the very last sectors are filled with junk here :)

LBA  :  101472

0000 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................  <- silence
SUBC :  01 14 01 02 12 22 00 22  34 72 00 00 00 00 00 00  ....."."4r......

LBA  :  101473

0000 :  6A 82 AF 21 BC 18 71 CA  A4 57 3B 7E 93 60 6D E8  j..!..q..W;~.`m.  <- junk
SUBC :  01 14 01 02 12 23 00 22  34 73 00 00 00 00 00 00  .....#."4s......

LBA  :  101474

0000 :  6A 82 AF 21 BC 18 71 CA  A4 57 3B 7E 93 60 6D E8  j..!..q..W;~.`m.  <- more junk
SUBC :  01 14 01 02 12 24 00 22  34 74 00 00 00 00 00 00  .....$."4t......

Notice how Q sub properly identifies this as audio track 14, index 1, time 22:34:74 - which is (22*60 + 34)*75 + 74 = 101624. With the usual TOC offset of 150 we get LBA 101474. Figures.
Also, TOC says track 14 should be 2:14:25 long and sure enough this is the last audio section, 2:12:24, according to Q (these counters start at zero).
Now, since there should be a 2-second long pregap for a data track that follows an audio track, and the data track starts at LBA 101625, so that means the first Mode 1 sector should be at LBA 101625 - 2*75 = 101475:

LBA  :  101475

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 00 01  ............"5..  <- Mode 1 header, first sector of a pregap
SUBC :  01 14 01 02 12 23 00 22  34 73 00 00 00 00 00 00  .....#."4s......  <- Q address goes back!

And we have a Mode 1 header! Except Q sub still says it's audio track #14-1 :) Also, data header address is 22:35:00 and Q has 22:34:73.
It doesn't take a genius to notice that 22:34:73 was also the first audio sector with "junk" instead of proper silence. Gee, I wonder what would happen if we extracted that junk and run it through descrambler. Yup. It is, in fact, the same sector we are looking at now and the header is 00 FF FF FF FF FF FF FF FF FF FF 00 22 35 00 01.
So... two last sections of track 14 and two first sectors of track 15 are overlapping. Depending which track you're reading, the drive will return either raw bytes for audio or descrambled sectors for data. In fact, it's even worse:

0000 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0010 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0020 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
...
0440 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0450 :  00 00 00 00 00 00 00 00  00 00 00 00 00 FF FF FF  ................
0460 :  FF FF FF FF FF FF FF 00  23 B5 00 61 00 28 00 1E  ........#..a.(..
0470 :  80 08 60 06 A8 02 FE 81  80 60 60 28 28 1E 9E 88  ..`......``((...
0480 :  68 66 AE AA FC 7F 01 E0  00 48 00 36 80 16 E0 0E  hf.......H.6....
0490 :  C8 04 56 83 7E E1 E0 48  48 36 B6 96 F6 EE C6 CC  ..V.~..HH6......

The data sector 22:35:00 starts in the middle of audio section 22:34:72. Which is perfectly OK. Except it cannot be properly ripped when the drive already cooks the raw data into sectors - since you can sync data to headers but there is no way to sync the audio.
And this is why I said it's not always possible to rip a CD-ROM track by track, even using subcodes as a reference. It only works for discs where audio sections match data sectors perfectly and even then it's down to drive logic to figure out how to join subchannel data with post-CIRC stream. So, claiming that any dump is a perfect copy of the master image is just... silly.

Let's see what happens next with our dump:

LBA  :  101476

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 01 01  ............"5..
SUBC :  01 14 01 02 12 24 00 22  34 74 00 00 00 00 00 00  .....$."4t......  <- note that Q still says track type is audio

LBA  :  101477

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 02 01  ............"5..  <- M1 address: 22:35:02
SUBC :  41 15 00 00 01 74 00 22  35 00 00 00 00 00 00 00  A....t."5.......  <- Q address: 22:35:00 (index 0, so a pregap)

LBA  :  101478

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 03 01  ............"5..
0920 :  00 00 00 00 00 00 00 00  00 00 AD 90 29 C9 83 16  ............)...
SUBC :  41 15 00 00 01 73 00 22  35 01 00 00 00 00 00 00  A....s."5.......

LBA  :  101625

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 00 01  ............"7..  <- First sector with actual data
SUBC :  41 15 00 00 00 01 00 22  36 73 00 00 00 00 00 00  A......"6s......  <- Q still says pregap

LBA  :  101626

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 01 01  ............"7..
SUBC :  41 15 00 00 00 00 00 22  36 74 00 00 00 00 00 00  A......"6t......

LBA  :  101627

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 02 01  ............"7..
SUBC :  41 15 01 00 00 00 00 22  37 00 00 00 00 00 00 00  A......"7.......  <- Q finally switches to index 1

Q never matches the header address and in fact even the track type is wrong for some sectors. But don't assume that this 2-section offset is constant for all the audio/data tracks on the disc. While it could be the case, it's not guaranteed. The spec allows up to +/- 1 second offset so the difference could be anywhere from zero to 150 and, depending on what the drive logic thinks is the start of audio data, the header position might end up in different part of the audio section. In fact it's allowed for the header to start well inside a frame, as long as it's on 4-byte boundary.

So... all these frames, sections, offsets, sectors, tracks and gaps start to confuse you? Well, keep in mind the CD, while digital in nature, was meant to carry audio. Data sectors were added later and kinda slapped onto existing audio layer. This let the manufacturers reuse their designs by just adding header detection and descrambling hardware (along with some RAM for data buffering). But thanks to that we have this mess on our hands now :)

KEEP off my lawn

2012-02-13T13:57:56Z

Funny how sometimes things happen, seemingly completely irrelevant to each other, and at some point you realize it's all connected.

I've been asked to look at PC-Engine and new methods of dumping CD-ROM based games for this system. Turns out there are some issues with current dumps, and most importantly some games are still unplayable. We're not sure if it's due to bugs in emulators or not but surely having a correct dump would help a lot.
The big word here is "correct". Dumping unprotected CDs compatible with Yellow Book is easy, however, when you get data sectors inside an audio track it's another story. There's been a lot of deliberation on the subject of CD-ROM dumps, especially when it comes to audio tracks and so-called "audio offset" - something that depends on particular drive/firmware and causes the actual data to be slightly shifted. This is usually way less than 1/75th of a second but many dumpers find it unacceptable that the same disc produces two different results, which cannot be directly compared.

The way I see it audio offset can't be fully accounted for when you dump the tracks on a PC drive. Sure, you can stick to one particular drive model to be able to compare the dumps but this cannot be really called "correct approach". Another problem is inability to directly read and dump special areas on the CD, like lead-in, lead-out, and some track pregaps (which might contain data - as part of a protection scheme or just due to shoddy mastering). I know that some people insist on dumping subchannel data for analysis and then proceed to read the disc according to that - but such methods are flawed as well. First of all, the subs are not protected by correction codes and contain bit glitches. Many are easy to spot and correct, either "by hand" or with a partial redump, but one can never be sure that the Q subchannel was read and reported as-is, or was it perhaps recreated by drive firmware when CRC checksum mismatched. Second, Q subs are not required for each and every section of the track. The drive must be able to deal with missing or damaged subs and this is also why data sectors carry their own address fields. Audio can only be addressed with Q sub, hence the seek function is only required to get as close as possible to requested audio chunk but it might not be a perfect hit. And this is why early drives had so many problems with stuttering audio rips - something that is mostly fixed on current models. Mostly :) Lastly though, many people don't realize that Q sub packets are stored for CIRC-encoded data, that in turn is a shuffled version of the actual raw data. So, in other words, when you remove CIRC and fix any C1/C2 errors, the order of data changes back and it's hard to tell what Q address it really belongs to. Again, not a problem for data sectors that have their own header and address field but it makes audio positioning rather inaccurate - hence the offset.

One method of dumping CDs that is not plagued with these issues is to try and grab raw frames, after EFM processing and before these enter CIRC decoder. We can have 1:1 mapping of data and subchannels, except we now have to process everything software-side in the emulator. Not only that, we need to actually get dumps like that first, and that is not possible with unmodified consumer equipment. Even when we do, we might end up with bit glitches (perfectly fixable with CIRC but still present in the dump) that will require a redump of perhaps several discs to get rid of. There are also weak EFM patterns in some protection schemes but these we can't do anything about, since even capturing analog stream coming directly from the optical pickup would not let us read those properly. So, no point in bothering. This new idea certainly has it's own problems but these can be overcome, so it's at least worth proper research.

So, what's this connection I mentioned? Well, I got a link to a document from KEEP. It stands for "KEEPING EMULATION ENVIRONMENTS PORTABLE" and it's EU-funded project aimed at preserving old games (well, multimedia in general). In theory it's great news that there are organizations like that and at least some politicians and/or civil servant could be convinced to fund them. In reality though the situation looks pretty grimm still.

Here's a link: http://www.keep-project.eu/ezpub2/index.php?/eng/About-KEEP/News/Newsletter-4

I wrote about this several times already but I feel like I need to keep reminding you, and myself, every now and then - so that you don't take things for granted. I urge you people to read this document, it's not all that long. After I did, I came to realise three things:

1) These guys have money.
Apparently got more than 3 million euro so far. You can tour the world with it and have open, free meetings for people. Make no mistake, I'm not saying it's bad, far from it - we need people to understand what's at stake and what the problems are. I just hope this is not the only thing the money were used for.

2) All they did is code UI in JAVA.
Yes, the greatest achievement so far is some silly-looking UI for a couple of open-source emulators. I mean, seriously, most of the document praises this... thing... as if it was the ultimate solution to game preservation. I've seen high-school projects more ambitious than this. Are the authors of WinUAE or VICE aware that KEEP is using their work? I'm not sure so I might be somewhat out of line here, but it really angers me when open-source projects are used like this, just because these are free. Free as in free beer in this case.

3) They are fully aware it's illegal.
At least they worked that out. Even went as far as to specifically point out legal status of bypassing copy protection in Germany and France, and the fact that there are no exceptions, whatsoever, for individuals doing research. I'm willing to bet there were games for Amiga and C64 that were copy-protected, and that authors of WinUAE and VICE had to study those in order to properly emulate. Now, if you can play a game in emulator that was stored on a floppy or a tape in a way that prevented copying, then surely this system has now been bypassed. So, it's illegal. And yet... no-one cares if it's KEEP? I mean, if I was to throw a conference like this and show Makaron working, then I'd most likely have lawyers (and possibly police) knocking on my door the next morning.

Please spread this little rant of mine. I'd like to have these KEEP people made aware of my own existence, and all the other people that were doing emulation and preservation long before they even had the idea. And we are all individuals working for free in our spare time, to beat the clock. Because some of the systems are becoming dust as we speak and soon there won't be anything left to preserve. Whether it's dumping or emulation, we often have to bypass original protection systems to get results, which is illegal. As is storing software you're not licensed to have, even if it's the last copy on Earth. We have the skills and knowledge to do these things and must keep a low-profile, whereas organizations like KEEP do very little to actually change the law but have no problems boasting success when using existing emulators.

If you think I'm overreacting then just answer yourself this little question: What would KEEP be able to show during these meetings if they did not use free emulators? That's right, nothing. Not even that UI since there would be no point to it without actual emulation. And systems and games that were researched and dumped long before they started. I really love the idea of raising awareness on horrible laws we have and all, but to me it feels like KEEP is trying to grab the money while reinventing the wheel. Rather than try and help/fund/coordinate existing projects, they take what is free and claim to have done all the dirty work. As if.

Don't fail me again

2011-12-27T11:21:14Z

Hi guys. And gals. Do we have any gals here?

I promised some people that there will be a Christmas release - but as you can see, it didn't happen. There are a few reasons for that but the last nail in the coffin was last Wednesday, when I slipped on ice and fell in a very unlucky way, with both of my hands still in my pockets. I hurt my left arm and probably cracked a rib or two since I still feel pain in my chest and back.

Somehow it seems that 2011 was rather unlucky when it comes to Makaron, I had very little free time this year, especially during the second half of it. I think next year should be a little less busy (though probably not until February, or even March).

I'm feeling a bit better now, can move around almost freely, so I will try to hack together something just to release it before midnight of 31 December. It won't be the T13 version though, I've made it DX11-only and there are still some issues I haven't worked out yet so it just won't be ready.

As always, thanks for your support.

By the way, some people have been asking about my custom JVS I/O and GD emulator. Both projects are now paused, but I haven't abandoned them or anything. I would still very much like to sell both devices to raise some money - software and hardware I need are pretty expensive and I'm stingy. There are 3 main problems with those right now:

1) Neither device is complete (though both are working well enough).
2) I'm still uncertain what features should be included or omitted.
3) How do I go about setting up a shop when I'm done :)

Please provide some input. I need to explain that I'm favouring simple and cheap devices - while even a more advanced ones would be cheap if mass-produced, I really doubt there is a market big enough for that. So, for example, my JVS I/O lacks the ability to daisy-chain it with other I/O boards. It's not really that much of a problem to add more functionality but it's another MCU port tied up, a couple more elements are needed, one more USB socket, more PCB space, etc. It seems like a small thing but the costs do add up. The original idea was to have just enough inputs for one player. A bigger MCU, like ATmega128 could easily support two players with analog inputs as well but it's again more costly. So, what do you guys think, a super-simple I/O or just a cheaper version of a standard two-player one? Because, again, I don't think there is market for both.
Same thing with the GD emulator. It runs off cheap SD cards and has no fancy user interface or anything. I don't really see it as a problem, someone could just code a GUI to run on Dreamcast and boot that as the default image. The device will support disc swapping properly you know. I already said that before but this approach means I can only do about 2MB/s when transferring data, which is just tad bit faster than the original GD drive. There are some games that will break when you transfer the data too slow, or too fast, so I think it's actually best to stay close to original specs. If I were to go for a faster solution the whole thing would need to be re-designed - which is possible but I'd rather not have to do it. Then again, if nobody wants to buy the device as it is now then what's the point, right?

So, talk to me. I'm open for new ideas, I just wanted to make sure you understand that adding things is costly and some of this stuff is actually pretty hard to implement because of various limitations of the hardware I want to use.

architecture syn of GDEMU is

2011-08-07T15:40:41Z

And they said imitation diamond wasn't good enough :)

So, does it work? Sure. It's a very early version though. I've hit some metastability problems so I switched from external ARM7 MCU to NIOS2 running on FPGA to speed up debugging. After I reversed my main clock polarity (yeah, Star Trek style) and it worked better I finally realised that I'm running a synchronous system with asynchronous inputs. Which is basically the same as having two different clock domains since I have no control over setup/hold times... So I've added two-stage synchronizers to /DIOR and /DIOW but that's additional latency and Dreamcast has a bad habit of deasserting /CS signals shortly after rising /DIOx. These things always work so well on paper :)

In the end the hardware side of GD interface is pretty small, should fit in EP2C5 (that's Cyclone II FPGA with 5k logic elements) and that's pretty cheap. The downside is I only have so much internal RAM so the main data buffer is just 8kB. While external SRAM could help here, I'm not yet sure it's worth the trouble. We'll see.

Digital audio is completly not supported yet (but is part of the design, so it will be added eventually) and I just wanted to test it out ASAP so I went with slow, PIO-only SD card access and very inefficient CPU buffering. Also, external MCU needs to be connected to FPGA with some sort of data bus and this becomes a bottleneck for the transfers, as it turns out. For example my ARM7 doesn't have a dedicated external memory interface so I have to do everything myself using a PIO port. With only 30 pins (minus a few for SPI and clock output) all I could manage was 8-bit shared address/data bus. Not very fast, unfortunately.

Because of the slow transfers games exhibit various issues, like missing textures, slowdowns, stuttering sound. This will get better as the project matures. In fact, with proper buffering I'm sure I can get it working as well as original GD drive and perhaps even faster - up to some 2x, which is the limit of what one can do with SD cards in SPI mode. Well, there's always the USB route I suppose.

By the way - I get simply tons of spam in the comments now. I've enabled LJ CAPTCHA but that only cut it in half or so. Worst of all, the spam looks (at the first sight) as proper comments, pretty nice English, capital letters, periods. I might accidentaly delete some actual comments while cleaning so keep that in mind when posting here. And if the situation gets even worse I'll probably disallow anonymous comments completly... though that's the last resort.

EDIT: Okay, a small explanation on what this does.

I started this project long time ago but lost interest after hitting some walls. Recently I had a few good ideas and decided to work on it a bit.

What you see on the photo is Dreamcast with it's cover off and the GD drive assembly removed. I cut some holes and soldered wires directly to the mainboard to avoid messing with the original connector. This way I can always plug the drive back in and use it as before - or even better, I can use FPGA as logic analyzer to watch the traffic.

In this configuration there is no real drive and FPGA runs a soft-core CPU that emulates it. Obviously there's some glue logic in there as well or I wouldn't need an FPGA in the first place :) Data is being pulled from SD card - you can see it inserted just over the flat cables. With this I can run any dumped game, and unlike CD rips I actually emulate a GD media so the Dreamcast can't tell the difference. The USB is used to program the FPGA and I can't disconnect it because I don't have a license for that NIOS2 soft-CPU core, it will stop working after the PC uplink is lost. Other than that I can't use it for data transfers unfortunately.

The idea is to have a much smaller (and cheaper) FPGA here with fast external MCU. Data would be stored on SD media or pulled via USB 2.0 uplink to PC.

So far I've tested a couple of games for EU region, and a few JP ones after I hacked them to be multi-region :) I do have Japanese Dreamcast mainboard (well, two actually) but this is the only one I have modified for the project. Once this goes out of prototype phase I'd like to find a matching connector and just make it a plug-in replacement for the GD drive.

So... Skies of Arcadia works, at least EU version. Hacked US one shows no picture but I can hear it running. It works on Makaron but I'm starting to wonder if there is a problem with this particular dump... Anyway, the GDEMU is good enough to not freeze the intro sequence like the ripped version does. There is about 3 seconds of video/audio desync at the end of the intro due to the transfer speed being a bit too low. Same things happens in Dead or Alive 2 intro, which is also pretty long. But other than that it seems to work. Street Fighter Alpha 3 has some sound issues in the attract mode but not during actual fights. This is all after a few improvements I made today, I still need to run SPI link closer to 25MHz to get better transfers from SD card. Buf for many games, like Soul Reaver 2 this is already enough to play without any problems. MPEG movies work OK as well.

DX11 status report

2011-06-09T10:37:13Z

So, there is this guy who keeps asking about DX11 renderer progress and I figured I might as well explain it with some pictures :)

Direct3D 11 is very different to D3D9, and this is the source of my problems. While I want the functionality it offers, I need to properly rewrite every part of the code. In fact the change is so big I created a separate project that is based on NAOMI emulator code but all it does is render one frame of previously captured data. Makes my life so much easier.

D3D11 is a functional extension to version 10 interface except you work on context rather than the device itself, which is only used for creating new resources. This lets you create command lists for things like deferred rendering, though I stil use immediate mode. Unlike in 10, the reference counters are incremented by various Get* methods and this is more like it was in D3D9, so the behaviour was clearly reverted. And for the better I suppose - makes much more sense this way.

The main problem is the grouping of various settings that I need to change individually, this was much easier on D3D9. While the new approach certainly speeds up things for games, I cannot make any assumptions about the 3D data I'm processing and I can't sort it nicely either, so I have to prepare many object with all possible settings well in advance and select the right one when the time comes. Doesn't seem that complicated, eh? Well, it's actually is much more bothersome than it sounds.

Anyway, I decided to start with the basics and then, once it all more or less works, try to add the layer peeling system for per-pixel depth sorting. There will be another major obstacle on the way, namely the fact that I can have different blending operations mixed in together. So I will need to store the blending instruction data along with the pixels and have pixel shader combine the fragments properly. Which is not going to be easy or super fast :) Still, no point in worrying about that now. It's either that or full software rendering if you want proper PVR2 emulation.

Here are some sample screenshots I took along the way, so that you could see there is progress. Note that I already ported and tested the text display class - though with just one frame being rendered the log is much better target for debug output.

These two are different becase I had to change the capture data format a few times:

Anyone wants to guess what game is this? Should be pretty easy by now :)

EDIT: We have a winner! It's Zero Gunner 2.

Uncensored chip porn

2011-06-05T18:21:35Z

Just some stuff I got today. Should provide plenty of fun - I got me 4 Sierra game packs some time ago, plus I own some Lucas Arts adventures as well :)

The only trouble now is my old SB128 (that I need for MIDI port) is no longer supported in 64-bit Windows. So I'm waiting for USB to MIDI converter to arrive. Not to mention half a dozen mono jack plugs... I just hope I can make ScummVM and DOSBox recognize and use the device, if not I'm going to ressurect an old DOS machine I have around here, somewhere.

Roland CM-300 (though modern wavetable based General MIDI emulation ain't half bad either):

Roland CM-64 (current MT-32 emulation is rather lacking, can't beat the real hardware):

Maybe I'm getting old but I'm having more fun with older games, that I have already finished at least once, than with the new ones. Oh, there are notable exceptions but this is the general trend these days..

Software design 101

2011-06-04T20:33:21Z

PSN shop is back - as broken as it was before. You'd think SONY took the time and fixed the interface along with the security issues, but no, apparently it's the best we're gonna get.

The annoying background adverts are back. Those are way too bright and make the white text nearly unreadable. Seriously guys, how difficult it is to QA these things? You could've made the windows more opaque or have the font outlined - these are the first two things that come to mind. Or better yet, have the colors/layout customisable with user settings or skins... So many options here.

Or, we could just do away with these silly adverts/wallpapers. Haha, yeah, that would be a small miracle right there if it happened.

So, there is this No More Heroes demo that I wanted to try. It's not under N though. Or H. In fact, it's not in the Demos section at all, it's in the Latest -> Demos. The most logical explanation is these entries are not links but entered and maintained separately - and that has consequences, more on this below.

The Secret of Monkey Island is not under M, it's under S. Okay, that makes perfect sense if you follow proper conventions, since it'd be Secret of Monkey Island, The then. But Monkey Island 2 is under S as well, for whatever reason... Maybe because it's Special Edition. God only knows. The allmighty might also know where the hell Tales of Monkey Island are because I just couldn't find them. At all. Telltale Games should totaly sue, they are loosing sales because of this blunder. Fortunately the search option will find what you want - but you can't navigate back to the point of origin so it remains a mystery where PSN guys put those.

The in-shop search function is a blessing but really, who uses that on daily basis? Not to mention it's far from perfect, as the rest of the shop interface, and with all the heavy load lately it's prone to time-outs. It'll just wait forever trying to fetch the results and will not work after that. What's more, if you wait too long you will get booted out of the PSN shop alltogether - and not while doing the search but later when navigating.

There are tons of other things wrong with the shop, and item naming is the most obvious and annoying. So what is, say, Epilogue and why it's under P? Well, that 'cause it's Prince of Persia add-on. The World Of Recycled Vessel is also under N because it belongs to Nier. Trust me, this list goes on. It's not just DLCs, HOARD game is under L for example. Sackboy's Prehistoric Moves is under L too, since it's Little Big Planet offspin I suppose - even if the title doesn't even contain the letter L at all.

And don't get me started on ALL CAPS names or Extremly Long Game Titles: First And Only Part There Was Or Ever Will Be But Just In Case We Made This Subtitle In Advance. Or weird version numbers in demo titles. Or how some games have the demo, full game and addons nicely grouped while others require deep knowledge of the PSN store layout. Monkey Island remake is one such game, you'd best remember there is a bundle now that has both of the games, and cheaper too. Too bad it's not pointed out anywhere in the descriptions of the individual downloads. Frankly those are never, ever updated once put in the store anyway.

Ah, about descriptions. Those mostly contain the legal mambo jumbo, rarely anything else or of use. Definitely not the download size, to get to know that you need to purchase first. I have 250G model and fast internet connection but I suppose not all PSN users do. Some descriptions will list supported resolutions and HDD requirements for saves, but that's it.

Oh, and the non-linked entries I mentioned? It's very easy to forget one when updating something, and so we have Worms that, depending on which section you're browsing, have the PSN+ price drop or not. I couldn't be bothered to look for more examples. Point is, these guys are getting paid for it. Maybe it's not enough, I don't know, I sure could do a better job in my spare time though.

The download history is pretty useful, or rather faster than navigating the darn shop. But it's just one big list, can't be searched either. Most importantly you can only redownload items on one by one basis, and if you happened to delete your 40+ free Ridge Racer 7 addons, you're going to weep. I sure did.

Well, now that I got that out of my system I feel much better. I needed that after the Might & Magic: Clash of Heroes demo got me this close to punching my monitor. It should be called Might & Magic: Now Loading - I'm willing to bet the Nintendo DS version works faster... And DS has what, 4MB of RAM?

Qui non proficit, deficit, SONY

2011-05-18T20:58:19Z

So... SONY manages to have their PSN data stolen. And not just any data - the most sensitive kind, their customers' personal data. SONY calls that a high-profile malicious attack but in reality their system had gaping security holes. Known vulnerabilities were not patched for months. Very proffesional, let me guess, it was all outsourced to cut the costs?

How bad is it? So bad that they don't even know WHAT was taken since the logs were compromised as well. Because SONY not only failed to run a proper server stack, they failed to setup a logging system that would run independently. Apparently if it wasn't for the attackers clumsy attempts to cover their tracks, nobody would even know the intrusion took place!

To add insult to injury I was away from country as it got reported and could not block my card or else I'd end up with no money to pay bills and return home...

Words fail to properly convey my feelings about this, I'll try though. Let's start with PSP - it got hacked, with a spoofed "service battery" of all things. Way to include a backdoor into your secure product, guys. Just brilliant. PS3 - got hacked with a stupid buffer overflow in USB layer, again with the help of a backdoor system known as "service JIG". Then it got owned completly because SONY decided to use curve-based crypto rather than the usual RSA-like system, and nobody understood the new algorithms well enough to actually use them properly. I was laughing so hard when this was published... And what does SONY do? Blame the hackers.

Next thing SONY fails at is running a secure database with all of their clients data. And not just PSN, apparently the SOE one was raided too. You couldn't do worse if you tried. Hell, SONY might have as well just copied and send out the data themselves - at least that way we wouldn't have to endure a lenghty downtime of their services. But wait, that's not all of it yet. Now they got their collective asses moving and decided to actually upgrade the Apache to the latest version - and maybe (though doubtful) to keep doing that from now on. And with the claim that it's secure now they made it into grand reopening. Some lesser CEO even apologized. Wow. It would be much more impressive if not for another exploit that was discovered, what, less then 24 hours after the restart? This seems to be the extent of their new security - they just can't code shit, even if their lives depended on it.

Also, as it turns out, Japanese government is not happy with the way SONY handled the situation and forbid PSN reopening until all the issues are properly resolved, and explanations are made as to what steps were undertaken to prevent this mess from happening again in future. Now, this might just be a political struggle but it does kinda make you wonder just how secure this "new" system really is. It sure bothers me.

We are talking about 100 million accounts, this is THE BIGGEST data leak in history. Ever. SONY fails time and time again, and there seem to be no consequences whatsoever. I'm really unhappy with EU and USA authorities not demanding formal apology and proper compensation. I suppose the unnamed hackers will again take all the blame, since apparently big companies can do no evil. Nobody died, right? Move along folks, nothing to see here.

Oh and don't get me started on the Welcome Back Programme - it's just as insulting as the half-hearted apology. Rather than get a wallet bonus or something to the effect, we are given old games that many already have. And this is no accident, SONY knows well what games we have and play (via PSN reports) so this way they just pretend to give us something without actually investing too much money. Think of it this way, you just exchanged all of your personal data, including possibly credit card info, for 2 lousy games and 30 days of PSN+. Was it worth it?

I'm pretty sure kids won't mind. When you're 16 or so, what do you care who knows what your name and date of birth is. But proper adults should realize the potential consequences of identity theft. I've seen it happen to someone who had ID card stolen and let me tell you - it ain't pretty. First of all it won't hit you right away, usually takes months, but then it will haunt you for many months to come. And there is nothing you can do about it, except work hard to clear your name. In the meantime it's quite possible you'll be visited by debt collectors asking for money you never had, and even receive threats from people who think you owe them. Banks might lock your accounts every now and then, your cards will stop working due to all the stuff happening, and so on. And the worst thing is you know none of this is your fault. Welcome to your personal hell.

I won't ask anybody to boycott SONY. You do what you think is right in that situation. I will probably keep using my PS3 since the milk is spilt and nothing will change that. I sure won't enter my card detais into the console again, though, and if there is anything interesting on PSN then I will look for alternative payment methods. That's assuming I will feel like buying from SONY anytime soon. I'm simply not going to trust that company with anything, ever again. Because, let me repeat that one more time, not only they can't protect my data, they don't even feel like compensating me for my loss. So, I don't feel like spending my money on their products.

EDIT: And this just in, though still not properly confirmed: It seems that firmware 3.61 causes some of the consoles to overheat and turn off while playing. Seriously SONY, I couldn't come up with this stuff if I tried.

Ups and downs

2011-05-02T14:05:19Z

I was away for a few weeks, out of the country, and now I'm back again. Couple of plane flights, had to get up early way too often :)

Sorry for the recent lack of updates - I'm really busy lately with various things, both life and work related. I'm still not out of the woods yet so can't promise anything.

Having fun with my ARM

2011-03-11T21:21:00Z

I have ARM7 dev board (AT91SAM7S256 MCU from Atmel) that I've used in a few projects, though none of which was ever completed. It's a kind of testbed, for developing code that will be eventually run on a different CPU. One of the things I used this MCU for was SPI-based SD card support, for simple FAT16/32 library. And that in turn was a part of GD emulator project. Once I was happy with the code I moved all the software to Nios2 soft-CPU running on FPGA.

Recently though I decided to pursue the original idea, that is interfacing FPGA with a separate MCU. In this case the FPGA would only handle ATA/ATAPI stack and I2S protocol, plus data buffering through external SRAM chip. This could probably be done with a CPLD device but I already have an FPGA and the entry-level devices are cheap enough should I decide to stick with Cyclone II series. This way I can have logic analyzer (for debugging) in the project as well. So, I fetched the ARM box from the shelf, unpacked the kit, connected the power supply, and realized I can't do anything with it :)

As you know I'm using Windows 7 Pro x64 now, had it for a few months already. I quite like it, and the only thing that irks me is the lack of low-level parallel port support. I have software written for Windows 9x that requires direct I/O access, something that was quickly fixed with giveio.sys in 2000/XP but will no longer work on 64-bit OS. While I do realize that every piece of hardware and software will eventually become obsolete, and direct I/O access belongs to DOS era, I really miss that functionality. There are some solutions available but none of them is compatible with giveio. So I ended up with non-functional chip programmer (old piece of junk but it works and has some unique features that I use) and it also turned out that my Wiggler-clone JTAG adapter is now unusable as well.

Without JTAG adapter I can't update the ARM7 program. Sure, there is the SAMBA boot manager, but I don't like it. Running the code from RAM is faster but there is 128k of FLASH on this MCU and 64k of RAM, and I need as much RAM as I can get for buffers and structures involved in accessing filesystem on the SD card. Also, since I run the part at 48MHz, I need to add one wait cycle for FLASH memory access (it can go full-speed only up to 30MHz), so I want to be running from FLASH to have accurate representation of how fast the final code will be. Why not downclock the MCU to 30MHz? Well, internal SRAM can still work full-speed and this is important for SPI DMA, and I'm also planning on having USB uplink with a PC so I need clock rate to be 48MHz. I can clock the SPI bus at 24MHz (the limit for SD cards is 25), which is important too.

Anyway, turns out there are some cheap DIY USB JTAG adapters out there, based on FT2232 chips. I gotta say, FTDI did a good job there, as there are other USB serial port chips but surely theirs are the most popular. Also, cheap and easy to get too. This time though I was too lazy to solder everything myself, I figured I'd buy a clone of a well-known JTAG adapter and have it delivered. While putting together your own tools and hardware is fun, I really wanted to have a go at that ARM and quite frankly I wouldn't be able to make my own adpter cheaper than some 25 Euro - which is how much I paid. It's Amontec JTAGkey clone, does the voltage level shifting too though the range is a bit more limited, but 2.7-5V is enough for my needs.

First problem was the FT2232 in the adapter sports different than default PID, so FTDI drivers will not recognize it. Amontec haven't prepared a 64-bit driver it seems and also I didn't want to use theirs anyway, who knows how old it is and what bugs it has. Instead I modified INF files in newest FTDI driver package and added all the necessary entries to it, along with correct product names. Obviously this will not work well with the signature inside the driver binaries so Windows complained - but all I had to do was confirm and it installed properly. I've heard about some lengthy procedures, involving switching 64-bit Windows to some test mode and what not, to install unsigned drivers, but I didn't need that. With this approach I get to use one driver for all connected FTDI devices, I can always update it, and most importantly - it works perfectly.

Second problem was OpenOCD. While free to use - something I really like in my software - it's no longer possible find a precompiled binary for FTDI drivers. It seems that sometime in 2009 they figured that linking with 3rd party closed source library, even if dynamic, violates their GPLv2 license. Picture me doing a facepalm here. There are GPL replacements for USB libs but - figures - these don't work well with 64-bit Windows OSes. I know that license stuff is important, but why shoot yourself in the foot like that... One could argue it's Microsoft fault for requiring the drivers to be signed and disliking open-source software in general. That's just one side of the coin, though. The signing process is not exactly extremly difficult or expensive, and the authors could have just made an exception for the FTDI libs. It's their code after all. To me this looks more like a case of open-source people not liking Microsoft :)
Well, I can always compile the damn thing myself. My personal build can be linked to FTDI libs without violating GPL. It's just... the sources are missing Makefiles and the configuration script needs to be run in POSIX-like environment. Picture me doing another facepalm here. I'm kinda alergic to Cygwin and the various bits and pieces of informations I found suggested that OOCD can be natively built on Windows, so I got creative. I installed MinGW64 - well, unpacked really. Once you have several compilers in the system, each targeting a different CPU, you need to be careful of what ends up in the PATH variable :) For ARM development I use devkitPro, which is also the first choice for Nintendo DS homebrew as it comes with nice system libs, and it sport MSYS environment. I married that MSYS with MinGW paths and finally got OpenOCD to configure and compile itself. And hey, it works too :) I only wish this method was at least mentioned somewhere in the docs, because README and INSTALL are pretty much targeted for *nix platforms only - again, maybe it's just me, but it sure looks like someone doesn't like Microsoft.

Third problem was my ancient OOCD configuration scripts didn't work with version 0.4.0 and I had to rewrite that stuff. I strongly recommend you read the docs here, it's much easier once you realize you can reuse the ready-made scripts and example files. At first I copy-pasted some of the stuff I googled but ended up with a pretty big script I didn't understand much. Then I found the TCL folder in OOCD sources and most of the configuration data I needed was already there :) Long story short, my script is now working and I can program the ARM7 FLASH memory with my code.

The whole purpose of this endavour was to try out a few ideas I had, one of them being running MCU in Thumb mode rather than ARM all the time, which should improve code execution speed. Another one was to try multiple block reads. Single reads are easier but each time you issue a request the card will waste about 1-2ms preparing itself for the transfer. Multi-read obviously works only for continuous memory areas but there is next to no delay between each sector, so unless the files are very fragmented this will result in a significant speedup. There is a small issue of having to properly stop the multi-read sequence if the the address you want is not the next one (and it will happen every now and then since FAT structures need to be read and processed as well) but I got that worked out. The end result for a sequence of 1000 sectors:

- single block reads: 1100kB/s
- multible block read: 2500kB/s

Effective file read speed (12MB file on FAT16) was well over 1800kB/s. With these figures I am finally back on target for x4-x12 CD read spead that real GD-ROM can do.

SONY Wars: The Empire Strikes Back

2011-02-25T21:57:20Z

One Graf Chokolo had his house raided by police, and now SONY is apparently seeking to sue him for 1 million Euros.

What exactly do they want from Graf is unclear - at least to me. I'm no lawyer and English is not my primary language so the legal speak confuses me somewhat. The way I understand the issue at hand is that SONY has yet to establish any proof of guilt, and the quoted 1 million is just their estimate of the damages incurred. I think it's been worded like that on purpose, to discourage anyone from helping Graf in any way. Whether or not he will have to pay anything and how much is yet to be seen.

I don't think it's smart of him to continue releasing stuff right now. Sure, most folks see this as the ultimate fuck you directed at SONY, but that's not the real issue here. Problem is, the guy has apparently been court-ordered not to do anything of the sort until the matter is resolved properly. Going against court order is just plain stupid because it's punishable and SONY could even drop the case now and Graf would still have to face charges.

Obviously SONY has the upper hand here. They can hire lawyers and ask for court orders - and this will, in theory at least, shut people up. If the lawyers somehow find a way to sue hackers and win, it's good for SONY. If not, they'll just say "Oh, sorry mate, carry on." - and get away with that. You see, their tactics might simply be to stall, it would take years to clear things up and by that time PS3 will become obsolete. So it's a win for SONY as well, said hackers are unlikely to have the resources to properly fight back and, as I just said, going against a court order is a bit suicidal.

So, we determined that the court order might be just a gag and it's issue unjustified. Don't hate the judge though, he is just doing his job. Sure, these people have their own views as well, and that does affect their actions (even if it's not supposed to), but they are mostly bound by laws. It does take a good knowledge of the law to make a compelling argument and SONY can afford to hire entire teams. A "hacker" will seek the cheapest legal council :) Though, I must say that Geohot has a good lawyer on his side and can sleep better, but SONY is not yet done with him.

Anyway, I think it's now a bit more clear why I decided to stay off SONY's turf and not, for example, release VMU emulator for PS3 system. Call me a chicken, but do understand that it's not just about SONY and their precious "next-gen" console. I could as easily get sued by SEGA. While reverse engineering of owned hardware is legal, it's pretty much impossible without breaking some software anti-piracy laws. I wrote about that in detail some time ago so just to remind you - it's like with DVDs. You can make a backup copy of the media, in case it gets damaged or something. But to make a copy you have to overcome the copy-protection systems and that's illegal. So, in the end, your right to make a lawful copy is just plain dead.

By the way, I had a good laugh at people who got angry with SONY latest PSN licence/TOS. The wording isn't all that different now, you know, most of the nasty stuff has been there since day one. And you take offence now? Well, I suppose now you actually read the terms. Or at least it has been pointed out to you. Maybe this will improve the awerness in general populace that this stuff is legally binding agreement. Don't bitch about your rights being violated, you accepted this willingly. All your base are belong to us, sucka, mwahahaha.

Electrons, and their use

2011-02-15T15:57:34Z

Since you asked so nicely I'm going to post some bits and pieces about M1. But first some necessary background.

There are several approaches to figuring out encryption / compression schemes, all involve deep analysis of the data before and after the decoding step.

Unless the system is very simple you won't get far by just looking at the (usually very few) patterns available. In fact it's pretty easy to come up with some hypothesises that will turn out to be false later and will derail the whole research, taking up days to weeks to disprove. This is what accounts for most of the time, and all the failures tend to be quite discouraging. This is the pure analytical approach. Andreas Naive does statistical analysis, it's easier in the sense that it gives you some hints and directions of what to try. However, to even attempt that you need to have the knowledge, tools and experience - things that I lack :) You also need to have plenty of test data if the results are to have any meaning.

We got a bit lucky with M2/M3 protection. I still have no idea how Andreas managed to identify the encryption so fast and reverse it (though if I was to hazard a guess then I suppose the work he did on Atomiswave carts helped a bit - it's not the same system but bears resemblance). I explained what I know about NAOMI to him and then we had some brainstorming (which was mostly him shooting down my silly suggestions), and eventually came up with some ideas on how to generate the test patterns he needed. I modified my dumper tools to output the data.

I should explain that M3 turned out to be just a variant of M2 that uses cart's own RAM buffer. This is also the major weakness of this system that we exploited. See, it's easy to say "generate test data", but how exactly do you present your own patterns to the decoder if all it can do is access read-only memory? Well, one way would be to inject the pattern into EPROM chip that serves as game program carrier. It's possible but you're limited by available space (up to some 4MBs), while erasing and reprogramming takes a lot of time. Not to mention EPROMs have rather limited number of erase cycles they can go through (about 100) before errors start to show up. Obviously you'll also need all the necessary equipment to do the job in the first place :)
Another problem was that M2/M3 is actually position-dependant. In other words, the same value at address A and A+n will decode to different result. Data is decrypted in 16-bit words, so to fully test just one address location I'd have to reprogram the EPROM over 65 thousand times. Not really my idea of fun.

But, as I said, there is a RAM buffer on the cart. We figured it will help us crack M3 and then we'll worry about M2. Having the ability to actually upload data to the cart and read back the result reduced the hardware testing times to mere minutes. Obviously Andreas needed very specific patterns but that was just a few changes in the program code on my side. This also allowed the tests to be run on pretty much every cart out there (not just those in my possesion) without resorting to any hardware modifications. It's thanks to this feature that we were able to crack the encryption keys on most games so quickly.

This was M2/M3. M1 is completly different system. The complexity of the encryption is nowhere near that of M2/M3 so it's much weaker from cryptographic point of view - but it took much more work and many more months of head scratching to figure it out. Kinda funny, now that I think about it, that I identified and named the methods used on NAOMI carts as M1 through M3, and it was M3 to go down first and M1 last. Well, we have M4 now too :)

Anyway, M1 didn't work the same way M2/M3 did. I had only one cart, with only one encrypted sequence on it, so not exactly a lot of data to work on. Trying to "decrypt" other ROM areas only produced confusing results. What's worse, trying to decrypt data from the EPROM area gave completly different and incoherent results. The only way to inject custom patterns into decoder was to spoof a ROM chip.
Soon it became obvious why EPROM approach doesn't work - on this cart type ROMs are paired and read two at once, to produce 2*16 = 32 bits of data for each address location. EPROM output is only 16 bits wide - and while cart compensates for that during normal data transfers, decryption engine just treats the other 16 bits as pulled up. Thus completly changing any patterns we'd want to test. I had to create a device that would make up for the lack of any writable memory on the cart. That's where Altera Cyclone II FPGA comes in again :)

Just to sum this up, I needed 32 bits for data, tristatable and preferably bidirectional for logic analyzer. Add at least 16 bits of address to that, and some additional control signals (chip selects mostly). Tons of soldering. To add insult to injury the ROMs are 5V and FPGA inputs can handle 3V3 tops (some people abuse the protection resistors and claim 5V tolerance but it's a bit too pricey chip for me to fry just because I'm lazy). I had to add additional overvoltage protection circuitry, that would still allow me to have the bus bidirectional. The final result was this wire mess:

It allowed me to to test a few things and establish a working theory or two. I figured that the data is clearly XORed with some unknown pattern, to obfuscate what looked like a dictionary-based compression system. It's not like I didn't do any testing at all without the FPGA, but only now I could clearly verify my hypothesises. My initial guess was that the XOR mask is rather short, up to 8 bytes, perhaps different length on every cart using M1. I also suspected that some of that pattern actually "leaks out" to the ouput when you try to decode streams full of zeroes or ones. That later turned out be true but quite frankly it's not predictable enough to make use of this weakness. I started collecting M1 data from people who dumped carts with my tools and had some more stuff to look at. I also got GG2K cart from Roberto Malone, which he bought specifically to aid M1 cracking process. At some point I ecountered an output sequence that did not look cyclic - this scared me somewhat, for it was now possible that the XOR mask is generated on-the-fly by decoder with some pseudo-random sequencer. Now this would be a real show-stopper. I guess I need to thank SEGA for not doing that :)

Still, figuring out the mask sitting on top of compression was a very tedious task. I didn't know much about the compression itself at the time, and fighting on two fronts is never easy. Also, after M2/M3 success I was trying to think strategically - if I could simplify the discovery method than I could use other people and their carts to aid me in my research. So far I'm the only one who actively worked on NAOMI anti-piracy protection methods. Well, perhaps some other guys tried too, but kept it secret since their objectives were more... money related let's call it :) In the end I had a bright idea that worked out pretty nicely. Take a look at this photo:

What you see here is Actel FPGA (not to be confused with the one I was using) that does everything on the cart: addressing, reading and buffering, DMA and also M1 decryption. All that inside one, small black box. But it does have a weakness, namely it lacks necessary amount of memory for all that functionality. So, there are two smaller chips below it: On the left there's a 9-bit asynchronous FIFO queue, 256 entries long, that serves as ouput buffer for DMA mode. 9th bit is unused, and rather than have two of those chips (external cart data bus is 16-bit wide) it stores words as byte pairs. It's fast enough to do that, it's latency is less than 20ns. On the right there is a SRAM chip. It's pretty fast too, just 15ns, and has 32k x 8 capacity, though closer inspection will reveal that only A0-A6 address lines are used. That limits it to 128 usable cells, and once you know that the compression dictionary can have only up to 111 entries... rings any bells yet?

The bright idea was to tap into the address lines of the SRAM chip to monitor dictionary usage. This would greatly help with the compression part so that I could focus more on XOR mask. Since it's just 7 address lines, 8 data ones and two control signals, I decided to wire it up completly. It helps this chip is 3V3 so no need for the overvoltage protection, the transient snubbing resistors on my FGPA board will suffice. Imagine my surprise when I started up logic analyzer project and saw that the data stored in SRAM is already de-XORed... What a find :) It took just few minutes and a piece of paper to figure out the complete XOR mask for Oh! My Goddess cart and learn it's only 4-byte long. A bit more time passed and a prototype decoder was ready and working PC-side.

So I went ahead and soldered some wires to GG2K cart as well. This is pretty clean compared to OMG, wouldn't you say. And some screenshots from SRAM usage monitor for two different input patterns:

This time the XOR mask was also easy to calculate but then the decoder failed to produce expected result. It only got two first bytes (one output word) right. A moment later I discovered that there are two modes of operation, one direct and one based on substraction. OMG uses substraction mode and only dictionary data (and not even all of it), while GG2K uses direct mode and has some data bytes stored in the compression command stream. A few fixes and the code was working in all cases. Mystery solved. Time elapsed... eh, let's say months. Obviously not spent entierly on this problem but I kept poking it and thinking about it all the time.

It's possible that M1 would be broken sooner if I had GG2K instead of OMG to work on - but that is just another theory, one that is never going to be verified :) Fact is I was extremly lucky to get just 3 carts and happen to hit all protection methods (M1 for OMG, M2 for DOA2 and M3 for CVSSNK).

No NAOMI carts were harmed during research or making of this document.

And a small post scriptum, yes, I know that MVSC2 does not work well on T12/7 for some people. I even know why. I did say stick to T12/5 in case of problems. I also confirm that second player input will not work since I forgot to add one line of code. Happens. Stop bugging me about it.

XOR ROX

2011-02-05T01:33:17Z

This is what M1-type NAOMI cart looks like:

And that is what it took to figure out how the FPGA handles the protected data:

Cracking this nut took plenty of time and hard work. Originally I planned on explaining this in great detail, with photos and screenshots, but in the end decided not to. Sadly, most people couldn't care less (as long as they get to play "free" games) and/or consider this black magic that is best left to nerds and otherwise smelly people. So, I'm going to post the source code. Those of you that can parse C should find it somewhat interesting, those that can't are probably reading wrong blog.

/*

This is a concept version of M1 decoder. It's fully functional, except for
clarity it doesn't handle cases where the data stream ends and you want to
keep reading - to add that functionality just move input buffer cursor to next
4-byte boundary and restart decoding (by re-reading dictionary again).

The source includes a brute-force exhaustive cracker (takes about an hour
for a decent CPU). Pattern compare with just 4 bytes might return some very
close but not correct XOR masks so I decided on 6. Due to the simplification
I explained above it's theoreticaly possible for the cracker to fail (if there
is EOS symbol in the first 6 decoded bytes), but that is highly unlikely.

D.

*/

#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <time.h>

#define CRACKER

uint8_t ReadByte ();
void StoreByte (uint8_t b);
void ShiftIn ();
void Decode (uint32_t mask);

  //-------------------------------------------------------------------------
int s_in_len, s_in_pos;
uint8_t* s_input;
uint8_t s_xor [4];

uint8_t s_dict [111];
int s_subst;

int s_out_len, s_out_cnt;
uint8_t* s_output;

int s_shift, s_bits;

  //-------------------------------------------------------------------------
uint8_t ReadByte ()
  {
  uint8_t v;

  switch (s_in_pos & 3)
    {
    case 0:
      v = s_input [s_in_pos + 3];
      v ^= s_input [s_in_pos + 1];
      break;
    case 1:
      v = s_input [s_in_pos + 1];
      v ^= s_input [s_in_pos - 1];
      break;
    case 2:
      v = s_input [s_in_pos - 1];
      break;
    case 3:
      v = s_input [s_in_pos - 3];
      break;
    }
  v ^= s_xor [s_in_pos & 3];
  s_in_pos++;
  return v;
  }

  //-------------------------------------------------------------------------
void StoreByte (uint8_t b)
  {
  if (s_subst && s_out_cnt >= 2)
    b = s_output [s_out_cnt - 2] - b;
  s_output [s_out_cnt] = b;
  s_out_cnt++;
  }

  //-------------------------------------------------------------------------
void ShiftIn ()
  {
  s_shift <<= 8;
  s_shift |= ReadByte ();
  s_bits += 8;
  }

  //-------------------------------------------------------------------------
void Decode (uint32_t mask)
  {
  int i, eos;

  s_xor [0] = (uint8_t)mask;
  s_xor [1] = (uint8_t)(mask >> 8);
  s_xor [2] = (uint8_t)(mask >> 16);
  s_xor [3] = (uint8_t)(mask >> 24);

  // byte dictionary
  s_in_pos = 0;
  for (i = 0; i < 111; i++)
    s_dict [i] = ReadByte ();

  // control bits
  s_subst = (s_dict [0] & 64) ? 1 : 0;

  // command stream
  s_out_cnt = 0, eos = 0;
  s_shift = 0, s_bits = 0;    
  while (!eos && s_in_pos < s_in_len)
    {
    int code, addr, t;

    if (s_bits < 2)
      ShiftIn ();
    code = (s_shift >> (s_bits - 2)) & 3;
    switch (code)
      {
      case 0:
        // 00-aa
        if (s_bits < 4)
          ShiftIn ();
        addr = (s_shift >> (s_bits - 4)) & 3;
        s_bits -= 4;
        if (addr == 0)
          {
          // quotation
          if (s_bits < 8)
            ShiftIn ();
          t = (s_shift >> (s_bits - 8)) & 255;
          s_bits -= 8;
          StoreByte (t);
          break;
          }
        StoreByte (s_dict [addr]);
        break;

      case 1:
        if (s_bits < 5)
          ShiftIn ();
        t = (s_shift >> (s_bits - 3)) & 1;
        if (t == 0)
          {
          // 010-aa
          addr = (s_shift >> (s_bits - 5)) & 3;
          addr += 4;
          s_bits -= 5;
          }
        else
          {
          // 011-aaa
          if (s_bits < 6)
            ShiftIn ();
          addr = (s_shift >> (s_bits - 6)) & 7;
          addr += 8;
          s_bits -= 6;
          }
        StoreByte (s_dict [addr]);
        break;

      case 2:
        if (s_bits < 7)
          ShiftIn ();
        // 10-aaaaa
        addr = (s_shift >> (s_bits - 7)) & 31;
        addr += 16;
        s_bits -= 7;
        StoreByte (s_dict [addr]);
        break;

      case 3:
        if (s_bits < 8)
          ShiftIn ();
        // 11-aaaaaa
        addr = (s_shift >> (s_bits - 8)) & 63;
        addr += 48;
        s_bits -= 8;
        if (addr == 111)
          // end of stream
          eos = 1;
        else
          StoreByte (s_dict [addr]);
        break;
      }
    }
  }
 
  //-------------------------------------------------------------------------
int main (int ia, char *ta [])
  {
  char* name;
  FILE* f;
  uint32_t m, x;
  time_t t1, t2;
  uint8_t pattern [6];
  int t;

  name = (ia < 2) ? (char*)"input.bin" : ta [1]; 
  f = fopen (name, "rb");
  if (f == NULL)
    {
    printf ("File open error: %s\n", name);
    return 1;
    }

  fseek (f, 0, SEEK_END);
  s_in_len = ftell (f);
  if (s_in_len < 112)
    {
    printf ("File too short: %s\n", name);
    return 1;
    }
  s_input = new uint8_t [s_in_len];
  fseek (f, 0, SEEK_SET);
  fread (s_input, 1, s_in_len, f);
  fclose (f);

  s_out_len = (s_in_len - 111) * 2;
  s_output = new uint8_t [s_out_len];

  name = (ia < 3) ? (char*)"pattern.bin" : ta [2]; 
  f = fopen (name, "rb");
  if (f == NULL)
    {
    printf ("File open error: %s\n", name);
    return 1;
    }
  if (fread (pattern, 1, 6, f) < 6)
    {
    printf ("File too short: %s\n", name);
    return 1;
    }
  fclose (f);

  // 840-0030 / AH! MY GODDESS QUIZ GAME: 0xCD9B4896
  // 840-0039 / Giant Gram 2000: 0x7F805C3F
  // 840-0084 / Virtua Tennis 2: 0x2D2D4743
  // 840-0098 / Shootout Pool: 0xA0F37CA7
  // 840-0106 / Virtua Fighter 4 Evolution: 0x1E5BB0CD
  // 840-0128 / Shootout Pool Prize: 0x9DBDE9CD
  // 840-0136 / Shootout Pool Medal: 0x9DBDE9CD
  // 840-0140 / Kick'4'Cash: 0x820857C9
  // 840-0150 / MKG TKOB 2K3 2ND VER1.003-: 0x3892FB3A
  // 841-0007 / Marvel vs. Capcom 2: 0xC18B6E7C

#ifdef CRACKER
  x = 0, time (&t1);
  t = s_in_len;
  s_in_len = 0x88;
  for (m = ~0; m != 0; m--)
#else
  m = 0xCD9B4896;
#endif
    {
    Decode (m);

#ifdef CRACKER
    if (memcmp (s_output, pattern, 6) == 0)
      {
      printf ("XOR mask: 0x%08X\n", m);
      break;
      }

    if (x - m > 1024)
      {
      x = m;
      time (&t2);
      if (t1 != t2)
        {
        t1 = t2;
        printf ("0x%08X...\n", m);
        fflush (stdout);
        }
      }
#endif
    }

#ifdef CRACKER
  s_in_len = t;
  Decode (m);
#endif

  f = fopen ("output.bin", "wb");
  if (f == NULL)
    return 1;
  fwrite (s_output, 1, s_out_cnt, f);
  fclose (f);
  return 0;
  }

Also, NAOMI Test 12/7. I don't have time to work on T13 lately and I'm not going to rush it, so you get this instead. I experimented a lot on this code - it might be actually less stable than the previous version. On the bright side, Power Stone is playable now. Should be, anyway :)

Shaken, not stirred

2011-01-27T22:57:06Z

Might as well follow up on my last entry - geohot got served with TRO just now.

I read the papers he got served and I'm confused. While I fully expected DMCA to be used at some point... how exactly does SONY get to use it in a CIVIL lawsuit? You can do that in US courts? If so, your legal system is completly wacky.

At first I thought SONY was just trying to present a case, and DMCA was only mentioned to express urgency and necessity of the TRO. You know, like by saying "He is a bad guy, Your Honor, he must be stopped before he does more damage". So I can fully understand statements like "breach of contract" and "interference with contractual relations". This is what SONY claims and must prove - obviously it remains to be seen if they can do that.

But the main reason for ordering TRO was mostly DMCA violation. How the hell does that work? Violating DMCA is penal offence/crime. Someone, please, explain it to me how does SONY get to motion anything in a civil case based on that? And more importantly, how does a judge rule in a civil case with penal law? Because this blows my mind...

Judgment day

2011-01-12T23:16:15Z

SONY just sued Geohot and fail0verflow team guys, and I can't help but to comment on that :)

Basically there are two things to consider here, the DMCA (Digital Millenium Copyright Act) and EULA (End User License Agreement). Violating DMCA will get you jailed, though that pretty much applies to USA only. We EU folk are not bound by that - but we have other laws that govern ownership and copyright issues. For example, around here it's illegal to possess any software that you do not have a license for. And I emphasise "possess", so it doesn't matter if you used said software or profited from it. Yup, that means games downloaded for emulators are illegal by the way, and calling them dumps, ROMs, images or whatever doesn't change that. And don't kid yourself thinking that it doesn't apply in your country - even if the company that made the game went bust, the copyright is still maintained for some 50-70 years. Fortunately most game developers figure it's too much hassle and waste of money to slap everyone playing 10 year old software with legal cease & desist notice. But they have that right, and much more.

Now, EULA on the other hand is a legal binding agreement that you accept of your own volition. In other words, you say "I promise to behave and you can beat me up if I don't.". You agree not to reverse engineer any software you get from SONY, and that you will not modify it or otherwise intrefere with it's intended operation. You agree not to use any software or hardware that SONY did not intend to be used with their own software (so it doesn't matter that it was, or was not compiled with SONY SDK). You also acknowledge that SONY will seek to be compensated for any damages you might cause.
What do you mean you didn't agree to all that? Sure you did. When you clicked the "I accept" button under that wall of text you didn't read when you first run your PS3. And you should have, because next time you run a software it's EULA might say "You agree to pay us 1000 Euro within 10 business days from now" and once you click that button you will have to pay. If you don't, a court order will make sure that you comply.

You know, I used to think that Microsoft Windows license was evil but now I know better. There are other, much more restrictive EULAs out there and SONY's PS3 one isn't that bad either. It's a standard legal notice preventing you from causing them harm, one should actually be expecting to see that. What I didn't like was how they treat personal data (name, address, etc) - we have laws in EU to protect those but since PS3 is global product you waive some of the rights you have once you agree to it's EULA. Well, haven't been spammed yet so it's not that bad I suppose.

And since I mentioned Microsoft, their Visual C++ Express license is neat. All they ask is not to mess with the binaries and that the produced executables be targeted for Windows operating systems (would be difficult, though not impossible to change that). And you get a free compiler, and you can even sell the programs you make with it. I only wish they dropped that silly requirement to register for 2010 version...

Anyway - my guess is SONY will try to establish that all the guys they sued have accepted the EULA. It varies a bit depending on where you live but I'm pretty sure each version of it has a clause like this one: "SCE and its licensors reserve the right to bring legal action in the event of a violation of this Agreement." It doesn't say what exactly they will do to you but it gives them plenty of room to maneuver.
SONY will also most likely try to prove that any encryption keys they've used are copyrighted by them - now that would be a bit tricky but doable, especially in US courts :)

The guys might claim they never agreed to PS3's EULA but that might land them in position of possesing illegal sofware. If that isn't a problem then it'd be worth it. AFAIK fail0verflow only published their findings and explained tools/methods involved - and reverse engineering hardware is not a crime, neither is writing your own software. Geohot might be in more trouble, he leaked the key rather than explain how to attack metldr.

A crypto key is just a bunch of numbers, you can't claim ownership of math. On the other hand though it's a pretty particular set of numbers, used in a specific way within a software, so it might be unique enough to copyright it. I guess we will see soon :)

UPDATE:

I figure this would make a great "donate" button :)

Too bad I'm not going to have one, huh :]

Flashing

2011-01-10T14:52:18Z

Name that cart:

That's right, -MAZAN- Flash of the Blade :) Many thanks to Hans from Sweden for lending it, and to Tomas for doing the actual dump. Unfortunately I broke JVS layer in my code so I'm unable, at the moment, to run it in emulator. It boots, sure, but dies with "I/O PCB ERROR" shortly after. So, the best I can show right now is this (don't quote me on the program checksum though, I got a bit creative with it before I made this screenshot):

Wait a minute, I do have a NAOMI2 box, right? A few emails later and some photos & manual exchanged, I've added another profile to my JVS I/O and it now spoofs the custom boards that came with the game. Still no go as there is a sanity check run on the sensors and I couldn's pass that, but at least I got the game test mode working:

Another hour or two spent patching the game program and finally this:

Yup, looks ugly. Sorry, the photo wasn't that great in the first place and you can tell some stuff is missing - like, the sky in background and half of the textures. I either messed up de-protecting the game or it just doesn't like being run from DIMM, much like HOTD2. Well, DIMM-enabled HOTD2 runs perfectly in emulator so I suspect addressing problem that only manifests itself on hardware. Whatever the cause, it's fixable - as soon as I figure it out :)

I've been also asked to do something about GDI format. The main issue is it keeps all the tracks in separate files and there are games with tons of digital audio. It doesn't carry any information about the game title either (especially if it's not an English one) or ring code - sometimes the only thing that allows you to tell GD-ROMs apart. Well, unless you can read the disc contents just by looking at it :)
It's primary purpose is to store Dreamcast-playable discs so it can handle GD-ROMs and MIL CD-ROMs (mostly because CDI format used so far is not exactly free, legally speaking). It's not all-purpose substitute for all optical media there is, thus simple and fast where it counts.

This is probably going to raise a few eyebrows in "What, another format?!" fashion, but don't worry. I'm not dropping support for GDI/CDI and ISO files, just adding one more. Whether or not this new format will become even remotely popular is not really my problem, the people who want to use it will do so. Some folks will also ask "Why not CHD then?" - well, I think CHD is great for MAME. And let's leave it at that.

Here's how a converter looks like (still WIP). Notice that you can store a small image within the file (256x256x32) so that it would be easier to identify the game. By default the image is that of GD texture used by BIOS in CD-player mode. Some games don't have it though, or there might be a better suited cover scan or photo, so it's possible to load an external one. It's completly optional but recommended.