My name is Bond

Disclaimer: This entry is not emulator related. If you're here only for Makaron news, please ignore it.

I found a very interesting paper a few days ago, here's a link: Breakthrough silicon scanning discovers backdoor in military chip

This is just a brief summary of a side attack on Actel/Microsemi ProASIC3 chips. The power analysis idea alone is mighty interesting - for such a big silicon structure as FPGA it seems impossible at first, but then it's not the whole chip that has been put under scrutiny. It's just the part responsible for JTAG and some of the internal addressing structrure, but even then measuring the subtle current fluctuations is not exactly trivial. While simple on paper, this task requires a number of fast and well-calibrated ADCs and most importantly a software capable of processing the collected data. I could probably come up with a good enough electrical component of such testbed but that software part is like magic to me - and I'm no stranger to programming :) Ah well, I suppose it's better for me to accept that certain math problems are way above my head, less stress that way (I guess ignorance IS a bliss after all).
Anyway, ProASIC3 series have been well designed, with countermeasures in place to thwart this type of attacks. The paper mentions fuzzy clock sources and very low leakage transistors, as well as carefull structure design - all these factors contribute greatly to the chip's security. It is, after all, touted the most secure FPGA available on the market today. But there's a twist: apparently the chip has some undocumented features that allow one to read it's configuration and internal memories even after it has been secured with secret key.

You see, when it comes to JTAG it's pretty much normal that most (if not all) of the protocol commands intended for factory testing are kept secret. This is to protect the manufacturer secrets, nothing strange to it. However, it's one thing when the design has, say, additional structures to improve yields and some of them are disabled even if fully operational in order to keep all chips with the same specs. It's a dirty little trick but in the end acceptable. It's quite another story with intentional backdoors placed into the design.

Hidden features like these are handy for the people who make the product, in case there is a problem that needs to be fixed after the production has been completed. It doesn't have to be a hardware thing - remember the first X360 DVD drives? Those could be reprogrammed if you just knew how to do it, sometimes even via the SATA link so there wasn't even any need to open the cover. This was a firmware backdoor and once discovered it was ultimately used enable the drives to accept recordable media as original games. I seriously doubt Microsoft had any knowledge of this prior to discovery by hackers, this was most likely something the drive manufacturer added on it's own to be able to reuse returned/repaired drives. This 'cost saving' in company A caused company B much grief and money loss.

So, the big question: Was it Actel that put this 'feature' in the design, or was this added by the Chinese factory that actually made the chips? Either way this will have far reaching consequences as ProASIC3 series is often used in military equipment. So I wonder if it's just a blunder or a some sort of espionage attempt. Actel does claim that their products are secure and the contents of the chip cannot be easily recovered as there is no way of doing so. Clearly, that is a lie. Obviously the company now says that there isn't any backdoor but you can't prove that something doesn't exist. On the other hand, the researchers will need to verify their claim by showing a succesful attack attempt to the public. So I guess we wait and see what happens. If it turns out to be true Actel's whole CPLD/FPGA branch could be finished.

In other, somewhat related news: I've taken some interest in software defined radios. I have to say that just by researching the subject my understanding of modern signal processing technology has gone up quite a bit. The best SDRs out there use FPGAs for signal processing - ADC interface, sampling rates, decimation, digital filtering, etc. A proper design can cover a big chunk of frequency spectrum while sampling at many Ms/s. The best way to go about it is to use quadrature demodulators to produce I/Q signals. These can be then freely processed PC-side (as long as you have CPU power to do so) to obtain anything from AM radio audio to satellite imagery. Problem is, these toys are costly :) I'm not about to throw 3-4k$ at my 'new hobby' so the only other options are:

1) Self-made QAM mixer/demodulator that uses audio frequency output. This can be plugged into modern PC soundcard and sampling rates of 96kHz or even 192kHz are avaiable, at a very nice 16-bit resolution. Some cards can even go as high as 24 bits, though in reality the SNR is probably not going to exceed some 20-22 bits even in best of conditions. I've actually designed a PCB based on YU1LM DR2A design, which is pretty simple and anyone can make it cheaply since there aren't any hard to come by parts. It needs external VFO at 4x the local oscillator frequency but I can make that too. Worst case scenario I'll buy a cheap DDS kit based on AD9850 or something like that.

Here's what the PCB will look like:
2012-06-05 DR2A PCB

2) For some 20$ you can buy a USB DVB-T dongle with Realtek RTL2832 chip inside. That chip model is important, apparently it can be put into a 'dumb' mode where it samples and passes the I/Q data unmodified to the PC. With a proper tuner attached it will cover 80-1700MHz range, with some holes though, and the SNR depends on the frequency. The best tuner seems to be Elonics E4000 but other ones (FC0012/FC0013 for example) will also work, though with more holes so less frequency coverage. This mode was meant for software processing of FM and DAB/DAB+ radio signals, and it seems to be undocumented feature not present in other designs. RTL2832 can only provide 8-bit data but it makes up for that in the bandwidth of 1Ms/s up to 3.2Ms/s. The best setting is an integer multiple of the on-board resonator, usually 28.8MHz, so that there are no sample drops. So, while far from perfect it's a great bargain for 20 bucks.

And the best thing, even though it's called a radio, is that such setup is pretty much a PC-based spectrum analyzer. These things are quite expensive as standalone hardware so it'll be interesting to see if a self-made project like this can be useful for something other than listening to police radio traffic :)
  • Current Music
    Smooth Jazz
  • Tags

Still cranking

I didn't get into finer details of the low-level CD dump in my previous post because it would get really long and boring - some people however took that as a lack of proper understanding of the complexity of such project on my part. Rest assured I did give it a lot of thought :)

I'm fully aware that a raw bitstream from the optical pickup contains many bit glitches, as EFM is far from perfect. After all this is what we have CIRC for. If the drive logic can deal with those glitches, so can we. The only difference is the drive will correct (where possible) the bad symbols in the stream and produce a correct output, whereas we want to come up with a clean input stream that would not even need correcting. This is more problematic but still doable.

What is a bit glitch? It's exactly as it sounds, a single bit having wrong value (1 instead of 0, or the opposite) somewhere in the stream. It's usually just one bit per byte, and doesn't happen very often, but more serious read errors happen as well from time to time. How often is "not very" you might wonder. Well, let's analyze Ranma dump I've made.

To keep things a bit less messy I will only use tracks 3-55 of the dump here, all of which are audio. 257060 audio sections to be precise, for a somewhat round number. Since each section is accompanied by exactly one Q sub packet, we have exactly 257060 of those as well. Knowing that this particular subchannel has a CRC we can easily check the integrity of all of them. And it turns out 151 are glitched.

In case you're not clear on why we stick to subchannels and not actual data, it's because subs are not protected by CIRC so a dump will not be repaired on the fly. Or at least should not be, some drives can actually do that for P & Q subs. It's not all that difficult (might be a bit slow though) - knowing that it's usually one bit that goes wrong we just keep flipping all the 96 bits in the Q packet and recalculate the CRC. Eventually we find the right one and have repaired the stream. Now, I do realize that we could end up with a bad repair and CRC collision but that is very unlikely for a single bit change. Also, keep in mind that it might be the CRC area that has the glitched bit :)

Sure enough, out of those 151 bad packets 150 were fixed by flipping one bit. One case was two bits going wrong next to each other, so it could be a fluke, or a case of a "walking bit" problem. These are also easily fixed but require another pass over the data and more complicated algorithm. In the end all packets were repaired though. CIRC is more complicated in principle but also designed for fixing the errors, not just detecting them. This is why we could have a dump that has many glitches but can be used since it auto-repairs itself when processed properly. But, obviously, we want to have clean one - and the question is how do we obtain one.

We could just repair the image after the dump is done, that will probably fix most of the glitches. Some might be tricky enough to warrant a partial redump to make sure, but that could be automated, we could even do two passes in all cases to have more material to work with. Unless the disc is damaged and keeps returning errors for one particular area, the glitches are otherwise pretty random, caused mostly by various harmonic frequencies in the rotation speed introduced by the spindle motor. Pickup positioning and focus is also not perfect and subject to both vibrations and thermal noise in the coil drivers.

Ranma, by the way, is also not exactly an example of good CD mastering. The data LBA to Q sub offset is maybe not as big as on Lady Phantom but it's there. Also, there are mode-2 Q sub packets present on the disc, even if empty, so it makes one wonder why didn't they delete them altogether if Catalog Number was not recorded. I'm mentioning that because mode-2 Q does not carry addressing information except the fraction field.

And, just to put things into perspective here, since you migth think that 151 glitches per (pretty much) a whole disc is not a big deal: This was just the Q subchannel data. There are 2352 bytes in the sector (when dumped raw or as audio section) and another 96 in the subchannels (98 actually but 2 bytes are used internally as a sync field, and not reported). A raw sector plus sub packet, when rejoined together, is 2448 bytes total. The Q subchannel alone is just 12 bytes. Since we got 151 errors by analyzing just 12 bytes out of 2448, then it means the total number of glitches would be about 30 thousands for the whole dump. And that is what I call a problem, and why we don't have a low-level dumper yet.
  • Current Music
    Lotus 3 remixes
  • Tags

Cranking old engines

Lookit what goodies came in the mail today:

There is a reason why I'm showing you Lady Phantom - this is one of the somewhat problematic CD-ROMs for PCE. First, the TOC:

-> Track 01  (Audio, 00:47:65, LBA: 0 / 00:02:00)
-> Track 02  (Mode 1, LBA: 3590 / 00:49:65)
-> Track 03  (Audio, 02:34:63, LBA: 5157 / 01:10:57)
-> Track 04  (Audio, 02:23:59, LBA: 16770 / 03:45:45)
-> Track 05  (Audio, 02:48:09, LBA: 27554 / 06:09:29)
-> Track 06  (Audio, 02:30:08, LBA: 40163 / 08:57:38)
-> Track 07  (Audio, 02:48:02, LBA: 51421 / 11:27:46)
-> Track 08  (Audio, 02:27:37, LBA: 64023 / 14:15:48)
-> Track 09  (Audio, 00:33:26, LBA: 75085 / 16:43:10)
-> Track 10  (Audio, 00:09:50, LBA: 77586 / 17:16:36)
-> Track 11  (Audio, 00:33:54, LBA: 78311 / 17:26:11)
-> Track 12  (Audio, 02:13:59, LBA: 80840 / 17:59:65)
-> Track 13  (Mode 1, LBA: 90874 / 20:13:49)
-> Track 14  (Audio, 02:14:25, LBA: 91550 / 20:22:50)
-> Track 15  (Mode 1, LBA: 101625 / 22:37:00)
-> Track 16  (Audio, 02:55:64, LBA: 102325 / 22:46:25)
-> Track 17  (Mode 1, LBA: 115514 / 25:42:14)
-> Track 18  (Audio, 02:05:73, LBA: 116214 / 25:51:39)
-> Track 19  (Mode 1, LBA: 125662 / 27:57:37)
-> Track 20  (Audio, 01:54:28, LBA: 126384 / 28:07:09)
-> Track 21  (Mode 1, LBA: 134962 / 30:01:37)
-> Track 22  (Audio, 01:43:38, LBA: 135678 / 30:11:03)
-> Track 23  (Mode 1, LBA: 143441 / 31:54:41)
-> Track 24  (Audio, 02:30:03, LBA: 144149 / 32:03:74)
-> Track 25  (Mode 1, LBA: 155402 / 34:34:02)
-> Track 26  (Audio, 01:52:62, LBA: 156088 / 34:43:13)
-> Track 27  (Mode 1, LBA: 164550 / 36:36:00)
-> Track 28  (Audio, 02:09:10, LBA: 165266 / 36:45:41)
-> Track 29  (Mode 1, LBA: 174951 / 38:54:51)
-> Track 30  (Audio, 00:37:56, LBA: 175575 / 39:03:00)
-> Track 31  (Mode 1, LBA: 178406 / 39:40:56)
-> Track 32  (Audio, 01:21:72, LBA: 179070 / 39:49:45)
-> Track 33  (Mode 1, LBA: 185217 / 41:11:42)
-> Track 34  (Audio, 01:43:24, LBA: 185889 / 41:20:39)
-> Track 35  (Mode 1, LBA: 193638 / 43:03:63)
-> Track 36  (Audio, 01:11:69, LBA: 194338 / 43:13:13)
-> Track 37  (Mode 1, LBA: 199732 / 44:25:07)
-> Track 38  (Audio, 01:31:39, LBA: 200368 / 44:33:43)
-> Track 39  (Mode 1, LBA: 207232 / 46:05:07)
-> Track 40  (Audio, 03:42:69, LBA: 208632 / 46:23:57)
-> Track 41  (Audio, 01:01:60, LBA: 225351 / 50:06:51)
-> Track 42  (Mode 1, LBA: 229986 / 51:08:36)
-> Track 43  (Audio, 04:11:57, LBA: 230654 / 51:17:29)
-> Track 44  (Mode 1, LBA: 249536 / 55:29:11)
-> LeadOut  (LBA: 250814 / 55:46:14)

It's very rare for any disc to have so many data and audio tracks mixed together. PCE doesn't use ISO9660 so the devs just split the game data as they saw fit.
If this was a well-mastered Yellow Book compliant CD-ROM then we'd just rip the tracks, ignore pre/postgaps and it'd be a complete, working dump. It's not so easy in this case. Let's analyze the transition from track 14 to track 15.

-> Track 14  (Audio, 02:14:25, LBA: 91550 / 20:22:50)
-> Track 15  (Mode 1, LBA: 101625 / 22:37:00)

The last sections of track 14 are silent, this is normal except for cases where you don't want gaps in audio. Since the next track is data it's a good idea to do this properly. Except, you know, the very last sectors are filled with junk here :)

LBA  :  101472

0000 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................  <- silence
SUBC :  01 14 01 02 12 22 00 22  34 72 00 00 00 00 00 00  ....."."4r......

LBA  :  101473

0000 :  6A 82 AF 21 BC 18 71 CA  A4 57 3B 7E 93 60 6D E8  j..!..q..W;~.`m.  <- junk
SUBC :  01 14 01 02 12 23 00 22  34 73 00 00 00 00 00 00  .....#."4s......

LBA  :  101474

0000 :  6A 82 AF 21 BC 18 71 CA  A4 57 3B 7E 93 60 6D E8  j..!..q..W;~.`m.  <- more junk
SUBC :  01 14 01 02 12 24 00 22  34 74 00 00 00 00 00 00  .....$."4t......

Notice how Q sub properly identifies this as audio track 14, index 1, time 22:34:74 - which is (22*60 + 34)*75 + 74 = 101624. With the usual TOC offset of 150 we get LBA 101474. Figures.
Also, TOC says track 14 should be 2:14:25 long and sure enough this is the last audio section, 2:12:24, according to Q (these counters start at zero).
Now, since there should be a 2-second long pregap for a data track that follows an audio track, and the data track starts at LBA 101625, so that means the first Mode 1 sector should be at LBA 101625 - 2*75 = 101475:

LBA  :  101475

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 00 01  ............"5..  <- Mode 1 header, first sector of a pregap
SUBC :  01 14 01 02 12 23 00 22  34 73 00 00 00 00 00 00  .....#."4s......  <- Q address goes back!

And we have a Mode 1 header! Except Q sub still says it's audio track #14-1 :) Also, data header address is 22:35:00 and Q has 22:34:73.
It doesn't take a genius to notice that 22:34:73 was also the first audio sector with "junk" instead of proper silence. Gee, I wonder what would happen if we extracted that junk and run it through descrambler. Yup. It is, in fact, the same sector we are looking at now and the header is 00 FF FF FF FF FF FF FF FF FF FF 00 22 35 00 01.
So... two last sections of track 14 and two first sectors of track 15 are overlapping. Depending which track you're reading, the drive will return either raw bytes for audio or descrambled sectors for data. In fact, it's even worse:

0000 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0010 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0020 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0440 :  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
0450 :  00 00 00 00 00 00 00 00  00 00 00 00 00 FF FF FF  ................
0460 :  FF FF FF FF FF FF FF 00  23 B5 00 61 00 28 00 1E  ........#..a.(..
0470 :  80 08 60 06 A8 02 FE 81  80 60 60 28 28 1E 9E 88  ..`......``((...
0480 :  68 66 AE AA FC 7F 01 E0  00 48 00 36 80 16 E0 0E  hf.......H.6....
0490 :  C8 04 56 83 7E E1 E0 48  48 36 B6 96 F6 EE C6 CC  ..V.~..HH6......

The data sector 22:35:00 starts in the middle of audio section 22:34:72. Which is perfectly OK. Except it cannot be properly ripped when the drive already cooks the raw data into sectors - since you can sync data to headers but there is no way to sync the audio.
And this is why I said it's not always possible to rip a CD-ROM track by track, even using subcodes as a reference. It only works for discs where audio sections match data sectors perfectly and even then it's down to drive logic to figure out how to join subchannel data with post-CIRC stream. So, claiming that any dump is a perfect copy of the master image is just... silly.

Let's see what happens next with our dump:

LBA  :  101476

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 01 01  ............"5..
SUBC :  01 14 01 02 12 24 00 22  34 74 00 00 00 00 00 00  .....$."4t......  <- note that Q still says track type is audio

LBA  :  101477

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 02 01  ............"5..  <- M1 address: 22:35:02
SUBC :  41 15 00 00 01 74 00 22  35 00 00 00 00 00 00 00  A....t."5.......  <- Q address: 22:35:00 (index 0, so a pregap)

LBA  :  101478

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 35 03 01  ............"5..
0920 :  00 00 00 00 00 00 00 00  00 00 AD 90 29 C9 83 16  ............)...
SUBC :  41 15 00 00 01 73 00 22  35 01 00 00 00 00 00 00  A....s."5.......

LBA  :  101625

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 00 01  ............"7..  <- First sector with actual data
SUBC :  41 15 00 00 00 01 00 22  36 73 00 00 00 00 00 00  A......"6s......  <- Q still says pregap

LBA  :  101626

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 01 01  ............"7..
SUBC :  41 15 00 00 00 00 00 22  36 74 00 00 00 00 00 00  A......"6t......

LBA  :  101627

0000 :  00 FF FF FF FF FF FF FF  FF FF FF 00 22 37 02 01  ............"7..
SUBC :  41 15 01 00 00 00 00 22  37 00 00 00 00 00 00 00  A......"7.......  <- Q finally switches to index 1

Q never matches the header address and in fact even the track type is wrong for some sectors. But don't assume that this 2-section offset is constant for all the audio/data tracks on the disc. While it could be the case, it's not guaranteed. The spec allows up to +/- 1 second offset so the difference could be anywhere from zero to 150 and, depending on what the drive logic thinks is the start of audio data, the header position might end up in different part of the audio section. In fact it's allowed for the header to start well inside a frame, as long as it's on 4-byte boundary.

So... all these frames, sections, offsets, sectors, tracks and gaps start to confuse you? Well, keep in mind the CD, while digital in nature, was meant to carry audio. Data sectors were added later and kinda slapped onto existing audio layer. This let the manufacturers reuse their designs by just adding header detection and descrambling hardware (along with some RAM for data buffering). But thanks to that we have this mess on our hands now :)
  • Current Music
    The Witcher OST
  • Tags

KEEP off my lawn

Funny how sometimes things happen, seemingly completely irrelevant to each other, and at some point you realize it's all connected.

I've been asked to look at PC-Engine and new methods of dumping CD-ROM based games for this system. Turns out there are some issues with current dumps, and most importantly some games are still unplayable. We're not sure if it's due to bugs in emulators or not but surely having a correct dump would help a lot.
The big word here is "correct". Dumping unprotected CDs compatible with Yellow Book is easy, however, when you get data sectors inside an audio track it's another story. There's been a lot of deliberation on the subject of CD-ROM dumps, especially when it comes to audio tracks and so-called "audio offset" - something that depends on particular drive/firmware and causes the actual data to be slightly shifted. This is usually way less than 1/75th of a second but many dumpers find it unacceptable that the same disc produces two different results, which cannot be directly compared.

The way I see it audio offset can't be fully accounted for when you dump the tracks on a PC drive. Sure, you can stick to one particular drive model to be able to compare the dumps but this cannot be really called "correct approach". Another problem is inability to directly read and dump special areas on the CD, like lead-in, lead-out, and some track pregaps (which might contain data - as part of a protection scheme or just due to shoddy mastering). I know that some people insist on dumping subchannel data for analysis and then proceed to read the disc according to that - but such methods are flawed as well. First of all, the subs are not protected by correction codes and contain bit glitches. Many are easy to spot and correct, either "by hand" or with a partial redump, but one can never be sure that the Q subchannel was read and reported as-is, or was it perhaps recreated by drive firmware when CRC checksum mismatched. Second, Q subs are not required for each and every section of the track. The drive must be able to deal with missing or damaged subs and this is also why data sectors carry their own address fields. Audio can only be addressed with Q sub, hence the seek function is only required to get as close as possible to requested audio chunk but it might not be a perfect hit. And this is why early drives had so many problems with stuttering audio rips - something that is mostly fixed on current models. Mostly :) Lastly though, many people don't realize that Q sub packets are stored for CIRC-encoded data, that in turn is a shuffled version of the actual raw data. So, in other words, when you remove CIRC and fix any C1/C2 errors, the order of data changes back and it's hard to tell what Q address it really belongs to. Again, not a problem for data sectors that have their own header and address field but it makes audio positioning rather inaccurate - hence the offset.

One method of dumping CDs that is not plagued with these issues is to try and grab raw frames, after EFM processing and before these enter CIRC decoder. We can have 1:1 mapping of data and subchannels, except we now have to process everything software-side in the emulator. Not only that, we need to actually get dumps like that first, and that is not possible with unmodified consumer equipment. Even when we do, we might end up with bit glitches (perfectly fixable with CIRC but still present in the dump) that will require a redump of perhaps several discs to get rid of. There are also weak EFM patterns in some protection schemes but these we can't do anything about, since even capturing analog stream coming directly from the optical pickup would not let us read those properly. So, no point in bothering. This new idea certainly has it's own problems but these can be overcome, so it's at least worth proper research.

So, what's this connection I mentioned? Well, I got a link to a document from KEEP. It stands for "KEEPING EMULATION ENVIRONMENTS PORTABLE" and it's EU-funded project aimed at preserving old games (well, multimedia in general). In theory it's great news that there are organizations like that and at least some politicians and/or civil servant could be convinced to fund them. In reality though the situation looks pretty grimm still.

Here's a link:

I wrote about this several times already but I feel like I need to keep reminding you, and myself, every now and then - so that you don't take things for granted. I urge you people to read this document, it's not all that long. After I did, I came to realise three things:

1) These guys have money.
Apparently got more than 3 million euro so far. You can tour the world with it and have open, free meetings for people. Make no mistake, I'm not saying it's bad, far from it - we need people to understand what's at stake and what the problems are. I just hope this is not the only thing the money were used for.

2) All they did is code UI in JAVA.
Yes, the greatest achievement so far is some silly-looking UI for a couple of open-source emulators. I mean, seriously, most of the document praises this... thing... as if it was the ultimate solution to game preservation. I've seen high-school projects more ambitious than this. Are the authors of WinUAE or VICE aware that KEEP is using their work? I'm not sure so I might be somewhat out of line here, but it really angers me when open-source projects are used like this, just because these are free. Free as in free beer in this case.

3) They are fully aware it's illegal.
At least they worked that out. Even went as far as to specifically point out legal status of bypassing copy protection in Germany and France, and the fact that there are no exceptions, whatsoever, for individuals doing research. I'm willing to bet there were games for Amiga and C64 that were copy-protected, and that authors of WinUAE and VICE had to study those in order to properly emulate. Now, if you can play a game in emulator that was stored on a floppy or a tape in a way that prevented copying, then surely this system has now been bypassed. So, it's illegal. And yet... no-one cares if it's KEEP? I mean, if I was to throw a conference like this and show Makaron working, then I'd most likely have lawyers (and possibly police) knocking on my door the next morning.

Please spread this little rant of mine. I'd like to have these KEEP people made aware of my own existence, and all the other people that were doing emulation and preservation long before they even had the idea. And we are all individuals working for free in our spare time, to beat the clock. Because some of the systems are becoming dust as we speak and soon there won't be anything left to preserve. Whether it's dumping or emulation, we often have to bypass original protection systems to get results, which is illegal. As is storing software you're not licensed to have, even if it's the last copy on Earth. We have the skills and knowledge to do these things and must keep a low-profile, whereas organizations like KEEP do very little to actually change the law but have no problems boasting success when using existing emulators.

If you think I'm overreacting then just answer yourself this little question: What would KEEP be able to show during these meetings if they did not use free emulators? That's right, nothing. Not even that UI since there would be no point to it without actual emulation. And systems and games that were researched and dumped long before they started. I really love the idea of raising awareness on horrible laws we have and all, but to me it feels like KEEP is trying to grab the money while reinventing the wheel. Rather than try and help/fund/coordinate existing projects, they take what is free and claim to have done all the dirty work. As if.
  • Current Music
    Astral Projection
  • Tags

Don't fail me again

Hi guys. And gals. Do we have any gals here?

I promised some people that there will be a Christmas release - but as you can see, it didn't happen. There are a few reasons for that but the last nail in the coffin was last Wednesday, when I slipped on ice and fell in a very unlucky way, with both of my hands still in my pockets. I hurt my left arm and probably cracked a rib or two since I still feel pain in my chest and back.

Somehow it seems that 2011 was rather unlucky when it comes to Makaron, I had very little free time this year, especially during the second half of it. I think next year should be a little less busy (though probably not until February, or even March).

I'm feeling a bit better now, can move around almost freely, so I will try to hack together something just to release it before midnight of 31 December. It won't be the T13 version though, I've made it DX11-only and there are still some issues I haven't worked out yet so it just won't be ready.

As always, thanks for your support.

By the way, some people have been asking about my custom JVS I/O and GD emulator. Both projects are now paused, but I haven't abandoned them or anything. I would still very much like to sell both devices to raise some money - software and hardware I need are pretty expensive and I'm stingy. There are 3 main problems with those right now:

1) Neither device is complete (though both are working well enough).
2) I'm still uncertain what features should be included or omitted.
3) How do I go about setting up a shop when I'm done :)

Please provide some input. I need to explain that I'm favouring simple and cheap devices - while even a more advanced ones would be cheap if mass-produced, I really doubt there is a market big enough for that. So, for example, my JVS I/O lacks the ability to daisy-chain it with other I/O boards. It's not really that much of a problem to add more functionality but it's another MCU port tied up, a couple more elements are needed, one more USB socket, more PCB space, etc. It seems like a small thing but the costs do add up. The original idea was to have just enough inputs for one player. A bigger MCU, like ATmega128 could easily support two players with analog inputs as well but it's again more costly. So, what do you guys think, a super-simple I/O or just a cheaper version of a standard two-player one? Because, again, I don't think there is market for both.
Same thing with the GD emulator. It runs off cheap SD cards and has no fancy user interface or anything. I don't really see it as a problem, someone could just code a GUI to run on Dreamcast and boot that as the default image. The device will support disc swapping properly you know. I already said that before but this approach means I can only do about 2MB/s when transferring data, which is just tad bit faster than the original GD drive. There are some games that will break when you transfer the data too slow, or too fast, so I think it's actually best to stay close to original specs. If I were to go for a faster solution the whole thing would need to be re-designed - which is possible but I'd rather not have to do it. Then again, if nobody wants to buy the device as it is now then what's the point, right?

So, talk to me. I'm open for new ideas, I just wanted to make sure you understand that adding things is costly and some of this stuff is actually pretty hard to implement because of various limitations of the hardware I want to use.
  • Current Music
    Pendulum - Immersion

architecture syn of GDEMU is

And they said imitation diamond wasn't good enough :)

2011-08-07 Dreamcast GD-ROM drive hardware emulator

So, does it work? Sure. It's a very early version though. I've hit some metastability problems so I switched from external ARM7 MCU to NIOS2 running on FPGA to speed up debugging. After I reversed my main clock polarity (yeah, Star Trek style) and it worked better I finally realised that I'm running a synchronous system with asynchronous inputs. Which is basically the same as having two different clock domains since I have no control over setup/hold times... So I've added two-stage synchronizers to /DIOR and /DIOW but that's additional latency and Dreamcast has a bad habit of deasserting /CS signals shortly after rising /DIOx. These things always work so well on paper :)

In the end the hardware side of GD interface is pretty small, should fit in EP2C5 (that's Cyclone II FPGA with 5k logic elements) and that's pretty cheap. The downside is I only have so much internal RAM so the main data buffer is just 8kB. While external SRAM could help here, I'm not yet sure it's worth the trouble. We'll see.

Digital audio is completly not supported yet (but is part of the design, so it will be added eventually) and I just wanted to test it out ASAP so I went with slow, PIO-only SD card access and very inefficient CPU buffering. Also, external MCU needs to be connected to FPGA with some sort of data bus and this becomes a bottleneck for the transfers, as it turns out. For example my ARM7 doesn't have a dedicated external memory interface so I have to do everything myself using a PIO port. With only 30 pins (minus a few for SPI and clock output) all I could manage was 8-bit shared address/data bus. Not very fast, unfortunately.

Because of the slow transfers games exhibit various issues, like missing textures, slowdowns, stuttering sound. This will get better as the project matures. In fact, with proper buffering I'm sure I can get it working as well as original GD drive and perhaps even faster - up to some 2x, which is the limit of what one can do with SD cards in SPI mode. Well, there's always the USB route I suppose.

By the way - I get simply tons of spam in the comments now. I've enabled LJ CAPTCHA but that only cut it in half or so. Worst of all, the spam looks (at the first sight) as proper comments, pretty nice English, capital letters, periods. I might accidentaly delete some actual comments while cleaning so keep that in mind when posting here. And if the situation gets even worse I'll probably disallow anonymous comments completly... though that's the last resort.

EDIT: Okay, a small explanation on what this does.

I started this project long time ago but lost interest after hitting some walls. Recently I had a few good ideas and decided to work on it a bit.

What you see on the photo is Dreamcast with it's cover off and the GD drive assembly removed. I cut some holes and soldered wires directly to the mainboard to avoid messing with the original connector. This way I can always plug the drive back in and use it as before - or even better, I can use FPGA as logic analyzer to watch the traffic.

In this configuration there is no real drive and FPGA runs a soft-core CPU that emulates it. Obviously there's some glue logic in there as well or I wouldn't need an FPGA in the first place :) Data is being pulled from SD card - you can see it inserted just over the flat cables. With this I can run any dumped game, and unlike CD rips I actually emulate a GD media so the Dreamcast can't tell the difference. The USB is used to program the FPGA and I can't disconnect it because I don't have a license for that NIOS2 soft-CPU core, it will stop working after the PC uplink is lost. Other than that I can't use it for data transfers unfortunately.

The idea is to have a much smaller (and cheaper) FPGA here with fast external MCU. Data would be stored on SD media or pulled via USB 2.0 uplink to PC.

So far I've tested a couple of games for EU region, and a few JP ones after I hacked them to be multi-region :) I do have Japanese Dreamcast mainboard (well, two actually) but this is the only one I have modified for the project. Once this goes out of prototype phase I'd like to find a matching connector and just make it a plug-in replacement for the GD drive.

So... Skies of Arcadia works, at least EU version. Hacked US one shows no picture but I can hear it running. It works on Makaron but I'm starting to wonder if there is a problem with this particular dump... Anyway, the GDEMU is good enough to not freeze the intro sequence like the ripped version does. There is about 3 seconds of video/audio desync at the end of the intro due to the transfer speed being a bit too low. Same things happens in Dead or Alive 2 intro, which is also pretty long. But other than that it seems to work. Street Fighter Alpha 3 has some sound issues in the attract mode but not during actual fights. This is all after a few improvements I made today, I still need to run SPI link closer to 25MHz to get better transfers from SD card. Buf for many games, like Soul Reaver 2 this is already enough to play without any problems. MPEG movies work OK as well.

DX11 status report

So, there is this guy who keeps asking about DX11 renderer progress and I figured I might as well explain it with some pictures :)

Direct3D 11 is very different to D3D9, and this is the source of my problems. While I want the functionality it offers, I need to properly rewrite every part of the code. In fact the change is so big I created a separate project that is based on NAOMI emulator code but all it does is render one frame of previously captured data. Makes my life so much easier.

D3D11 is a functional extension to version 10 interface except you work on context rather than the device itself, which is only used for creating new resources. This lets you create command lists for things like deferred rendering, though I stil use immediate mode. Unlike in 10, the reference counters are incremented by various Get* methods and this is more like it was in D3D9, so the behaviour was clearly reverted. And for the better I suppose - makes much more sense this way.

The main problem is the grouping of various settings that I need to change individually, this was much easier on D3D9. While the new approach certainly speeds up things for games, I cannot make any assumptions about the 3D data I'm processing and I can't sort it nicely either, so I have to prepare many object with all possible settings well in advance and select the right one when the time comes. Doesn't seem that complicated, eh? Well, it's actually is much more bothersome than it sounds.

Anyway, I decided to start with the basics and then, once it all more or less works, try to add the layer peeling system for per-pixel depth sorting. There will be another major obstacle on the way, namely the fact that I can have different blending operations mixed in together. So I will need to store the blending instruction data along with the pixels and have pixel shader combine the fragments properly. Which is not going to be easy or super fast :) Still, no point in worrying about that now. It's either that or full software rendering if you want proper PVR2 emulation.

Here are some sample screenshots I took along the way, so that you could see there is progress. Note that I already ported and tested the text display class - though with just one frame being rendered the log is much better target for debug output.

2011-06-08 DX11 test #1

2011-06-08 DX11 test #2

2011-06-08 DX11 test #3

2011-06-08 DX11 test #4

These two are different becase I had to change the capture data format a few times:

2011-06-08 DX11 test #5

2011-06-08 DX11 test #6

Anyone wants to guess what game is this? Should be pretty easy by now :)

EDIT: We have a winner! It's Zero Gunner 2.

2011-06-08 DX11 test #7

Uncensored chip porn

Just some stuff I got today. Should provide plenty of fun - I got me 4 Sierra game packs some time ago, plus I own some Lucas Arts adventures as well :)

The only trouble now is my old SB128 (that I need for MIDI port) is no longer supported in 64-bit Windows. So I'm waiting for USB to MIDI converter to arrive. Not to mention half a dozen mono jack plugs... I just hope I can make ScummVM and DOSBox recognize and use the device, if not I'm going to ressurect an old DOS machine I have around here, somewhere.

Roland CM-300 (though modern wavetable based General MIDI emulation ain't half bad either):
2011-06-04 Roland CM-300 PCB2011-06-04 Roland CM-300 chips

Roland CM-64 (current MT-32 emulation is rather lacking, can't beat the real hardware):
2011-06-04 Roland CM-64 PCB #1 (CM-32L)2011-06-04 Roland CM-64 PCB #2 (CM-32P)

Maybe I'm getting old but I'm having more fun with older games, that I have already finished at least once, than with the new ones. Oh, there are notable exceptions but this is the general trend these days..
  • Current Music
    Unreal Tournament mods :)
  • Tags

Software design 101

PSN shop is back - as broken as it was before. You'd think SONY took the time and fixed the interface along with the security issues, but no, apparently it's the best we're gonna get.

The annoying background adverts are back. Those are way too bright and make the white text nearly unreadable. Seriously guys, how difficult it is to QA these things? You could've made the windows more opaque or have the font outlined - these are the first two things that come to mind. Or better yet, have the colors/layout customisable with user settings or skins... So many options here.

Or, we could just do away with these silly adverts/wallpapers. Haha, yeah, that would be a small miracle right there if it happened.

So, there is this No More Heroes demo that I wanted to try. It's not under N though. Or H. In fact, it's not in the Demos section at all, it's in the Latest -> Demos. The most logical explanation is these entries are not links but entered and maintained separately - and that has consequences, more on this below.

The Secret of Monkey Island is not under M, it's under S. Okay, that makes perfect sense if you follow proper conventions, since it'd be Secret of Monkey Island, The then. But Monkey Island 2 is under S as well, for whatever reason... Maybe because it's Special Edition. God only knows. The allmighty might also know where the hell Tales of Monkey Island are because I just couldn't find them. At all. Telltale Games should totaly sue, they are loosing sales because of this blunder. Fortunately the search option will find what you want - but you can't navigate back to the point of origin so it remains a mystery where PSN guys put those.

The in-shop search function is a blessing but really, who uses that on daily basis? Not to mention it's far from perfect, as the rest of the shop interface, and with all the heavy load lately it's prone to time-outs. It'll just wait forever trying to fetch the results and will not work after that. What's more, if you wait too long you will get booted out of the PSN shop alltogether - and not while doing the search but later when navigating.

There are tons of other things wrong with the shop, and item naming is the most obvious and annoying. So what is, say, Epilogue and why it's under P? Well, that 'cause it's Prince of Persia add-on. The World Of Recycled Vessel is also under N because it belongs to Nier. Trust me, this list goes on. It's not just DLCs, HOARD game is under L for example. Sackboy's Prehistoric Moves is under L too, since it's Little Big Planet offspin I suppose - even if the title doesn't even contain the letter L at all.

And don't get me started on ALL CAPS names or Extremly Long Game Titles: First And Only Part There Was Or Ever Will Be But Just In Case We Made This Subtitle In Advance. Or weird version numbers in demo titles. Or how some games have the demo, full game and addons nicely grouped while others require deep knowledge of the PSN store layout. Monkey Island remake is one such game, you'd best remember there is a bundle now that has both of the games, and cheaper too. Too bad it's not pointed out anywhere in the descriptions of the individual downloads. Frankly those are never, ever updated once put in the store anyway.

Ah, about descriptions. Those mostly contain the legal mambo jumbo, rarely anything else or of use. Definitely not the download size, to get to know that you need to purchase first. I have 250G model and fast internet connection but I suppose not all PSN users do. Some descriptions will list supported resolutions and HDD requirements for saves, but that's it.

Oh, and the non-linked entries I mentioned? It's very easy to forget one when updating something, and so we have Worms that, depending on which section you're browsing, have the PSN+ price drop or not. I couldn't be bothered to look for more examples. Point is, these guys are getting paid for it. Maybe it's not enough, I don't know, I sure could do a better job in my spare time though.

The download history is pretty useful, or rather faster than navigating the darn shop. But it's just one big list, can't be searched either. Most importantly you can only redownload items on one by one basis, and if you happened to delete your 40+ free Ridge Racer 7 addons, you're going to weep. I sure did.

Well, now that I got that out of my system I feel much better. I needed that after the Might & Magic: Clash of Heroes demo got me this close to punching my monitor. It should be called Might & Magic: Now Loading - I'm willing to bet the Nintendo DS version works faster... And DS has what, 4MB of RAM?
  • Current Music
    Tangerine Dream - Tyranny of Beauty
  • Tags
    , ,

Qui non proficit, deficit, SONY

So... SONY manages to have their PSN data stolen. And not just any data - the most sensitive kind, their customers' personal data. SONY calls that a high-profile malicious attack but in reality their system had gaping security holes. Known vulnerabilities were not patched for months. Very proffesional, let me guess, it was all outsourced to cut the costs?

How bad is it? So bad that they don't even know WHAT was taken since the logs were compromised as well. Because SONY not only failed to run a proper server stack, they failed to setup a logging system that would run independently. Apparently if it wasn't for the attackers clumsy attempts to cover their tracks, nobody would even know the intrusion took place!

To add insult to injury I was away from country as it got reported and could not block my card or else I'd end up with no money to pay bills and return home...

Words fail to properly convey my feelings about this, I'll try though. Let's start with PSP - it got hacked, with a spoofed "service battery" of all things. Way to include a backdoor into your secure product, guys. Just brilliant. PS3 - got hacked with a stupid buffer overflow in USB layer, again with the help of a backdoor system known as "service JIG". Then it got owned completly because SONY decided to use curve-based crypto rather than the usual RSA-like system, and nobody understood the new algorithms well enough to actually use them properly. I was laughing so hard when this was published... And what does SONY do? Blame the hackers.

Next thing SONY fails at is running a secure database with all of their clients data. And not just PSN, apparently the SOE one was raided too. You couldn't do worse if you tried. Hell, SONY might have as well just copied and send out the data themselves - at least that way we wouldn't have to endure a lenghty downtime of their services. But wait, that's not all of it yet. Now they got their collective asses moving and decided to actually upgrade the Apache to the latest version - and maybe (though doubtful) to keep doing that from now on. And with the claim that it's secure now they made it into grand reopening. Some lesser CEO even apologized. Wow. It would be much more impressive if not for another exploit that was discovered, what, less then 24 hours after the restart? This seems to be the extent of their new security - they just can't code shit, even if their lives depended on it.

Also, as it turns out, Japanese government is not happy with the way SONY handled the situation and forbid PSN reopening until all the issues are properly resolved, and explanations are made as to what steps were undertaken to prevent this mess from happening again in future. Now, this might just be a political struggle but it does kinda make you wonder just how secure this "new" system really is. It sure bothers me.

We are talking about 100 million accounts, this is THE BIGGEST data leak in history. Ever. SONY fails time and time again, and there seem to be no consequences whatsoever. I'm really unhappy with EU and USA authorities not demanding formal apology and proper compensation. I suppose the unnamed hackers will again take all the blame, since apparently big companies can do no evil. Nobody died, right? Move along folks, nothing to see here.

Oh and don't get me started on the Welcome Back Programme - it's just as insulting as the half-hearted apology. Rather than get a wallet bonus or something to the effect, we are given old games that many already have. And this is no accident, SONY knows well what games we have and play (via PSN reports) so this way they just pretend to give us something without actually investing too much money. Think of it this way, you just exchanged all of your personal data, including possibly credit card info, for 2 lousy games and 30 days of PSN+. Was it worth it?

I'm pretty sure kids won't mind. When you're 16 or so, what do you care who knows what your name and date of birth is. But proper adults should realize the potential consequences of identity theft. I've seen it happen to someone who had ID card stolen and let me tell you - it ain't pretty. First of all it won't hit you right away, usually takes months, but then it will haunt you for many months to come. And there is nothing you can do about it, except work hard to clear your name. In the meantime it's quite possible you'll be visited by debt collectors asking for money you never had, and even receive threats from people who think you owe them. Banks might lock your accounts every now and then, your cards will stop working due to all the stuff happening, and so on. And the worst thing is you know none of this is your fault. Welcome to your personal hell.

I won't ask anybody to boycott SONY. You do what you think is right in that situation. I will probably keep using my PS3 since the milk is spilt and nothing will change that. I sure won't enter my card detais into the console again, though, and if there is anything interesting on PSN then I will look for alternative payment methods. That's assuming I will feel like buying from SONY anytime soon. I'm simply not going to trust that company with anything, ever again. Because, let me repeat that one more time, not only they can't protect my data, they don't even feel like compensating me for my loss. So, I don't feel like spending my money on their products.

EDIT: And this just in, though still not properly confirmed: It seems that firmware 3.61 causes some of the consoles to overheat and turn off while playing. Seriously SONY, I couldn't come up with this stuff if I tried.
  • Current Music
    Universe at War OST
  • Tags
    , ,