Go Back   Project64 Forums > General Discussion > Open Discussion

Thread Tools Display Modes
Old 15th April 2016, 03:20 AM
Falkoner Falkoner is offline
Junior Member
Join Date: Mar 2015
Posts: 11
Default Hey You Pikachu! HLE Implementation

Starting a new thread here, since I believe the NRage plugin is the closest currently to providing the support for raw VRU input, which we'll need to get this working.

The BlissBox developers have acquired a VRU and are hoping to implement the necessary hardware/driver solution to get owners with a VRU capable of playing on emulator. Here's their comment in the EmuTalk forum:

Whats up guys. Ulao from Bliss-Box here... So I'm going to add the VRU to my Bliss-Box. I found the main commands to make the units start working. I'll be working on this to try and get it to recognize words and see what codes it wants to send. The best I can tell so far is that it sends 11 bytes. I'm guessing that is ASCII data or just some hex codes.
example (11 bytes) all zeroes.
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
If each byte was a word then you could send lots of data, if each byte is an ASCII char it would take a lot more. If its just 11 bytes I can send all of that down the USB interface as I have a total of 13 packets exposed to HID DX.

At this point I just need to know what goes on in the rom. I dug up some old MESS emulator dev talking about it. He saw this at boot up.
The 0x0a command sends 16 bytes to the controller and expects 1 byte in return.
The 0x0b command sends 2 bytes to the controller and expects 3 bytes in return.
The 0x0c command sends 6 bytes to the controller and expects 1 byte in return.
The 0x0d command sends 2 bytes to the controller and expects 1 byte in return.

I see most of that on my end as well but 0x0b returns 11 bytes for me.

I think we need to know more about what is going on in game. I'm hopping the game is a listen only device, so no game instruction needs to talk to the adapter. If it does need to send data to the adapter we could always use the rumble data. Just send either custom rumble or unused effect types. If anyone wants to help just jump in. If there are any Bliss-Box owners out there and want to beta test let me know.
I also have already created a voice recognition prototype that could act as a decent start for our HLE version. Once we have it working through BlissBox, we can capture the commands and then have a modern voice recognition system send the correct command to the system based on what it thinks the player said into their computer's mic.

Need some help from developers to get support for the VRU implemented in the NRage plugin.
Reply With Quote
Old 15th April 2016, 10:30 AM
Falkoner Falkoner is offline
Junior Member
Join Date: Mar 2015
Posts: 11

Based on the description of the device here, it appears that the only major instructions are an instruction from the CPU to the game to begin recording(when player holds down Z), the instruction to retrieve the captured data and word guess from the VRU, when Z is released, and the initialization instructions sent to the VRU, which appear to essentially be the word dictionary, which the VRU uses to guess which word the player said. The captured data itself doesn't appear to ever be sent back to the N64, just what word the VRU thought it was.

Issues opened up in both Mupen64Plus and Project64 repositories, since I believe we need modifications to controller plugins on both to get it working. Project64's implementation of NRage's plugin likely is the closest, since it has some kind of RAW input support implemented for Adaptoids, and has gotten controller paks working as well but I'm uncertain that is sufficient for the VRU to work.
Mupen64+ Issue
Project64 Issue

I'm fairly certain it needs to be done, because I purchased a Wishtec Adaptoid, installed a WinXP environment to get the driver working, and found that there was still no support in the emulators for the VRU.

I PM'd MdkCheatz back in December, and he responded, here's what he said he knew:
The VRU was tech designed originally to be compatible for use with future games.

It's been a while since I've been involved in the Hey You Pikachu project but I reckon you must have come across the research I had once published back when I was trying to reverse engineer it.

I see someone is trying to do achieve what I was once hoping to do.

Here's the deal... Anything I had done was surface fluff. All that I know is theoretical based on interpreting developer data I found online regarding the VRU.

Regarding the VRU... you have 4 very basic concepts.

1. When instructed by the game (any game), the VRU hardware begins writing and translating analogue data taken from the mic itself.

2. The VRU goes through a matching process inside the VRU and not the game. This word or phrase is translated into instructional code back to the game.

3. The word phrase is calculated as a probability of match. I forget if this happens in the VRU or the game itself.

4. The game has a "tolerance" range of probability it is willing to accept depending on a difficulty setting.

Basically, the VRU turned out to be a psuedo voice recognition unit. It isn't true recognition. thats why I stopped doing research on it. To emulate the VRU would involve learning how to make a true voice recognition unit and trying to water it down to a simple set of instructions. IMO a complete waste of time. But if you are ambitious enough to do this, hats off to you.

A feasible means to emulate the VRU would likely be forcing a probability range that is accepted by the game. you could theoretically play the game to the end but you would loose the value of actually playing it. It would be as if someone is playing it for you.

Or, you could offer a solution specific to the game where the player can select input based on random options of words (like multiple choice),therefore adding the possibility of more involvement with the player. but the real magic would still be lost.
Doing a little more digging, here's the official N64 Programming Manual regarding the VRU. I think that's actually perhaps the most useful piece of info I've found yet. Not certain if everything is accurate, since this appears to apply to the Japanese VRU, but I think it should be a very similar process.

If I'm interpreting it correctly, each word registered is an 8-byte character, here's the relevant code for the entire process, from initialization to creating the dictionary, to adding words, to then starting the recording, to receiving the result back.
s32 osVoiceInit(OSMesgQueue *siMessageQ, OSVoiceHandle *hd, int channel);
typedef struct {
OSMesgQueue *__mq; /* SI message queue */
int __channel; /* Controller port No. */
s32 __mode; /* Used within the OS */
u8 cmd_status; /* Command status */
} OSVoiceHandle;
s32 osVoiceSetWord(OSVoiceHandle *hd, u8 *word);
s32 osVoiceClearDictionary(OSVoiceHandle *hd, us words);
s32 osVoiceSetWord(OSVoiceHandle *hd, u8 *word);
s32 osVoiceStartReadData(OSVoiceHandle *hd);
s32 osVoiceGetReadData(OSVoiceHandle *hd, OSVoiceData *result);
typedef struct {
u16 warning; /* Warning */
u16 answer_num; /* Candidate number (0~5) */
u16 voice_level; /* Voice input level */
u16 voice_sn; /* Relative voice level */
u16 voice_time; /* Voice input time */
u16 answer[5]; /* Candidate word number */
u16 distance[5]; /* Distance value */
} OSVoiceData;
s32 osVoiceStopReadData(OSVoiceHandle *hd);
//Registered Words:
u8 *registration_word[] = {
What interested me is this says that 255 syllables can be registered, including multiple syllables per word, but 640 commands appear to be in the game's code, with 459 of them being unique(I assume the repeats are pronunciation options). I think they must reregister the dictionary depending on what section of the game you're in, pinata commands in the pinata game, vegetable names in the soup-making level, Pokemon names are only registered as needed(ie, for Who's That Pokemon! game), etc. That potentially complicates the HLE implementation a bit, but if we can get the Sphinx dictionary accurate enough I figure it should work fine.

Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

All times are GMT. The time now is 11:51 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.