Archive for the 'Blackberry' Category

Screenshots of Our Upcoming Blackberry Game!

December 24th, 2010 Comments(4)

Well – I’m very excited. If you’ve been following the Twitter feed at all, you know that I’ve been working at a breakneck pace trying to get Galactic Blast completed. You might recognize the title from the demo game featured in our tutorial on creating a Blackberry game. The commercial release of Galactic Blast is built off a very similar framework to what’s featured in the demo, only with a LOT of extra stuff added – pre-rendered 3d graphics, bonus rounds, weapon upgrades, etc.

So without further ado, screenshots!

The Gamma-3 base - the final boss.

The demolition controller destroys the derelict ships in decommissioned shipyard A-3. In this case, however, he's trying to destroy you! Third boss in the game.

The Sutoran Nebula is home to the Phoraxian Shadow Fleet. They hide within the clouds of the nebula to make detection more difficult.

Every 5 waves your SR-13 kicks it into hyperdrive, and you enter the bonus round. Pass through as many rings as possible for maximum points!

Main menu. Start/Resume game, Instructions, View High Scores, Change settings (Sound/Music/Vibration, etc), and Quit.

Title Screen

I just submitted the application to RIM for approval on App World, so hopefully it checks out and it will be available for download soon! More to come…

Unfortunate News for a Realtime Blackberry Mixer

November 23rd, 2010 Comments(1)

Overview

Sometimes I’m not sure if it’s a good thing or a bad one, but I have the kind of personality that when I get involved trying to solve a problem, I can’t stop until it’s solved. This works out well as eventually the issue at hand is fixed, but it also has the side effect that I forsake pretty much everything else until I can resolve things. When I can’t find a solution, or there isn’t a good one, it drives me nuts.

And unfortunately, I’m in that irritated mindset now. If you read my post on Mixing Two or More Sounds Together on the Blackberry, you saw the video I posted of a software sound mixer I had written. While the mixer did indeed work, the problem was I had only tested one scenario – where the user has chosen to start mixing all the sounds together at the same time. This use case is no problem.

The Ongoing Problem

The use case that’s an issue (and honestly the whole point of a realtime mixer), is where you start playing one sound, and then mix another one in at an arbitrary time later. The reason why this is an issue is fairly easily explained (and unfortunately not apparent to me until I had written most of the code).

As you know if you’re a Blackberry developer, all user code is written in Java – the BB runs a JVM where all the software runs – that’s how the phone is designed. This creates a layer of abstraction, where the user doesn’t have direct access to the hardware, and where the code runs relatively slow (compared to native instructions). Because of this, RIM provided mappings from the javax package’s Player class to the Blackberry sound hardware.

The Problem of Buffers and Fetching

Player does a number of things – but the core of it retrieves data from a sound stream (a wav file in our case), massages it as necessary, and then loads the sound hardware buffers with the data. The big problem is at the stage when Player retrieves data. It does so in chunks of data – specifically (from my experiments and as other people on the web have reported) in 58000 byte chunks. That means that is takes in a few seconds of sound at a time so the hardware buffers don’t underrun during playback. You can do some tricks with blocking the stream and force it not to load in this full 58000 – and you will immediately hear the effects – choppy audio. The 58000 byte buffer is not to be annoying – it’s necessary for smooth and clean audio.

And this is why we can’t do realtime mixing. We don’t have direct access to the hardware buffers, we only have access to the buffer streams we can feed to Player. And if Player demands a few seconds of audio at each read, we cannot insert a sound into the few seconds that we feed to it after the fact.

An Example of the Problem

I have two sounds. Drums.wav which is a 20 second wav, and flute.wav which is a 5 second wave. I would like to play drums over and over again, and then immediately mix the flute in when the user presses the ‘F’ button.

Player immediately reads in, let’s say 5 seconds of drum.wav. Then at second 3, the user presses the ‘F’ button. Our program can immediately queue in flute.wav into the input stream, but it won’t be played until second 5, since the Player already read in 5 seconds. Even if we do some creative thread blocking, we can never guarantee a real time mix at all times.

The Code (Could Still Be Useful in Certain Applications)

Below I have included all my work on this project. It consists of a “MixerStream”, which is meant to be a replacement to InputStream. You feed an instantiated MixerStream object to Player, and then call the MixerStream “addStream” method to mix other audio files, in real time, into the Player. It does work – but there will be large delays between when the sound is added, to when it is heard, due to the problem described above.

Note – this code is currently hardcoded to only work with one format of PCM stream right now (I put two hardcoded headers of a 44.1khz, 16bit, stereo, and a 44.1khz, 8bit, mono) – in addition to doing a very low tech way of mixing streams together. The method “getMixedByte” is all setup for future functionality (Allowing any format of PCM, doing better mixing, preventing clipping, etc) – but I’m not up to doing any more work with this right now. It only works as a semi-real time mixer, which I don’t think is too useful – the only place where it might be nice is if you’re developing a Blackberry audio-editing application, or something to that effect – where realtime isn’t necessary. If you are doing this – feel free to use the code below – I’ll provide any help I can.

So for now – this project is abandoned. Frustrating, but necessary for now. Hopefully with the new QNX based OS coming out, there will be more support for directly accessing the hardware buffers.

package com.synthdreams;

import java.io.IOException;
import java.io.InputStream;
import java.util.Vector;

import net.rim.device.api.ui.component.Dialog;

// The MixerStream class allows us to add multiple PCM wave streams to mix together in real time
public class MixerStream extends InputStream{
   private Vector _streamVector; // All wave streams currently managed by the mixer
   int[] _headerArray;
   int _headerPos;
   int _totalRead;

   // WaveStream class is an InputWrapper stream that stores wav audio information
   private class WaveStream extends InputStream {
      private InputStream _fileStream; // The input stream containing the wav data
      private int _sampleRate; // Samplerate of the wave
      private int _bitRate; // Bitrate of the wave
      private int _channels; // Channels in the wave 

      public int getSampleRate() { return _sampleRate; }
      public int getBitRate() { return _bitRate; }
      public int getChannels() { return _channels; }

      public WaveStream(InputStream passStream) throws IOException {
         byte[] byteArray;
         String tempChunkName;
         int tempChunkSize;

         // Assign the file stream, then read in header info
         _fileStream = passStream;

         // 4 - ChunkID (RIFF)
         _fileStream.read(byteArray = new byte[4]);
         if (new String(byteArray).equals("RIFF") == false) {
            throw new IOException("Not a valid wave file (ChunkID).");
         }

         // 4 - ChunkSize
         _fileStream.read(byteArray = new byte[4]);

         // 4 - Format (WAVE)
         _fileStream.read(byteArray = new byte[4]);
         if (new String(byteArray).equals("WAVE") == false) {
            throw new IOException("Not a valid wave file (Format).");
         }

         // 4 - SubchunkID (fmt)
         _fileStream.read(byteArray = new byte[3]);
         if (new String(byteArray).equals("fmt") == false) {
            throw new IOException("Not a valid wave file (SubchunkID).");
         }
         _fileStream.read(byteArray = new byte[1]);

         // 4 - Subchunk1Size
         _fileStream.read(byteArray = new byte[4]);
         tempChunkSize = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);

         // 2 - AudioFormat(1)
         _fileStream.read(byteArray = new byte[2]);
         if (byteArray[0] != 1) {
            throw new IOException("PCM Compression not supported.");
         }

         // 2 - NumChannels
         _fileStream.read(byteArray = new byte[2]);
         _channels = ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff); 

         // 4 - Sample Rate
         _fileStream.read(byteArray = new byte[4]);
         _sampleRate = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);

         // 6 - Byte Rate, Block Align
         _fileStream.read(byteArray = new byte[6]);

         // 2 - Bitrate
         _fileStream.read(byteArray = new byte[2]);
         _bitRate = ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff); 

         // variable - Read in rest of chunk 1
         _fileStream.read(byteArray = new byte[tempChunkSize-16]);

         // Burn through unneeded chunks until we get to data
         tempChunkName = "";
         tempChunkSize = 0;
         while (tempChunkName.equals("data") == false) {
            // Read in name and size of chunk
            _fileStream.read(byteArray = new byte[4]);
            tempChunkName = new String(byteArray);
            _fileStream.read(byteArray = new byte[4]);
            tempChunkSize = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);

            // Burn through non-data chunks
            if (tempChunkName.equals("data") == false) {
               _fileStream.read(byteArray = new byte[tempChunkSize]);
            }
         }
         // End of header
      }

      public int read() throws IOException {
         return _fileStream.read();
      }
   }

   public MixerStream()  {
      _headerPos = 0;
      _totalRead = 0;

      // A constructed wav header for a 44100hz, 16bit, stereo wav of maximum length
      /*_headerArray = new byte[] {'R','I','F','F', (byte)0xFF, (byte)0xFF, (byte)0xFF, (byte)0xFF,
                                 'W','A','V','E', 'f','m','t', 0x20, 0x10, 0x00, 0x00, 0x00,
                                 0x01, 0x00, 0x02, 0x00, 0x44, (byte)0xAC, 0x00, 0x00,
                                 0x10, (byte)0xB1, 0x02, 0x00, 0x04, 0x00, 0x10, 0x00,
                                 'd','a','t','a', (byte)0xDB, (byte)0xFF, (byte)0xFF, (byte)0xFF};*/

      // A constructed wav header for a 4410hz, 8bit, mono wav of maximum length
      _headerArray = new int[] {'R','I','F','F', 0xFF, 0xFF, 0xFF, 0xFF,
            'W','A','V','E', 'f','m','t', 0x20, 0x10, 0x00, 0x00, 0x00,
            0x01, 0x00, 0x01, 0x00, 0x44, 0xAC, 0x00, 0x00,
            0x44, 0xAC, 0x00, 0x00, 0x01, 0x00, 0x08, 0x00,
            'd','a','t','a', 0xDB, 0xFF, 0xFF, 0xFF};

      _streamVector = new Vector();
   }

   // MixerStream will first present a wav header (as documented above), then will mix data
   // from all added streams.  If there aren't any streams, it will block.
   public int read() throws IOException {
      // Increase the total count of bytes read
      _totalRead++;

      // First present header
      if (_headerPos < _headerArray.length) {
         return _headerArray[_headerPos++];
      }
      else {
         // Mix any streams together
         if (_streamVector.size() > 0) {
            return getMixedByte();
         }
         else {
            // If no streams are available, normally block.
            // Return 0 during the prefetch process (58000 bytes) so prefetch
            // doesn't block forever
            try {
               if (_totalRead < 58001) return 0;

               synchronized(_streamVector) {
                  _streamVector.wait();
               }

               return getMixedByte();
            }
            catch (Exception e) {
               return 0;
            }
         }
      }
   }

   // This method will mix all the available streams together, keep track of current
   // playback byte position to accommodate different PCM formats, and normalize the
   // sound mixing.
   private int getMixedByte() throws IOException {
      int tempRead;
      int tempValue;

      tempValue = 0;

      // Loop through each stream
      for (int lcv = 0 ; lcv < _streamVector.size() ; lcv++) {
         tempRead = ((WaveStream)_streamVector.elementAt(lcv)).read();

         // If we're at the end of the stream, remove it.  Otherwise, add it
         if (tempRead == -1) {
            _streamVector.removeElementAt(lcv--);
         }
         else {
            tempValue += tempRead;
         }
      }

      // Normalize
      if (_streamVector.size() > 0) {
         tempValue /= _streamVector.size();
         return tempValue;
      }
      else {
         return 0;
      }
   }

   // Queue an audio stream to be mixed by MixerStream
   public int addStream(InputStream passStream) throws IOException {
      WaveStream tempStream;

      tempStream = new WaveStream(passStream);

      // Make sure this WaveStream is compatible with MixerStream
      // Check for sample rates
      if (tempStream.getSampleRate() != 11025 && tempStream.getSampleRate() != 22050 && tempStream.getSampleRate() != 44100) {
         throw new IOException("Sample rate not supported.  (11025, 22050, 44100)");
      }

      // Check for bitrates
      if (tempStream.getBitRate() != 8 && tempStream.getBitRate() != 16) {
         throw new IOException("Bit rate not supported.  (8, 16)");
      }

      // Check for channels
      if (tempStream.getChannels() != 1 && tempStream.getChannels() != 1) {
         throw new IOException("Number of channels not supported.  (1, 2)");
      }

      // Wave Stream checks out, let's add it to the list of streams to mix
      _streamVector.addElement(tempStream);

      try {
         // Notify the read method that there's data now if it's currently blocked
         synchronized(_streamVector) {
            _streamVector.notify();
         }
      }
      catch (Exception e) {
         Dialog.inform("Error notifying _streamVector");
      }
      return 0;
   }
}

Blackberry – Mixing 2 or More Sounds Together (Concept Video)

September 02nd, 2010 Comments(3)

An Audio Issue

One thing that I (and many others judging by forum posts) have run into is the fact that the Blackberry isn’t very good at mixing more than one sound together. You’ll be listening to music, or playing a game with music, and suddenly you’ll get a text message, and it will simply terminate the current sound and play the new one. If you’ve got a friendly application, it will restart/resume the original audio, but it’s a jarring audio experience.

What I’m about to post won’t fix that. However – developers also run into the issue when they want to mix sounds together in their applications and games. Play 2 or more sound effects simultaneously and/or while music is playing. I’ve seen official RIM developers comment that certain devices can achieve two simultaneous sounds by instantiating the Player class twice – but from what I gather, this is on GSM only phones – and still limited to two. On CDMA devices, you’re completely out of luck.

The Cheap and Quick Workaround

In my 6 part tutorial on writing a Blackberry game, in the audio article, I mentioned how the Alert.startAudio method can be used for simple sound effects (tone/duration pairs) that will play simultaneously while your midi or mp3 music plays in the background. For many applications, this is enough if you don’t need sophisticated sound effects.

The Start of a Mixer

I’m guessing that the limitation is imposed by the audio chipset/DAC inside of CDMA devices, and perhaps RIM realizes the CPU time involved in a software based audio mixer would slow the phone down too much. However, I think the software option should be there, and developers can use it if they’d like – they might not need a large amount of CPU, or might be dealing with low quality sound files – there are a few scenarios where a software mixer would just be a nice options.

Tonight I sat down and wrote a quick and dirty proof of concept application showing a PCM audio mixer in action. I loaded in 3 audio 44.1khz 8bit mono files and mixed them together in real time. I didn’t normalize the audio at all, so its a little soft, but the code works, and could be expanded on quite a bit to make a full featured mixer. Maybe I’ll run into problems down the road that RIM already has, but it’ll be interesting nonetheless.

Here is the video of the test:

« Previous Page — Next Page »