Re: [Csnd-dev] Status of Oboe

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: [Csnd-dev] Status of Oboe

Michael Gogins-2
I have gotten the Csound for Android app to do duplex audio using the
Oboe library. I have pushed the code to the feature/oboe branch of
https://github.com/csound/csound. The code is all in one file,
https://github.com/csound/csound/blob/feature/oboe/android/CsoundAndroid/jni/csound_oboe.hpp

Your advice was very helpful. Thank you very much.

The Csound repository has been reorganized, and the Csound for Android
app has actually moved to https://github.com/gogins/csound-extended
along with some other extensions of Csound that I will be maintaining.
However, the main Csound repository feature/oboe branch does build and
run for me. I will now merge these changes into csound-extended and
get everything building there.

I will let you know when the new version of the Csound for Android app
with duplex Oboe audio is available on the Google Play store. There is
no doubt more work to be done.

So far, it looks like Oboe is a big help towards implementing decent
audio on Android, not only the lower latency on Oreo, but also the
increased simplicity on all SDK levels.

Best,
Mike

-----------------------------------------------------
Michael Gogins
Irreducible Productions
http://michaelgogins.tumblr.com
Michael dot Gogins at gmail dot com


On Wed, Jan 24, 2018 at 5:21 PM, Michael Gogins
<[hidden email]> wrote:

> Yes, I can do that in my own code.
>
> Thanks,
> Mike
>
> -----------------------------------------------------
> Michael Gogins
> Irreducible Productions
> http://michaelgogins.tumblr.com
> Michael dot Gogins at gmail dot com
>
>
> On Wed, Jan 24, 2018 at 5:13 PM, Phil Burk <[hidden email]> wrote:
>> I don't think that is guaranteed.
>>
>> For MMAP mode, probably. But for AAudio Legacy mode and OpenSL ES, if the
>> underlying code is doing sample rate conversion, or is dealing with
>> converting between odd buffer sizes, then you could get back-to-back
>> callbacks for just input or output.
>>
>> The code I am adding will put a FIFO between the input and output to smooth
>> out some of these rough edges.
>>
>>
>> On Wed, Jan 24, 2018 at 10:46 AM, Michael Gogins <[hidden email]>
>> wrote:
>>>
>>> Are the input stream callbacks synchronized with the output stream
>>> callbacks?
>>>
>>> In other words, is it always the case that Oboe calls:
>>>
>>> inputstreamcallback
>>> outputstreamcallback
>>> inputstreamcallback
>>> outputstreamcallback
>>> ...
>>>
>>> Or can one fall behind or get ahead of the other?
>>>
>>>
>>>
>>> -----------------------------------------------------
>>> Michael Gogins
>>> Irreducible Productions
>>> http://michaelgogins.tumblr.com
>>> Michael dot Gogins at gmail dot com
>>>
>>>
>>> On Wed, Jan 24, 2018 at 12:38 PM, Phil Burk <[hidden email]> wrote:
>>> > The channel handling is a bit awkward.  Channel support, particularly
>>> > for
>>> > input, varies by device. I am planning to add code to automatically
>>> > convert
>>> > one channel to two within Oboe and AAudio.
>>> >
>>> >> The Oboe documentation is quite unclear. I read it as starting one
>>> >> output callback, then using a blocking read from within that callback
>>> >> of the input stream.
>>> >
>>> >
>>> > That should be a non-blocking read.  But read() is not working yet in
>>> > Oboe.
>>> > Only callback works. I am working on the read() and write() code now.
>>> >
>>> >> However, I will do as you suggest, which I now
>>> >> understand means starting both streams, using one callback for both
>>> >> streams, doing a blocking read when the callback is for input, and
>>> >> doing what I now do when the callback is for output.
>>> >
>>> >
>>> > I'm not sure I understand that. For now, in Oboe, use two callbacks.
>>> > You'll
>>> > have to figure out how to get data from one to the other.
>>> >
>>> > Phil Burk
>>> >
>>> >
>>> >
>>> > On Tue, Jan 23, 2018 at 6:25 PM, Michael Gogins
>>> > <[hidden email]>
>>> > wrote:
>>> >>
>>> >> In Csound, input and output can each have a different number of
>>> >> channels. The Samsung Note 8 appears only to support 1 channel of
>>> >> input but I am generating 2 channels of output. So the setup and
>>> >> callback have to handle cases where the input has one number of
>>> >> channels and the output has a different number of channels.
>>> >>
>>> >> The Oboe documentation is quite unclear. I read it as starting one
>>> >> output callback, then using a blocking read from within that callback
>>> >> of the input stream. However, I will do as you suggest, which I now
>>> >> understand means starting both streams, using one callback for both
>>> >> streams, doing a blocking read when the callback is for input, and
>>> >> doing what I now do when the callback is for output.
>>> >>
>>> >> I will verify just exactly what is in spout when the Csound scaling
>>> >> parameter 0dbfs is set to 1. Then I can rescale by your instructions.
>>> >>
>>> >> Thanks for your prompt response,
>>> >> Mike
>>> >>
>>> >>
>>> >>
>>> >> -----------------------------------------------------
>>> >> Michael Gogins
>>> >> Irreducible Productions
>>> >> http://michaelgogins.tumblr.com
>>> >> Michael dot Gogins at gmail dot com
>>> >>
>>> >>
>>> >> On Tue, Jan 23, 2018 at 5:16 PM, Phil Burk <[hidden email]> wrote:
>>> >> > Hello Michael,
>>> >> >
>>> >> > When copying from spout[] to the float or short buffer, one of those
>>> >> > will
>>> >> > need scaling by 32768 or 32767 depending on how spout is declared.
>>> >> > Float samples are normalized [-1.0, +1.0)
>>> >> >
>>> >> > In general:        shortsample = (int16_t)(floatSample * 32768)
>>> >> >
>>> >> > The input code handling code use output_channel_count. is that
>>> >> > correct?
>>> >> >
>>> >> > You are using the same callback function for both input and output.
>>> >> > So
>>> >> > that
>>> >> > code is executing twice!
>>> >> > The input and output callbacks are not combined.
>>> >> > Either use two callbacks or check direction.
>>> >> >
>>> >> > if (oboeStream->getDirection() == oboe::Direction::Input) {
>>> >> >     // handle input
>>> >> > } else {
>>> >> >     // handle output
>>> >> > }
>>> >> >
>>> >> > @Don, can we make the Oboe docs more clear? Some systems use float
>>> >> > samples
>>> >> > that range between -32768.0 and +32768.0.
>>> >> >
>>> >> > Phil Burk
>>> >> >
>>> >> >
>>> >> > On Tue, Jan 23, 2018 at 12:38 PM, Michael Gogins
>>> >> > <[hidden email]>
>>> >> > wrote:
>>> >> >>
>>> >> >> Sorry for the premature send...
>>> >> >>
>>> >> >> I have heard the distortion in Drone IV. This is just the signal
>>> >> >> being
>>> >> >> too loud. Not sure why it's a problem with Oboe but no OpenSLES but
>>> >> >> it's probably very simple. If you reduce the signal level in the
>>> >> >> Csound code it sounds fine. Probably I got the level normalization
>>> >> >> wrong and I will fix that.
>>> >> >>
>>> >> >> Regarding the Lindenmayer example, the "unknown tag" messages are
>>> >> >> warnings only. The syntax errors are from JavaScript and can be
>>> >> >> ignored. The level here also is too loud, but if you reduce it the
>>> >> >> piece sounds OK. I admit this is sloppy and I will fix this in the
>>> >> >> next release. You have to first play the piece, then click on the
>>> >> >> generate button to create the score and actually begin performance.
>>> >> >>
>>> >> >> About duplex, I coded it as you said but I don't hear it working,
>>> >> >> though there are no error messages. I don't hear anything even if I
>>> >> >> monkey with the levels. With the OpenSLES driver I do hear it.
>>> >> >>
>>> >> >> I have included the setup code and the audio callback code here in
>>> >> >> case you see something obviously wrong. Keep in mind the Csound
>>> >> >> audio
>>> >> >> samples are floating point numbers.
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Mike
>>> >> >>
>>> >> >> Setup:
>>> >> >>
>>> >> >>     virtual int Start()
>>> >> >>     {
>>> >> >>         Message("CsoundOboe::Start...\n");
>>> >> >>         // If and only if -odac, enable host-implemented audio.
>>> >> >>         std::string output_name = GetOutputName();
>>> >> >>         auto position = output_name.find("dac");
>>> >> >>         int csound_result = 0;
>>> >> >>         if (position != std::string::npos) {
>>> >> >>             SetHostImplementedAudioIO(1, 0);
>>> >> >>             csound_result = Csound::Start();
>>> >> >>             if (csound_result != 0) {
>>> >> >>                 Message("Csound::Start returned: %d.\n",
>>> >> >> csound_result);
>>> >> >>                 return csound_result;
>>> >> >>             }
>>> >> >>             oboe::Result result;
>>> >> >>             // Create the oboe input and output streams.
>>> >> >>             audio_stream_builder.setCallback(this);
>>> >> >>
>>> >> >> audio_stream_builder.setSharingMode(oboe::SharingMode::Exclusive);
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> audio_stream_builder.setPerformanceMode(oboe::PerformanceMode::LowLatency);
>>> >> >>             audio_stream_builder.setSampleRate(GetSr());
>>> >> >>             frames_per_kperiod = GetKsmps();
>>> >> >>
>>> >> >> audio_stream_builder.setFramesPerCallback(frames_per_kperiod);
>>> >> >>             if (input_channel_count > 0) {
>>> >> >>                 spin = GetSpin();
>>> >> >>                 input_channel_count = GetNchnlsInput();
>>> >> >>                 spin_size = sizeof(MYFLT) * frames_per_kperiod *
>>> >> >> input_channel_count;
>>> >> >>
>>> >> >> audio_stream_builder.setChannelCount(input_channel_count);
>>> >> >>
>>> >> >> audio_stream_builder.setDirection(oboe::Direction::Input);
>>> >> >>                 result =
>>> >> >> audio_stream_builder.openStream(&audio_stream_in);
>>> >> >>                 if (result != oboe::Result::OK){
>>> >> >>                     Message("CsoundOboe::Start: Failed to create
>>> >> >> Oboe
>>> >> >> input stream stream. Error: %s.\n", oboe::convertToText(result));
>>> >> >>                     return -1;
>>> >> >>                 }
>>> >> >>                 // We assume that Oboe's input format is always the
>>> >> >> same
>>> >> >> as
>>> >> >>                 // its output format! But input and output may open
>>> >> >> without
>>> >> >>                 // the other.
>>> >> >>                 oboe_audio_format = audio_stream_in->getFormat();
>>> >> >>                 Message("CsoundOboe::Start: Audio input stream
>>> >> >> format
>>> >> >> is: %s.\n", oboe::convertToText(oboe_audio_format));
>>> >> >>                 Message("CsoundOboe::Start: Starting Oboe audio
>>> >> >> input
>>> >> >> stream...\n");
>>> >> >>                 // Is this correct?
>>> >> >>                 audio_stream_in->start();
>>> >> >>             }
>>> >> >>             spout = GetSpout();
>>> >> >>             output_channel_count = GetNchnls();
>>> >> >>             spout_size = sizeof(MYFLT) * frames_per_kperiod *
>>> >> >> output_channel_count;
>>> >> >>
>>> >> >> audio_stream_builder.setChannelCount(output_channel_count);
>>> >> >>
>>> >> >> audio_stream_builder.setDirection(oboe::Direction::Output);
>>> >> >>             result =
>>> >> >> audio_stream_builder.openStream(&audio_stream_out);
>>> >> >>             if (result != oboe::Result::OK){
>>> >> >>                 Message("CsoundOboe::Start: Failed to create Oboe
>>> >> >> output stream stream. Error: %s.\n", oboe::convertToText(result));
>>> >> >>                 return -1;
>>> >> >>             }
>>> >> >>             // Start oboe.
>>> >> >>             oboe_audio_format = audio_stream_out->getFormat();
>>> >> >>             Message("CsoundOboe::Start: Audio output stream format
>>> >> >> is:
>>> >> >> %s.\n", oboe::convertToText(oboe_audio_format));
>>> >> >>             Message("CsoundOboe::Start: Starting Oboe audio output
>>> >> >> stream...\n");
>>> >> >>             audio_stream_out->requestStart();
>>> >> >>         } else {
>>> >> >>             csound_result = Csound::Start();
>>> >> >>             if (csound_result != 0) {
>>> >> >>                 Message("Csound::Start returned: %d.\n",
>>> >> >> csound_result);
>>> >> >>                 return csound_result;
>>> >> >>             }
>>> >> >>         }
>>> >> >>         // Start Csound.
>>> >> >>         is_playing = true;
>>> >> >>         return 0;
>>> >> >>     }
>>> >> >>
>>> >> >> Callback:
>>> >> >>
>>> >> >>     /**
>>> >> >>      * This is the Oboe audio callback. It fires when the audio
>>> >> >> output
>>> >> >> stream
>>> >> >>      * needs audio data, i.e. once per kperiod. The audio _input_
>>> >> >> stream
>>> >> >> is
>>> >> >>      * read in a blocking fashion.
>>> >> >>      */
>>> >> >>     oboe::DataCallbackResult onAudioReady(oboe::AudioStream
>>> >> >> *oboeStream,
>>> >> >>             void *audio_data,
>>> >> >>             int32_t frame_count)
>>> >> >>     {
>>> >> >>         int csound_result = 0;
>>> >> >>         int frames_read = 0;
>>> >> >>         if (input_channel_count > 0 && audio_stream_in) {
>>> >> >>             frames_read = audio_stream_in->read(audio_data,
>>> >> >> frame_count, timeout_nanoseconds);
>>> >> >>             // If ksmps input audio frames have not yet been
>>> >> >> consumed,
>>> >> >> tell the
>>> >> >>             // Oboe driver to continue.
>>> >> >>             if (frames_read == frame_count) {
>>> >> >>                 return oboe::DataCallbackResult::Continue;
>>> >> >>             }
>>> >> >>             // Otherwise, copy the input stream audio to Csound.
>>> >> >>             // The Oboe's audio sample format may differ by
>>> >> >> platform.
>>> >> >>             if (frames_read >= 0) {
>>> >> >>                 if (oboe_audio_format == oboe::AudioFormat::Float){
>>> >> >>                     float_buffer = static_cast<float *>(audio_data);
>>> >> >>                     for (int i = 0; i < frames_per_kperiod; i++) {
>>> >> >>                         for (int j = 0; j < output_channel_count;
>>> >> >> j++)
>>> >> >> {
>>> >> >>                             spin[i * output_channel_count + j] =
>>> >> >> float_buffer[i * output_channel_count + j];
>>> >> >>                         }
>>> >> >>                     }
>>> >> >>                 } else {
>>> >> >>                     short_buffer = static_cast<int16_t
>>> >> >> *>(audio_data);
>>> >> >>                     for (int i = 0; i < frames_per_kperiod; i++) {
>>> >> >>                         for (int j = 0; j < output_channel_count;
>>> >> >> j++)
>>> >> >> {
>>> >> >>                             spin[i * output_channel_count + j] =
>>> >> >> short_buffer[i * output_channel_count + j];
>>> >> >>                         }
>>> >> >>                     }
>>> >> >>                 }
>>> >> >>             }
>>> >> >>         }
>>> >> >>         // Consume one kperiod of audio input from spin,
>>> >> >>         // and produce one kperiod of audio output to spout.
>>> >> >>         csound_result = PerformKsmps();
>>> >> >>         // If the Csound performance has finished, tell the Oboe
>>> >> >> driver
>>> >> >>         // to stop.
>>> >> >>         if (csound_result) {
>>> >> >>             is_playing = false;
>>> >> >>             return oboe::DataCallbackResult::Stop;
>>> >> >>         }
>>> >> >>         // Otherwise, copy the Csound output audio to the Oboe
>>> >> >> output
>>> >> >>         // buffer. Oboe's audio sample format may differ by
>>> >> >> platform.
>>> >> >>         if (oboe_audio_format == oboe::AudioFormat::Float){
>>> >> >>             float_buffer = static_cast<float *>(audio_data);
>>> >> >>             for (int i = 0; i < frames_per_kperiod; i++) {
>>> >> >>                 for (int j = 0; j < output_channel_count; j++) {
>>> >> >>                     float_buffer[i * output_channel_count + j] =
>>> >> >> spout[i * output_channel_count + j];
>>> >> >>                 }
>>> >> >>             }
>>> >> >>         } else {
>>> >> >>             short_buffer = static_cast<int16_t *>(audio_data);
>>> >> >>             for (int i = 0; i < frames_per_kperiod; i++) {
>>> >> >>                 for (int j = 0; j < output_channel_count; j++) {
>>> >> >>                     short_buffer[i * output_channel_count + j] =
>>> >> >> spout[i * output_channel_count + j];
>>> >> >>                 }
>>> >> >>             }
>>> >> >>         }
>>> >> >>         return oboe::DataCallbackResult::Continue;
>>> >> >>     }
>>> >> >>
>>> >> >>
>>> >> >> -----------------------------------------------------
>>> >> >> Michael Gogins
>>> >> >> Irreducible Productions
>>> >> >> http://michaelgogins.tumblr.com
>>> >> >> Michael dot Gogins at gmail dot com
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> -----------------------------------------------------
>>> >> >> Michael Gogins
>>> >> >> Irreducible Productions
>>> >> >> http://michaelgogins.tumblr.com
>>> >> >> Michael dot Gogins at gmail dot com
>>> >> >>
>>> >> >>
>>> >> >> On Tue, Jan 23, 2018 at 1:31 PM, Phil Burk <[hidden email]>
>>> >> >> wrote:
>>> >> >> > Hello Michael,
>>> >> >> >
>>> >> >> >> What is the current status of the Oboe library?
>>> >> >> >
>>> >> >> > Oboe is in active development.
>>> >> >> >
>>> >> >> > We recommend that when doing full duplex that the output stream
>>> >> >> > should
>>> >> >> > use a
>>> >> >> > callback and the input should use non-blocking read().  I am
>>> >> >> > working
>>> >> >> > on
>>> >> >> > the
>>> >> >> > non-callback read/write mode now. That is a high priority for us.
>>> >> >> >
>>> >> >> > I tried CSound with Oboe. Thanks so much for trying out Oboe.
>>> >> >> > That
>>> >> >> > will
>>> >> >> > really be helpful for us for testing the new OS versions.
>>> >> >> >
>>> >> >> > I'm running on OC-MR1, which has support for the MMAP mode in
>>> >> >> > AAudio.
>>> >> >> > I was able to run "Chime Pad" with Oboe. It sounded good and
>>> >> >> > responded
>>> >> >> > quickly to slider changes.
>>> >> >> >
>>> >> >> > I did run into a few problems. I get lots of syntax errors when I
>>> >> >> > try
>>> >> >> > to
>>> >> >> > run
>>> >> >> > the Lindenmayer example.
>>> >> >> > The first time I got "missing ) parentheses" errors.  Now I get
>>> >> >> > lots
>>> >> >> > of
>>> >> >> > errors like:
>>> >> >> >
>>> >> >> > unknown CSD tag: <html>
>>> >> >> >
>>> >> >> > It looks like a code file got converted to HTML.
>>> >> >> >
>>> >> >> > ------------------------------------
>>> >> >> > Also, I am getting major distortion using Oboe
>>> >> >> > To reproduce:
>>> >> >> >
>>> >> >> > plug in headphones
>>> >> >> > launch CSound
>>> >> >> > load example Drone IV or Xanadu
>>> >> >> > press Start button
>>> >> >> > turn up sliders
>>> >> >> > turn up volume until you hear a gentle drone
>>> >> >> > press Stop button
>>> >> >> > open Settings
>>> >> >> > select Oboe Audio Driver
>>> >> >> > IMPORTANT Leave headphones plugged in but remove them from ears to
>>> >> >> > be
>>> >> >> > safe!
>>> >> >> > IMPORTANT
>>> >> >> > Did you remove your headphones?
>>> >> >> > press Start button
>>> >> >> >
>>> >> >> > Expect: same gentle drone
>>> >> >> >
>>> >> >> > Actual: deafening distorted roar
>>> >> >> >
>>> >> >> >
>>> >> >> > ---------------------------------
>>> >> >> > I also get a crash if I change Audio Driver without stopping. To
>>> >> >> > reproduce:
>>> >> >> >
>>> >> >> > plug in headphones
>>> >> >> > launch CSound
>>> >> >> > load example Drone IV
>>> >> >> > press Start button
>>> >> >> > turn up sliders
>>> >> >> > turn up volume until you hear a gentle drone
>>> >> >> > open Settings
>>> >> >> > select Oboe Audio Driver
>>> >> >> > press Stop button
>>> >> >> >
>>> >> >> > I suspect there are bugs in CSound, Oboe and AAudio. I am happy to
>>> >> >> > work
>>> >> >> > with
>>> >> >> > you to fix all of these.
>>> >> >> >
>>> >> >> > Thanks,
>>> >> >> > Phil Burk
>>> >> >> >
>>> >> >> >
>>> >> >> > On Tue, Jan 23, 2018 at 8:02 AM, Michael Gogins
>>> >> >> > <[hidden email]>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> What is the current status of the Oboe library? Is it now
>>> >> >> >> possible
>>> >> >> >> to
>>> >> >> >> run duplex audio (input stream synchronized with output stream)?
>>> >> >> >> If
>>> >> >> >> so, I suspect the documentation needs updating; how should this
>>> >> >> >> be
>>> >> >> >> used (if indeed it is now implemented)?
>>> >> >> >>
>>> >> >> >> Also, did either of you try running the Csound for Android app
>>> >> >> >> for
>>> >> >> >> which I sent you the link with the Oboe driver enabled? The
>>> >> >> >> version
>>> >> >> >> of
>>> >> >> >> the app now in the Google Play Store also can run the Oboe driver
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> (https://play.google.com/store/apps/details?id=com.csounds.Csound6&hl=en).
>>> >> >> >> I will be updating this in the next day or week.
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> Mike
>>> >> >> >>
>>> >> >> >> -----------------------------------------------------
>>> >> >> >> Michael Gogins
>>> >> >> >> Irreducible Productions
>>> >> >> >> http://michaelgogins.tumblr.com
>>> >> >> >> Michael dot Gogins at gmail dot com
>>> >> >> >
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>