Evan Laforge | 2 Sep 21:50 2008
Picon

merging midi streams

I brought this up a while back but didn't pursue it too far because I
foresaw the need for this, but hadn't actually arrived at the point
where I did need it.  So now I've arrived there, and I'm thinking
about it more seriously.

Basically, what I want is the ability to merge timestamped midi
streams.  Pm_Write insists that the timestamps be non-decreasing, and
merged streams would mean I could have two non-decreasing streams of
msgs.  Clearly this has to be done at the driver level, since this is
where the timing is handled, so I have to do some spelunking into
portmidi.  Here are some possibilities I came up with:

1- Make a non-portable extension to portmidi that provides a new
function, like "WriteThru".  Unfortunately I can't re-use any of the
portmidi implementation (I'm on a mac so it would be
pm_mac/pmmacosxcm.c) since everything is static, so I'd basically be
writing another non-portable MIDI interface, at which point why use
portmidi at all?

2- Do like the above, but modify portmidi itself.  I could either do
something like Pm_WriteThru, or an explicit merge like
Pm_OpenMergedOutput(PortMidiStream **new_stream, PortMidiStream
*merge_to_stream).  Since I only know how to implement this for Core
MIDI, it would have #error "not implemented" for the other platforms
until someone added them.  It's kinda complicated but it is explicit.

3- Modify portmidi to provide merged streams implicitly for 0
timestamp events: Pm_Write works normally with non-decreasing
timestamps, but 0 timestamps go out immediately.  This is both the
simplest to implement (and it's what I've already done) and I think is
(Continue reading)

Evan Laforge | 3 Sep 00:38 2008
Picon

Re: merging midi streams

[ re-adding media_api, I assume you meant to reply-all? ]

On Tue, Sep 2, 2008 at 2:23 PM, Roger Dannenberg <rbd@...> wrote:
> Evan,
>   Merging MIDI streams with timestamps is a difficult problem, at least from
> an API point of view. As far as I know, Windows MIDI drivers and the MME API
> do not support merging, and since a goal of PortMidi is to run on multiple
> platforms, there is no built-in merging capability. One thing you can do is

Well, it's pretty easy on platforms that support merging, though I
certainly appreciate the desire to be cross platform (well, support
windows mme really).  Actually, I heard some noise about vista wanting
to be more reasonable about audio stuff so maybe they have a driver
now that does that.  Not that many people use vista...

It seems like kind of a shame though to have to write a whole separate
thread in C and figure out system specific priority and timing stuff
when this is exactly what is already provided by the driver...
especially if it's just to support windows mme, which I have no plans
of ever supporting anyway.  I notice my windows sequencer (cubase)
uses DirectMusic and seems to consider mme legacy anyway (dunno if
DirectMusic provides merging).  I guess if someone really cared they
would have written a DirectMusic backend for portmidi though.

I guess you would consider it beyond the portmidi mandate to use the
OS's scheduler for the core midi, alsa, and maybe vista-whatever, and
fall back to creating a high prio thread to schedule msgs in windows
mme?  This is after all the system specific hackery that portmidi is
supposed to be saving me from.

(Continue reading)

Roger Dannenberg | 3 Sep 15:12 2008
Picon

Re: merging midi streams

Thanks for your input. There are so many questions, I'm resending the discussion and adding more to it below...

Evan Laforge wrote:
[ re-adding media_api, I assume you meant to reply-all? ] On Tue, Sep 2, 2008 at 2:23 PM, Roger Dannenberg <rbd-ETDLCGt7PQU3uPMLIKxrzw@public.gmane.org> wrote:
Evan, Merging MIDI streams with timestamps is a difficult problem, at least from an API point of view. As far as I know, Windows MIDI drivers and the MME API do not support merging, and since a goal of PortMidi is to run on multiple platforms, there is no built-in merging capability. One thing you can do is
Well, it's pretty easy on platforms that support merging, though I certainly appreciate the desire to be cross platform (well, support windows mme really). Actually, I heard some noise about vista wanting to be more reasonable about audio stuff so maybe they have a driver now that does that. Not that many people use vista... It seems like kind of a shame though to have to write a whole separate thread in C and figure out system specific priority and timing stuff when this is exactly what is already provided by the driver... especially if it's just to support windows mme, which I have no plans of ever supporting anyway. I notice my windows sequencer (cubase) uses DirectMusic and seems to consider mme legacy anyway (dunno if DirectMusic provides merging). I guess if someone really cared they would have written a DirectMusic backend for portmidi though.
I don't think DirectMusic provides any advantage over MME, but I'd be happy to know if I'm wrong. There haven't been any requests for a DirectMusic version of PortMidi that I know of.

There's an example, pm_test/midithru.c, that uses porttime to set up a high-priority thread and provide MIDI THRU along with communications to a non-real-time thread. This was written when I took out some support for MIDI THRU that turned out to have some synchronization problems.

After thinking about this a bit more, I'm surprised that a merging Pm_Write() would actually help you. With PortMidi, you need to actively request/poll for incoming MIDI, so to provide a MIDI thru capability, you need to quickly read any incoming message and send it back out. That implies that you have a high-priority thread running to poll for MIDI, but if you have that, you probably don't need timestamped messages and merging.

It was my impression that every sequencer uses a user-level thread to do MIDI scheduling and merging. In particular, most sequencers have some options on THRU, e.g. filter certain channels, and unless everything the sequencer wants to offer is supported by some underlying THRU API, the application has to do the work.
I guess you would consider it beyond the portmidi mandate to use the OS's scheduler for the core midi, alsa, and maybe vista-whatever, and fall back to creating a high prio thread to schedule msgs in windows mme? This is after all the system specific hackery that portmidi is supposed to be saving me from.
You make a good point. The counter-argument is that a portmidi user should be able to get the same quality of accurate timing as someone going directly to the OS-specific API. If we "extended" the Win MME API by putting the scheduling in user-land, this would make it impossible to get timing at the driver level. (Or at least there would be a confusing API offering two implementations.)
In short, timed messages are good when sent by a process that is doing other things and does not have good timing accuracy, but if you are doing anything interactive, you are probably better off just running a high priority thread to do MIDI processing with "zero" latency and no timestamps.
Well, no process is going to have good timing accuracy on a multitasking system. The only thing that will is a high-prio scheduler thread, whether that be implemented by the application, as a small loop in a separate thread, or by the OS driver, probably also as a small loop in a separate thread. So it seems to me that *all* apps fall into the "doing other things" camp, unless it really is just a midi file player in which case the whole app *is* the high-prio scheduler thread.
[Sorry, I was using "process" in a more generic sense. In this context, I should have said "timed messages are good when sent by a thread that is doing other things and does not have good timing accuracy...".]
I would not be violently opposed to adding a Pm_WriteThru() call that sends messages immediately, in front of pending timestamped messages, but this would not work for Windows unless we moved all the timestamp processing into the application space, and I don't think that is a good idea. (Correct me if I'm wrong about Windows -- I can't say that I've tried this, so maybe I'm not interpreting the spec correctly.)
Wait, weren't you just advocating moving all the timestamp processing into application space for all platforms (well, except non-interactive "freeze while playing music" type ones I guess)? Wouldn't it be strictly better to do the timestamp processing in the app for the single relatively primitive driver that doesn't support it, and in the driver for the rest of the drivers that can handle it?
I think the choices are: (1) retain the potential for driver-based timing in Windows and possibly add a non-portable call like Pm_WriteThru for non-windows users, or (2) than take away the best-possible timing for Windows and make fully cross-platform API that supports merging (even though most applications that need merging will probably do their own merging anyway)

I stated these options in a biased way to indicate the reasons why (1) seems preferable to me. I'm not  enthusiastic about adding a non-portable WriteThru call, but it seems pretty harmless and better than encouraging you to maintain a separate version.
Anyway, I'll probably continue with the patch I have for a while (portmidi is not exactly actively developing so it won't get out of date), and then maybe send a patch for Pm_WriteThru once I have some more experience with this. I still like the way I have it because it requires no special support from the app, but a separate function won't be a big burden either. Here's a more minor issue: Is there a particular reason Pm_Abort specifies that the ports will have to be all reopened after use? This makes it awkward for me to use. The CoreMIDI equivalent MIDIFlushOutput doesn't have that requirement, and it looks like nor does alsa_abort, though it's not even implemented in portmidi. It's actually silently not implemented for CoreMIDI either, though doing so would be trivial. From looking at the windows docs, it looks like midiStreamStop doesn't reset any connections either. So how about if I implement abort for CoreMIDI, and remove the stuff about "The caller should immediately close the output port"? If I make it send EOX if a sysex is in progress, I can probably remove the scary warning about partial midi messages too. Apparently no one is using Pm_Abort anyway, except maybe on windows. And hey, if abort has been not implemented on the majority of the platforms for all this time, what's wrong with midi merge being not implemented on one platform?
Well, abort should be implemented everywhere. I wasn't aware that it wasn't, but it's not the most useful function since it risks leaving note-on's, sending corrupt sysex messages, etc. I think it's only useful if the latency is very high and you don't want to wait for pending messages to be sent but instead need to shut down right away. I'm not sure why you would want to leave the midi connection open, since after Pm_Abort, messages may have been dropped, and you can't always know the state of the MIDI receiver.

To answer your question, I think Pm_Abort() could be made to just drop all pending messages but leave the stream in a usable state. The implementations may assume the stream will be closed, so I'd have to check very carefully that, e.g., the Windows stream buffers that get returned after Pm_Abort() are retained and prepared for reuse, etc.

I'm curious how you would use this. E.g. Windows documentation says after midiStopStream() that all on notes are turned off (so maybe they send all-note-off commands on all channels). Would you expect all implementations to do the same kind of clean-up, and what should they do?

_______________________________________________
media_api mailing list
media_api@...
http://lists.create.ucsb.edu/mailman/listinfo/media_api
O. P. Martin | 3 Sep 19:23 2008

Re: PortMidi linking MinGw


Hi, Roger,

How are you?

I apologize for taking so long to get back: I have been ill.

I do not have a makefile.  I used Eclipse + CDT to build PortMidi using 
MinGw.  If you want, I can send the project files that I used for the 
build (see link).

http://www.opmartin.com/bin/portmidi_cdt.zip

The two files .project and .cproject I believe must be in the root of 
the portmidi tree to work with Eclipse.  I have also included the 
resulting binaries if you are interested.  They have been successfully 
tested and are now in use by my project.  I have only compiled the 
relevant sources into one .dll here; I'm not sure if that is what you 
would recommend.

I am not very experienced yet in these matters, so I hope the attached 
files work for you.  Thank you.

There was an option to use Gnu Make files, but I was unable to test it 
because I do not have Gnu Make installed, preferring MSYS instead.

May the Lord bless you,
Philip

Roger Dannenberg wrote:
> That's great. If you'd like to contribute a makefile (or even fix one up
> for the test programs) let me know. -Roger
>
>
>   
Evan Laforge | 5 Sep 04:05 2008
Picon

Re: merging midi streams

> There's an example, pm_test/midithru.c, that uses porttime to set up a
> high-priority thread and provide MIDI THRU along with communications to a
> non-real-time thread. This was written when I took out some support for MIDI
> THRU that turned out to have some synchronization problems.

Ah, I should have looked in there long ago.  I didn't realize that
porttime handled the OS dependent parts of a high-prio thread.  Good
stuff in there.

> After thinking about this a bit more, I'm surprised that a merging
> Pm_Write() would actually help you. With PortMidi, you need to actively
> request/poll for incoming MIDI, so to provide a MIDI thru capability, you
> need to quickly read any incoming message and send it back out. That implies
> that you have a high-priority thread running to poll for MIDI, but if you
> have that, you probably don't need timestamped messages and merging.
>
> It was my impression that every sequencer uses a user-level thread to do
> MIDI scheduling and merging. In particular, most sequencers have some
> options on THRU, e.g. filter certain channels, and unless everything the
> sequencer wants to offer is supported by some underlying THRU API, the
> application has to do the work.

This is not entirely the case for me:  I have a normal thread polling
on Pm_Read that drops the msgs into a queue, which goes straight into
the main event loop (i.e. the queue is multiplexed between midi,
keyboard, mouse, and all the various other async inputs that go into
the event loop).  So midi thru is integrated with all other event
handling, and handled in the same way as any other event producing
midi (which are not only midi-in events).  Initially I thought it
would be too slow and I'd have to do a special case hack for thru, but
it's been fine so far.

So if I eventually did do a special case for thru, I suppose I could
tack it onto the input loop.  I would need to keep a priority buffer
and implement my own scheduling and abort and whatnot.  It might not
be rocket science, but it's all stuff that the OS already provides for
me, so it seems like I shouldn't have to do it.  And maybe I'll never
need the special case thru.

> You make a good point. The counter-argument is that a portmidi user should
> be able to get the same quality of accurate timing as someone going directly
> to the OS-specific API. If we "extended" the Win MME API by putting the
> scheduling in user-land, this would make it impossible to get timing at the
> driver level. (Or at least there would be a confusing API offering two
> implementations.)

You mean they would be tied into whatever timing accuracy portmidi
provided?  Yeah, that's true, and I agree users should be able to get
down to the driver.  Isn't MIDI timing basically one ms though?  Is
someone going to want more than that, given all the other latencies in
the system?  But then it would be an additional complicated detail,
and users couldn't plug their own things into the scheduling loop (not
that users of CoreMIDI or alsa can)... so I dunno.

> I think the choices are: (1) retain the potential for driver-based timing in
> Windows and possibly add a non-portable call like Pm_WriteThru for
> non-windows users, or (2) than take away the best-possible timing for
> Windows and make fully cross-platform API that supports merging (even though
> most applications that need merging will probably do their own merging
> anyway)
>
> I stated these options in a biased way to indicate the reasons why (1) seems
> preferable to me. I'm not  enthusiastic about adding a non-portable
> WriteThru call, but it seems pretty harmless and better than encouraging you
> to maintain a separate version.

If it's really true most apps do their own merging, then yeah, #2
would be silly.  But... aren't they all doing merging because the
underlying driver doesn't support it?  So if the library provided
that, wouldn't they not be doing their own merging anymore?  Granted I
don't have the source for any OS X sequencers on me, but why would any
of them do their own scheduling?  And why couldn't portmidi provide
best-possible timing for windows out of the box?

> Well, abort should be implemented everywhere. I wasn't aware that it wasn't,
> but it's not the most useful function since it risks leaving note-on's,
> sending corrupt sysex messages, etc. I think it's only useful if the latency
> is very high and you don't want to wait for pending messages to be sent but
> instead need to shut down right away. I'm not sure why you would want to
> leave the midi connection open, since after Pm_Abort, messages may have been
> dropped, and you can't always know the state of the MIDI receiver.

Well, you have to clear the state anyway when stopping playback, and
abort doesn't change that.  I just blast AllNotesOff on all channels
since I don't bother keeping track of channel state.  As far as sysex,
that's why I suggested the library sending an EOX if it's in the
middle of a sysex.  The synth will complain about a corrupt sysex but
won't get interminably stuck in some syx receive mode.  Actually, it
probably wouldn't anyway because it would see a high bit in the next
msg... except for running status.  So I guess EOX it is.

And I don't see how closing and reopening the connection after the
abort is supposed to change anything, if we're talking about external
synths getting in a wonky state.  How are they supposed to know you
reopened the connection?  And on CoreMIDI at least, there's really no
such thing as opening or closing an output port, so you couldn't close
the port even if you wanted to.

Anyway, if you are actually using the timestamp feature, and
scheduling notes say 30s or so in advance, it's kind of awkward to hit
stop and have the music roll on until the buffer exhausts itself.

So, I implemented Pm_Abort for OS X, and fixed a bug in the portmidi.c
function while I was at it.  It works for me, and now I don't have the
annoying delay when hitting stop.  The alsa implementation is still
all commented out.  Can I send you a patch directly?  I'll check the
portmidi page for instructions.

> To answer your question, I think Pm_Abort() could be made to just drop all
> pending messages but leave the stream in a usable state. The implementations
> may assume the stream will be closed, so I'd have to check very carefully
> that, e.g., the Windows stream buffers that get returned after Pm_Abort()
> are retained and prepared for reuse, etc.

I can't test the MME version, but the msdn documentation doesn't
mention anything about reopening handles.  Nor does the OS X version,
and I have tested it.  Not sure about alsa, but it's not implemented
anyway.

> I'm curious how you would use this. E.g. Windows documentation says after
> midiStopStream() that all on notes are turned off (so maybe they send
> all-note-off commands on all channels). Would you expect all implementations
> to do the same kind of clean-up, and what should they do?

I'd say leave cleanup to the app.  I do cleanup when stopping anyway
(wouldn't you have to even without abort?).  It's true MME does it for
you, but at least core midi doesn't and it's trivial to do in the app.
 And in my case, I want to do other things, like reset pitch bend.
Roger Dannenberg | 5 Sep 15:47 2008
Picon

Re: merging midi streams

Evan Laforge wrote:
After thinking about this a bit more, I'm surprised that a merging Pm_Write() would actually help you. With PortMidi, you need to actively request/poll for incoming MIDI, so to provide a MIDI thru capability, you need to quickly read any incoming message and send it back out. That implies that you have a high-priority thread running to poll for MIDI, but if you have that, you probably don't need timestamped messages and merging. It was my impression that every sequencer uses a user-level thread to do MIDI scheduling and merging. In particular, most sequencers have some options on THRU, e.g. filter certain channels, and unless everything the sequencer wants to offer is supported by some underlying THRU API, the application has to do the work.
This is not entirely the case for me: I have a normal thread polling on Pm_Read that drops the msgs into a queue, which goes straight into the main event loop (i.e. the queue is multiplexed between midi, keyboard, mouse, and all the various other async inputs that go into the event loop). So midi thru is integrated with all other event handling, and handled in the same way as any other event producing midi (which are not only midi-in events). Initially I thought it would be too slow and I'd have to do a special case hack for thru, but it's been fine so far.
I would not recommend this. You probably get away with it because MIDI is intermittent, so a message is only delayed when the delivery time (or for THRU, when the arrival time) coincides with some heavy computation like a long wait for disk I/O or a graphics update. Even then, it's hard to notice an occasional timing problem in a typical MIDI performance.
So if I eventually did do a special case for thru, I suppose I could tack it onto the input loop. I would need to keep a priority buffer and implement my own scheduling and abort and whatnot. It might not be rocket science, but it's all stuff that the OS already provides for me, so it seems like I shouldn't have to do it. And maybe I'll never need the special case thru.
I think you already have logic to generate MIDI messages in time order, so rather than generate them in advance to yet another data structure and then deliver them from there, how about just generating messages as you need them? This should not add much code at all. In fact, I would be a little concerned about generating 30s of MIDI in advance -- depending on the density of messages, that could be quite a lot of storage for device drivers, PortMidi, etc. If you reduce this to, say, 1s in advance, then you'll need logic in your app to generate data incrementally, and this is exactly the logic you need to eliminate the buffering entirely and just generate each message just-in-time. Once you are generating MIDI output just-in-time, you can easily merge a MIDI input stream.
You make a good point. The counter-argument is that a portmidi user should be able to get the same quality of accurate timing as someone going directly to the OS-specific API. If we "extended" the Win MME API by putting the scheduling in user-land, this would make it impossible to get timing at the driver level. (Or at least there would be a confusing API offering two implementations.)
You mean they would be tied into whatever timing accuracy portmidi provided? Yeah, that's true, and I agree users should be able to get down to the driver. Isn't MIDI timing basically one ms though? Is someone going to want more than that, given all the other latencies in the system? But then it would be an additional complicated detail, and users couldn't plug their own things into the scheduling loop (not that users of CoreMIDI or alsa can)... so I dunno.
To get timing accurate to 1ms from the application level (where PortMidi runs), you must at least wake up the application every 1ms and check for incoming data. Not all systems can do this, and simple applications with one thread handling a GUI, file IO, etc. are unlikely to do this. By passing timestamps down to the device driver (or even a real-time process or server), you can get the most accurate timing possible with the fewest assumptions about the kernel, the system configuration, etc. Of course, if you generate MIDI in advance with timestamps, it poses some limits on interactivity, but developers should have a choice of which way to go.
I think the choices are: (1) retain the potential for driver-based timing in Windows and possibly add a non-portable call like Pm_WriteThru for non-windows users, or (2) than take away the best-possible timing for Windows and make fully cross-platform API that supports merging (even though most applications that need merging will probably do their own merging anyway) I stated these options in a biased way to indicate the reasons why (1) seems preferable to me. I'm not enthusiastic about adding a non-portable WriteThru call, but it seems pretty harmless and better than encouraging you to maintain a separate version.
If it's really true most apps do their own merging, then yeah, #2 would be silly. But... aren't they all doing merging because the underlying driver doesn't support it? So if the library provided that, wouldn't they not be doing their own merging anymore? Granted I don't have the source for any OS X sequencers on me, but why would any of them do their own scheduling? And why couldn't portmidi provide best-possible timing for windows out of the box?
That may be true. I guess merge makes sense if you believe you need device-driver level timestamping for sequencer data, but it's good enough for the application to handle MIDI IN and send some messages to the "front of the line" of the MIDI output stream. I think (at least some) high-end sequencers feel they must implement their own MIDI THRU with low latency and once they have low latency at the application level, they might as well do all their scheduling there, eliminating timestamps on the midi output stream. Sequencers also have to respond to GUI events that might affect the MIDI output. You don't want to undo a bunch of buffered MIDI data and recompute it when the user hits the transpose function or edits a velocity of something that's already in the output stream.

Well, abort should be implemented everywhere. I wasn't aware that it wasn't, but it's not the most useful function since it risks leaving note-on's, sending corrupt sysex messages, etc. I think it's only useful if the latency is very high and you don't want to wait for pending messages to be sent but instead need to shut down right away. I'm not sure why you would want to leave the midi connection open, since after Pm_Abort, messages may have been dropped, and you can't always know the state of the MIDI receiver.
Well, you have to clear the state anyway when stopping playback, and abort doesn't change that. I just blast AllNotesOff on all channels since I don't bother keeping track of channel state. As far as sysex, that's why I suggested the library sending an EOX if it's in the middle of a sysex. The synth will complain about a corrupt sysex but won't get interminably stuck in some syx receive mode. Actually, it probably wouldn't anyway because it would see a high bit in the next msg... except for running status. So I guess EOX it is.
I suspect a lot of devices don't buffer SYSEX data to see if the message is well formed before starting to change internal state, so dropping SYSEX data just because a user stops a sequence seems like bad behavior to me. But this may be something you just have to live with if you are going to precompute stream data and then abort it.
And I don't see how closing and reopening the connection after the abort is supposed to change anything, if we're talking about external synths getting in a wonky state. How are they supposed to know you reopened the connection?
They wouldn't know. The only question is: What does PortMidi have to do internally to make abort work and to keep the port open for more data?
And on CoreMIDI at least, there's really no such thing as opening or closing an output port, so you couldn't close the port even if you wanted to.
So for CoreMIDI, the answer to my question above is "nothing" (that's good).
Anyway, if you are actually using the timestamp feature, and scheduling notes say 30s or so in advance, it's kind of awkward to hit stop and have the music roll on until the buffer exhausts itself. So, I implemented Pm_Abort for OS X, and fixed a bug in the portmidi.c function while I was at it. It works for me, and now I don't have the annoying delay when hitting stop. The alsa implementation is still all commented out. Can I send you a patch directly? I'll check the portmidi page for instructions.
Yes, please just send a patch to me.
To answer your question, I think Pm_Abort() could be made to just drop all pending messages but leave the stream in a usable state. The implementations may assume the stream will be closed, so I'd have to check very carefully that, e.g., the Windows stream buffers that get returned after Pm_Abort() are retained and prepared for reuse, etc.
I can't test the MME version, but the msdn documentation doesn't mention anything about reopening handles. Nor does the OS X version, and I have tested it. Not sure about alsa, but it's not implemented anyway.
I'm curious how you would use this. E.g. Windows documentation says after midiStopStream() that all on notes are turned off (so maybe they send all-note-off commands on all channels). Would you expect all implementations to do the same kind of clean-up, and what should they do?
I'd say leave cleanup to the app. I do cleanup when stopping anyway (wouldn't you have to even without abort?). It's true MME does it for you, but at least core midi doesn't and it's trivial to do in the app. And in my case, I want to do other things, like reset pitch bend.
I agree -- we could say that the underlying system may or may not "clean up" by sending all notes off messages, etc., so a cross-platform application should not make assumptions and explicitly restore the state.

_______________________________________________
media_api mailing list
media_api@...
http://lists.create.ucsb.edu/mailman/listinfo/media_api
Evan Laforge | 7 Sep 04:36 2008
Picon

Re: merging midi streams

Well, phooey, I had a whole response written up and then lost it when I sent
the patch.  I'll try to be more concise this time...

> I would not recommend this. You probably get away with it because MIDI is
> intermittent, so a message is only delayed when the delivery time (or for
> THRU, when the arrival time) coincides with some heavy computation like a
> long wait for disk I/O or a graphics update. Even then, it's hard to notice
> an occasional timing problem in a typical MIDI performance.

So I did some tests with a very high bandwidth midi controller, and going
through the whole event loop does lead to lagginess, but it's due to other
inefficiencies in the event handling.  I patched in a shortcut to handle thru
directly and I can't hear the lag anymore.

> I think you already have logic to generate MIDI messages in time order, so
> rather than generate them in advance to yet another data structure and then
> deliver them from there, how about just generating messages as you need
> them? This should not add much code at all. In fact, I would be a little
> concerned about generating 30s of MIDI in advance -- depending on the

Yeah, msgs are generated incrementally, and 30s is just an arbitrary amount of
time I set the generator to stay ahead of realtime.  I can take it down to 1-2s
if necessary.  I suspect on OS X it's user memory, from the microkernel thing,
so the OS scheduler is just as good a place as any to store the midi, but that
might not be true for alsa.

> density of messages, that could be quite a lot of storage for device
> drivers, PortMidi, etc. If you reduce this to, say, 1s in advance, then
> you'll need logic in your app to generate data incrementally, and this is
> exactly the logic you need to eliminate the buffering entirely and just
> generate each message just-in-time. Once you are generating MIDI output
> just-in-time, you can easily merge a MIDI input stream.

Logic to generate msgs incrementally is not the same as logic to do so in
realtime.  Generation is complicated, and I never know when it's time to go GC
or get paged in by the OS.  The point behind copying the midi into the driver
is that it will do stuff like wire the buffer into memory and run at a high
priority.  Yes, I can also write my own high prio process with wired memory,
but the point of the OS guys putting that in a library is that I don't have to.

> To get timing accurate to 1ms from the application level (where PortMidi
> runs), you must at least wake up the application every 1ms and check for
> incoming data. Not all systems can do this, and simple applications with one
> thread handling a GUI, file IO, etc. are unlikely to do this. By passing

The system in this case is windows, which can do that right?  And if the
library creates the thread, it doesn't matter if the application is
single-threaded.  In fact, putting the thread in the library is what allows the
application to be single threaded.

> That may be true. I guess merge makes sense if you believe you need
> device-driver level timestamping for sequencer data, but it's good enough
> for the application to handle MIDI IN and send some messages to the "front
> of the line" of the MIDI output stream. I think (at least some) high-end
> sequencers feel they must implement their own MIDI THRU with low latency and
> once they have low latency at the application level, they might as well do
> all their scheduling there, eliminating timestamps on the midi output

I don't have the source for, say, Logic handy, but I'll bet it uses CoreMIDI
directly, and not portmidi.  And since it uses CoreMIDI, it has access to the
midi scheduler along with the MIDI thru library.  In fact, I wouldn't be too
shocked if apple designed core midi with logic in mind.  Yes if it wants to do
more complicated thru handling than core midi has built in it has to put its
own callback in there, but that's just a "Message -> [Message]" function, not
a whole scheduler.

> stream. Sequencers also have to respond to GUI events that might affect the
> MIDI output. You don't want to undo a bunch of buffered MIDI data and
> recompute it when the user hits the transpose function or edits a velocity
> of something that's already in the output stream.

Sure you do.  In fact the way I look at it, you have to.  Either you convert
the midi directly from the GUI data structures (in which case probably GUI
locking will hurt you), or you insert a little buffer of converted MIDI that
the player reads from.  Maybe the buffer is only .5s ahead of the play point,
depending on how much work model->midi conversion is, but if you really want to
support editing stuff right next to the play point then you do have to clear it
and regenerate it, possibly stalling the output.  Or just don't pick up changes
which fall within the buffer, which is what I do.  And since in my case,
converting from model->midi is possibly a lot of work, I need a much larger
buffer.

Now, whether the buffer happens to be stored in the OS provided MIDI scheduler
or in your own home-grown one is irrelevant since they have to provide the same
interface: add timestamped msg to buffer, and abort buffer.

Ok, so suppose I wrote my own scheduler.  It would poll inputs and dump them in
a queue to give me a callback or blocking interface, merge thru and timestamped
msgs, and have an abort operation to clear the output queue.

If thru processing were deterministic I could run it inline, but it's not so
it's in a separate thread hanging off the input queue and feeding back into the
output queue (which is what I did above).

Not totally coincidentally, this is the interface that core midi provides.
>From looking at the docs, it looks like alsa works this way too, only with even
more features.  Also not coincidentally, this is the interface I wind up with
after a certain amount of code to adapt portmidi.  As I learn more about the
requirements just using core midi and alsa directly gets more appealing.  If
using portmidi means writing my own scheduler, I'd better spend the time by
writing my own interface to core midi (and alsa later).

> I suspect a lot of devices don't buffer SYSEX data to see if the message is
> well formed before starting to change internal state, so dropping SYSEX data
> just because a user stops a sequence seems like bad behavior to me. But this
> may be something you just have to live with if you are going to precompute
> stream data and then abort it.

Well, there's no real right way.  For short ones, you'd want to deschedule the
rest but finish the sysex.  For long ones, you'd want to cut it short (though
I doubt anyone does SDS anymore).  I'm fine with both behaviors.

> And on CoreMIDI at least, there's really no
> such thing as opening or closing an output port, so you couldn't close
> the port even if you wanted to.
>
> So for CoreMIDI, the answer to my question above is "nothing" (that's good).

Based on my reading of the alsa docs, it's nothing for alsa too.  The MSDN docs
don't say anything about having to reset any connections after an abort, but
I'd be pretty surprised if you have to, since they send note offs.

So I'm not sure where the reset thing came from... the old mac system?
Roger Dannenberg | 7 Sep 05:53 2008
Picon

Re: merging midi streams

Evan Laforge wrote:
Well, phooey, I had a whole response written up and then lost it when I sent the patch. I'll try to be more concise this time...
I would not recommend this. You probably get away with it because MIDI is intermittent, so a message is only delayed when the delivery time (or for THRU, when the arrival time) coincides with some heavy computation like a long wait for disk I/O or a graphics update. Even then, it's hard to notice an occasional timing problem in a typical MIDI performance.
So I did some tests with a very high bandwidth midi controller, and going through the whole event loop does lead to lagginess, but it's due to other inefficiencies in the event handling. I patched in a shortcut to handle thru directly and I can't hear the lag anymore.
My concern is not throughput but the worst case latency, e.g. how long does it take a thru message to go through the system? The normal case is going to be very fast, but it's the occasional case due to the application event loop doing other things (or the system doing other things) that causes problems.
I think you already have logic to generate MIDI messages in time order, so rather than generate them in advance to yet another data structure and then deliver them from there, how about just generating messages as you need them? This should not add much code at all. In fact, I would be a little concerned about generating 30s of MIDI in advance -- depending on the
Yeah, msgs are generated incrementally, and 30s is just an arbitrary amount of time I set the generator to stay ahead of realtime. I can take it down to 1-2s if necessary. I suspect on OS X it's user memory, from the microkernel thing, so the OS scheduler is just as good a place as any to store the midi, but that might not be true for alsa.
density of messages, that could be quite a lot of storage for device drivers, PortMidi, etc. If you reduce this to, say, 1s in advance, then you'll need logic in your app to generate data incrementally, and this is exactly the logic you need to eliminate the buffering entirely and just generate each message just-in-time. Once you are generating MIDI output just-in-time, you can easily merge a MIDI input stream.
Logic to generate msgs incrementally is not the same as logic to do so in realtime. Generation is complicated, and I never know when it's time to go GC or get paged in by the OS. The point behind copying the midi into the driver is that it will do stuff like wire the buffer into memory and run at a high priority. Yes, I can also write my own high prio process with wired memory, but the point of the OS guys putting that in a library is that I don't have to.
So I guess you are saying that your application is not realiably low-latency, but you are willing to put up with that in the case of midi thru, but not in the case of playing sequencer data.
To get timing accurate to 1ms from the application level (where PortMidi runs), you must at least wake up the application every 1ms and check for incoming data. Not all systems can do this, and simple applications with one thread handling a GUI, file IO, etc. are unlikely to do this. By passing
The system in this case is windows, which can do that right? And if the library creates the thread, it doesn't matter if the application is single-threaded. In fact, putting the thread in the library is what allows the application to be single threaded.
That may be true. I guess merge makes sense if you believe you need device-driver level timestamping for sequencer data, but it's good enough for the application to handle MIDI IN and send some messages to the "front of the line" of the MIDI output stream. I think (at least some) high-end sequencers feel they must implement their own MIDI THRU with low latency and once they have low latency at the application level, they might as well do all their scheduling there, eliminating timestamps on the midi output
I don't have the source for, say, Logic handy, but I'll bet it uses CoreMIDI directly, and not portmidi. And since it uses CoreMIDI, it has access to the midi scheduler along with the MIDI thru library. In fact, I wouldn't be too shocked if apple designed core midi with logic in mind. Yes if it wants to do more complicated thru handling than core midi has built in it has to put its own callback in there, but that's just a "Message -> [Message]" function, not a whole scheduler.
That's all plausible.
stream. Sequencers also have to respond to GUI events that might affect the MIDI output. You don't want to undo a bunch of buffered MIDI data and recompute it when the user hits the transpose function or edits a velocity of something that's already in the output stream.
Sure you do. In fact the way I look at it, you have to. Either you convert the midi directly from the GUI data structures (in which case probably GUI locking will hurt you), or you insert a little buffer of converted MIDI that the player reads from. Maybe the buffer is only .5s ahead of the play point, depending on how much work model->midi conversion is, but if you really want to support editing stuff right next to the play point then you do have to clear it and regenerate it, possibly stalling the output. Or just don't pick up changes which fall within the buffer, which is what I do. And since in my case, converting from model->midi is possibly a lot of work, I need a much larger buffer.
I would suggest a different structure: the GUI can write single-word parameters atomically (because memory words are atomic) to be read by the thread generating MIDI data from the sequencer data structures. In the case of more complex updates, data can be sent via messages to the generating thread, which can process messages atomically between polling for incoming MIDI. There's' no need to constantly flush messages, back up in time, and regenerate them as the user moves a slider.
Now, whether the buffer happens to be stored in the OS provided MIDI scheduler or in your own home-grown one is irrelevant since they have to provide the same interface: add timestamped msg to buffer, and abort buffer.
I was arguing for *no buffer*.
Ok, so suppose I wrote my own scheduler. It would poll inputs and dump them in a queue to give me a callback or blocking interface, merge thru and timestamped msgs, and have an abort operation to clear the output queue. If thru processing were deterministic I could run it inline, but it's not so it's in a separate thread hanging off the input queue and feeding back into the output queue (which is what I did above).
Keep in mind that PortMidi does not support access by multiple threads.
Not totally coincidentally, this is the interface that core midi provides. >From looking at the docs, it looks like alsa works this way too, only with even more features. Also not coincidentally, this is the interface I wind up with after a certain amount of code to adapt portmidi. As I learn more about the requirements just using core midi and alsa directly gets more appealing. If using portmidi means writing my own scheduler, I'd better spend the time by writing my own interface to core midi (and alsa later).
I suspect a lot of devices don't buffer SYSEX data to see if the message is well formed before starting to change internal state, so dropping SYSEX data just because a user stops a sequence seems like bad behavior to me. But this may be something you just have to live with if you are going to precompute stream data and then abort it.
Well, there's no real right way. For short ones, you'd want to deschedule the rest but finish the sysex. For long ones, you'd want to cut it short (though I doubt anyone does SDS anymore). I'm fine with both behaviors.
And on CoreMIDI at least, there's really no such thing as opening or closing an output port, so you couldn't close the port even if you wanted to. So for CoreMIDI, the answer to my question above is "nothing" (that's good).
Based on my reading of the alsa docs, it's nothing for alsa too. The MSDN docs don't say anything about having to reset any connections after an abort, but I'd be pretty surprised if you have to, since they send note offs. So I'm not sure where the reset thing came from... the old mac system?
On Windows, I think abort on a stream is going to return a bunch of buffers to the caller -- I think there's some special case code to clean up, but I'm really not sure without study whether there's an issue here or not.

_______________________________________________
media_api mailing list
media_api@...
http://lists.create.ucsb.edu/mailman/listinfo/media_api
Marco Fabiani | 12 Sep 12:05 2008
Picon
Picon

PortMIDI installation WinXP - missing pm_dll

Hi,

I am trying to compile and install portMidi on Windows XP using Visual 
C++ Studio Express 9. When I convert the project file from version 8 to 
9, I get an error because pm_dll.vcproj and relative files are missing 
from the pm_win directory. I tried both the svn latest version and the 
sourceforge zip file, but they are missing in both.

I need the libpmdll.a file in order to install pyPortMidi.
Any suggestion?

Thank you

Cheers
Marco
Roger Dannenberg | 12 Sep 14:29 2008
Picon

Re: PortMIDI installation WinXP - missing pm_dll

Marco,
    You should compile without pm_dll and  pm_dll.vcproj. These VC 
project files are really hard to maintain, and I'm not sure what 
happened, but the high-level story is that we have (experimentally) 
determined that Windows no longer crashes when you forget to close a 
MIDI Input port, so we decided that extra efforts (the pm_dll) to catch 
exiting programs and close open ports is no longer necessary. 
Furthermore, the dll caused problems under Vista. There was a 
pre-processor symbol to enable/disable use of the pm_dll, so if you have 
trouble compiling without the pm_dll library, be sure the project file 
is not turning on the option.
    I'm heading to ISMIR for a week, but I'll try to look things over 
and update the repository soon.
    -Roger

Gmane