Improve error handling for high MIDI throughput

  • I spent yesterday dealing with numerous Kemper crashes (Rig Manager not running), the first I've ever encountered in a platform that's overall highly stable. Comments from Kemper folk on various threads I've read hypothesize that a high number of MIDI messages, such as a burst that can result if your DAW is set for chase, could be the trigger.


    I consider the Kemper to be a mission critical application as no one can afford a crash in a live performance. With that in mind, I would respectfully submit that error handling should be a high priority when there's a known vulnerability. If the Kemper is unable to process a high volume burst of MIDI messages, it would be far preferable to simply ignore the ones that can't be processed rather than crashing the unit and causing a time consuming reboot in the middle of a song.


    Like a lot of the folks here I make a living as a software developer, so I'm well versed in the art of the possible. I do understand that you can overwhelm any system with too much input data. That said, I think it would be useful to anticipate this in regards to the MIDI input stream and handle the error more gracefully rather than allowing a buffer overrun or whatever it is that's crashing the OS.


    I realize error handling isn't as sexy as a new feature, as we all love shiny objects. Nonetheless, sometimes it's worth a little time on these things to keep the reliability factor high. I doesn't matter how many features there are if your Kemper crashes on a gig.


    Much respect to the devs and support folks. This is an excellent product, and it's the first time I've dealt with a crash. I know there are lots of other things slated for the next release, but I'd be most grateful if a review of the error handling for MIDI was a part of it.

  • After making this post, I realized I didn't offer the Kemper guys any context, which is always helpful when debugging things.


    People who use expression pedals are sending a ton more MIDI data that I am, so as I thought about it I think the problem, at least in my case, is related to the processing effort to load / unload performances, slots and stomps. That's mainly what I'm doing. Additionally, it occurred to me this morning that this is in the context of a show file, so Cubase has one project that has eight songs in it, each with a MIDI track pointing to the Kemper to handle this.


    Nobody likes back seat driving (especially programmers), but I wonder if my crashes might be less about the volume of actual MIDI messages and more about the error handling when you tell the Kemper to load a performance, then a slot, then a stomp, then turn the stomp off, load a different slot, the change to a new performance, repeated rapid fire half a dozen times in a short burst. Perhaps this causes the OS to try to unload an entity that hasn't completely finished loading, or some similar case.


    In any event, from my perspective it's still an error handling issue. If trying to do thing B before the consequences of thing A are complete, better to let the request fall on the floor than crash the system.


    I hope this provides more clarity.

  • It always drives me crazy when I get a bug report from a user and all the info I get is just, "it doesn't work." So, I've been trying to reproduce the crash and have finally succeeded.


    Attached is a MIDI file that is the sum total of all MIDI CC messages, 48 in total, I send to the Kemper in the aforementioned Cubase project that experienced the crash. I then combined them and put them in a ridiculously short timeframe, e.g. 1.1.1.1, 1.1.1.2, etc. so that they all fire by 1.1.1.48.


    I then set Cubase to cycle mode from 1.1.1.0 to 1.2.1.0 and let it run continuously. Much to my irritation, I got no crash. I stopped Cubase and went to the toaster and confirmed that it was still operating as normal.


    At the end of each song I send the CC to turn on the tuner so it's always available until the next song starts, and the beginning of each song turns it off. The remote is in the live room as it would be on stage, not so much to change performances but just for the tuner (and manual override should the computer go south on me). Sometimes, however, without thinking I step on the tuner or other footswitches while the Cubase sequence is still running.


    So, for the next test, I started the Cubase cycle above, walked into the live room, and stepped on the tuner and a couple of other switches. This caused the toaster lockup I've been experiencing.


    When I'm not sending MIDI to the Kemper, the remote works fine and unless Rig Manager is running (it's not in any of these cases) it's rock solid. It appears there's some level of contention that's not handled gracefully when the Kemper is receiving MIDI and there's activity on the remote.


    While this is a different scenario than the MIDI data alone causing the crash, I believe my initial request for more robust error handling is still valid. Unless you simply cannot send MIDI to the Kemper and also use the remote, this contention should be addressed so that the unit doesn't crash. Even if MIDI and the remote were mutually exclusive by design, remote input while receiving MIDI should be ignored rather than crashing the unit. Hopefully this is a reasonable request.


    In closing, a brief story to illustrate my point. Early in my development career, my wife came into my office and I was showing off the program I was working on. She also had a career in dev as a senior level software QA, so I suppose I should have known better. She smiled, reached over my shoulder and banged on the keyboard with both hands like a drunken chimpanzee. The program promptly crashed in glorious technicolor.


    Aghast, I asked why she would do such a thing. She simply smiled and said there was no guarantees that a user wouldn't do something stupid, so I should have better error handling. And she was right. If it can be done, someone will do it. The system still shouldn't crash. :)


    KemperMidiCrash.zip

  • My previous assumption that remote activity would trigger the crash was not entirely true. It turns out I can still crash it without stepping on the remote.


    I continue to have stability issues with my current workflow, and the conclusion I've come to is that I simply can't have chase for CC enabled without freaking out the Kemper and remote. This is a global Cubase feature and I was trying to avoid having to remember to turn it on or off depending on whether or not I'm working a show project with the Kemper. So, after yet another crash this evening I've disabled it and will see if I can make it through the rest of the night without any further lockups.


    I haven't seen any commentary from the Kemper folks here and I wonder if they intentionally don't participate in feature request threads. If that's the case then I regret posting all this information here as I was really hoping to hear their perspective. I could start another thread, but then I'm duplicate posting, which would be impolite.

  • yup, my experience as well. MIDI chase is not an option with Kemper (it is musical DDoS :-) ) which is a shame because it is super convenient on rehearsals, when sometimes we want to start "somewhere in the middle", but can't because patch was not changed...


    Let us know if disabling chase helped.

  • yup, my experience as well. MIDI chase is not an option with Kemper (it is musical DDoS :-) ) which is a shame because it is super convenient on rehearsals, when sometimes we want to start "somewhere in the middle", but can't because patch was not changed...


    Let us know if disabling chase helped.

    After disabling chase I made it another hour with no crash, so that's encouraging.


    I think you're on the right track with the DDoS analogy. Not all messages, of course, since expression pedals can send a flood of CCs, but rather hammering the system's loading / unloading of performances, slots and stomps. Each of those appear to be more resource intensive than a wah or volume pedal as it takes a second or two for them to load. Additionally, operating the remote at the same time as this is apparently also asking for trouble.


    So, as much as I wish it were otherwise, I can see a case for when a performance / slot / stomp is mid-load and several more messages land requesting that same operation but different selections, there's trouble. That makes chase a non-starter.


    That said, I believe my original feature request of "better error handling" is still valid.


    If I fire a bunch of performance changes and it takes a few seconds for it to land on the last one, that's understandable. Crashing, however, is obviously undesirable, and I think a little more attention to the error handling in that area would make the Kemper that much more robust.