...
There are a couple of downsides to using Matlab Engine. On systems other than Windows, you can't attach to an existing session of Matlab (though there are ways around that, like http://www.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix), and because of how it transfer input and output, you can't simultaneously control the engine from the typical matlab terminal and the GUI, which requires some sort of terminal.
The solution is to keep the engine running as a full client to the GUI, and input Matlab commands into the Open Ephys GUI and have them implemented in the engine (you can then save the workspace if you want to switch to full MATLAB later). This allows for full command line control over the engine, including running MATLAB GUI elements.
...