I thought my question and its answer might be of interest to the group:

From: Bernd Kohler <bekosoft@tibalt.supernet.ab.ca>
To: brill@oakland.edu¶
Subject: Pitch to Midi¶

Hello Mr. Pleatman

On Thu, 5 Oct 1995, you wrote:¶
> I noticed your recently uploaded program for detecting the frequency of
> played music, and thought you might be able to answer a question I've had
> for a number of years.  How difficult would it be to write a program to
> convert played music do midi?

In my humble opinion, conversion from pitch to midi is very difficult.

> In other works, could you record somebody playing the piano, and detect
> all the notes being played along with their amplitude and length, then
> convert the information to a midi file that could be edited and played
> back on a Disklavier or other such instrument?

There may be some hope for this specific instance, where only one instrument
is involved. To do a good job of it, I don't believe that it would be
possible in real time, however.
Of course, if you were using a midi keyboard instead of a 'normal' piano,
you can easily record the midi directly with a midi interface and a midi
sequencer package. Some companies also manufacture other midi instruments,
like midi guitars. There is even a company (DigiTech if I recall correctly),
that has come up with a system for vocalists, using special sensor to do this.

> I am sure this would be fairly difficult, but in principal it ought to be
> possible.  There would be interesting commercial possibilities, such as
> rejuvenating old 78 recordings made by famous artists; as well as playing
> a record through a real piano (if the transfer could be done in real-time).

In principal, it may indeed be possible, but the problems go much beyond
the detection of a pitch that I have accomplished with my products, Perfect
Pitch and In Tune.
I can assure you that just to do as much as measure a pitch accurately to
1 Cent, has been very difficult. Perhaps you can relate to this simply by
the fact, that there are very few programs on the market that can do this
at all.

I will try an explain the problems with pitch to midi a little bit further.
To measure even a single note of one instrument is difficult, because the
instrument's sound is made up of many frequency components. For example, when
you pluck the E0 string of a guitar, the sound will contain frequency
components for notes E0,E1,B1,E2,G#2,B2,D3,E3 (fundamental + 2nd to 8th
harmonic). Now most of the time, E0 and E1 will have the largest
amplitude so that my software, which assumes that only one note is played,
can determine the correct note for the string. If I couldn't assume that only
one note was being played, the task becomes impossible, because notes
B1 and E2 (3rd and 4th overtone for E0) also happen to coincide with the
B1 and E2 strings of the guitar. So, how do you tell a string's harmonic
apart from other strings ? The only way that this could be done, is if a
complete frequency spectrum which includes all of the overtones for each
string could be computed and analysed. This task would take considerably
more computer time than what I can currently achieve with In Tune for
example. Now, add to this the fact, that the spectrum for the guitar notes
are not constant, but vary over time, vary with the selected pickup, with
the position of where the string is plucked, and many more, and you can
perhaps appretiate, that the task becomes almost impossible.
To the best of my knowledge, you will see this if you look at the special
midi guitar pickups that one can buy. They appear to be dealing with each
string of the instrument separately, which makes the job easier again.

In some of my research and tests, I have found, that at this point in time,
most of the computer's time is required simply to compute the spectrum for
the recorded sound, and leaves no time to perform the complex analysis that
I am hinting about in my previous paragraph. This is where the human ear has
a distinct advantage over the computer. The ear's structure already relays
spectral information to the brain, it doesn't record sound and then converts
it into a spectrum, like the computer.

I hope this answers your question, or at least gives you a small appretiation
of the task, even if the result isn't a positive one. But people are
continually working in this field, and I am sure, that progress will be made.

Sincerely,

Bernd Kohler, bekosoft.