Hmmm... When I hummed Kururi's "WORLD'S END SUPERNOVA", it turned out to be a hit.
To use Hum Search, tap the microphone icon from your mobile device's latest Google app or Google Search, say "What's this song?" or launch it with the "Search Songs" button. Also, it is OK to talk to Google Assistant "What is this song?" After humming for about 10-15 seconds, search results will be displayed (sometimes there are no hits).
What's going on behind this humming search? On November 12th, Google gave a technical explanation about humming search.
Sequentialize humming melodies and match them with songs
In the humming search, timbres and instrumental sounds are eliminated from the studio sound source of the song, and the melody is converted into a numerical sequence using a machine learning model. Using this as a database, the melody that the user sings when searching is also converted into a numerical sequence, and by detecting from the database those that match both numerical sequences, it proposes the song that the user is looking for.
Number strings converted from the same melody consist of close numbers, and numbers converted from another melody consist of distant numbers. Information other than the melody, such as pitch and voice quality, is excluded when converting to a sequence. Another point is that since the matching is based on a sequence of numbers that represent the melody, there is no question of individual singing ability, which varies widely.
The development of the machine learning model is based on music recognition by the Google AI team. It is said that the technology has evolved from "What's this song? (Now Playing)" (2017), which detects music playing nearby, and "Sound Search" (2018), which expands the search range.
It sounds simple, but according to Google Senior Product Manager Krishna Kumar, "We spent a considerable amount of time training (for machine learning) and reached the release. In this humming search, there is a difference between face recognition using an actual face and face recognition using an illustration." At the initial stage, it was difficult to search by humming (although it was possible to search for singing voices), so we also implemented an approach to generate humming songs from actual songs sung by people. He also collected a wide range of samples for humming and repeated tests to improve accuracy.
Matching is done on the cloud side. When I actually used it, there were times when it didn't match well with just humming, but I have the impression that matching accuracy increases when you sing along with the lyrics. Currently, it takes about 10 to 12 seconds to sing, but Google says it wants to shorten the humming time as well.
Humming Search is currently only available on Android. iOS supports only English, but it is scheduled to be released on iOS in the future.
