I struggled with this for a few days while trying to convert a Silverlight video player to HTML5, and finally found an answer. Posting here in case anyone else is having trouble!
You need to specify the format as MPEG DASH to get it to smoothstream the MP4 file to the HTML5 video player. This is done by adding a format parameter to the manifest URL, as follows:
Note the (format=mpd-time-csf) at the end of the URL. There are a number of other formats you can stream, including the Silverlight SmoothStream, Adobe’s streaming format, Apple’s HTTP Live Streaming for iOS devices, and more. This is all done for you automatically by Azure’s Media Services. Pretty darn cool.
I struggled to find this, too, so quite happy I finally got things working. Here’s the source URL from Microsoft for more details:
Auri's Blog 