Hello and Welcome to Data Driven.

In this episode, Frank and Andy speak with researcher Matteo Interlandi about project Hummingbird.  

Audio file 

my.sharepoint.com/personal/andy_leonard_entdna_com/Documents/Transcribed%20Files/matteo-mixdown.mp3">matteo-mixdown.mp3 

Transcript 

00:00:00 BAILey 

Hello and welcome to dated driven. 

00:00:02 BAILey 

In this episode, Frank and Andy speak with researcher Matteo Interlandi about project Hummingbird. 

00:00:09 BAILey 

Now on with the show. 

00:00:10 Frank 

Second, hello and welcome to data driven. 

00:00:21 Frank 

The podcast where we explore the emerging fields of data science, machine learning and artificial intelligence. 

00:00:27 Frank 

If you'd like to think of data as the new oil, then you can consider us. 

00:00:30 Frank 

Car Talk because we focus on where the rubber meets the virtual road and with me on this epic Rd. 

00:00:36 Frank 

We're on the information superhighway as oh is Andy Leonard. 

00:00:39 Frank 

How you doing Andy? 

00:00:40 Andy 

I'm well Frank, how are? 

00:00:41 Frank 

You I'm doing alright. We're recording this on Wednesday, September 1st, 2021 and the the. 

00:00:51 Frank 

The the remnants of Hurricane Ida are ripping through the DC area. 

00:00:57 Frank 

Uh, so if, uh, if I suddenly get dropped, that's because we probably lost power. 

00:01:03 Frank 

But I do have the backup generator, the one that the professionals installed and my. 

00:01:10 Frank 

Duct taped together a solar generator so. 

00:01:15 Frank 

I will be offline. 

00:01:17 Frank 

For a short. 

00:01:18 Frank 

Bit and hopefully come back online. 

00:01:20 Frank 

How how you doing, Eddie. 

00:01:23 Andy 

I'm doing alright Frank. Well, we are you know I'm about gosh 250 miles South of UM we didn't get near the near the effects of Hurricane Ida as you did. 

00:01:34 Andy 

We're getting a little bit of rain now. 

00:01:36 Andy 

We've had some wind. 

00:01:37 Andy 

Gusts, but it's been really mild, and if you look on the radar. 

00:01:41 Andy 

Gotta watch it into track and I I do. 

00:01:43 Andy 

I'm a weather weenie and amateur but it it just kind of went around us to the to the West and it actually started the east when it got a little north of us and aimed right for your house. 

00:01:54 Andy 

I was looking outside that's where Frank lived, right? 

00:01:56 Andy 

And look, the eye is coming right for. 

00:01:58 Andy 

Frank what's left? 

00:02:00 Frank 

Well, fortunately we're safe. 

00:02:02 Frank 

There was some kind of flooding in Rockville and the small overnight, and some folks they got up. 

00:02:09 Frank 

No one, nobody died that I'm. 

00:02:10 Frank 

Aware of so. 

00:02:11 

It it says. 

00:02:12 Frank 

You know we're not. 

00:02:13 Frank 

Custom the floods or hurricanes or tornadoes up here in DC and and we're more used to the human threats of, you know, little things like terrorism and things. 

00:02:25 Frank 

Like that, but. 

00:02:26 Andy 

Yeah yeah, you guys got a little bit more to worry about that than we do here in FarmVille, right? 

00:02:32 Andy 

But you know these days. 

00:02:33 Andy 

Who knows? 

00:02:35 Andy 

The, uh, definitely our thoughts and prayers are with the folks in in Louisiana and Mississippi. 

00:02:40 Andy 

They were hit very hard. 

00:02:42 Andy 

I've got got friends in Georgia, Western Georgia were telling me that. 

00:02:47 Andy 

They they took a beating as well and you know it just it looks horrible I. 

00:02:53 Andy 

I you know, I've I've been in a few of those places after hurricanes have hit as part of like church efforts to help clean up and stabilize and stuff like that. 

00:03:04 Andy 

It looks like I don't know. 

00:03:06 Andy 

They people describe it as like a war. 

00:03:09 Andy 

I've never been in a war so I don't know. 

00:03:10 Andy 

I've seen pictures and. 

00:03:13 Andy 

There's a lot. 

00:03:14 Andy 

It looks like a lot of stuff is blowing over, and that sort of. 

00:03:16 Andy 

Stuff, it's just. 

00:03:18 Andy 

So, and they're talking weeks and weeks before power comes back on. 

00:03:22 Frank 

That's horrible, that's. 

00:03:23 Andy 

Similar places, yeah. 

00:03:25 Frank 

That's that's. 

00:03:26 Frank 

Probably going to be do more damage from for a lot of things. 

00:03:30 Andy 

Were you worried? 

00:03:30 

But on a. 

00:03:30 Frank 

More positive note, uh, a positive note. 

00:03:31 Andy 

Yes, on a positive note. 

00:03:35 Frank 

Uh, we are. 

00:03:37 Frank 

I am super excited to have a special guest and I say super excited because he's from Microsoft. 

00:03:42 Frank 

He's a senior scientist in Jelt at Microsoft, working on scalable machine learning systems. 

00:03:50 Frank 

Before he was at Microsoft, he was a postdoc scholar at the Computer Science department at UCLA, and this he was doing a lot of interesting stuff there. 

00:04:03 Frank 

He was doing research at Qatar or Qatar. 

00:04:05 Frank 

I'm not sure how to say that exactly, but he has a PhD in computer science. 

00:04:11 Frank 

In university. 

00:04:12 Frank 

Of Modena and or? 

00:04:15 Frank 

I'm going to botch this. 

00:04:15 Frank 

Reggio Emilia. 

00:04:17 Frank 

Welcome to the show, Mateo. 

00:04:22 Frank 

Awesome, so we are really excited to have you here. 

00:04:25 Frank 

We actually booked you a whole month in advance. 

00:04:27 Frank 

I've been looking forward to this. 

00:04:29 Frank 

Yeah, because you're coming by way of some of the folks at the Mlad conference. 

00:04:35 Frank 

And for those who don't know, I'm a I've mentioned this. 

00:04:37 Frank 

Mlad stands for machine learning and data science summit. 

00:04:40 Frank 

It used to be in person I think now it's entirely virtual for the foreseeable future. 

00:04:45 Frank 

Uh, but that why I attended M lads in 2016 summer of 2016 and it was uh, it was life altering like I don't say that. 

00:04:55 Frank 

Lightly so. 

00:04:56 Frank 

So Microsoft does amazing work in the machine learning and data science space. 

00:05:02 Frank 

Very much cutting edge stuff very much I. 

00:05:06 Frank 

I wouldn't say under the radar, but Microsoft does not do a great job putting its own horn, so we're very excited for you to come on Mateo and talk about this little project that you're working on. 

00:05:17 Frank 

And what is the is it have a code name or what? 

00:05:20 Frank 

What is it called? 

00:05:22 Matteo 

Hummingbird should the code name is actually I'm in. 

00:05:26 Matteo 

Don't have any specific internal names for. 

00:05:28 Matteo 

This for this. 

00:05:28 Frank 

OK, what what is GL stand for? 

00:05:32 Frank 

That was my that was my first question. 

00:05:33 Frank 

When I saw your bio. 

00:05:35 Matteo 

Uh is for Gray system lamp and is the after Jim Gray which. 

00:05:41 

Oh, OK. 

00:05:41 Matteo 

Is putting award yeah? 

00:05:45 

OK. 

00:05:46 Matteo 

So these are the search lab after this name yeah and use within the Azure data organization. 

00:05:49 

Oh, interesting. 

00:05:53 Frank 

And uhm, So what? 

00:05:56 Frank 

What what cool stuff does Hummingbird do? 

00:06:00 Matteo 

So, Hummingbird, uh? 

00:06:03 Matteo 

Is a little bit, uh, weird project in the sense that when we started this project we didn't know if it was going to. 

00:06:10 Matteo 

To be a success or not? 

00:06:12 Matteo 

Because what we try to do basically is to uhm translate traditional machine learning models and into neural networks. 

00:06:22 Matteo 

Actually not Internet format into tensor programs such that then we can run over tensor runtime, such as pipers. 

00:06:30 Matteo 

In terms of. 

00:06:32 Matteo 

Uhm, so when we started this project actually idea was hey there is a lot of investment in general pulling into this neural network frameworks and. 

00:06:45 Matteo 

Coming from the Azure data organization, instead, we are more interested in these traditional machine learning methods such as decision trees. 

00:06:52 Matteo 

Linear models were not encoding all those boring traditional algorithms. 

00:07:00 Matteo 

And so we look at this. 

00:07:01 Matteo 

The neural network system and say hey how we can take advantage of all this technology that is built. 

00:07:05 Matteo 

Into this domain so you can run neural. 

00:07:08 Matteo 

Network over CPU. 

00:07:10 Matteo 

Over the GPU, then you can use like fancy compilers to compile to generate the transfer programs. 

00:07:16 Matteo 

All those sort of techniques and we were. 

00:07:19 Matteo 

Kind of struggling. 

00:07:20 Matteo 

To see what we could do with the with this stack and and what we come up with with is this Amber project. 

00:07:27 Matteo 

So we basically take a. 

00:07:32 Matteo 

Traditional machine learning pipelines composed right feature iser and machine learning models. 

00:07:37 Matteo 

After the day trained. 

00:07:39 Matteo 

So first you need to train it using cycle ornamental net or. 

00:07:43 Matteo 

Uhm, uhm, one of those traditional machine learning platforms and then once it is trained we basically convert it into a set of tensor operations in. 

00:07:54 Matteo 

In the current version we use basically PY torch for doing this conversion and then basically you have a pipeline model so you can do whatever you can do with Python. 

00:08:03 Matteo 

Models so you can deploy it in in it into a PY torch. 

00:08:08 Matteo 

Uhm, deployments you can run over CPU ran over the GPU or you can do the torch script if you want to get rid of all the Python dependency and just have a C++ program you can. 

00:08:19 Matteo 

Do all those all those tricks. 

00:08:22 Frank 

Interesting, does it impact accuracy precision? 

00:08:26 Frank 

Does it improve it? 

00:08:27 Frank 

Keep it the same. 

00:08:29 Matteo 

We tried to keep it the same so we are able to keep. 

00:08:33 Matteo 

It The same up to floating point numbers roundings? 

00:08:36 Matteo 

So since we use, you know we use PY torch to run these programs and not like a socket or ornamental net. 

00:08:44 Matteo 

There are some differences in how they do you know, floating point operations. 

00:08:48 Matteo 

So the. 

00:08:49 Matteo 

Accuracy is up to roundings in the Floating Points, which sometimes are actually. 

00:08:54 Matteo 

It can be quite a bit, but most of the time is really small, almost not noticeable. 

00:09:00 Frank 

Interesting, interesting, uhm. 

00:09:03 Frank 

Do you would you know. 

00:09:05 Frank 

If there was like. 

00:09:06 Frank 

A discrepancy, or you Dutch as part of testing? 

00:09:09 Matteo 

It's part of testing. 

00:09:10 Frank 

Right, all software is tested, right Andy? 

00:09:11 Matteo 

So we have we have. 

00:09:13 Frank 

Sometimes intentionally is that the email. 

00:09:15 Andy 

That's right. 

00:09:17 Frank 

And he has a saying where all softwares I I forget exactly what it is. 

00:09:21 Frank 

But what is it? 

00:09:23 Andy 

Yeah, all software is tested, some intentionally. 

00:09:27 Frank 

There you go. 

00:09:30 Frank 

Uhm, so what's the? 

00:09:33 Frank 

What's the real? 

00:09:34 Frank 

What are? 

00:09:34 Frank 

What are the advantages of of of converting kind of a traditional model over to a tensor model? 

00:09:41 Frank 

Is it? 

00:09:41 Frank 

Is it portability? 

00:09:42 Frank 

Is it speed? 

00:09:43 Frank 

You did mention that you can run it on. 

00:09:45 Frank 

You could take advantage of GPU as well as CPU. 

00:09:51 Matteo 

Yes, exactly so you most mostly is related to speed, so you can basically run your socket, learn model on GPU end to end and and this user provides you know a little bit of quite a bit of speed up we for some of our example we even saw like 2 ordinal Magneto speedups. 

00:10:11 Matteo 

For some of the models. 

00:10:13 Matteo 

And uhm, and usually we try to show that. 

00:10:18 Matteo 

If you use GPU. 

00:10:19 Matteo 

Can be much faster, but on CPU we try to be kind of as close as possible scikit learn or the base or the base or diminished model. 

00:10:27 Matteo 

Sometimes we can, sometimes we are a little bit slower. 

00:10:31 Matteo 

Uh, but we. 

00:10:32 Matteo 

We had some really interesting result. 

00:10:34 Matteo 

Like for instance, we did some experiment with some. 

00:10:39 Matteo 

Some folks at the VM and we took some extra boost model and we compiled some training accuracy boost model. 

00:10:47 Matteo 

Uh, using Hummingbird anti VM into some uh, we basically do code generation and we show that the that model that was compiled to Python was even faster than they quoted the C++ implementation that they're having next used, but those CPU and GPU. Yeah, there was kind of OK. What's going on? 

00:11:06 Matteo 

This is not. 

00:11:08 Matteo 

This was not expected. 

00:11:08 Frank 

Wait, did you say it was faster than a C++ implementation? 

00:11:11 Matteo 

Yes, I mean if she used. 

00:11:13 Matteo 

Underneath C++ even scikit learn. 

00:11:15 Matteo 

You know they use like. 

00:11:16 Matteo 

From C++ library and yeah, using TVM for doing the code generation, they are able to do like a operator fusion which you don't normally have for like these traditional models. 

00:11:28 Matteo 

So we told these tricks bigger, basically that are coming from the neural network. 

00:11:31 Matteo 

Famous we were able to get like this. 

00:11:34 Matteo 

These surprising numbers. 

00:11:36 Frank 

Interesting, so that's a real performance boost, and probably if you scale that up into the cloud that probably. 

00:11:44 Frank 

Means a lot of money saving too in terms of on cloud computing things like, I imagine a company like the size of Microsoft would be very interested in getting better results faster with less cloud compute. 

00:11:56 Frank 

You did mention an acronym, I just wanna make sure folks know. 

00:11:59 Frank 

What that is? 

00:12:00 Frank 

Tyvm what is that? 

00:12:03 Matteo 

Uh, I don't know what is exactly for, uh, some tensor maybe? 

00:12:08 Frank 

Andy looks like he knows, but he's on mute. 

00:12:10 Andy 

I don't, yeah I I don't know. 

00:12:13 Frank 

OK, I'm just curious. 

00:12:13 Andy 

I'll go look it up. 

00:12:15 Frank 

There you go. 

00:12:16 Andy 

EVM acronym. 

00:12:19 Matteo 

I think is for tensor virtual machine, but I'm. 

00:12:21 Matteo 

Not sure if this is approach. 

00:12:22 Frank 

That sounds about right. 

00:12:23 Frank 

Tector,...