In this episode, Audrow Nash speaks to Matt Robinson, Program Manager for ROS-Industrial Americas at the Southwest Research Institute. ROS Industrial is a group that seeks to help industrial users, for example factories, leverage ROS and its ecosystem. In this conversation, they talk about Matt’s background, the need for the ROS-Industrial group and what problems they’re working to solve, ROS-Industrial’s consortium, and ROS-Industrial’s new working group.

Start

Introducing Matt and ROS-Industrial

Motivation in forming ROS-Industrial

Matt’s background

Getting introduced to ROS

Collaboration before Git

Getting to the Southwest Research Institute

Lowering the barrier to entry

Overview of SWRI

ROS-Industrial’s consortium

Just in time manufacturing

ROS-Industrial’s new working group

ROS-Industrial’s conferences

Links to share

This is a conversation with Matt Robinson, who is a program manager for ROS industrial Americas at the Southwest Research Institute. ROS industrial is a group that seeks to help industrial users for example, factories, leverage ROS and its ecosystem. ROS industrial has many parts, including a GitHub organization for code contributions, regional consortiums, and a newly announced working group. In this interview, Matt and I talk about how industry often solves problems and how those solutions can be improved with more modern tooling like ROS and get Matt's background, ROS industrials consortium and ROS industrials new working group. This is the sensing X Podcast. I'm Audrow Nash. Thank you to our founding sponsor, Open Robotics. And here is my conversation with Matt Robinson. Hi, Matt, would you introduce yourself?

Yes, absolutely. I'm Matt Robinson. I'm the grace industrial America's consortium program manager based here at Southwest Research Institute.

Yeah, would you give me a bit of background on ROS industrial?

Yeah. So. So ROS industrial is an open source project that seeks to extend ROS, robot operating system to industrial uses and applications in industrial relevant hardware. Right. So it was it was kind of conceived around 2011. And then around 2013, a consortium was established to kind of help give guidance to what we should be working on next.

And can you define industrial for me? Like what does it mean to be like, what are we targeting Exactly? With industrial? In this?

Yeah, no, that's a great question. Right. And we get that a lot, right. So there's a lot of sometimes assumptions, when you tag something as industrial in the context of the ROS industrial open source project. We are seeking to provide open source utilities that enable industrial applications on industrial relevant hardware. So that's the mean, the main components of the industrial tag, if you will, right, there's a lot of specific hardware that shows up in the automation space, in factories, and sometimes it's somewhat, they could have like a bit long in the tooth, right. I mean, like, it's very common for certain industrial industrial sectors to put hardware that they expected to be there for 1015 years. And so they're very proven pipelines to facilitate these automated processes that are somewhat unique, right? So we want to provide in your faces, needs to interact and take advantage of this robust ecosystem. And so that's where the extending of ROS into this industrial space was kind of the main component. Now that said, as part of like, kind of like our overriding themes and mission is to think about the, when people hear industrial, where are they sort of these things, I think, are they think of reliability, durability, quality, right? uptime, things of that nature? Right? So what are things we have to do is we think about the capabilities that we seek to deliver, and tools we seek to provide, how do we how do we realize those sort of attributes?

Very interesting. So going back a little bit industrial relevant hardware, how and you mentioned that it should be around for a long time, so people can plan to use it. How do you select your industrial relevant hardware?

Right? That's a good question, too. Right. So let's, let's talk about a couple of examples of what industrial relevant hardware may be right? So it's kind of an it can be somewhat ambiguous. So the industrial manipulator, right, so we think about, if you've seen like the big snapshot of like a place where they make cars and have all of the arms reaching in and there's sometimes sparks flying if it's a video, that's sort of the industrial manipulator of retargeting not like, behind me, we have a number of these other types of manipulators as well that are sort of set up to these in depending on the application can work around people. But that is also a class of industrial relevant hardware in the context of the manipulator. But we also look at things like PLCs, the programmable logic controller, you know, provided by companies like Rockwell and Bradley, and of course, Siemens. And, of course, looking at like different like sort of vision tools, both emerging contemporary 3d cameras, all the way back to Your traditional 2d cameras which have been running in factory lines for like last 30 years. That's sort of the those are just some examples of industrial hardware that we seek to help people be able to bring and merge with the capabilities that are kind of coming forward in the ROS ecosystem.

And what was kind of the problem or something that made it so you or whoever started off industrial? Like, why did it? Why did you begin it what problem kind of led to you wanting to create this organization?

Right, so, you know, my view like, and we can talk a bit about, like the whole layers and Thrall some industrial, the ROS industrial open source project was was started as inspired by actually a little bit similar to my background as well, with the group here at Southwest Research Institute, and also deliver solutions. Now, it's not like a solution provider integrator in the traditional sense, being a not for profit Institute here at Southwest Research Institute, but the group here develops like sort of like unique Hi, Max, one of a kind, oftentimes, prototype solutions that live and operate in factory environments or field environments that have been doing so since the early 80s. Up until or before they adopted ROS, very often, they would develop that advanced application with the industrial tools that were available at the time. Right, so the industrial manipulator and the control platform that comes with the Cuca, robot, or fanic, or ABB, they would develop those solutions, it gets something working that does something new and novel, but lo and behold, they had to do something similar. But someone specified a different mix, if you will, of the industrial hardware, they had to basically rewrite the software from scratch, right, because of the industrial ecosystem. The the whole stack from the hardware all the way back through how you develop applications for that hardware operates in a proprietary chain, if you will. And we'll call it a software stack. Exactly. But but all those communication, the middleware, if you will, as well as like the programming language and environments, they are all proprietary to that hardware set you're selecting. And then even certain vendors, like would kind of team up and have better bridges are complementing those companies. And so yeah, it wasn't like swappable

for all these different. So you'd make a solution. And we'd be like a mobile base, and then there'd be a robot arm on it. And the mobile phase and the robot arm, we both have kind of like their own ways of talking. So it was like a domain specific language often for using either of those. And then you would have to have software that is kind of translating between them, so that you can make them talk and do some sort of application. Is

that correct? Or you'd have to write in that specific domain? Sure, right, in that specific domain. Correct. And that's usually what happened. And that's from as well, I got very adept at scripting and, you know, vendor specific programming languages. And that's where a lot of us who were doing, you know, traditional industrial robot deployments, dating back to the early 90s, that's kind of how we came up. Right. And so, you know, the challenges is like, hey, this industrial robotic solution may offer some advantages. But it brings with it different software, different scripting languages, how how you implement applications, how you set up communications, you have to actually factor that if you were going to actually look at other other other solutions. And so obviously, the advent of this group here in front, Edwards is in the picture behind me when they went out to Willow Garage, and kind of that first discover motivator about somewhere around 2011, he created that first interoperability between Boston and Boston, but there was an Aha. So hey, I can develop an advanced application in a sort of hardware agnostic fashion, and not have to reinvent all this stuff. Because I have to do some other application with a different hardware stuff like that in the industrial robotic space. That was, that was like a real a huge one.

Yeah, yeah. So it was a reuse. real opportunity. Okay.

can you speak a bit more, so it was a big opportunity, because you could make it so that instead of using these domain specific languages for each hardware component, you could instead maybe abstract out that component and then connect to it with ROS or some other like, high level thing that you can use as middleware for this. Okay. And then that gives you the ability to just worry about you so you Your interface for this, whatever the device is, then you use it in a stack. And then you can swap it out with other components. You can swap new things in, as long as they have interfaces, you have one way of talking to them and controlling them. Right?

Yes, that is obviously the goal. Obviously, the devils in the details, always. But But yes, exactly right. And that was definitely new in industrial circles, right. And either became very good at working with like, you know, a Cuca, oriental scholar or offenen,

and maybe their own language in a sense, and a lot of

literally right, in different support. And they encourage people like when I came from industry, I mean, obviously, like, vendor X wanted us to buy all of their stuff, including their software programming environments, they were incentivized to not foster interoperability at the time. So fortunately, like, right, this notion of abstraction, which is obviously very common in the software world, was sort of not a new concept to develop industrial applications.

Now, going back a little bit, I would love to hear about your background, and kind of how you got started with all of us.

You've Right, right. So to catch up for the listeners, right, I have to confess that I'm not a software developer, right. So I came out of like, sort of like materials, joining materials, research, welding materials and metals, domain. In caught on with Caterpillar, the, the heavy industry equipment producer, and engine manufacturer, I worked in our corporate research group, and quickly got into developing the applications on robots, largely focusing in the early phases on specifically welding. Some of it pretty complex welding operations, were trying to control different things to realize very specific outcomes. And we had a lot of success in developing applications, you know, we had these this robot, right, and it was all set up and had all these extra bells and whistles. And we get something working really great. Like, okay, take that and go around the world and make it work. And then very quickly, we learned,

what do you mean, take that and go around the world and make it work? So like, deploy that scale? Or what do you mean,

but it scales relative for a caterpillar is not a, you know, not a Ford or GM, right. And so it'd be like, you know, they may make, you know, a given product, like a wheel loader, they may make it in five or six places. And the robots may be like, maybe three or four per site. Because you're only because you're not making as many wheel loaders as you're like, you know, border states. So, you know, not enough, not enough, let's say buying power to really put pressure on OEMs to play nice, not enough buying power to justify buying an exact same piece of equipment and shipping it around the world. So what would happen is

like this lighter spot, of where you're not able to scale incredibly effectively, because you don't have all the factories lined up for it. So it was fairly custom solutions. But it's also big enough that you want to deploy it 20 times or I don't know, some some larger number of times where it has to be you like all the all the bugs will come out. And it will be unreliable unless you kind of work it out. Well,

yeah, I, you know, if we scale back to just sort of a very manufacturing benefit, in particular for dealing with something that has to do meaningful work in the field, right. So like, we alert, right, it's got to lift heavy stuff. There's a tangible benefit if I can use robots to make it because the consistent quality actually leads to a better performing product that I can sell to people, right, it lifts more, it weighs less. So it consumes less gas, or diesel in the case of big equipment. And so there's a real incentive as a real marketing improved product, I can go out there and sell if I can leverage automation to produce it. Now the challenge was because we weren't we're not you know, that particular group I was working with was not, you know, they can't buy 1000s of robots, right? GM has a real benefit when they work with industrial robot providers, because they buy so many, they can make a lot of, you know, dictate some rules.

They say I want to do custom thing that's going to be a bunch of additional work. And because of the scale of it, the people making the robots are like, Okay, we'll do that. That sounds good to us.

Yeah. And I felt like obviously, obviously, there was a lot of other folks, if you will, in the same boat that I was, right where, hey, these different sites that could justify buying automation, but when it came that they couldn't necessarily buy exactly the same thing due to support like they would buy different brands because that's what was available and how

I could see why this was so funny. because if you're by different parts, because different brands are available in different regions, now you have to hook up to that domain specific thing. And really quick.

Yeah, we had an application running really well on a nice orange robot with this right sensor located on a totally different mix of hardware. And we literally showed up, I showed up in, you know, this other location and had to start from scratch. My, my, my one week trip turned into like a month. So that that ended up being one of the real resume business cases to when I was at Caterpillar to get involved with the ROS industrial open source projects. Well, I learned that aerospace companies, other heavy equipment, manufacturers, oil and gas companies, you know, ticking through a few even even tier one automotive, right? Because they have a slightly different problem than say your GM example.

We're here wanna automotive? Is that like, the really big car companies? Or I don't know what their

tier one whatever, there's automotive or, or any letters? Is anyone who provides the parts to the OEM or the person?

Okay, and OEM is original equipment manufacturing, manufacturing.

So GM, yep. Right? GM, you know, they make the car, they're the OEM and a company that's providing them like, say, the seats or the dashboard, assembly would be a two.

Okay, so the parts that go into the final product, that's the tier one, so it keeps going back. So tier two is the ones that provide the parts to tier one. Is it correct?

Okay, like, right. Now we're getting into supply chain, which is very resident in the pandemic era. Right? Yeah. So, but, uh, but but, uh, sorry, one of my speakers went out. But, but yeah, so So we ended up learning that a lot of other producers, manufacturers, people interested in adopting the benefits of robotics ran on the same challenges. And that was really insightful. And all of a sudden, we're starting to build this little industrial community,

it's like, the challenge was explicitly that most things were not able to talk well to each other because of domain specific languages or like custom hardware setup. So it was like, they were very reliable industrial devices, but they didn't play nice with other things. So developing kind of like custom automation solutions, were a was an extremely difficult task, because of this, the domain specific languages that they were

enrolled in. And I was, I would say, that was obviously very prescient, in your face a big challenge. But obviously, it also drove a lot of like, limited ability for reuse, you know, drove, so a lot of like, you know, hey, we'd fund some research in a university, and we can maybe get it to a point solution, but we couldn't really leverage it properly. Yeah. So, you know, we ended up learning a lot of this other sort of, you know, when you go back and look at, you know, how a lot of software companies have evolved over the years, the presence of open source, how it's able to be leveraged? What do we talk about? What do we mean by pre competitive IP, like all this was sort of new in some of the industrial circles, at least that I ran in? Before, before I got involved with the details with less industrial, and learning that a lot of other manufacturers had these same challenges. It was a learning experience.

Okay, and then so you were at Caterpillar? Where did you go from? So you and then you were building these kind of custom solutions for doing some sort of automation task that would improve a fairly small man, like you're making only? I don't know the numbers, but you had like 20 robots making them or something, some product, but a wheel loader, or whatever you have mentioned? Yeah. Where do you go from there?

So, you know, like I said, I worked in corporate research in we had a little automation team. And so we started basically collaborating with the team here at Southwest Research Institute to identify some potential near term use cases or business cases around like, what what does ROS really offer us? Right? Okay.

So at this time, you had been exposed to ROS or like, how did ROS come into the picture?

We ran into the ROS industrial team at a conference. Okay? And ask them we're like, Hey, what is this? Tell us about it? And my predecessor, Paul Voss, and Shaun Edwards, again, pictured over my shoulder for those watching the video. Their boss, Paul Evans, they came out to caterpillar and explained it to us, right, and then we found a way to kind of get involved and hence there was already this consortium established. And as Caterpillar, we join that consortium, right, so I was kind of the caterpillar face for the ROS industrial consortium. And I started like, kind of collaborating and sharing core stories with other members. But then how do we move the ball forward? leveraging open source and is this is this something we can really do in an industrial setting? That was still a question. Back in like, 20 2015 2014? Wow, I can't be like why we actually saw can we solve industrial problems with with ROS?

Source? Yeah, yeah. And what did to? And then how did that investigating that question go?

Great, right. So obviously, we got involved. The consortium has a bunch of other similar minded folks with similar challenges. You know, and it's not relevant. It's not narrow to like, say, construction equipment or whatever, I rattle off a couple of other industries, the Boeing, Boeing's and others, there was a documentary for our BMW not long ago. And so, you know, we got involved with a couple different pilot initiatives to kind of see if we could prove like, hey, what can we do? You know, to that point, everything we did was very, you know, I grabbed up the user interface, the Teach pendant on the robot, and I, you know, programmed it, I could script it on my computer and then loaded onto the robot. But I was still really, I'd still have to go through and tweak all those positions, like, right, you try their interface

for how to do it. Yeah. Right. So I've heard I've heard some horror stories about that. But I don't know that I really. So that's crazy. You would like it's like this big tablet? And you would like, would it be like a block programming kind of thing on some interface that the company may? Or?

It's it's scripting? Yeah, it's a scripting language. And, and this would be like, if I was more prepared, I would have probably wheeled over one of these robots who could have done it live. But but but no, it was very laborious. And they were introducing things like offline programming, but it was really just sitting at your desk or doing it on a computer. Lab and or on the shop floor?

Oh, yeah. Cuz you couldn't see what I was doing. You could just write the bulk of it from your computer.

Yeah. And you might have like, a visualization perfectly matches the physical system. Right. So that's why when we talk about the utilities that for industrial use cases, one of the ones is all the stuff around calibration. And so we ran some pilot programs during my time at Caterpillar and was like, Oh, wow, like, I could really not do this traditional industrial robot programming with this teaching with the Teach pendant or whatever, at an offline. You know, hey, we can basically do this notion of like, you know, use perception perceive things, leverage the data streams to make decisions, do motion planning, and actually execute processes that was mind blowing at the time, like in this 2013 2014 timeframe. And so we ran a couple pilot projects in parallel, like we'd already been doing work and perception, like, hey, wow, 3d cameras, this is this is really useful. And notions of segmentation, right? Like, oh, work on this and not that. In so we've been doing work with universities in that space. But again, it was really hard to scale. So bringing generalize it

and yeah, you said it was like a point solution. So it's solving one specific problem, but not, it doesn't generalize to other problems very easily.

Okay, we're x. So we started getting into like, what they called at the time, this focus technical project, which are these collaborative projects in the ROS industrial Consortium on blending, and blending is this notion of getting some sanding media. If you think about your, your wood furniture at home, like you're going to refinish it, you want to blend all those surfaces together, right? So you can repaint it, right. That's basically what we're doing with metal parts, we perceive them. If it's a flat surface, I apply some raster patterns. And then you can sit there and visualize the motion plan. And yeah, that looks reasonable go never got it. No teach pendant programming. I don't necessarily like drawing stuff on CAD files, which you know, doesn't necessarily save you time. Those are very legal demonstrations of proof of proof of concepts were able to do it Caterpillar they got really good by. That's awesome. Then from from there and continue to grow. This notion of collaborative projects and things like that. And so now, a handful of years later, I had the opportunity to kind of, you know, streaking it up to the open source Kool Aid and this idea of leveraging ROS 2 solve industrial problems. I had a chance to come here and take over the ROS industrial project here from my predecessor, Paul Voss, and I did that in 2017.

Gotcha. And how does it actually so one thing is when you're saying, like collaborative work on these, you mean basically using something like I don't know, some version control like Git or something so that programmers can work on it at the at different times in different places and kind of all collaborate towards one product. Yes. Is this what you mean? Oh,

my God, like back then, like in our life. Manufacturing Research that was all very new, like get get what? Like right? At the time like, right so that that idea of collaborating on software from different locations, we did that collaborative project and the ROS industrial sort of family, if you will, the consortium we partnered with Boeing was a performer on that particular project, as well as wolf robotics was like the solution provider out of Fort Collins, they're now part of Lincoln Electric, and of course, as Caterpillar, and we offer like little in kind contributions, like we developed our team developed a key driver for this laser line scanner. But yeah, but to your point, right, there was the first time that we had to script up stuff, put it someplace where other people can merge it in. And, and that sort of workflow. workflow is very new to a group that was very used to like, I'm developing right on the robot controller, right? I'm not, I'm not putting it someplace where it's really, truly accessible by others. And can be like, you know, you know, it could be reviewed version of, of that Git workflow, if you will. And that's, that's been a slow process, we still work with traditional industrial stakeholders that are still getting up to speed on that kind of get workflow. And that's been that's been a fun ride. As we talk about software and industrial settings, obviously, here in early 2020s. I mean, right, between the startup communities and the prevalence or emergence of ROS and people looking at industrial applications, it's becoming more commonplace. But like, in the early 2010s, it was non existent, almost. We had to figure out what GitHub was right? We have to get permission to use it. Yeah, right. Because the IRS corporate IT groups are like, Oh, the

internet? Yeah, the internet's bad. Yeah, you're putting your stuff on there. Yeah. It's kind of I mean, so I guess I have most of my background is in software, I would say. And it is shocking to me to hear that it was, it's like 2010 10 years ago, ish. That, like these kind of things that I associate, as modern workflows are not being used. But that's very interesting. And so the pipeline, but how it typically worked, it was you have several people that are working on different things, and you modularize the problem, so that each person can do or we, each entity can work on one thing and have that one thing, do it very well. And then you just combine them and in theory, it works perfectly. Is that is that kind of like the traditional way of doing things?

That's probably Yeah, so you'd have the person who's doing like the robot programming, you'd have a person who's doing like, let's say, there's some sensor on the robot that's gonna do some thinking or processing a lot, the one person who sets that up and programs it. And then you have like, say, an electrical engineer who who slash controls person who might do your PLC and control like electrical panel, and then yes, and then there's an integration sort of period of time, where all three of them will come together and try to like, make sure all their stuffs communicating properly. That's, that's actually, in some regards, still a similar workflow that we use here. And we do advanced systems as we bolted on also the software development piece, right, which, which enables us to kind of at least addressed interfaces, we can test a lot of the interfaces and message passing, you know, in simulation, we have a few more tools at our disposal than we had 10 years ago. And obviously, even though here, we're advanced, say compared to traditional manufacturers, you know, we still have to overcome some of the limitations of the industrial hardware, what it will just support, right, like, every every everyone, like no one has like it, this notion of a documented API. Not nice. Right?

So there's no application programmer interface to, like use the things kind of clearly.

Yeah, I mean, well, because again, we're going their scripting language, maybe exactly, just did not stay in our ecosystem,

right. That's kind of in our ecosystem. And that's kind of

a mantra, if you will, right. So but that's, it's making progress, right? Like, you know, we were at a conference one time, and this guy stood up and raised his hand, and he's lying. There was this time, I downloaded Solitaire and my computer got a virus. And now you're telling us to download download software off the internet and run our robots with it. And so that's where we were starting. The industrial sector is, I mean, they've been they've been doing what they do for an automation for 50 6070 years and so cases PLCs been around a long, long time. So it's right it was a real disruption to think about, you know, over and over again, some open source pieces layer on kind of just like the stuff of novel stuff to do this specific thing. But then like not reinvent the wheel. And like, Hey, we're gonna maintain it and make contributions back to the stuff in the open source. Because right rising tide lifts all boats. And there are some people that are just like, Wait, you're going to download stuff off the internet and run an industrial robot. I mean, we, it took a long time to just define open source get a consensus of understanding in the industrial community. And it was funny, I sat with a peer in Germany, he's like, You know what? Like, the whole, like, you can put open source on an industrial piece of equipment and have it do something meaningful in a factory and the whole place not melt or break down. We want that, like, we're doing that. And we are and lots of people, lots of great startups over the years, was it there's that one article, like 80% of new startups, whatever the robot they're putting out there. Like there's, there's ROS, there's a ROS something on Yeah, it's

crazy. I don't know the number either, but it's quite high, which is nuts. And it's

great, right? And that and that. And that was if we didn't done nothing else, we brought the industrial sector to the party, right, and generated some general acceptance for open source and like, Hey, you can do this, right. And this can work. And that's, that's been fun. It's been great to see that change over time, compared to where we were, we were like in the early teens, 20 times.

Yeah, I imagine it's pretty exciting to see kind of it coming along in this. And you can probably move a good bit faster when you adopt some of these kind of these different development paradigms like version control, and you can collaborate and then you can tie it with simulation, and you have all the parts integrated. So you can test them all together. Even if it is limited in like, say a simulation or something, but you still don't it's not like this one part. We're only going to test that. And then we're going to connect it to something that's also only tested by itself. Whereas you can like say all of these things are in the the gazebo simulator, and we're just going to run this control code on it and see if it does what we intended to do.

So you were at Caterpillar? Now, did you go? How did you end up at?

Southwest Research Institute? Yeah, so you know, I've been doing my job at Caterpillar, like 10% Was this open source? You know, motion planning, you know, no more teach better programming robot stuff. Like that was like, Okay. And that was the funnest part of my job, right? And I had all this other stuff. And I was like, I was kind of like, yeah, looking at different opportunities, right. So I mix it up. And the guys here heard, and they're like, Hey, why don't you come down here? And like, you know, I'm like, oh, like, run the ROS Industrial Program. Like, you got to be kidding me. I don't know anything about this stuff, right? I'm just some guy, like running some projects. Now. Like, no, no, you have that perspective of the person who hates programming. Okay, I definitely have plenty of that. And so, so yeah, I, we talked, and they found a role for me. And I was like, alright, we'll give it a shot. And so I came down and middle of 2017. And obviously, I brought a little different perspective, right? I mean, you know, I, I was putting robots on the factory floor and commissioning them and, and then like, trying to replicate successes on on certain hardware or development configurations, and trying to replicate that and really struggling. So I brought that in. So when other people had that pain, I was like, oh, yeah, we had that pain, too. And then like, obviously, I come from a pure group where, you know, when someone brings up a terminals, and they're like, bouncing between the terminals, I'm like, yeah, like people in the industry community can't do that now. Right? Because we're not software developers. Right? So there was this idea of like, what can we do to to lower the barrier to entry? And obviously, we've seen a lot of progress on that front, right between nice IDE ease between, you know, other, like just more Python, right, lowers the barrier to entry. You know, there wasn't a ton of Python stuff. When we first started poking around, but all of that helps, right? And so that the whole lower the barrier to entry is obviously, part of our mission is

just, I'm sure people know, mostly but so IDE is integrated development environment. And so that's like where you code. Often it makes things a lot easier. If you have a good IDE And then Python, maybe I don't need to define it, but just a programming language, which, for ROS, we are a lot of C++, which is low level. And we have some Python, which is like a nice scripting language. And so that's higher level easier, you can say like print, quotations Hi, and then close it, and it will print that out. And so it's like it's easier. So it makes it nicer for working, because people can just kind of see the logic right there. And it's almost like a human readable, like, paragraph kind of thing. Although it's death, almost. Okay,

so one of the big parts of ROS industrial, you're saying is lowering that barrier of entry? How are you doing that? Or like, what are some strategies you guys are working towards to try to lower the barrier of entry?

That's a great question. Right. So obviously, one of the things we do in our little consortium family is we provide training, right? And the training is constantly evolving, based on the feedback from the membership, right? So they want to they want more examples, and helps them there's like, sort of templates to do certain classes of things. Obviously, we've tried to give an example. Yeah, maybe, right. But setting up what a good example is like, we have a training on setting up a perception pipeline. Okay, right. And then we have a training class, where we leverage the perception pipeline to say, like, do a pick and place down, right, so you perceive this object, and you identify and grasp when you go get it? Right. So we have this whole lab exercise that takes place on the top, one of the pieces of hardware that are behind me. And then, so they get to like kind of actually build an application, but they have some skeleton to start with. Right? We went through an example, right, which can become sort of the template.

Yeah, that's how we can work with some something that kind of already works

correctly, obviously, like, sometimes the feedback is, well, this is a lot of copy and paste. I don't know what I'm copying and pasting. But so it's a continuous journey. And that, but that feedback only makes it better. But obviously, you know, the more you know, obviously, the issue is, you know, I mean, the tech world is rapidly evolving all the time. So new things are always coming out. And of course, what legs is the documentation that we're in the same boat where where we've been doing a lot of interesting new work and in reconstruction and different types of motion planners, we have an optimization based motion planning framework that's out there. So

are you for all these things? are you leveraging things that are in Rosu? For your navigation? Are you using like nav two, for your motion planning are using like move it? Or is it kind of custom solutions for industry things?

So where we want to, we want to stick with, with what's out there, emerging out of the community, right? We really want it's extending ROS into industrial applications. So creating like homegrown stuff kind of defeats the purpose. Now, obviously, there are cases where something special needs to emerge. And thereby, you will create that, in a lot of cases, we open sourced. Right, so one great example that comes up in our community is the emergence of Tesseract, which is motion planning environments very similar to move it, it was born out of a very specific use case. And it was chosen to open source it, because why not? Right. And it's still it's still we we leave, all of our training is still built on move it we start with movement. But then for those interested or specific applications, we have this alternative called Tesseract that, in certain situations offers some benefits. But we don't necessarily offer like, say training on it. It's sort of like, if your application sort of would benefit from some of the things here, you know, we can do some use on demand, right? But But in general, we start with, to your point, right, the NASDAQ move it and the tools they're in, we want people to because the benefit is the whole benefit of this whole open source idea is that once we start using them, we make contributions back. Right. So if we made a bunch of replicates of stuff, well, then we just spread out the contributions and they're less impactful. Right? So we want to start in use the most common tools as well. And so yes, so hence we've migrated our training over to ROS 2 as well we're trying to as I described earlier, right the in the industrial crowd is slow to move, right because they like what works. So we've been having to be very judicious and how we manage rolling out in the direction of ROS 2 but but but we're getting right. Most recently we also are trying to revisit our We're gonna phases because in the industrial side that's really lagged. But it's a good opportunity to revisit for consistency purposes, you can talk more about that. Yes, for

sure. Yeah, that's very interesting. I want to talk about that have you got before have you is one approach that you guys may do abstracting things further. So if, say, I want to move something from here to here, and from one location to another, maybe I abstract what you need to do in that to or subtract what you need to do and move it or something so that it's easier at kind of like, like, instead of like, I'm gonna do all like, I want to run this specific thing, or this set of commands in NAB to maybe like, more at a behavioral level or something like something that's, that's

like, that's actually it's interesting. Right? So that's some of the that sort of like, just kind of give a higher order kind of instruction. Yeah, exactly assist or empower the system to make the optimal decision. Yeah. Because like, let's say we have a mobile manipulator to your right, we're now getting involved and move it. I don't necessarily want to have to like, push out commands the boat, right? It needs to be a higher order. And then let the system's intelligence sort of figure out a bunch of

smart programmers figure it out once and then everyone gets to benefit from it

correct. And we are we are trying to bring to some proof of concept capabilities in that space to present to like, say the manufacturing engineer knows process, they know what they have to do in their factory, but is it going to, to your point, command driven or math, right, they're gonna have to tell it like, hey, you need to do it like this, just go do that. Right? When abstract away, if you will, that move that voice level, but that level. Next level down, we have some proof of concept initiatives going in that direction, we have put out some material on those, and some of the progress we're making. So for those interested, you know, if you when you share the LinkedIn stuff, you'll see posts about some of the outcomes from some of those group meetings that talk about some of the work going on what we call our development environments for industrial applications. And that's something we're looking to hopefully find some people to poke holes at and and try. And so we're looking for some beta testers now, we're hoping that help some of our industrial stakeholders lower the barrier to and again, that whole idea of lowering the barrier, particularly as the systems get more complex, of course, you'll not just mobile, but we'll do we do a lot with gantry based systems. And that's where a manipulator maybe inverted up that upside down. And it's an a huge gantry that has x, y, z. And it can take that manipulator through an entire volume.

I'm imagining a 3d printer with an arm instead of the printing spot.

Yeah, absolutely. Something like that. That's, that's a great example. And so obviously, of all the redundant axes, there's lots of solutions. How do you how do you search that space? Right. And so what are the tools for that? How do you prioritize using the axes? Sorry, gantry axes versus the manipulator axes? I mean, so, you know, we're trying to bring tools to kind of enable end users to

take advantage of that thing. That's interesting. I don't I don't know much about use cases for gantries with the robotic arms on them or anything. That seems quite cool. I have you. I don't. I did an interview earlier with Brett Aldrich. From Smak. Have you thought of there? Like, have you looked at SMAC for kind of like a higher level way of controlling robots for specific tasks? It kind of seems like their use case.

Yeah. So it's interesting. You mentioned Brett and snack. I had to today's Friday. So yesterday, I was at a Technical Steering Committee meeting and talking about hardware interfaces. And and how we'd like to do a new reference implementation to kind of herd the cats on hardware interfaces. In Brett Pinkney after that particular meeting, said like, Hey, we should we should talk about Smak. I was like, That's a great idea. So other than like reading, reading, like three lines about it, I have immersed myself derelict on catching up on some of my reading. I've only caught wind of Bratton his work since he was getting engaged with the Technical Steering Committee.

correct. Yeah. And obviously, like when I read his application, I was like, Yeah, this guy. This guy's awesome. For sure. So we're excited. And I'm glad he reached out to me and I look forward to learning more smack Yeah, it's probably it could be, again much. You know, we've had folks like Brett, come speak to our industrial audience. And so that those opportunities are existing and we're just trying to I have a hard time keeping up and I'm like in the ROS world more than I thought

so right. So where it makes sense. I tried to, in some of our organizations though, like stodgy or right kind of old school, they do have people that really are looking for solutions. And like to hear from these types of people, right, so we try to we do a lot of that matchmaking. Obviously, if there's a chance to collaborate with you, we'd love to see them.

Awesome. Yeah, if you maybe some help with the background, but if you're speaking with him anyway, there's a previous podcast interview with where we went into like, pretty deep detail on how SMAC works and where it might be useful. But so now you're at Southwest at the Southwest Research Institute, sweary. And you're leading ROS industrial from there, and you have a bunch of projects and a team, or what's the setup? And how does it work? And how does the organization

for great, great, right, so we have this manufacturing robotics department, here at the Institute. And they leverage all these tools in their day to day problem solving. Right? So they're kind of like, the ROS industrial developing team, if you will, are contributors. So the rest of us are open source project, but their users as well. And we try to walk the talk, right? So we take all these things, and we go solve problems with that. Now, how large

is the business model? And is it tied to a company and a bunch of things like this, they don't quite understand.

Essential, essentially, the Institute at a high level, right, which has about 11 operation operating divisions. One of the most well known is the Space Science Division, right? Because they actually have a mission control here and lead missions for NASA. There was a couple different the New Horizons was one where they kind of went by and did some shots of Pluto. Space Science, and then here in our division, right, so I'm in a department or division is three of us what unmanned ground systems are applied sensing. And then critical systems does a lot of NASA software. And of course, we collaborate with them as well to share tips and practices and where ROS makes sense. But anyways, so there's, it's all applied engineering contract r&d. Actually, when I was at Caterpillar, we spent a lot of time here doing dyno testing. Right, so about half of caterpillars dyno testing occurs here at Southwest Research Institute. It's totally vital. Sorry, yeah. So it's really taking the engine. And they basically try to run it, like so they put all these they feel they pump fuel into it and make sure the exhaust is handled effectively. But it just sits there on a stand and runs, they put it through its paces, they may, they may make the room, they may make the room very cold, and they make

very hot so they can test all the different conditions while running. I see

my anywhere. Correct, I was supporting on Highway tear three emissions at the time for modeling trucks. Tier three was at the time they were trying to address diesel particulates. So I was the materials person involved with a lot of the creation of some of the components for the that particular exhaust system at the time. And so they were running a lot of those tests here. So but it's but it's multi discipline, right? So I mentioned space science, obviously, we're manufacturing robotics, we have unmanned ground systems. So we have all these different groups. And so

an effective thing around these. Yeah, you have like, cuz i You say different domains, but I assume it's like, fairly similar competencies between the different domains or at least complementary in some sense.

Yeah. So you know, I mean, you're there's a lot of computer science across the Institute, a lot of material science folks like myself, mostly in the mechanical engineering division. A lot of mechanical engineers, a lot of chemists, we have a whole chemistry, division as well. So so but we basically think back to my group at Caterpillar, I worked at corporate research. This group had unique skills that I didn't have software skills in my Manufacturing Research Group, explicitly, I had some people could do some Python. So we would work with this team, we'd write a little contract, then they worked as like an extension, corporate r&d team. I had worked to some statement of work and help my team out. And in very much like I like to describe it, we work with somebody, we want to lead them, not just with something we give them, but their staff is a little bit better using the tools than they were before we show them. Right. So we don't just deliver them something we teach them while we work with them as part of the delivery to teach them as ons. Yeah, I mean, here we are using all these open source tools we want. We don't want to just give you a software that we will but ideally you have someone in your team You can learn it as well and be self sufficient. So you don't necessarily have to us. Oh, definitely, that's that's sort of the model here and proliferates around the institute. Right? The whole there's a tagline about benefiting science in humanity. So we try to walk that talk, right? We try to help our partners teach our partners, as well as like deliver novel solutions. The cool thing about unique about our department specifically, and our division is we don't just deliver the novel solution, but we also try to make contributions or offer things in the open source. So hence, a lot of the robot drivers, some of the calibration utilities, they they're born out of this group, specifically, as well as some of our partner groups. ROS industrial is stewarded by three organizations you've mentioned Southwest Research Institute, but I'd be remiss not to mention Fraunhofer IPA in Stuttgart, Germany and the advanced remanufacturing Technology Center in Singapore, Stuart ROS, industrial Europe and ROS industrial Asia Pacific, respectively. Those organizations also make great contributions to open source and of course, ROS ROS, too, and ROS industrial.

Can you tell me a bit about the consortium? Around ROS industrial?

Right. So in 2013, and 2013, is when it first got started, like, there's a lot you can work on, right, like so we had all these problems, just getting the robot like getting my application to work on all these different robots. But there's all this other stuff like right, should we be doing motion? To be What about perception, what to focus

on, basically, writers out there early on things that you could possibly focus on?

Okay. Yeah, my apprentice, my predecessor, my predecessor, Paul Voss, now and plus one. He, he basically was like, started, like, basically, surveying industry is like, Hey, what are some of your pain points and work? What's really working on it just naturally, sort of like, like, hey, what if we started a consortium, and you chip in a little bit, and we start building up this repository and maintain it. And for tools to enable you

to chip in a little bit, you mean, you fund a little bit so that, like, you have some sort of stake, and it funds like a few people to work on this full time, most of their time.

That's the intent of the consortium, right. So there's some level of maintenance, to keep things at least running to a degree. And so our membership dues are different. Yeah. And then, and then at enables, like some level of technical support. So one of the benefit benefits for people who are involved in the consortium, is they get some call, like, hey, help me like tech support. As well as like, you know, we do a lot of maintenance. And, of course, obviously, pushing out some new stuff fixes, new versions, etc, we maintain the Qt plugin for ROS. And different functions like that. And put on obviously, the training, it supports the training events. But that's really what the dues cover any sort of collaborative project is separately specked out and if people want to contribute income, and software or, or just fun labor, they can do that with veggie spun out separate. So the consortium seeks to provide the direction for what we do in the open source. The first thing, sorry,

its direction in a sense that they inform you of what they're working on what their needs are. But it also comes with these added benefits of you, you can you support them. And then you also provide these training things. Drink training, I don't know the courses or we may refer to it

to training events, right? Yeah, we enable them to sort of like kind of influence the agenda, like, oh, it'd be nice to have more more on navigation. Right? Oh, we generally, like we've started pushing out ROS 2 training and we got a lot of feedback that like, Hey, this is fine. But like, how do we make it run with our ROS 1 stuff? Right. So we ended? Yeah, add in like, yeah, added, like best practices with the bridge, pulling exercises, and hey, what are some things for to help us port stuff? And so that's, and that that's a key role, right? I mean, Oh, for sure. consortium, the membership, like provides this sort of, well, and because they are paying membership dues, they will definitely express their opinions. And that's right. I mean, we're sure

if it's the best match desires of industry, represented by the consortium.

Yeah. So we have like this, like, baked in voice of voice of the customer. Right. So that's been really good. And then shortly after we started the Americas consortium, Fraunhofer IPA stood up a ROS industrial consortium in Europe. They have a subset of members and Asia Pacific. Like I said, let a Singapore RTC Singapore. Were there one of your colleagues, a couple of your colleagues in Singapore I think just down the road from RTC. They support the Asia Pacific console issue and when we work all off, like a common agreement, so we have reciprocal benefits. And it's really very uniform from either outside, as well as inside. We tried to make it consistent, uniform experience, like very often often Asia Pacific member who has someone sitting here in the States, and they'll attend our training events. That's okay, that's nice. How do you?

How does it work when companies join the consortium? Is it? So is it like a fixed do or can companies though? Like, if? I don't know, I like say it's a $5,000 for one seat, or something? Does it is Can someone contribute a whole bunch and basically monopolize how it works? Or how does all I guess how do you balance different companies?

ROS-Industrial

ROS-Industrial's Twitter

ROS-Industrial's Discourse

Southwest Research Institute

Accelerating Industrial Workflows with Open Source

Rising Tide Lifts All Boats

In this episode, Audrow Nash speaks to Steve Macenski, who is the Open Source Robotics Engineering Lead at Samsung Research America. Steve leads the Nav2 project, which is an open source robot navigation framework. In this conversation, they talk about the problem and challenges of robot navigation, how Nav2 works at a high level, hybrid planners, and about Steve’s experience working with the Nav2 community.

Introducing Steve and Nav2

History of the navigation stack and Nav2

What problems does Nav2 solve?

How Nav2, MoveIt, and ROS Control relate

Challenges in navigation

Working in a grid world

Motion primitives

Potential fields

Unknown maps

Relating paths to wheel velocities

When is Pathplanning used?

Hybrid planners

Changing environments

Behavior trees

Behavior tree example

Behavior trees versus alternatives

Nav2 and SLAM

Nav2 for different types of surface environments

Experience with the Nav2 community

Open Source as a business model

Getting involved

Career advice

This is a conversation with Steve Macenski, who is the Open Source Robotics Engineering lead at Samsung Research America. Steve leads the Nav2 two project, which is an open source robot navigation framework. In this interview, Steve and I talk about the problem and challenges in robot navigation, how Nav2 works at a high level, hybrid planners, and about Steve's experience working with the Nav2 community. This is the Sense Think Act Podcast. I'm Audrow. Nash. Thank you to our founding sponsor Open Robotics. And now here's my conversation with Steve. Steve, would you introduce yourself?

Hi, yeah, I'm Steven Macenski. And the open source engineering lead for a box at Samsung research. So I'm the lead developer on the ROS to navigation stack. And I sit on the TSC. For us, too, as well. And the TSC is the Technical Steering Committee. Correct? Yeah. So it's like the the board for ROS, let's say. So it's a bunch of different companies that kind of come together with Idris and ROS 2 to make sure that we can see the project the right direction.

So yeah, tell me a bit about the navigation stack or not. And then how that relates to NAV two.

So I'm going to navigate stack was the the, you know, the software built at Willow Garage to do you know, mobile based application, you know, primarily for the PR to robot. But now even in those early days, they understood that a lot of the same technology could be applied to other mobile platforms. So I made at the time they made it, it was awesome. The fact that there was a software out there that was free and open source and with a strong community around it with documentation and examples and all that kind of good stuff. But you know, over time, you know, that was, you know, in the very early days of robotics, where basically the light robot you could buy if your home was like, you know, an iRobot Roomba. And there really were few of any kind of service robots, you know, in application. What year do you think this was? Was this like, 22,000? Or when do you think it was like 12 ish is when started. And then there's a big push to update some algorithms. And I think the summer of 2011 2012, when a bunch of insurance guy came in, and absolute new new variations of the algorithms.

Okay, so you had this open source navigation stack? And that was with Willow Garage. And then how did that evolve? How did that how did we go from there? Yeah,

so it's after we'll closed in the 2013 timeframe, kind of all major developments on the navigation, kind of kind of halted. So you had folks like David Liu and Mike Ferguson that continued to maintain things over time, to date. So David was, you know, working at Lucas robotics and working as a consultant doing doing various navigation work. And then Mike Ferguson was starting, what was the precursor to Fetch robotics, then later, Fetch robotics. And so he was using in the early days, some of the navigation stuff, I think, these days, I have to assume everything's been stripped out their systems at this point, but in the early days to try and build that MVP to get the venture capital money, that that's the he was working on on the napkin stuff, as well. But then, you know, nothing really major change from from 2013, basically, until about, you know, 2019 or so when, you know, ROS 2 started hitting kind of more mature milestones. So I think with without see is when Matt Hanson, in the open source robotics team at Intel, had a big post a discourse saying that, hey, there, they want to work on this roster navigation stack. They didn't want it to just be a straight port for boss, one that wants to redesign from the ground up based on like, the current requirements that are basically is asking people to air their grievances about the boss one stack, or what kinds of things they were swapping out the stack because they didn't meet the requirements. And so that that actually, that discourse posts, I still today occasionally open up and go down a list of refers to it to make sure Yeah, to make sure like, okay, oh, like I forgot about this, we still haven't addressed this point. And I'll add that to my queue. But at this point, a lot of those lob abroad, things that people said are have not been handled for the most part. So really, Matt, and the team at Intel, really put like the legwork in to get this get the ball rolling. And when they did that's kind of when I when I jumped on on the bandwagon to start working on things. But that's essentially how that app to project kind of got going. And how we got a lot of the architectural elements we see today.

Gotcha. So if I understand correctly, so it was started at Willow Garage in 2010. Ish. And the as a general navigation library, and then ROS 1 came out. And it was used with ROS 1. So the general navigation code was using ROS 1. It continued and people were using that. And then ROS 2 came out in an early version, he said bouncy, and then you then ask the community for what was wrong or what could be improved. And then this is how we have come to NAV to

Yeah, I mean, that's it It wasn't me that that was Matt and the team at Intel, they were really the ones in the early days when you're porting stuff over and starting to do some of the core architecture elements that were taking the lead. So I think they had like six people working on it. So they there was a lot of professional software engineering time put into it. And in those early

days, nice, and how did you get involved?

Yeah, so I've been kind of doing open source stuff for a bit before that point. So when I was doing the robotics team at semi robotics, I built the STL layer, as well as the SAM toolbox library. And so I've already done a couple of ROS con talks on those topics. So Samsung was looking to hire somebody to work with the open source group on various open source box technologies. And they've reached out to Matt Hanson asking kind of who they might recommend that would be good for this. And they said that, you know, I might be good, good, good candidate. At the time, I was leaving Sydney robotics, and so is really good transition to go over to Samsung research, to work on this open source technology full time, versus just, you know, working on these problems that are specific to Cindy, for the end products, and then open sourcing the little bits here and there that makes sense open source. You know, now I can start working on open source things and make big infrastructure changes that are that are fully in the open.

Yeah, it's awesome. It's an awesome position to be in. It's fun. Oh, yeah. With. So tell me kind of at a high level, what problems does not have to solve?

For work to solve? Yeah, so now, nav two is basically sitting between you have a bunch of motors and a motor controller that can take take philosophy commands to go forward and your autonomy application where you want robots, you say I want robot to go there. So it kind of deals with everything in the middle from the the mapping and localization systems, which I consider to be slightly separated from navigation, I consider the navigation, one problem set and localization to be another problem set. But we do have reference implementations that work with AMC LSF toolbox for that localization element. And then on the navigation side, and we say, well, now that we know where we are within some global representation of the world, and we have this global representation of the world, how do we, you know, navigate the space in order to, you know, go to our to our final goal. So this could be in a warehouse, like the position on a shelf, where you want to have go pick a box, or with your Roomba say, you know, what's this little pattern I want to nap on the floor in order to clean my space? So we solved that problem of saying, how do we take some sensor data from from just hardware and populate a representation of our environment, given the current state of the world? And then once we have the state of the world, how do we make a route that goes from where I am to where I want to be within that environment. And then finally, dealing with the the lower level system of okay, how I now follow this path that goes from where I am to where I want to be accurately efficiently. And there's obviously more little bells and whistles on widgets, here and there to help make that you know, either more efficient or more configurable or deal with fault tolerances and things like that. But that'd be kind of the core problem navigation and actually tries to solve.

So it's, it's sounds like several things. How do you? How do you find a good delimiter? Between math two and other ROS libraries? Like ROS to control for example, or move it or any of these other ones?

Yeah, so I guess these all become parallel pillars. I mean, these are all by themselves really significant applications that have their own specific developer code bases and don't own individual ecosystems, and are they themselves you know, equivalently complex challenges to solve. So move that I say in the most simplistic terms is our navigation, which was actually the the the library name from Willow Garage that became move it was application. So it's often the same kinds of problems as as the mobile robot navigation that nav two solves, but for arms, so it's taking potentially sensor data to build an environmental representation. In the case of movement, they're using Okta map and with 3d Octrees, then they're using sampling based motion planning in order to get a a rough plan from point A to point B. And using various smoothing techniques usually optimization base to smooth those out to be a little bit less clunky on the on the motors earlier while you're going through the sampling base planner would give you naturally and then finally, then having those the same kind of local trajectory controllers, we have Neptune now with the hyperplane schema. So there's somebody a very, very analog problem, pretty much everything they exist in that one nephew, rather, and move and have some sort of direct analog to each other for the most part. And then your ROS control is is kind of sitting in that lower level on the actual electronics solving the motor control issue of saying, you know, I have this I have this motor and I want to control this motor to do something interesting for me. So if you have a differential drive robot, how do I take this this block The command I get for navigation and and follow that with, with my my motor efforts, or for manipulation, which is frankly, more of what ROS control is used for has, you know more advanced impedance controllers and other other type of modeling to deal with, you know, the kind of manipulation problem. For the most part, the navigation control systems at the lower level are, you know, very simple, right, this is the kind of stuff that undergraduates might do in their, their, their intro to controls lab, it's not not overly complex,

like a, like a proportional integral derivative or PID controller kind of thing or, yeah, to try to get it to a position.

Yeah, pictures are pretty, pretty common, as well as it potentially if you're working on warehousing robot where you have, you know, different different weights of opposite objects on top of it. You there's other slightly more advanced control scheme you might use, that's based on on efforts rather than just just photocurrents

Petrit. Now, why is can you tell me some of the challenges and making something perform navigation? Like, what are some of the problems that nav two is solving, and there's some flexibility around the ways it salts but so

yeah, so, you know, one of the biggest challenges is just taking both the representation of your environment you have from from mapping or localization or you know, other previous data that you've collected and somehow assembled. And then the current data you have about the environment. So if you have a 2d laser scanner on it, or you have a RGB camera on it, which provides you depth information, as well as color camera information, in, you know, radars, sonars tall sensors, I mean, there's, there's a large variety of different sensors that you can utilize that robot. And most robot systems actually have multiple these types of sensors on them. So the problem is dealing with how do we take all that sort of information and populate something that is now useful to solve the navigation problem. And so we do that with a navigation stack with the the cost of 2d package, which is essentially a direct port. For last one, we have some additional plugin layers, we've added for more advanced capabilities, but you know, high level, exact same package, long term, we'd like to actually replace that with something a bit a bit more advanced. But at the moment, that's where we're at. So that's one of the big problems. Another problem is saying, Okay, well, now that we have this populated representation of my environment with current sensor data is being fused into it. How do I actually plan within that space so that my robot can can, you know, achieve its goal? And so there's a variety of different methods of doing that, obviously. So there's, there's, you know, holonomic planners that work well with like 2d, or grid, basically techniques that work very well with

holonomic. It means it's like a car or something, right? And non holonomic. robot can drive like sideways and things where a car has to go forward or backward. That's what all means, right? Or?

Yeah, it's a hot holonomic means it can move in any direction. So think about a robot with Swiss wheels, or whatever they're called, or yeah, those little roller wheels. Yeah. Or, or potentially, where you can pick the wheels themselves to be able move sideways. Yep, those are considered to be holonomic robots. On a first order approximation, we also say that differential drive robots, if they're circular are also holonomic. Because if you can pit it perfectly in place as a different drive robot, and you're circular, so your your footprint isn't moving, you can you can achieve the same kind of behavior as with a truly holonomic robot,

because the angle is facing and then you can drive anyway. I see. Yeah.

Yeah, precisely. So for those kind of robots, it's really easy to leverage just like, you know, your usual a storage actress algorithm that you might learn in, like an intro to algorithms class, but those are perfectly serviceable, you probably want to do some smoothing on it, if you are doing a direct research so that you don't have like little jaggedy lines on thing. But you know, it's there's there's different ways we do that through navigation functions, which are basically generating these potential fields of cost that you're you're tracing on instead of just like directly just doing a grid search. But I mean, at a at a conceptual level, they're, they're, you know, the same kind of class of algorithm. As a star. Yeah, it is. They solve the same kind of problems. Yeah, they're, they're giving you a path through space.

Yep. And if I remember correctly, so for a star, it's a heuristic, a search that you say the goal is that way and I'm going to try to search in that direction. And then you find the optimal in a grid world you'll find the optimal solution of what's the shortest path to get there. And what was extras? Because it is it more general thing or forget how it relates?

Just breath first, it just goes in every direction at the same time. You don't have there's not a heuristic that that saying, like go in that direction, it's just gonna go out and expand its hole, find the optimal solution.

Okay, and then we feel for these if you have a start and an end spot, and so it's going to search the space, and it's going to find it. And we're representing space in a grid world, or some sort of representation. And so that's why you would have these kind of jagged edges, rather than a smooth way to get there, per se, if you go up into diagonal to the grid.

Yeah, so So you had like, basically, the motion model motion model, so to speak, that you're using is, is the the search neighborhood. So you're saying, I can search, you know, left, right up down for like the four connects neighborhood or say uptown, you know, up, down, left, right, and then the diagonal? So that's the eight connected space, and so that they'll let you then work at, you know, how do you plan to go 45 degrees, but you're still restricted only to these, these, you know, exactly the angle increments of either going, you know, zero 90 or 45 degrees for your planner. So this works, you know, fairly well, when you're working with just a simple, you know, simple base that like you might see on your on your on your Roomba over this kind of breaks down, it becomes more of a significant issue is when you're working with your larger noncircular robots. So think about like a forklift, or you're thinking about like, I think that's robotic cells, like a big platform robot, I don't, I don't know they call it. It's like something 1500 bucks. The thing is, and those are obviously like, very noncircular. And so even though they are a differential drive robot and get pivot in place, if you need to do a bit bit more work to make sure that it's a viable the plant or a space when you're in a narrow aisle way where maybe the robot can just trivially rotate in place,

because it could swing and hit something because it Yeah, exactly. It's rectangular or something. Okay,

yeah. And the other big kind of class of problem that that you have working with those techniques is, when you have drive trains aren't, you know, trivial that you can treat essentially as holonomic. So you mentioned before, like cars like the Ackermann model, which is the car model we talked about. So that that's, that's one example of where if you just plan a path as is go here, go straight to the right, you know, your car can't do that, right. That's not, that's not a feasible way of getting from point A to point B for the actual drive train at the robot. So you have to then take into account, not just like the the emotion model of saying this for a cap space, but instead, using in your search space, user space planner, actual motion primitives that better describe the capabilities of your drive train. And that's where we have some of the new algorithms that Neptune provides in the snap planner, which is the hybrid, a star planner, and the state lattice planner, which would enable us to set different search patterns, so that you can create physically feasible paths that robots with with constrained drive trains can actually navigate exactly and not not Not, not not not treat as holonomic.

So by search patterns, human kind of simulating the position in space, to try to see if it's valid from one position to another. So like, I have a car, I'm going to try to move it from one spot to another. And I'm going to kind of project it out into the future and try to see if that path gets us there, or what do you mean by search?

Yeah, so So imagining in this this 2d example, we said, If I'm at the cell, I can, the cells I can recently traverse are either up down and left or right in this forecasted space. So that that will be the motion model that we're using to, to to expand search. Because then once you go up, we apply the same for then you just keep doing that recursively. And you get, you know, eventually like a chessboard span basically.

So with for instance, let's, let's talk about the simpler one first. So for hybrid, a star, which the name kind of breaking that down you a hybrid. So you know, there's something different about this than a star. So you're still using this, like heuristic search algorithm at its core. And what that's doing is, is what we're first who is hybrid sampling, which is that instead of just naively saying we're going to, we're going to visit each of these grid cells just as the grid structure. Instead, we're going to treat our search neighborhood as primitives of actual velocity commands the robot can achieve. So for a car like model, you can either turn left and drive a bit, you can go straight and drive bit or you can turn right and those are the premises.

The turn left a little bit turn right a little bit go straight. Yeah, I suppose backwards as well. And backwards, or whatever. Okay, precisely.

Yeah. So those are the vacancies. Yeah. And then, you know, for the sake of implementation, the length of these primitives you want to set is something interesting and not, you know, arbitrarily just like driving the future a bunch because then you'd otherwise you'd end up going in circles on the left circles on the right and so straight forever. You have to decide, like, what are the lengths of these things I want to work with.

Okay, and you want to swap it basically, so that you can find things okay. So you don't go all the way in that circle, if you turn your wheel a little bit to the left kind of thing. Yeah.

But because we're also still working. So to go back, we're still working in this grid space, though. So we need to make sure that we go small amount, but needs to be enough. So that guarantees that we leave our current cells. So as we're expanding the search over time, we don't have situations where we're revisiting the same cell we currently had, was certainly something that can be dealt with. Yep. Yeah. Or actually, they, it's all geometry, you have read the code. And this is like, you know, six lines, where I go through step by step how to mentally drive this, this this idea. But while the paper itself of the hybrid star from from Thron, and folks from the DARPA Urban Challenge, or grand challenge, one of the challenges, they they, I think, allow you to do that to have some sort of solution space for how they select which one to use. I instead say I don't want to deal with that problem. I'm assuming I printed sufficiently long, so we always guaranteed to leave a cell. So there are a minimum square root of two cell lengths away from each other. So that way, we're guaranteed for any time that the proof hits a cell, it's always guaranteed to leave that cell in the next iteration.

Yeah, um, well, I guess they don't don't have to be. But I mean, the first I have to they are, yeah, gotcha. They could be circles. They could be hexagons, I've actually I did some, some research and a couple papers I've, I've just not written for publication, but written terms of like, you know, to take in regarding actually having an environmental Model B, you have a hex hexagonal cells, rather than grid cells. But that ends up having a whole lot of other issues. But it's interesting to think

about. Okay, so you have this grid, and you have a way of describing how your robot moves in this grid. If you have a robot that's like a car or something you have some idea for, if you tell it to drive forward at left for this angle, or something, how it goes into another cell? How do you just it sounds like a hard representation, because I'm imagining, if I turn the wheel 10 degrees to the left, like, and I drive, I'm not in the middle of a cell, but I guess I still am in a cell. And you can use that for planning your trajectory.

So rather than just so with these grid based methods, or essentially visiting, let's say, like, we go up, down, left and right, yeah, we could say that, that, you know, you're you're not busy and arbitrary place in that cell, right. Because if you blow that any, any any space, we blow it up infinitely, to go to look at it. So we look a little cell and make it really big. It's like, well, we're visiting any random place we're visiting, let's say the center or an edge, depending on how you want to find it, of that cell and each cell are visiting the center of, but when you're working with these motion primitives, you're right, we don't actually have exact angle, these aren't computers that were ending exactly in the center of every single cell every single time. So we do actually have the cash in the planning sequence, where within that, that, that that that cell, we've actually landed, and then when we go back, and we visit that cell to expand it, again, we're actually not using the center of that cell, we're again using that that position within the cell that we actually visited, which then, you know, if you compound that over and over again, to build a path, that means that it's exactly drivable in this grid

Yeah, um, so if you were, if you were to have, for instance, like a naive implementation of this, you might think about saying, let's approximate that I had this this this primitive that I'm projecting forward at a curve, and it ends at within a cell, but not in the center of the cell, it might be natural for you to think if you come from an from like a 2d grid lands space to say, Oh, well, you know, these, these, these cells, these B cells are sufficiently small that I can just store that as the as I've entered the cell. And then the next iteration, when you when you visit that cell to expand it to get its neighbors, you could then say, Okay, well, let's just pretend like we started at the center and expand it again. But if you keep doing that, then the path you get at the end of it actually isn't then respecting the constraints on that motion model. Because you're you're not accurate, you're that the small errors compound, each of each individual primitive. So instead of, yeah, so So doing that when we when we, we visit, we visit a cell and we're trying to find its neighbors, we're not using an approximation of the position that we're using. As like the center, we're using the real position of the actual cell coordinates used, so that as that compounds together that's going to be guaranteed to be drivable within the kinematics of the of the the robot system.

Interesting. So how does it work if I want to go from one spot to another spot, and I have these motion primitives, which say I can turn and for these motion primitives, if I have a car? Do you have like basically infinite motion primitives because you have the whole degree of angles that the car can go or how Is it? How do you go from one spot to another? Do you know what I'm asking? With? Motion permitted? Yeah, yeah. So you have the continuous range between, say, minus 45 and positive 45, that the car can travel in, that the wheels could face.

Yeah, so so we subsample that space. So essentially, we have for the hybrid star, and in the current implementation that that exists today, you have a left turn, you have a right turn either straight turn, and then you have the reverse equivalents of those things. So you have either three or six motion primitives you're working with, that are the most extreme terms, right? The the your your robot model can obtain, and it randomly samples in

that space, or just uses one of one of the six will expand.

So expand all of them, right. So every time we visit a cell from an A star, we expand every single cell, and we set a heuristic cost at the end value of that cell that goes into a priority queue to sort who I the the most optimal trajectory possible. And then when we go back through and visit the next cell in that priority queue, you're getting back out that that next cell that you want to go to, so you're not, that's not a goal. It's not like a the, the, it was not a heuristic search, and you're using a set A dexterous algorithm on this kind of thing. And right, every single aisle six, aisle six, aisle six, and you just be spreading and spreading and spreading out, which is problematic. That's where your priority,

so which is very clever. We're Yeah, it seems like you're going to do it.

Yeah. That's pretty standard for a star's that that's kind of, at least to my understanding, that's how most people do it.

Gotcha. So you, but it's not. So you're only using these primitives of like extreme angles. So it's all the way this all the way left all the way right or straight? If we're just doing the forward case. In this case, if I How do you arrive at a solution where the robot doesn't drive like incredibly manically doing all the way left all the way? Right, on its way towards its goal?

Yeah, so that's where we have a smoothing step at the end of this. So this we do this for

the house space, basically, this is this is for you arriving at a solution. And then from that solution, you smooth it.

Yeah. So so we have. So we thought through this, we have now a path that is rival maybe, yeah, little suurvey. But drivable. And so what we do is we take that path, and we put it into a smoothing algorithm, which will, you know, take out some of those extreme turns and make them a little bit more, or I guess, less aggressive. And then also goes through the boundary conditions to make sure that those remain tight during the entire entire process. So that part

of boundary conditions and make sure, yeah, what do you mean by that?

So sorry, to type that word. But so essentially,

we mean, optimal at night, but what do you mean, like a good solo error? Or I don't know what you mean, actually.

Yeah, so we have. So in the hyperstar, paper that is published that, you know, an autonomous driving company might use, for instance, has a pretty advanced move there. And in fact, this is the some of the information we used to use within the spec planner, that takes into account, you know, keeping spacing, the sames, which helps us smoothness, using our Fermoy diagram to help you drive away from obstacles and then limiting the curvature so that you you keep those constraints that you had during the planning process. And I think a couple other terms as well. And those then spin through an optimization problem to give you an optimal solution. Given those those cost functions, this ends up being a very computationally expensive thing to do. And it's ultimately why I removed that, and instead, Swift changed up a bit of the internal systems of how the search the search, or restrict operates, so that we don't have to use that kind of smoothing technique. But what we use instead is essentially like a super, super naive, smoother, like you might find in like, I don't know, like, one of those Udacity nano degree kind of things, we just go through very simply and say, like, let's make these things as close together as possible. And then iterate until we have very little change left in our in our in our outside path between

these things, which actually things have close to each other. What I mean by that

each of the the points on the path. Okay, so

in opposition to land, it's maybe not as intuitive to somebody who isn't who's worked with it. But it's something that that's generally a very successful way of smoothing is just say, all my points my path, I want to make sure that they're each as close they're evenly distributed from each other. And when you iterate over that and try to minimize that cost function, you end up actually smoothing out the path from from turn to such which is a convenient and easy Oh, that's a waste smoothie. I

see. It's basically like the shortest path from something is straight to it. And so if you're making all your points closer to each Other, you probably are working towards that. Because they will be further. That is a cool heuristic to use to smooth. Yeah. Interesting. So if

you had no no other, other things doubting that solution, and essentially what you're right, and just being exact, perfect. Just ignore everything. Yeah, yeah. But that's why I have to add in things like, collision looks otherwise Yeah, definitely all into it, for sure. Yeah. And then, because we derive it kinematics, these will path we also want to add some weights on the original path itself. So that, like, if you have a little squiggle that could be made into a straight line, we want that to smooth out. But we don't necessarily want something to turn from a nice smooth turn, just like you know,

Yeah, where it's no longer actually also valid by the current motion model. And so I have, I tried this more in the actual the readme file. And I don't think it's worth getting to the specific details of this at the moment. But essentially, you can, you can, through logical proofs, find that the only time when you break feasibility is during sharp turns at the beginning and the end of the path. And so those are the boundary conditions, okay, so the boundary conditions totally break during this process. So if you were to try and drive the robot using just that pseudo described, both the start and the end would not be guaranteed to be drivable. So we do is we have an additional post processing step on the beginning and the end of it, where we use the Kinect motion model of the robot that we use previously during that planning technique to take the Star Pose and then sample across the new smooth path until we find a the shortest path that goes from the start the actual starting point to a point on the smooth path, or to give us a drivable connection point between those two areas. And so by doing that, then we end up with a smooth path that also then then meets the boundary conditions of drivability, and therefore the end path is

drivable. So I don't know that I fully understand that. But can you think of it as just if the start and end will be wrong because of some weird boundary conditions in the optimization, you just extend them a little bit. So you consider like a little before the start and a little after the end? Assuming the motion primitive, just continue with something like this or?

So I actually I tried that it didn't work. But yeah, I did. I did try that I tried a bunch of stuff before I found the solution that we currently use. So essentially, so this is where it gets it gets a little out into the weeds. So within these planners, we also have a concept called analytic expansion, where occasionally during our search process, we take our search position and our goal position. And we use a library called Oh MPL, the open motion planning library, which is commonly used for sampling bass planning implementations. They also have a very good implementation of the Ackermann models that you can use both for sampling but also in our

5 posts with the industry tag