U of T prof seeks higher order construction robotics using AI

The use of robotics on construction sites has failed to reach its potential, says a University of Toronto researcher, but with more research, says Daeho Kim, a higher-order fully autonomous mobile robot could be one step closer to the realization.

Kim, an assistant professor in the U of T College of Applied Sciences and Engineering, says most of today’s so-called robots patrolling construction sites should be more accurately referred to as tools that repeat some pre-programmed tasks.

A few success stories aside, what is missing is full robotic digitization and automation that uses human-level visual artificial intelligence (AI) to fully understand the construction sites where they are deployed.

To reach the high level of visual AI that robotics will power on sites, millions of images are required, but for a variety of reasons, getting that number is impractical. What Kim and her team propose are two novel techniques: synthesizing virtual building images and generating miniature-scale building images.

“As we develop new forms of construction robots, the hardware part has come a long way, for example, Spot from Boston Dynamics, but the software development, that artificial intelligence part, still has a long way to go. Kim said.

“The problem is that we are missing training data for construction scenes. DNN, deep neural network, the core engine of visual AI, is a supervised model, which naturally becomes greedy for data. To develop a well-trained AI… we need a large number of well-diversified training images for construction scenes.”

Kim’s research program was one of 251 university initiatives announced as recipients of a total of $64 million in funding from the Canadian Foundation for Innovation’s John R. Evans Leaders Fund in September.

Their project brief, submitted to the innovation hub, said: “AI-enhanced robotic solutions will collaborate with field workers safely, improving productivity and profitability while compensating for growing labor shortages. The proposed research project is essential to making this vision a reality, as it delivers optimized and field-applicable DNN models, a critical next step in the development of autonomous construction robots.”

The robotics will collect, analyze and document site information, enabling the creation of live digital twin models of ongoing construction sites.

Kim explained that synthesizing images to be developed into visual AI is required, first because it is difficult to collect the data in person.

Surveillance cameras and drones have occlusions, are very expensive (Kim mentioned $2-$10 per image), and have other problems.

Collecting a million images would take a long time, and there are various problematic regulations and confidentiality issues.

Trading and data sharing in a competitive construction environment are other issues.

Work is progressing quickly in Kim’s U of T lab, and the team uses five tensor processing units and Google Cloud software. More computational resources are required.

To develop a well-trained artificial intelligence, researchers need a large number of well-diversified training images of construction scenes, says University of Toronto assistant professor Daeho Kim.
SCREEN GRAB ALI TOHIDI VIDEO: To develop a well-trained artificial intelligence, researchers need a large number of well-diversified training footage of construction scenes, says University of Toronto assistant professor Daeho Kim.

“We are fully focused on developing simulation software that can automatically synthesize non-real but realistic-looking construction images, and a few weeks ago, we started actively generating one million construction training images. This is exciting news for me as, to my knowledge, we have never had the opportunity to use a million training images in construction DNN training before,” he said.

The steps of the synthesis include the creation of a 3D human model, followed by the input of worker motion capture data; create a 3D construction worker avatar by mapping the 2D or 3D clothing map onto the 3D human model; set the imaging conditions randomly, including the distance from the camera and the lighting conditions; and synthesize and generate construction images or videos by overlaying the virtual construction worker avatar on 3D construction backgrounds.

Later comes the prototyping of a fully autonomous mobile robot for digital construction twinning that implements the higher order DNN models.

Construction robots will need to be able to monitor and analyze the location, speed and direction of movement, pose, proximity, and other factors that pick up the presence of construction workers.

“It is not yet clear how effective synthetic images are in training AI visual models for a construction scene, which is highly dynamic and unstructured.” Kim said. “We may or may not need our own unique solution.”

For the final step, Kim will need partners in the private sector: he is looking for an innovative construction company that will financially support the research.

Follow the author on Twitter @DonWall_DCN

Leave a Reply

Your email address will not be published. Required fields are marked *