AdaFold: Adapting Folding Trajectories of Cloths via Feedback-loop Manipulation

Alberta Longhini1, Michael C. Welle1, Zackory Erickson2, Danica Kragic1,
1KTH Royal Institute of Technology, , 2Carnegie Mellon University
Paper arXiv Code

AdaFold adapts folding trajectories of cloths with different physical propreties,
whitouth knowing the properties in advance.

Abstract

We present AdaFold, a model-based feedback-loop framework for optimizing folding trajectories.

AdaFold extracts a particle-based representation of cloth from RGB-D images and feeds back the representation to a model predictive control to re- plan folding trajectory at every time-step. A key component of AdaFold that enables feedback-loop manipulation is the use of semantic descriptors extracted from geometric features. These descriptors enhance the particle representation of the cloth to distinguish between ambiguous point clouds of differently folded cloths.

Our experiments demonstrate AdaFold’s ability to adapt folding trajectories of cloths with varying physical properties and generalize from simulated training to real- world execution.

AdaFold

AdaFold decouples the problem of feedback-loop manipulation into cloth perception and trajectory optimization.

Description of image

AdaFold optimizes the best folding action at each time-step t, given a set of pick-and-place positions. Leveraging RGB-D observations from multiple calibrated cameras, it extracts point cloud representations augmented with semantic descriptors. These descriptors, Upper and Bottom, are derived from geometric features. The optimal folding action is determined using Model Predictive Control (MPC), which employs a learned model of the cloth to evaluate candidate trajectories (light blue) and update the optimal control sequence (dark blue).

Perception

Given a cloth in a deformed configuration (a), unstructured 3D representation such as point cloud (b) become ambiguous. AdaFold addresses this limitation by using semantic descriptors (c) to enhance the representation of the cloth.

Description of image

The descriptors are derived from geometric features, which define folding lines identifying the upper and bottom layers of the cloth. The layers can be seen as two specific parts of the cloth, one to be folded over the other, varying depending on the cloth type and the desired fold.

Description of image

The descriptors are extracted at time 0, and tracked over the manipulation using XMeM as a pre-trained video object segmentation module. Here we show an example of the output for the upper descriptors of two different objects.

Trajectory Optimization

To optimize the folding trajectory, AdaFold uses sampling-based MPC over a finite horizon H using a learned model of the cloth dynamics. Specifically, at each time-step t, AdaFold samples a set of candidate trajectories an and evaluates the trajectory predicted by the model rollout. Only the first action of the candidate trajectory that minimizes a cost function is executed. The un-executed portion of the trajectory is used to warm-start the optimization at the next time-step.

The cost function is designed to select actions that maximize the alignment between the two halves of the cloth. This alignment is quantitatively assessed using the Intersection over Union (IoU) metric, calculated based on the ratio of the areas occupied by the upper and lower half of the cloth when projected to the folding plane. We additionally incorporate in the cost function a term discouraging actions resulting in large displacements of the cloth.

Visualization of the trajectory optimization process.

Baselines comparison

Visualization of the folding trajectory of AdaFold compared to a baseline executing a predefined trajectory, and an ablation of AdaFold where the semantic descriptors of the upper and bottom layer are discarded.

We present the result on three rectangular cloths with different physical properties.

Fixed Trajectory

Final results:

Description of image 1

AdaFold - No Descriptors

Final results:

Description of image 2

AdaFold

Final results:

Description of image 3

Generalization to different sizes and objects

Visualization of the folding trajectory of AdaFold for objects with different sizes and shapes. Specifically, we test on a rectangular cloth larger than previous ones, a pair of pants already partially folded to demonstrate the method's generalizability to different stages of the folding process, and flattened pants to show generalization on different shapes.

Bigger cloth

Description of image 1

Folded pants

Description of image 2

Leg of the pants

Description of image 3

BibTeX

@article{longhini2024adafold,
  author    = {Longhini, Alberta and Welle, Michael C and Erickson, Zackory and Kragic, Danica},
  title     = {AdaFold: Adapting Folding Trajectories of Cloths via Feedback-loop Manipulation},
  journal   = {IEEE Robotics and Automation Letters},
  year      = {2024},
  publisher={IEEE}
}