You can apply a soft edge to masks and keylines using the Filter Browser.
- Open the Filter Browser and add a new filter set by pressing the + button highlighted below. This will duplicate the filter which was selected at the time.
- Double click on the new filter to rename it to something more appropriate, like ‘Soften.’
- Edit the filters to a Gaussian Blur with appropriate strength, and choose which direction the blur effect should move in.
- Close the Filter Browser.
- Select an appropriate Brush to use from the Brushes Palette, and from the popup menu, choose your newly created added filter.
- Begin to paint along the edges of a shape or key line to smooth the edges.
This technique can be particularly effective for cleaning up rough, 1 bit layer. In the snapshot below, the original information is on the left, and the cleaned up mask is on the right,