Displacement maps can be used for Tessellation, Parallax Occlusion Mapping and Offset Bump Mapping. Textures are setup the same way which allows easy iteration and choice on which type of displacement to use and which suits your assets/scene best.
Displacement maps (or Height maps) have to be explicitly stored using the _displ suffix (i.e. mytex_displ.tif).
Do NOT attach the height map as an alpha channel of the normal map as this mode is deprecated.
Save the displacement map in the alpha channel of your _displ texture. The RGB channels can be left empty.
Use the DisplacementMap preset to store _displ textures. Height maps will be converted to A8 textures.
If you don't see any displacement, double check the format in the Editor's texture file dialog preview. If it isn't A8 fix the preset, save and reload.
For information on how to setup Displacement maps in Sandbox, please see the following articles: