Case study 1: commercial publisher
Bloomsbury, well-known for publishing the Harry Potter series, has launched an academic imprint, Bloomsbury Academic. This will publish academic monographs and they will be Open Access. The business model to be used is to offer free digital access while charging for print copies, e-book bundles to libraries and royalties for using the content commercially, such as in student packs. For more on this venture, see http://www.researchinformation.info/features/feature.php?feature_id=197
Case studies: university presses
For some case studies of university presses publishing Open Access books see here and here.