Apple faces a potentially heavy fine and may have to open up its mobile payments system to competitors, the latest development in a long-running saga between the European Union and the iPhone maker.