Overhauled media player

[s-kaliberda]

Hi team,

Our investigation helped us identify the following approach for implementing Picture-in-Picture (PiP) on iOS.
It is proposed to use VLC as the main player and AVKit for PiP mode.

A seamless transition between full-screen playback and PiP will not be possible. When switching modes, playback will need to be stopped in one player and started in the other, which will result in a short delay.

Key considerations:

• AVKit supports fewer formats, so in some cases PiP may not be available and an error message will need to be shown.
• Additional logic will be required to preserve the selected audio track when switching between players.
• Subtitles may not be supported in the PiP floating window (please confirm if they are required).
• Media queue behavior in PiP may need to be limited due to AVKit format restrictions.
• Supporting external audio tracks in PiP is technically complex and requires confirmation whether it is needed.
• The PiP UI is fully system-controlled and cannot be customized.
• We also investigated VLC 4 (based on VLCKit 4), which already supports PiP. However, it is still in alpha and not production-ready, so it cannot be used at this stage.

[mpivchev]

Hi, thanks for the details!

"It is proposed to use VLC as the main player and AVKit for PiP mode."

• Can the same player UI be achieved no matter which player we are using?

"AVKit supports fewer formats, so in some cases PiP may not be available and an error message will need to be shown."

• We can show a subtle message or some indicator that PIP is not available for the selected video

"Additional logic will be required to preserve the selected audio track when switching between players."

• Is this already in the code you shared?

"Subtitles may not be supported in the PiP floating window (please confirm if they are required)."

• I think subtitles are supported in iOS PIP as an accessibility feature? Are there any current issues you are facing?

[mpivchev]

Just as a side note, would be great if you could sign future commits so DCO passes: https://github.com/nextcloud/ios/pull/4007/checks?check_run_id=64965106498

I set it to passed manually for the current commits.

[s-kaliberda]

Hi, thanks for the details!

"It is proposed to use VLC as the main player and AVKit for PiP mode."

• Can the same player UI be achieved no matter which player we are using?

"AVKit supports fewer formats, so in some cases PiP may not be available and an error message will need to be shown."

• We can show a subtle message or some indicator that PIP is not available for the selected video

"Additional logic will be required to preserve the selected audio track when switching between players."

• Is this already in the code you shared?

"Subtitles may not be supported in the PiP floating window (please confirm if they are required)."

• I think subtitles are supported in iOS PIP as an accessibility feature? Are there any current issues you are facing?

1. UI in the main player will be one, and UI in the floating PiP window will be other
2. We need to load at least file metadata to define if AVKit can play a video and so If we can play the video file in PiP
3. We haven't implemented PiP in shared code, we only implemented media queue there.
4. Yes, subtitles are supported in PiP as accessibility feature

[marinofaggiana]

Hi,
My idea was to completely rewrite the SwiftUI viewer for images and videos. As you can see, the code is quite outdated and in need of a thorough overhaul.

Regarding the video part, and considering that the video simply runs within a normal UIView or View, you can check whether the video is playable with AVFoundation (a very simple function to write and quick to execute). If it doesn't (for example, a MKV or WebM etc.), it will open VLC within the view.

Releasing a version that can no longer open videos with VLC-view is definitely not a choice I feel comfortable making, given that AVFoundation is very poor in codecs.

[s-kaliberda]

Hi,

My idea was to completely rewrite the SwiftUI viewer for images and videos. As you can see, the code is quite outdated and in need of a thorough overhaul.

Since AVKit, AVFoundation, and VLCKit all rely on UIKit, the PiP feature doesn’t require SwiftUI. For now, it’s simpler to stick with UIKit.

Regarding the video part, and considering that the video simply runs within a normal UIView or View, you can check whether the video is playable with AVFoundation (a very simple function to write and quick to execute). If it doesn't (for example, a MKV or WebM etc.), it will open VLC within the view..

That’s feasible — we can implement that.

Releasing a version that can no longer open videos with VLC-view is definitely not a choice I feel comfortable making, given that AVFoundation is very poor in codecs.

Understood — thanks for the clarification.

So the plan is to support opening video files with AVKit, and if AVKit doesn’t support the format, we’ll fall back to opening the video with VLCKit.

[marinofaggiana]

So the plan is to support opening video files with AVKit, and if AVKit doesn’t support the format, we’ll fall back to opening the video with VLCKit.

Yes, exactly, AVFoundation offers all the possibilities to verify this.

About swiftUI It's still a step that needs to be taken, the current code is far too tangled (even if it works), even if the videos use UIKit we can easily embed it inside a simple SwiftUI View with minus line of code, I just need to find the right timeline to start ...