Oh well, turns out that you basically have to set a Role with sufficient privileges for the shared item.
That’s the role assumed by the access_token. That way you could limit access to the shared item, by creating a role with limited access. Even in case you set it to Administrator, access_token still remains scoped to that single shared item.
After setting a role with sufficient privileges access works either using the share link to the Directus Studio App, or you can use the token returned by /shares/auth to access the resource at /items/collection/shared_item_id