Hi all,
As stated in some previous OpenAMP meeting, we encountered a limitation with remoteproc resource table. Indeed, all resources are encoded using 32 bits fields whereas the memory accessible on our processor is above the 32 bits boundary.
For instance, when the physical address is 0x1_0000_0000, then this PA can't be inserted in the resource table or it will be truncated to 0. This can temporarily be workarounded by using device tree dma-ranges (at least on Linux) to have a correct DA but the PA will still be truncated. Moreover, this only work if you can remap the 64 bits memory zone to a 32 bits one (this almost implies to have an IOMMU in front of the device).
Since 64 bits systems are now pretty common, it seems clear that we need to modify the resource table to support 64 bits addresses. In the same time, it will also be necessary to use 64 bits storage for virtio device features. Some of them already need more than 32 bits (virtio-net features for instance).
Both modifications requires to handle versionning in the resource table.
The following suggestions are mainly a RFC ! Please feel free to comment any of the proposed modifications.
Versionning ----------- Currently, the supported version is "1". I propose to increment this number (version 2 ?) which will allow to add necessary evolutions for 64 bits support. This version should probably support both old format and new 64 bits support (comments are welcomed).
64 bits addresses -----------------
Since most of the systems are now using 64bits addresses, it is necessary to support such configuration
There are various problems which occurs when modifying these fields - Retro compatibility (We probably must keep it) - Fields alignment (Some architecture might not handle misalignment) - 32 bit remote vs 64 bits master
Regarding the retrocompatibility, it appears clear that modifiying existing resources is probably not feasible due to missing reserved fields. It will probably be necessary to create a secondary type of resources which are almost the same as the 32bits one. For instance for a vdev, we will have the "classic" rsc flavor:
struct fw_rsc_carveout { u32 da; u32 pa; u32 len; u32 flags; u32 reserved; u8 name[32]; } __packed;
And the 64 bits one:
struct fw_rsc_carveout_64 { u64 da; u64 pa; u32 len; u32 flags; u32 reserved; u8 name[32]; } __packed;
This does not seems really ideal and adds a lot of duplication but I do not have (yet) any better idea (again, comments are welcomed !).
However, what should we do with the other fields ? Let them on 32 bits or change them all to use 64 bits ? If keeping some of them on 32bits, this can lead to misaligned members (since they are packed, the compiler will force the alignment). Since some architecture do not support that well, it's probably better to ensure all fields will be aligned on their natural type boundary. it also means that all structures sizes must be a multiple of 8 bytes.
If we don't want to guarantee alignment, then I do not have yet a clear picture of what it would involve on various drivers which accesses the memory of the resource table (rproc, virtio for the config space, etc) and what are their assumptions about that.
If this can be accepted, then the question of the 64 bits master versus the 32 bits remote can be addressed. I think this is probably orthogonal to the fact we can have 64 bits addresses in the resource table. Indeed, if both PA and DA are stored using 64 bits, then, there is no limitation on what can be done on theses addresses.
For instance, on Linux Kernel, the device tree can allow remapping 64 bits to 32 bits using dma-ranges.
Features -------- In virtio, the set of features is a bitfield of 64 bits. The current virtio-rproc implementation only allows for 32bits features. This is already limiting since virtio-net needs more than 32 bits (VIRTIO_NET_F_STANDBY and VIRTIO_NET_F_SPEED_DUPLEX). So these fields (gfeatures, etc) will also need to be udated to 64 bits variants. It might also be a good idea to keep in mind that future evolutions will probably extend the available features.
Comments are welcomed !
Regards,
Clément Léger