Re: Keystone Issue

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi,

> On 5 Jun 2020, at 03:29, CodeWiz2280 <codewiz2280@xxxxxxxxx> wrote:
> 
> On Thu, Jun 4, 2020 at 2:24 PM Julien Grall <julien@xxxxxxx> wrote:
>> 
>> Hi,
>> 
>> On 04/06/2020 13:07, CodeWiz2280 wrote:
>>> On Thu, Jun 4, 2020 at 6:16 AM Julien Grall <julien@xxxxxxx> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> On 04/06/2020 10:08, Bertrand Marquis wrote:
>>>>> I would have thought that linux would have need some memory, even small 
>>>>> in the 32bit space in order to boot.
>>>> 
>>>> Yes it needs some, but then they are switching to use the high memory
>>>> alias after the MMU has been switch on.
>>>> 
>>>>  From my understanding, the only difference is the page-tables will
>>>> point to the high memory alias address rather than the low memory one.
>>>> Linux will still be located at the same place but now accessed from the
>>>> high memory alias rather than the low one.
>>>> 
>>>> Note that AFAICT the secondary CPUs will still be brought-up using the
>>>> low memory alias.
>>>> 
>>>>> I could understand that some memory in the low address space needs to be 
>>>>> reserved by Linux as DMA area for peripherals not supporting 36-bit 
>>>>> addresses, but the whole low memory sounds like a big restriction.
>>>> Many platforms have devices only supporting 32-bit DMA, but none of them
>>>> require such aliasing. So this doesn't look to be the issue here.
>>>> 
>>>> TBH, this code is only used by Keystone and switching address space is
>>>> expensive (you have to turn off the MMU, updates page-tables, flush the
>>>> cache...). I find hard to believe a developper would have come up with
>>>> this complexity if it were possible to always use the low memory address
>>>> range. It is even harder to believe Linux community would have accepted it.
>>>> 
>>>>> 
>>>>> Would it be possible to have a bit more information on the “problem with 
>>>>> peripherals” here ?
>>>> 
>>>> I am curious as well, so I looked in more depth :). Going through the
>>>> Linux history, one of the commit message [1] suggests they are switching
>>>> to a coherent address space. The datasheet [2] (page 75) also confirm
>>>> that the low region is not IO coherent.
>>>> 
>>>> So I think you would not be able to do DMA without flush the cache which
>>>> can be pretty expensive. For a PoC, it might be possible to force Linux
>>>> flushing the area before and after each DMA request. This should be
>>>> possible by marking the devices as not coherent.
>>>> 
>>>> Although, I am not entirely sure if there is any fallout.
>>>> 
>>>> @Dave, do you think it is possible for you to have a try? I can provide
>>>> the patch for Linux to disable DMA coherency if possible.
>>> I attempted to do that, where I removed the "dma-coherent" flags from
>>> the device tree.  There are likely other issues, but the most glaring
>>> problem that I ran into is that the ethernet does not work.  Eth0
>>> shows up in ifconfig but there is no activity on it after a small
>>> handful of message exchanges, whereas booting without Xen it seems to
>>> work fine even if left in 32-bit mode (with the dma-coherent
>>> disabled).  I don't know what implications behind the scenes there are
>>> trying to stay in the lower 0x8000_0000 alias range either though.
>> 
>> Thank you for the answer. As wrote, Linux is working fine in 32-bit mode
>> when dma-coherent is left in 32-bit mode. So this suggest a different
>> issue on the platform.
>> 
>> Given that you receive an handful of packet and then nothing, this would
>> lead to maybe an interrupt problem. Can you check whether the number of
>> interrupts increments the same way on baremetal and on Xen?
>> 
>> Dumping /proc/interrupts should be sufficient.
>> 
> I am able to ping the board from itself, do you think it could still
> be an interrupt issue?  It just cannot seem to ping out to a different
> host (or ping from
> my pc).  Unfortunately, the interrupts for the netcp Ethernet driver
> on this board don't show up in the cat /proc/interrupts output under
> the non-Xen kernel or
> Xen loaded kernel from what I can tell.  I'm not sure how I would confirm 
> that.

Could you check the content of /proc/interrupts ?

I did raise an issue several years ago on the keystone 2 related to interrupts 
and virtualization (no with Xen but the context should still be right):
https://e2e.ti.com/support/processors/f/791/t/462126?Keystone-2-no-interrupts-received-out-of-80-and-92-

There might be something to check in regards to level vs front interrupts for 
forwarded interrupts.

Regards
Bertrand



> 
>>> I
>>> would rather run it as intended by switching to the upper
>>> 0x8_0000_0000 alias region.
>> 
>> I agree this would be ideal :).
>> 
>> Cheers,
>> 
>> --
>> Julien Grall