|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] oxenstored memory leak? seems related with XSA-38
On 04/07/13 03:48, Liuqiming (John) wrote:
> Hi all,
>
> Continue my test about oxenstored:
>
> I switch to original C xenstored and test my "broken" vm. The cxenstored do
> not have the "memory leak" issue.
> So I compared the IO ring handler logic between cxenstored and oxenstored and
> find out the difference:
>
> In Cxenstord, after got the cons and prod value, a index check will be
> performed
>
> if (!check_indexes(cons, prod)) {
> errno = EIO;
> return -1;
> }
>
> static bool check_indexes(XENSTORE_RING_IDX cons, XENSTORE_RING_IDX prod)
> {
> return ((prod - cons) <= XENSTORE_RING_SIZE);
> }
>
> So any connection has prod - cons > XENSTORE_RING_SIZE will be treated as
> "bad client", and cxenstored will not handle its IO ring msg any more.
>
> But in oxenstored, there is just a simple comparison between prod and cons
> if (prod == cons)
> return 0;
>
> so there leaves a security hole to a guest vm user who can manipulate prod
> and cons to make oxenstored increasing memory usage and out of service.
>
> I managed to create a patch to fix this and I'm testing it on xen4.2.2. Will
> send out soon.
Are you aware of
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=eeddfad1b339dcaa787230f519a19de1cbc22ad8
which is a second patch for XSA-38 ?
~Andrew
>
>> -----Original Message-----
>> From: Liuqiming (John)
>> Sent: Monday, July 01, 2013 9:47 PM
>> To: 'xen-devel@xxxxxxxxxxxxx'; 'ian.jackson@xxxxxxxxxxxxx';
>> 'ian.campbell@xxxxxxxxxx'
>> Cc: Yanqiangjun
>> Subject: oxenstored memory leak? seems related with XSA-38
>>
>> Hi all,
>>
>> I test starting vm using xen-4.2.2 release with oxenstored, and got a problem
>> may be related with XSA-38
>> (http://lists.xen.org/archives/html/xen-announce/2013-02/msg00005.html).
>>
>> When vm started, oxenstored memory usage keep increasing, and it took
>> 1.5G memory at last. Vm hanged at loading OS screen.
>>
>> Here is the output of top:
>>
>> top - 20:18:32 up 1 day, 3:09, 5 users, load average: 0.99, 0.63, 0.32
>> Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 4.5 us, 1.8 sy, 0.0 ni, 93.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
>> st
>> KiB Mem: 46919428 total, 46699012 used, 220416 free, 36916
>> buffers
>> KiB Swap: 2103292 total, 0 used, 2103292 free, 44260932
>> cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 806 root 20 0 955m 926m 1068 R 99.9 2.0 4:54.14
>> oxenstored
>>
>>
>> top - 20:19:05 up 1 day, 3:09, 5 users, load average: 0.99, 0.67, 0.34
>> Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 4.6 us, 1.6 sy, 0.0 ni, 93.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
>> st
>> KiB Mem: 46919428 total, 46708564 used, 210864 free, 36964
>> buffers
>> KiB Swap: 2103292 total, 0 used, 2103292 free, 44168380
>> cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 806 root 20 0 1048m 1.0g 1068 R 100.2 2.2 5:27.03
>> oxenstored
>>
>>
>>
>> top - 20:21:35 up 1 day, 3:12, 5 users, load average: 1.00, 0.80, 0.44
>> Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 4.7 us, 1.6 sy, 0.0 ni, 93.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
>> st
>> KiB Mem: 46919428 total, 46703052 used, 216376 free, 37208
>> buffers
>> KiB Swap: 2103292 total, 0 used, 2103292 free, 43682968
>> cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 806 root 20 0 1551m 1.5g 1068 R 100.2 3.3 7:56.10
>> oxenstored
>>
>> And oxenstored log got these over and over again:
>>
>> [20130701T12:27:14.290Z] D8 invalid
>> device/suspend/event-channel
>>
>> ..
>> [20130701T12:27:14.290Z] D8.1937077039 invalid
>> /event-channel
>>
>> ..
>> [20130701T12:27:14.290Z] D8.1852727656
>> invalid
>>
>> ..
>> [20130701T12:27:14.290Z] D8 debug
>> [20130701T12:27:14.290Z] D8 debug
>> [20130701T12:27:14.290Z] D8 debug
>> [20130701T12:27:14.290Z] D8 debug
>> [20130701T12:27:14.290Z] D8 debug
>> [20130701T12:27:14.290Z] D8 debug
>>
>> My vm is a windows guest and has GPL PVDriver installed. This problem is
>> hard to reproduce, and after a hard reboot, everything looks normal.
>>
>> I guess it's something wrong with the xenbus IO Ring, so I investigated the
>> code:
>>
>> 1) oxenstored and xenbus in vm using a shared page to communicate with
>> each other
>> struct xenstore_domain_interface {
>> char req[XENSTORE_RING_SIZE]; /* Requests to xenstore daemon. */
>> char rsp[XENSTORE_RING_SIZE]; /* Replies and async watch events. */
>> XENSTORE_RING_IDX req_cons, req_prod;
>> XENSTORE_RING_IDX rsp_cons, rsp_prod;
>> };
>>
>> 2) xenbus in vm put request in req and increase req_prod, then send a event
>> to oxenstored
>> 3) oxenstored calculates how many to read using req_cons and req_prod,
>> and after read oxenstored increase req_cons to make it equals req_prod
>> which means no request pending.
>> 4) oxenstored put responds in rsp and increase rsp_prod, then send a event
>> to vm, xenbus in vm using similar logic to handle the rsp ring.
>>
>> Am I correct?
>>
>> So, I'm curious about what happened when req_cons larger than req_prod
>> (this can be caused by buggy PV Driver or malicious guest user), it seems
>> oxenstored will fell in a endless loop.
>>
>> Is this what XSA-38 talk about?
>>
>> I built a pvdriver which will set req_prod to 0 after several xenstore
>> operation, and test it on xen-unstable.hg make sure all XSA-38 patches
>> applied.
>> It seems that the problem I first met reproduced. Oxenstored will took a lot
>> of memory eventually.
>>
>> Could anyone help me about this issue?
>>
>>
>>
>>
>>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |